Beauty in Details: HSE University and AIRI Scientists Develop a Method for High-Quality Image Editing
Researchers from the HSE AI Research Centre, AIRI, and the University of Bremen have developed a new image editing method based on deep learning—StyleFeatureEditor. This tool allows for precise reproduction of even the smallest details in an image while preserving them during the editing process. With its help, users can easily change hair colour or facial expressions without sacrificing image quality. The results of this three-party collaboration were published at the highly-cited computer vision conference CVPR 2024.
Artificial intelligence is already able to generate and edit images using generative adversarial networks (GANs). The architecture consists of two independent networks: a generator that creates images and a discriminator that distinguishes between real and generated samples. These networks compete with each other, and a new stage in their development is the StyleGAN model. This model can generate images and modify specific parts based on user requests, but it has not been able to work with real photos or images before.
Researchers from the HSE AI Research Centre, the Artificial Intelligence Research Institute (AIRI), and the University of Bremen have proposed a method to quickly and efficiently edit real images. This StyleFeatureEditor approach consists of two modules: the first inverts (reconstructs) the original image, and the second edits this reconstruction. The results of these two steps are passed to StyleGAN, which generates the edited image based on the internal representations. The developers addressed some challenges that had been encountered in previous research. With a small set of representations, the network could edit the image well, but it lost some details from the original. However, with a larger set, all the details were preserved, but the network had difficulty transforming them correctly according to the task.
To solve this, the researchers proposed a new solution: the first module finds both large and small representations, while the second learns how to edit the larger ones using the smaller ones as reference.
However, to train these modules to accurately edit the representations, the neural network requires both real images and their edited versions.
‘We needed examples, such as the same face with different expressions, hairstyles, and details. Unfortunately, such image pairs do not exist at the moment. So, we came up with a trick: using a method that works with small representations, we created a reconstruction of a real image and an example of editing this reconstruction. Although the examples were relatively simple and without details, the model clearly understood how to make the edits,’ explains Denis Bobkov, one of the authors of the article, a research intern at the Centre of Deep Learning and Bayesian Methods of the AI and Digital Science Institute (part of the HSE Faculty of Computer Science), and a Junior Research Fellow at AIRI’s Fusion Brain Lab.
However, training only on generated (simple) examples leads to a loss of detail when working with real (complex) images. To prevent this, the researchers added real images to the training dataset, and the neural network learnt to reconstruct them in detail.
Thus, by showing the model how to edit both simple and complex images, the scientists created conditions under which the network could edit complex images more effectively. In particular, the developed approach handles adding new elements of style while preserving the details of the original image better than other existing methods.
In the case of simple reconstruction (first row), StyleFeatureEditor accurately reproduced a hat, while most other methods almost completely lost it. The developed method showed the best results with additional accessories (third row): most methods could add glasses, but only the StyleFeatureEditor retained the original eye colour.
‘Thanks to this training technique on generated data, we have obtained a model with high editing quality and a fast processing speed due to the use of relatively lightweight neural networks. The StyleFeatureEditor framework requires only 0.07 seconds to edit a single image,’ says Aibek Alanov, Head of the Centre of Deep Learning and Bayesian Methods of the AI and Digital Science Institute (part of the HSE Faculty of Computer Science), and leader of the research group ‘Controlled Generative AI’ at AIRI's Fusion Brain Lab.
The research was funded by a grant from the Analytical Centre under the Government of the Russian Federation for AI research centres.
The research results will be presented at the Fall into ML 2024 conference on artificial intelligence and machine learning, which will take place at HSE University on October 25–26, 2024. Leading AI scientists will discuss the best papers published at top-tier (A*) flagship AI conferences in 2024. A demo of the developed method can be tried out on HuggingFace, and the source code is available on GitHub.
See also:
HSE Researchers Teach Neural Network to Distinguish Origins from Genetically Similar Populations
Researchers from the AI and Digital Science Institute, HSE Faculty of Computer Science, have proposed a new approach based on advanced machine learning techniques to determine a person’s genetic origin with high accuracy. This method uses graph neural networks, which make it possible to distinguish even very closely related populations.
HSE Economists Reveal the Secret to Strong Families
Researchers from the HSE Faculty of Economic Sciences have examined the key factors behind lasting marriages. The findings show that having children is the primary factor contributing to marital stability, while for couples without children, a greater income gap between spouses is associated with a stronger union. This is the conclusion reported in Applied Econometrics.
Fifteen Minutes on Foot: How Post-Soviet Cities Manage Access to Essential Services
Researchers from HSE University and the Institute of Geography of the Russian Academy of Sciences analysed three major Russian cities to assess their alignment with the '15-minute city' concept—an urban design that ensures residents can easily access essential services and facilities within walking distance. Naberezhnye Chelny, where most residents live in Soviet-era microdistricts, demonstrated the highest levels of accessibility. In Krasnodar, fewer than half of residents can easily reach essential facilities on foot, and in Saratov, just over a third can. The article has been published in Regional Research of Russia.
HSE Researchers Find Counter-Strike Skins Outperform Bitcoin and Gold as Alternative Investments
Virtual knives, custom-painted machine guns, and gloves are common collectible items in videogames. A new study by scientists from HSE University suggests that digital skins from the popular video game Counter-Strike: Global Offensive (CS:GO) rank among the most profitable types of alternative investments, with average annual returns exceeding 40%. The study has been published in the Social Science Research Network (SSRN), a free-access online repository.
HSE Neurolinguists Reveal What Makes Apps Effective for Aphasia Rehabilitation
Scientists at the HSE Centre for Language and Brain have identified key factors that increase the effectiveness of mobile and computer-based applications for aphasia rehabilitation. These key factors include automated feedback, a variety of tasks within the application, extended treatment duration, and ongoing interaction between the user and the clinician. The article has been published in NeuroRehabilitation.
'Our Goal Is Not to Determine Which Version Is Correct but to Explore the Variability'
The International Linguistic Convergence Laboratory at the HSE Faculty of Humanities studies the processes of convergence among languages spoken in regions with mixed, multiethnic populations. Research conducted by linguists at HSE University contributes to understanding the history of language development and explores how languages are perceived and used in multilingual environments. George Moroz, head of the laboratory, shares more details in an interview with the HSE News Service.
Slim vs Fat: Overweight Russians Earn Less
Overweight Russians tend to earn significantly less than their slimmer counterparts, with a 10% increase in body mass index (BMI) associated with a 9% decrease in wages. These are the findings made by Anastasiia Deeva, lecturer at the HSE Faculty of Economic Sciences and intern researcher in Laboratory of Economic Research in Public Sector. The article has been published in Voprosy Statistiki.
Scientists Reveal Cognitive Mechanisms Involved in Bipolar Disorder
An international team of researchers including scientists from HSE University has experimentally demonstrated that individuals with bipolar disorder tend to perceive the world as more volatile than it actually is, which often leads them to make irrational decisions. The scientists suggest that their findings could lead to the development of more accurate methods for diagnosing and treating bipolar disorder in the future. The article has been published in Translational Psychiatry.
Scientists Develop AI Tool for Designing Novel Materials
An international team of scientists, including researchers from HSE University, has developed a new generative model called the Wyckoff Transformer (WyFormer) for creating symmetrical crystal structures. The neural network will make it possible to design materials with specified properties for use in semiconductors, solar panels, medical devices, and other high-tech applications. The scientists will present their work at ICML, a leading international conference on machine learning, on July 15 in Vancouver. A preprint of the paper is available on arxiv.org, with the code and data released under an open-source license.
HSE Linguists Study How Bilinguals Use Phrases with Numerals in Russian
Researchers at HSE University analysed over 4,000 examples of Russian spoken by bilinguals for whom Russian is a second language, collected from seven regions of Russia. They found that most non-standard numeral constructions are influenced not only by the speakers’ native languages but also by how frequently these expressions occur in everyday speech. For example, common phrases like 'two hours' or 'five kilometres’ almost always match the standard literary form, while less familiar expressions—especially those involving the numerals two to four or collective forms like dvoe and troe (used for referring to people)—often differ from the norm. The study has been published in Journal of Bilingualism.