Reducing redundancy in the bottleneck representation of autoencoders
Laakom, F., Raitoharju, J., Iosifidis, A., & Gabbouj, M. (2024). Reducing redundancy in the bottleneck representation of autoencoders. Pattern Recognition Letters, 178, 202-208. https://doi.org/10.1016/j.patrec.2024.01.013
Published in
Pattern Recognition LettersDate
2024Copyright
© 2024 The Authors. Published by Elsevier B.V.
Autoencoders (AEs) are a type of unsupervised neural networks, which can be used to solve various tasks, e.g., dimensionality reduction, image compression, and image denoising. An AE has two goals: (i) compress the original input to a low-dimensional space at the bottleneck of the network topology using an encoder, (ii) reconstruct the input from the representation at the bottleneck using a decoder. Both encoder and decoder are optimized jointly by minimizing a distortion-based loss which implicitly forces the model to keep only the information in input data required to reconstruct them and to reduce redundancies. In this paper, we propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. To this end, we propose an additional loss term, based on the pairwise covariances of the network units, which complements the data reconstruction loss forcing the encoder to learn a more diverse and richer representation of the input. We tested our approach across different tasks, namely dimensionality reduction, image compression, and image denoising. Experimental results show that the proposed loss leads consistently to superior performance compared to using the standard AE loss.
...
Publisher
ElsevierISSN Search the Publication Forum
0167-8655Keywords
Publication in research information system
https://converis.jyu.fi/converis/portal/detail/Publication/202074852
Metadata
Show full item recordCollections
Related funder(s)
Funding program(s)
Public research networked with companies, BFAdditional information about funding
This work has been supported by the Academy of Finland Awcha project DN 334566 and NSF-Business Finland Center for Big Learning project AMALIA. The work of Jenni Raitoharju was supported by the Academy of Finland (projects 324475 and 333497).License
Related items
Showing items with similar title or keywords.
-
Unsupervised representation learning of spontaneous MEG data with nonlinear ICA
Zhu, Yongjie; Parviainen, Tiina; Heinilä, Erkka; Parkkonen, Lauri; Hyvärinen, Aapo (Elsevier BV, 2023)Resting-state magnetoencephalography (MEG) data show complex but structured spatiotemporal patterns. However, the neurophysiological basis of these signal patterns is not fully known and the underlying signal sources are ... -
Unsupervised feature analysis of real and synthetic knee X-ray images
Tuomikoski, Joonas (2023)Generatiiviset mallit ovat parantuneet valtavasti viime vuosina, ja tämä on luonut tarpeen automaattisille validointitekniikoille synteettiselle datalle. Tässä pro gradu -työssä testatiin menetelmää synteettisten kuvien ... -
A method for anomaly detection in hyperspectral images, using deep convolutional autoencoders
Penttilä, Jeremias (2017)Menetelmä poikkeavuuksien havaitsemiseen hyperspektrikuvista käyttäen syviä konvolutiivisia autoenkoodereita. Poikkeavuuksien havaitseminen kuvista, erityisesti hyperspektraalisista kuvista, on hankalaa. Kun ongelmaan ... -
Additive autoencoder for dimension estimation
Kärkkäinen, Tommi; Hänninen, Jan (Elsevier BV, 2023)Dimension reduction is one of the key data transformation techniques in machine learning and knowledge discovery. It can be realized by using linear and nonlinear transformation techniques. An additive autoencoder for ... -
Dimensionality reduction framework for detecting anomalies from network logs
Sipola, Tuomo; Juvonen, Antti; Lehtonen, Joel (CRL Publishing, 2012)Dynamic web services are vulnerable to multitude of intrusions that could be previously unknown. Server logs contain vast amounts of information about network traffic, and finding attacks from these logs improves the ...