Parameter smoothness in neural networks

DegreeMaster
Status
Supervisor(s)Alexander Schlögl, MSc

Description

Some compression algorithms benefit greatly from smoothness (i.e., closeness to a polynomial) of their input data. We know that this is the case for some physics-based simulations. What is unknown is whether we can constrain the parameters of a neural network in a similar fashion, thus enabling high throughput compression for distributed learning algorithms.

Your goal in this thesis is to find out whether parameter updates, or parameters themselves, can be constrained to be reasonably well approximated by a polynomial. This can happen either by using an additional loss or by quantizing parameter updates/weights. We are interested in the compression ratio, as well as the accuracy change incurred due to the additional constraints.

Note that due to the highly experimental nature of this thesis, we cannot guarantee success. Negative experiment results (i.e., it cannot be done in this way) are also completely valid, and do not stand in the way of a very good grade if they are well documented.

Prerequisites

Theoretical and practical machine learning knowledge

References

  • Koutnik, J., Gomez, F., and Schmidhuber, J. Evolving neural networks in compressed weight space. In Annual Conference on Genetic and evolutionary computation. 2010, pp. 619–626.
  • Fabian Knorr, T.F., Peter Thoman. ndzip: a high-throughput parallel lossless compressor for scientific data. 2021. https://sigport.org/documents/ndzip-high-throughput-parallel-lossless-compressor-scientific-data#files.
  • Ibarria, L., Lindstrom, P., Rossignac, J., and Szymczak, A. Out-of-core compression and decompression of large n-dimensional scalar fields. In Computer Graphics Forum. 22, Wiley Online Library, 2003, pp. 343–348.