EPFL researchers have made a significant breakthrough in the field of deep learning with the development of an algorithm that overcomes the limitations of physical neural networks. This algorithm allows analog neural networks to be trained as accurately as their digital counterparts, presenting a new avenue for developing more energy-efficient alternatives to power-hungry deep learning hardware.
Deep neural networks have revolutionized data processing by utilizing algorithmic learning instead of traditional programming. However, as these networks have become larger and more complex, they have also become significant consumers of energy. This has raised concerns about their contribution to global carbon emissions.
In response to these concerns, researchers at EPFL’s Laboratory of Wave Engineering in the School of Engineering, led by Romain Fleury, have developed an algorithm for training physical systems that offers improved speed, enhanced robustness, and reduced power consumption compared to existing methods. The algorithm was successfully tested on wave-based physical systems that use sound waves, light waves, and microwaves to carry information.
The traditional method of neural network training involves a forward pass and a backward pass, known as backpropagation. However, backpropagation is energy-intensive and ill-suited for physical systems, often requiring the use of a digital twin for training. The EPFL researchers replaced the backpropagation step with a second forward pass through the physical system, allowing each network layer to be updated locally. This not only reduced power consumption and eliminated the need for a digital twin but also better resembled human learning.
The researchers utilized their physical local learning algorithm (PhyLL) to train experimental acoustic and microwave systems, as well as a modeled optical system, to classify data such as vowel sounds and images. The results showed comparable accuracy to backpropagation-based training methods while demonstrating robustness and adaptability in the face of external perturbations.
While the approach is currently a hybrid training method that still requires some digital updates, the researchers aim to minimize digital computation in the future. They plan to implement their algorithm on a small-scale optical system and ultimately scale up to neural networks with hundreds of layers and billions of parameters. However, this will require overcoming technical limitations associated with physical systems.
The research opens up new possibilities for the development of analog neural networks that are more energy-efficient and have a reduced environmental impact. By harnessing the power of physical systems, researchers are taking a step towards a future where deep learning technologies can operate sustainably on a large scale.
Reference:
Ali Momeni et al, Backpropagation-free training of deep physical neural networks. Science 0, eadi8474 DOI: 10.1126/science.adi8474
