Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks.

Feng Chen Daniel Kunin Atsushi Yamamura Surya Ganguli

Published in: CoRR (2023)

Keyphrases

dynamic model
monte carlo
noise level
dynamical systems
noise reduction
noise model
missing data
gradient estimation
edge orientation
additive noise
median filter
signal to noise ratio
denoising
noisy data
input data
gaussian noise
least squares
noise free
random noise
gradient direction
state space
feature selection