Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence.
Nicolas LoizouSharan VaswaniIssam H. LaradjiSimon Lacoste-JulienPublished in: CoRR (2020)
Keyphrases
- step size
- learning rate
- convergence rate
- stochastic gradient descent
- convergence speed
- global convergence
- rapid convergence
- faster convergence
- update rule
- gradient method
- convergence theorem
- hessian matrix
- variable step size
- global optimum
- adaptive learning rate
- primal dual
- convergence analysis
- conjugate gradient
- steepest descent method
- weight vector
- line search
- training algorithm
- differential evolution
- evolutionary algorithm
- learning algorithm