Abhishek Panigrahi

Abhishek Panigrahi

Graduate Student at Princeton University

Random Reshuffling converges to a smaller neighborhood than SGD

less than 1 minute read

Posted on:

This article is on the recent work by Ying et. al. 2018, in which the authors show that SGD with Random Reshuffling outperforms independent sampling with replacement by showing that the MSE of the iterates at the end of each epoch is of the order \(O(\eta^2)\) for constant step-size \(\eta\). This is a significant improvement compared to the traditional SGD with i.i.d. sampling where the same quantity is of the order \(O(\eta)\).

The complete PDF post can be viewed here.

Analysis of Newton’s Method

less than 1 minute read

Posted on:

In optimization, Netwon’s method is used to find the roots of the derivative of a twice differentiable function given the oracle access to its gradient and hessian. By having super-liner memory in the dimension of the ambient space, Newton’s method can take the advantage of the second order curvature and optimize the objective function at a quadratically convergent rate. Here I consider the case when the objective function is smooth and strongly convex.

Deriving the Fokker-Planck equation

less than 1 minute read

Posted on:

In the theory of dynamic systems, Fokker-Planck equation is used to describe the time evolution of the probability density function. It is a partial differential equation that describes how the density of a stochastic process changes as a function of time under the influence of a potential field. Some common application of it are in the study of Brownian motion, Ornstein–Uhlenbeck process, and in statistical physics. The motivation behind understanding the derivation is to study Levy flight processes that has caught my recent attention.