An Overview of Basic Gradient Descent Optimization Algorithms

Gradient Descent is a popular optimization algorithm used to minimize a differentiable multivariate function in machine learning and artificial intelligence. It works by taking repeated steps in the opposite direction of the gradient of the function at the current point, leading to a path that eventually minimizes the objective function. There are three variants of gradient descent algorithm - Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-batch Gradient Descent.

Batch Gradient Descent, also known as vanilla gradient descent, computes the gradient of the cost function with respect to the parameters using the entire training dataset before making one parameter update. This method is guaranteed to converge to the global minimum for convex error surfaces and to a local minimum for non-convex surfaces. However, it can be very slow and intractable for large datasets that do not fit into memory, and it cannot be used for online learning with new examples on-the-fly.

Stochastic Gradient Descent, on the other hand, performs parameter updates for each random training example and label. This method performs one update at a time, making it much faster than Batch Gradient Descent and allowing for online learning with new examples as they come in. However, the updates performed by SGD have high variance, which can cause the objective function to fluctuate during optimization.

Mini-batch Gradient Descent is a compromise between Batch Gradient Descent and SGD, where a small batch of training examples (smaller than the entire dataset) is used to compute the gradient of the objective function, and the parameters are updated accordingly. This method balances the trade-off between the accuracy of parameter updates and the time it takes to perform an update, making it a popular choice for optimizing advanced neural networks.