What Is Gradient Descent in Machine Learning?

Key takeaways

Gradient descent is an algorithm you can use to train models in both neural networks and machine learning.

• The “gradient” in the gradient descent formula represents the slope of the cost function at a specific point. 

• Convergence in gradient descent is the point where the algorithm has minimized the cost function enough that parameter updates are almost negligible.

• You can strengthen your math skills to better understand and implement gradient descent. 

Find out what gradient descent is, why it’s vital for machine learning, its main types, and potential limitations. Afterward, if you’re ready to build foundational skills in machine learning, consider enrolling in DeepLearning. AI’s Mathematics for Machine Learning and Data Science Specialization. Along with machine learning methods, this program offers guidance on data preprocessing, statistical analysis, applied mathematics, and more. 

What is gradient descent?

Gradient descent is an algorithm you can use to train models in both neural networks and machine learning. It uses a cost function to optimize its parameters, showing the accuracy of a machine learning model under training at each parameter. Gradient descent existed as a mathematical concept before the emergence of machine learning. 

A gradient in vector calculus is similar to the slope but applies when you have two or more variables. It is the vector of partial derivatives for all independent variables and is denoted \(\nabla f\). The vector (\(\nabla f\)) points in the direction of the maximum rate of increase of the function, while -\(\nabla f\) points in the direction of the maximum rate of decrease.

How does gradient descent work?

Gradient descent works with differentiable functions (often optimized for convex functions, but not restricted to these functions) and finds the lowest point of a surface by moving in the direction of the steepest descent, optimizing the path as it progresses. Let’s go over a couple of terms that inform gradient descent before examining how they work:

• Parameters: The coefficients of the function that minimize the cost.

• Cost function: Also called the “loss function” in machine learning, a loss funtion computes the error for a single training example. A cost function averages the errors over the entire dataset. A model stops training once this function gets as close as possible to 0.0.

• Learning rate: Sometimes referred to as the step rate or alpha, it is the magnitude of the steps the function takes as it minimizes the cost.

The primary function of gradient descent is to find the parameters that best minimize the cost by having the coefficients make the cost equal to or as close to 0.0 as possible 

How is the “gradient” used in the gradient descent formula?

In the gradient descent formula, the “gradient” is the slope of the cost function at a specific point. It shows the direction and rate of the steepest descent, using it to move toward the minimum of the function.

Basics of performing gradient descent

The goal is to have the cost = 0.0 or the closest acceptable minimum. To calculate this, start by writing the cost function. 

1. Write the cost function as cost =f(x) with x as the coefficient. 

2. Use a starting coefficient of 0.0 or any small number. 

3. Take the derivative, or partial derivative if multiple variables are present, to find the gradient to know which direction to move in on the curve. 

4. Once you have a gradient (the derivative of the cost function) and know which way to move, use your learning rate to tell how much the value of the coefficient changes every calculation. 

5. Repeat until the cost is zero or as close as it can reach. 

Gradient descent involves knowledge of calculus, but its implementation is always the same series of steps. 

What is convergence in gradient descent​?

Convergence in gradient descent is the point where the algorithm has minimized the cost function enough that parameter updates are almost negligible. This is essentially when the model reaches the “best” solution, and more iterations are unlikely to improve findings. 

What is gradient descent in machine learning? Two types to explore

Machine learning uses two main types of gradient descent: 

• Batch gradient descent (BGD): Provides an update to the machine learning model's parameters once per iteration by computing the gradient (average error) of the cost function using the entire training dataset. 

• Stochastic gradient descent (SGD): Calculates the error and updates the coefficients for only one randomly selected training sample per iteration. This makes it much fast and more efficient memory-wise for larger datasets.

Comparing batch gradient descent and SGD

Batch gradient descent and stochastic gradient descent have unique advantages and disadvantages when calculating gradient descent in machine learning. Let’s take a look at each:

Batch gradient descent is a common approach to machine learning, but stochastic gradient descent performs better on larger data sets. 

Mini-batch gradient descent

If you need to use aspects of both batch gradient descent and SGD, consider using a method called mini-batch gradient descent (MBGD) that combines them. It still uses batches but breaks up a data set into small batches, each providing the updates from SGD as they perform gradient descent.

It makes the learning of each batch quick while also remaining computationally fast. This method is standard in machine learning, the training of neural networks, and deep learning applications. 

Common gradient descent problems

While gradient descent is an efficient way to optimize machine learning algorithms, you will find some common problems the algorithm runs into that may leave you with models that aren’t fully optimized. With non-convex optimization landscapes, points other than the global minimum can make the gradient of the cost function equal to 0.0, which causes the algorithm to stall before the optimal solution is reached. These two points are:

• Local minima: These give a slope of 0.0 and seem like global minimum points to the algorithm, but are just local minimum points before the cost function increases again before reaching the global minimum.

• Saddle points: Points where the cost function has a slope (gradient) of 0.0, but which are neither a local minimum nor a local maximum. The function decreases along some directions while increases along other directions.

In applying gradient descent to deep learning neural networks, two issues arise:

• Vanishing gradients: Occur during the backpropagation of neural networks, making the gradient too small, leading to an eventual coefficient of zero, resulting in the network stopping learning.

• Exploding gradients: Occur when a model becomes unstable due to a gradient that is too large, leading to coefficients that are no longer calculable because of a complex machine learning algorithm.

Read more: Neural Network Weights: A Comprehensive Guide

Check out our free resources on machine learning

Subscribe to Career Chat on LinkedIn to stay current with the latest trends in machine learning. Continue your learning journey with our other free digital resources:

• Explore certifications: 6 machine learning certificates + how to choose the right one for you

• Watch on YouTube: Machine Learning Classification | Python Diabetes Prediction Model

• Plan your career trajectory: Machine Learning Career Paths: Explore Roles & Specializations

Accelerate your career growth with a Coursera Plus subscription. When you enroll in either the monthly or annual option, you’ll get access to over 10,000 courses.