(BackBack)
ML//3 min read

Adaptive Moment Estimation (ADAM)

ADAM is the use of Gradient Descent with Momentum and RMSProp at the same time.

Getting to Know Gradient Descent

Before we dive in, let's touch on the basics of Gradient Descent. This is a method we often use to minimize our cost function, or error, in machine learning algorithms. Sounds good, right? But there's a catch - sometimes, it can be a slow process.

This is where the learning rate comes into play. We can speed up convergence by increasing the learning rate, but in some scenarios this might cause the cost function to oscillate due to the shape characteristics of the function itself. If we're not careful, the cost function could even diverge. This means we often have to be patient and wait, even if the learning speed seems sluggish.

Figure 1. Gradient Descent
Figure 1. Gradient Descent

Let's look at Figure 1, where the parameter b oscillates before it eventually converges to the minimum value. This is due to the gradient vector's nature which points in the direction perpendicular to the contour line.

Figure 2. What we want
Figure 2. What we want

The ideal situation? We want to converge slowly in the vertical direction and quickly in the horizontal direction (check out Figure 2).

Adding Momentum to Gradient Descent

Gradient Descent with Momentum is like an upgrade! It's designed to slow down the updates if the direction keeps changing in every iteration, and to speed up if the direction remains the same.

Momentum Algorithm:

We start with Vdw(0)=0⃗,Vdb(0)=0⃗V_{dw(0)} = \vec{0}, V_{db(0)} = \vec{0}

During the t-th iteration:

  1. We calculate dW(t),db(t)dW_{(t)}, db_{(t)} for the current batch.

  2. Next, we compute:

    Vdw(t)=β1Vdw(t−1)+(1−β1)dW(t)Vdb(t)=β1Vdb(t−1)+(1−β1)db(t)V_{dw(t)}=β_1V_{dw(t−1)}+(1−β_1)dW_{(t)}\\V_{db(t)}=β_1V_{db(t−1)}+(1−β_1)db_{(t)}

  3. Finally, we update our Weight and Bias as follows:

    W:=W−αVdw(t)b:=b−αVdb(t)W := W - αV_{dw(t)}\\b:=b-αV_{db(t)}

    (where α is the learning rate)

The core of the Momentum algorithm is Equations (1) and (2), but since the form is almost the same, let's solve Equation (1) a little more and think about it.

Equation (1) is a recursively calculated term, which will be as follows if we look at it step by step from iteration 1.

iteration 1

Vdw(1)=β1Vdw(0)+(1−β1)dW(1)V_{dw(1)}=β_1V_{dw(0)}+(1−β_1)dW_{(1)}

iteration 2

Vdw(2)=β1Vdw(1)+(1−β1)dW(2)=β1(β1Vdw(0)+(1−β1)dW(1))+(1−β1)dW(2)=β12Vdw(0)+β1(1−β1)dW(1)+(1−β1)dW(2)V_{dw(2)}=β_1V_{dw(1)}+(1−β_1)dW_{(2)}\\ =β_1(β_1V_{dw(0)}+(1−β_1)dW_{(1)})+(1−β_1)dW_{(2)}\\ =β^2_1V_{dw(0)}+β_1(1−β_1)dW_{(1)}+(1−β_1)dW_{(2)}

iteration 3

Vdw(3)=β1Vdw(2)+(1−β1)dW(3)=β1{β12Vdw(0)+β1(1−β1)dW(1)+(1−β1)dW(2)}+(1−β1)dW(3)=β13Vdw(0)+β12(1−β1)dW(1)+β1(1−β1)dW(2)+(1−β1)dW(3)V_{dw(3)}=β_1V_{dw(2)}+(1−β_1)dW_{(3)}\\ =β_1\{β^2_1V_{dw(0)}+β_1(1−β_1)dW_{(1)}+(1−β_1)dW_{(2)}\}+(1−β_1)dW_{(3)}\\ =β^3_1V_{dw(0)}+β^2_1(1−β_1)dW_{(1)}+β_1(1−β_1)dW_{(2)}+(1−β_1)dW_{(3)}

Normalize

Vdw(k)=βk1Vdw(0)+β1k−1(1−β1)Vdw(1)+β1k−2(1−β1)Vdw(2)+⋯+β10(1−β1)Vdw(k)=β1kVdw(0)+∑ki=β1k−i(1−β1)Vdw(i)V_{dw(k)}=β_{k1}V_{dw(0)}+β^{k−1}_1(1−β_1)V_{dw(1)}+β^{k−2}_1(1−β_1)V_{dw(2)}+⋯+β^0_1(1−β_1)V_{dw(k)}\\ =β^k_1V_{dw(0)}+\displaystyle∑_k^i=β^{k−i}_1(1−β_1)V_{dw(i)}

normally Vdw(0)=0V_{dw(0)} = 0

∑ki=β1k−i(1−β1)Vdw(i)\displaystyle∑_k^i=β^{k−i}_1(1−β_1)V_{dw(i)}

As we iterate, we can see that gradients in the b-axis that change up and down will gradually approach zero, while those in the W-axis will continue to speed up. This shows that the momentum algorithm adjusts the learning speed for each parameter appropriately.

RMSProp: Harnessing the Power of Propagation

RMSProp, although not officially presented in an academic paper, has proven its worth in the machine learning field. It's quite similar to Gradient Descent with Momentum, but uses the size of the gradient to adjust the learning speed for each parameter.

RMSProp Algorithm:

We start with Sdw(0)=0⃗,Sdb(0)=0⃗S_{dw(0)} = \vec{0}, S_{db(0)} = \vec{0}

During the t-th iteration:

  1. We calculate dW(t),db(t)dW_{(t)}, db_{(t)} for the current batch.

  2. Next, we compute:

    Sdw(t)=β2Sdw(t−1)+(1−β2)dW(t)2Sdb(t)=β2Vdb(t−1)+(1−β2)db(t)2S_{dw(t)}=β_2S_{dw(t−1)}+(1−β_2)dW^2_{(t)}\\S_{db(t)}=β_2V_{db(t−1)}+(1−β_2)db^2_{(t)}

  3. Finally, we update our Weight and Bias as follows:

    W:=W−αdW(t)Sdw(t)+ϵb:=b−αdb(t)Sdb(t)+ϵW := W - α\frac{{dW(t)}}{\sqrt{S_{dw}(t)+ϵ}}\\ b:=b-α\frac{{db(t)}}{\sqrt{S_{db}(t)+ϵ}}

    (where α is the learning rate)

RMSProp adjusts the learning rate for each parameter based on the size of the gradient, thus allowing for efficient learning.

ADAM: The Best of Both Worlds

ADAM (Adaptive Moment Estimation) is like a supercharged method that uses both Gradient Descent with Momentum and RMSProp.

The ADAM algorithm:

We start with Vdw(0)=0⃗,Vdb(0)=0⃗,Sdw(0)=0⃗,Sdb(0)=0⃗V_{dw(0)} = \vec{0}, V_{db(0)} = \vec{0},S_{dw(0)} = \vec{0}, S_{db(0)} = \vec{0}

During the t-th iteration:

  1. We calculate dW(t),db(t)dW_{(t)}, db_{(t)} for the current batch.

  2. Next, we compute:

    Vdw(t)=β1Vdw(t−1)+(1−β1)dW(t)Vdb(t)=β1Vdb(t−1)+(1−β1)db(t)V_{dw(t)}=β_1V_{dw(t−1)}+(1−β_1)dW_{(t)}V_{db(t)}=β_1V_{db(t−1)}+(1−β_1)db_{(t)}

    Sdw(t)=β2Sdw(t−1)+(1−β2)dW(t)2Sdb(t)=β2Vdb(t−1)+(1−β2)db(t)2S_{dw(t)}=β_2S_{dw(t−1)}+(1−β_2)dW^2_{(t)}S_{db(t)}=β_2V_{db(t−1)}+(1−β_2)db^2_{(t)}

  3. Finally, we update our Weight and Bias as follows:

    W:=W−αVdw(t)Sdw(t)+ϵb:=b−αVdb(t)Sdb(t)+ϵW := W - α\frac{V_{dw(t)}}{\sqrt{S_{dw}(t)+ϵ}}\\ b:=b-α\frac{V_{db(t)}}{\sqrt{S_{db}(t)+ϵ}}

    (where α is the learning rate)

The creators of the ADAM algorithm suggest using β1:0.9β1:0.9, β2:0.99β2:0.99 and ϵ:10−8ϵ:10^{−8} for best results. With ADAM, you're combining the advantages of two powerful algorithms for efficient optimization!