Stable vs. Unstable Equilibrium

An object in stable equilibrium will return to its original position when shifted a small amount from its equilibrium position. For example, we have this pendulum:

When we move the bob a little bit, the net force acting on the bob accelerates the bob towards its equilibrium position. On the other hand, an inverted pendulum is clearly in an unstable equilibrium: one in which the object is accelerated away from the equilibrium position if it is shifted slightly.

Whether a system is in stable or unstable equilibrium can be determined with the second derivative of the function describing the system's potential energy-U.

At the critical points of the function U(x) where the first derivative of the function is zero, the system experience a net force of zero and is at a state of equilibrium. However, the second derivatives at these critical numbers are required to determine if it's stable or unstable.

If the second derivative of the function U(x): dU^2/d^2x is greater than zero, the system is at a stable equilibrium, and the function U(x) concaves upward.

If the second derivative of the function U(x): dU^2/d^2x is less than zero, the system is at an unstable equilibrium, and the function U(x) concaves downward.

At the top of the hill, dU^2/d^2x is less than zero, and the function concaves downward. The ball has the tendency of rolling down along the ramp if the ball is given a light push. Therefore it's evident that the system is at a unstable equilibrium at the top of the hill.

At the bottom of the hill, dU^2/d^2x is greater than zero, and the function concaves upward. As obvious from the graph of imagination, even if a slight push is given to the ball the ball has the tendency of rolling back to the bottom and return to stable equilibrium.