Whether a system is in stable or unstable equilibrium can be determined with the second derivative of the function describing the system's potential energy-U.
At the critical points of the function U(x) where the first derivative of the function is zero, the system experience a net force of zero and is at a state of equilibrium. However, the second derivatives at these critical numbers are required to determine if it's stable or unstable.
If the second derivative of the function U(x): dU^2/d^2x is greater than zero, the system is at a stable equilibrium, and the function U(x) concaves upward.
If the second derivative of the function U(x): dU^2/d^2x is less than zero, the system is at an unstable equilibrium, and the function U(x) concaves downward.
At the top of the hill, dU^2/d^2x is less than zero, and the function concaves downward. The ball has the tendency of rolling down along the ramp if the ball is given a light push. Therefore it's evident that the system is at a unstable equilibrium at the top of the hill.
At the bottom of the hill, dU^2/d^2x is greater than zero, and the function concaves upward. As obvious from the graph of imagination, even if a slight push is given to the ball the ball has the tendency of rolling back to the bottom and return to stable equilibrium.