Gradient… Jacobian… Hessian…
Three words that initially seem intimidating, but in reality, they're just three ways of measuring change.
➡️ Which are the ways and how it works?
1. scaling function:
f : ℝⁿ → ℝ
Returns the vector of first partial derivatives.
It answers the question:
"In which direction does the function f grow the fastest?"
That's why gradients are the foundation of optimization.
Gradient descent goes in the opposite direction because the gradient points to the direction of maximum growth.
Backpropagation efficiently calculates gradients during training.
2. vector-valued function:
F : ℝⁿ → ℝᵐ
Returns the m × n matrix of first partial derivatives.
It answers:
"How does each output depend on each input?"
The Jacobian is a local linear mapping of a vector function.
It appears in:
→ sensitivity analysis
→ variable substitution
→ automatic differentiation
→ forward-mode AD
→ reverse-mode AD / backpropagation
In simple terms:
forward-mode AD uses Jacobian–vector products.
reverse-mode AD uses vector–Jacobian products.
3. scalar function:
f : ℝⁿ → ℝ
Returns the n × n matrix of second partial derivatives.
It answers:
"How does the gradient itself change?"
That is, the Hessian measures curvature.
When the second partial derivatives are continuous, the Hessian is symmetric.
At a critical point:
→ positive-definite Hessian → strict local minimum
→ negative-definite Hessian → strict local maximum
→ indefinite Hessian → saddle point
A pure mental model
Gradient = first derivatives of a single output
→ shows direction
Jacobian = first derivatives of many outputs
→ shows sensitivity
Hessian = second derivatives of a single output
→ shows curvature
And the connection between them is simple:
The Hessian is the Jacobian of the gradient.
For a scalar output, the Jacobian contains the same partial derivatives as the gradient, up to the convention on rows/columns.
Same idea: measuring change.
Different objects: direction, sensitivity, curvature.
When this becomes clear, optimization stops looking like a set of formulas. It starts looking like a map of the task.
•••••••••••••••••••••••••••••••••••
🤖 Data & ML | @DataXplore