Finite Difference Methods
Learning Objectives
- Approximate derivatives using the Finite Difference Method
Finite Difference Approximation
Motivation
For a given smooth function \(f(x)\), we want to calculate the derivative \(f'(x)\) at a given value of \(x\).
However, sometimes we do not know how to compute the analytical expression \(f'(x)\) or it is computationally too expensive.
Finite difference methods can help find an approximation for the value of \(f'(x)\).
Definition
For a differentiable function \(f(x):\mathbb{R} \rightarrow \mathbb{R}\), the derivative is defined as
We define the forward finite difference approximation, \(df(x)\), to the first derivative as
where \(h\) is often called a "perturbation", i.e., a "small" change to the variable \(x\) (small when compared to the magnitude of \(x\)).
By Taylor's theorem, we can write
Rearranging the above, we get the derivative \(f'(x)\) as a function of the forward finite difference approximation \(df(x)\):
Error
Therefore, the truncation error of the forward finite difference approximation is bounded by:
where \(M\) is a constant bound on the value of \(h\).
If \(h\) is very small, we will have cancellation errors, which is bounded by:
where \(\epsilon_m\) is machine epsilon.
To find the \(h\) that minimizes the total error:
Using the forward finite different approximation on \(f(x) = e^x - 2\), we can see the values of total error, truncation error, and rounding error depending on the chosen perturbation \(h\) in the graph below.

Therefore, we can see that the optimal \(h\) that minimizes the total error is where the truncation error and rounding error intersect.
Using a similar approach, we can summarize the following finite difference approximations:
Forward Finite Difference Method
\[df(x) = \frac{f(x+h)-f(x)}{h}\]In addition to the computation of \(f(x)\), this method requires an extra cost of one function evaluation \(f(x+h)\) for a given perturbation, and has truncation order \(O(h)\).
Example
Assume \(f(x) = 2x^2 + 15x + 1\) and we are trying to find \(df(x)\) at \(x = 10, h = 0.01\).
We can find the absolute truncation error by:
Backward Finite Difference Method
\[df(x) = \frac{f(x)-f(x-h)}{h}\]In addition to the computation of \(f(x)\), this method requires and extra cost of one function evaluation \(f(x-h)\) for a given perturbation, and has truncation order \(O(h) \).
Example
Assume \(f(x) = 2x^2 + 15x + 1\) and we are trying to find \(df(x)\) at \(x = 10, h = 0.01\).
We can find the absolute truncation error by:
Central Finite Difference Method
\[df(x) = \frac{f(x+h)-f(x-h)}{2h}\]This method requires an extra cost of two function evaluations (\(f(x+h)\) and \(f(x-h)\) ) for a given perturbation, and has truncation order \(O(h^2) \). So, we can see that the Central Finite Difference approximation provides better accuracy with possible increased computational cost.
Example
Assume \(f(x) = 2x^2 + 15x + 1\) and we are trying to find \(df(x)\) at \(x = 10, h = 0.01\).
We can find the absolute truncation error by:
Gradient Approximation
Consider a differentiable function \(f(x_1, \dots, x_n):\mathbb{R^n} \rightarrow \mathbb{R}\), where the derivative is defined as the gradient, or
We define the gradient finite difference approximation as
using the forward finite difference method, where \(\delta_i \) is a vector with a \(1\) at position \(i\) and \(0\) elsewhere.
Note: \(df(x) \) can be defined using any finite difference method.
Example
Assume \(f(x_1, x_2) = 2x_1 + x_1^2x_2 + x_2^3\) and we are trying to find the forward finite difference gradient approximation at \(x_1 = 1.3, x_2 = 4.9\) when \(h = 0.05\).
We can find the absolute truncation error by:
Jacobian Approximation
Consider a differentiable function \(f = \begin{bmatrix} f_1(x) & f_2(x) & \dots & f_n(x) \end{bmatrix}:\mathbb{R^n} \rightarrow \mathbb{R^m}\), where the derivative is defined as the Jacobian matrix, or
We define the Jacobian finite difference approximation as
where \(df_i(x_j) \) is the approximation of \(f_i\) at \(x_j\) using any finite difference method.
Example
Assume \(f = \begin{bmatrix} f_1(x) = 2x^2 + 6x_1x_2 & f_2(x) = 3x_1 + 7x_2 \end{bmatrix}\) and we are trying to find the forward finite difference gradient approximation at \(x_1 = 3, x_2 = 7\) when \(h = 0.1\).
We can find the absolute truncation error by:
Reference text: "Scientific Computing: an introductory survey" by Michael Heath
Review Questions
-
What is the general form of a Taylor series?
-
How do you use a Taylor series to approximate a function at a given point?
-
How can you approximate the derivative of a function using Taylor series?
-
How can you approximate the integral of a function using Taylor series?
-
Given an function and a center, can you write out the \(n\)-th degree Taylor polynomial?
-
For an \(n\)-th degree Taylor polynomial, what is the bound on the error of your approximation as a function of distance from the center?
-
For simple functions, can you find the constant \(C\) in the Taylor error bound?
-
Be able to determine how many terms are required for a Taylor series approximation to have less than some given error.