Analytical vs numerical differentiation

[Walter]

The question has come up about the difference between taking a derivative (analytical) vs numerical differentiation.

Let me begin by saying it's not a case of one being superior to the other. While they are different tools for doing a similar job there are advantages and disadvantages to each.

The most notable difference is that we can use numerical methods when we have no idea of what the underlying function is. Given a set of data points obtained through measurement we can gain some insights into what we have measured but measurement comes with inherent errors. We can reduce the amount of error by improving measurement accuracy and averaging over several datasets but it's still a statistical issue. I'm sure we've all seen poll results that include the caveat "95% accuracy 9 times out of 10". What does the "9 times out of 10" signify? In the case of a poll, it signifies that we have taken a poll of a subset of a larger population and the make up or our sample subset may not be representative of the larger population. For example, if our population consists of 5,000 Republican voters and 5,000 Democratic voters it is possible that our same size of 1,000 voters is significantly skewed one way or the other. In the case of motion analysis, it signifies that we have no idea what happens in between measurements. What may appear as a smooth curve in our data set may actually have missed some important data point. Remember, it's a statistical issue.

That's not to say that we've always missed something, it just means that we may have missed something. In addition, when we clean the data set and apply a smoothing function, we are introducing other potential errors. A square wave could become a sine wave, or we could miss a peak value as shown in this diagram:

We also need to be aware that if we perform numerical differentiation on our dataset that the errors are cumulative. If we differentiate displacement to determine velocity, we introduce additional error and in differentiating for acceleration we introduce even more error.

If I have an actual function for something there is no error introduced when I take the derivative.

Why would I even use numerical methods then? The main reason is that I may not have any idea what the underlying function is and, without that underlying function I can't use analytical methods. Taking the derivative of a non-existent function gives me a non-existent function. Also, I may not be able to take the derivate of my function at all points. The derivative of my square wave is undefined at the corner points. Another issue is that the derivative may be difficult to determine or use for analysis purposes. In some cases, an analyst may know the underlying function but find it difficult to work with so they will generate a dataset from the underlying function and use numerical methods and tools to do their analysis.

Sampling, measurement and numerical analysis are important aspects of the scientific method. The laws of physics don't become laws of physics until we confirm them with numerical methods. The laws are typically stated as actual functions though (e.g. work = force x distance), not data sets.

[James9118]

I just chuck the Runge-Kutta at everything these days because I've forgotten how to do maths

[gordonjudd]

We can reduce the amount of error by improving measurement accuracy and averaging over several datasets but it's still a statistical issue.

Walter,
Can you give an example of taking the numerical derivative of a known function (sin(x)) for example and compare its results with the close form value?

I think you will find that the difference between the "approximation" given by the numerical routine and the closed form expression is very, very small, nowhere near 5%.

Smoothing is another issue, but I will take the derivative produced by a smoothed spline fit to the data compared to the noisy derivative that is produced with the adjacent points approach used by Tracker any day. In high speed videos with small delta t time intervals between points that sampling "noise" is overwhelming.

Gordy

[gordonjudd]

I just chuck the Runge-Kutta at everything these days because I've forgotten how to do maths

James,
I am always looking for better numerical differentiation routines. Is there a Runge-Kutta approach for doing numerical differentiation, or does that powerful technique just apply to doing the numerical integrations involved in solving ODE's?

Thanks,
Gordy

[Walter]

Walter,
Can you give an example of taking the numerical derivative of a known function (sin(x)) for example and compare its results with the close form value?

Gordy,

You want to start with a known function such as sin(x) and determine a numerical derivative?

If I start with a known function I can make it as accurate as I want. That’s one of the strengths of numerical analysis. As I mentioned previously, if I know the underlying function I can generate an arbitrarily accurate dataset and then use tools like mathcad to analyze that dataset. Another strength is that I don’t need to know the underlying function to analyze a dataset but if I’m working from a random sample from an experiment and I don’t know the underlying function then I’m basically doing a statistical analysis.

[gordonjudd]

If I start with a known function I can make it as accurate as I want.

Walter,
How about an example to illustrate what you are saying?

and I don’t know the underlying function then I’m basically doing a statistical analysis.

I thought you would be a numerical derivative of the data points rather than some sort of statistical analysis. Can you explain the difference?
Gordy

[Walter]

Gordy,

If I told you my sample set was the points (0,0) and (1,0) can you tell me what the underlying function is? Suppose I were to change my sampling interval and now my sample set was (0,0), (.5 .5), (1,0) can you tell me the underlying function? Now I do different set with greater accuracy and get (0, .1), (.5, .49), and (0,1) am I any closer to knowing the underlying function? If I repeat my experiment 100 times do I pick one arbitrary run that I particularly like or do I use an average or possibly a range of the results? If in 99 of my runs that center value is .49 +/- .3 and get one sample with .6 do I throw out the outlier? Can you see a difference between analyzing based on measurement and analyzing based on knowing the actual underlying function?

[gordonjudd]

Can you see a difference between analyzing based on measurement and analyzing based on knowing the actual underlying function?

Walter,
I don't.

Since we don't have a closed form "actual function" in casting I don't see how you could make that distinction in the way we compute velocity (or) speed data from digitized distance values in casting.

All we can do is to take a numerical derivative (and there are many approaches to choose from) of the measured data to come up with relevant x and y velocity (or speed) values along that measured path.

For example, speed and velocity are time based derivatives of distance and displacement respectively. A derivative is a function, not a value, and in order to determine a derivative you would need to know the function you are deriving it from.

You seem to imply that taking a numerical derivative does not give give accurate speed vs time values. In fact numerical derivatives give the same results as the closed form expression when the underlying data points are the same. I don't think numerical derivative approaches have anything to do with statistical analysis. They just give values that shows how some variable varies with time.

It should be noted that minimizing the ripple that different numerical derivative routines produce is a whole other topic. I much prefer the different derivative approaches that are available in Matlab to the simple discrete delta_x/delta_t approach that is available in Tracker.

Gordy

[Walter]

Since we don't have a closed form "actual function" in casting I don't see how you could make that distinction in the way we compute velocity (or) speed data from digitized distance values in casting.

Gordy,

Exactly.

If I have an actual function, then why do I need to measure anything? If I don't have a formula, then we need to measure.

The distinction in computing speed or velocity is that if I have an actual function describing distance (or displacement), (f(t) = distance) then the process to determine speed (or velocity) is to take a derivative of that function (f'(t)=speed). How to calculate that derivative is based on limit theory calculating rise/run as run approaches zero (not the same thing as dividing by zero). That gives me another function. I use that function to determine the precise speed at any given point in time. At no time do I need to measure anything. The derivative of sine, for example, is cosine. The derivative of cosine is -sine and so on. I didn't need to measure anything to determine that.

If I don't have an actual function, then at some point I need to do some actual measurement to determine distance at various points in time. Numerical differentiation, given that set of measured points, involves computing rise/run between every two sets of points to create another set of points. It cannot compute rise/run at each specific point because dividing by zero is undefined so it's an estimate of the actual derivative.

We don't have actual functions for much of this stuff. I will be using actual formulae in some of my analyses discussing the haul, but it should be obvious when and where I do. Where we don't have formulae then it is necessary to do measurement. We have no other choice in that case. With luck we may develop actual formulae but until we do we have to rely on measurement.

Another issue, as I pointed out, is that given an actual function I may not be able to take a derivative at every point in the function. Numerical differentiation won't care. It will provide some value regardless.

I'm not sure why any of that confuses you. The only area of clarification I was looking for earlier is that you were using the expression "taking the derivative of" which, to me, implies that you have a known function that you are working with. I'm aware that programs that do numerical differentiation, such as tracker and mathcad, may call the function that they are using a derivative function but it's not the same thing as taking a derivative of an actual function.

[gordonjudd]

Can you give an example of taking the numerical derivative of a known function (sin(x)) for example and compare its results with the close form value?
and
In fact numerical derivatives give the same results as the closed form expression when the underlying data points are the same.

Walter,
If you can't provide an example of the expected error then I will.

One case where the measured data should fits with an analytical function is when measuring the natural frequency of a clamped rod. In that test the y tip values will follow the path expected for a damped SHO.

Thus the expected y values will nominally be equal to exp(-zeta*t)*amp*sin(omega*t) where amp is the initial deflection amplitude, zeta is the damping coefficient, and omega is equal to 2*pi*fn where fn is the natural frequency of the rod oscillations. As an example lets assume those y positions for a rod with a natural frequency of 2Hz and an amplitude of 1m are given by y(t)=exp(-.1*t)*sin(4*pi*t).

If I have applied the chain rule correctly (always a big if when I am doing calculations) then the expected y tip velocities would be :
vy(t)=-.1*exp(-.1*t).*sin(omega*t)+omega*(exp(-.1*t).*cos(omega*t)) where omega=4*pi.

A comparison of those closed form velocity values with the corresponding values given by the B-form spline fitting derivative that I use in Matlab is
shown below.

compare_deriv_values.jpg (54.54 KiB) Viewed 121 times

You can see those two curves (closed form in blue), (numerical derivative in green) are essentially overlays of each other.

There is a slight amount of error if you take the difference of those two curves as shown below.

derivative_error.jpg (49.55 KiB) Viewed 121 times

However you can see the relative error is on the order of 1.6e-10 so for all practical purposes the numerical derivative values are the same as the closed form values.

Tracker and mathcad, may call the function that they are using a derivative function but it's not the same thing as taking a derivative of an actual function

As shown above I don't think there is any real practical difference between the values you get from a closed form function and its numerical derivative equivalent. Your assessment is truly a distinction without a difference.

Gordy