function to compare analytic and numeric derivatives

This function compares analytic and numerical derivative and prints related diagnostics information. It is intended for testing and debugging code for analytic derivatives for maximization algorithms.

compareDerivatives(f, grad, hess=NULL, t0, eps=1e-6,
                   printLevel=1, print=printLevel > 0,
                   max.rows=getOption("max.rows", 20),
                   max.cols=getOption("max.cols", 7),
                   ...)

Arguments

f

function to be differentiated. The parameter (vector) of interest must be the first argument. The function may return a vector, in that case the derivative will be a matrix.

grad

analytic gradient. This may be either a function, returning the analytic gradient, or a numeric vector, the pre-computed gradient. The function must use the same set of parameters as f. If f is a vector-valued function, grad must return/be a matrix where the number of rows equals the number of components of f, and the number of columns must equal to the number of components in t0.

hess

function returning the analytic hessian. If present, hessian matrices are compared too. Only appropriate for scalar-valued functions.

t0

numeric vector, parameter at which the derivatives are compared. The derivative is taken with respect to this vector. both fm grad (if function) and hess (if present) must accept this value as the first parameter.

eps

numeric. Step size for numeric differentiation. Central derivative is used.

printLevel

numeric: a positive number prints summary of the comparison. 0 does not do any printing, only returns the comparison results (invisibly).

print

deprecated (for backward compatibility only).

max.rows

maximum number of matrix rows to be printed.

max.cols

maximum number of columns to be printed.

...

further arguments to f, grad and hess.

Details

Analytic derivatives (and Hessian) substantially improve the estimation speed and reliability. However, these are typically hard to program. This utility compares the programmed result and the (internally calculated) numeric derivative. For every component of f, it prints the parameter value, analytic and numeric derivative, and their relative difference $$\textrm{rel.diff} = \frac{\textrm{analytic} - \textrm{numeric}}{\frac{1}{2}(|\textrm{analytic}| + |\textrm{numeric}|)}.$$ If \(\textrm{analytic} = 0\) and \(\textrm{numeric} = 0\), then rel.diff is also set to 0. If analytic derivatives are correct and the function is sufficiently smooth, expect the relative differences to be less than \(10^{-7}\).

Value

A list with following components:

t0

the input argument t0

f.t0

f(t0)

compareGrad

a list with components analytic = grad(t0), nmeric = numericGradient(f, t0), and their rel.diff.

maxRelDiffGrad

max(abs(rel.diff))

If hess is also provided, the following optional components are also present:

compareHessian

a list with components analytic = hess(t0), numeric = numericGradient(grad, t0), and their rel.diff.

maxRelDiffHess

max(abs(rel.diff)) for the Hessian

Author

Ott Toomet otoomet@ut.ee and Spencer Graves

See also

numericGradient deriv

Examples

## A simple example with sin(x)' = cos(x)
f <- function(x) c(sin=sin(x))
Dsin <- compareDerivatives(f, cos, t0=c(angle=1))
#> -------- compare derivatives -------- 
#> Note: analytic gradient is vector.  Transforming into a matrix form
#> Function value:
#> [1] 0.841471
#> Dim of analytic gradient: 1 1 
#>        numeric          : 1 1 
#>       
#> param  theta 0  analytic   numeric      rel.diff
#>   [1,]       1 0.5403023 0.5403023 -5.129895e-11
#> Max relative difference: 5.129895e-11 
#> -------- END of compare derivatives -------- 
##
## Example of normal log-likelihood.  Two-parameter
## function.
##
x <- rnorm(100, 1, 2) # generate rnorm x
l <- function(b) sum(dnorm(x, mean=b[1], sd=b[2], log=TRUE))
gradl <- function(b) {
    c(mu=sum(x - b[1])/b[2]^2,
    sigma=sum((x - b[1])^2/b[2]^3 - 1/b[2]))
}
gradl. <- compareDerivatives(l, gradl, t0=c(mu=1,sigma=2))
#> -------- compare derivatives -------- 
#> Note: analytic gradient is vector.  Transforming into a matrix form
#> Function value:
#> [1] -228.008
#> Dim of analytic gradient: 1 2 
#>        numeric          : 1 2 
#> t0
#>      mu sigma
#> [1,]  1     2
#> analytic gradient
#>            [,1]     [,2]
#> [1,] -0.2964324 16.79941
#> numeric gradient
#>              mu    sigma
#> [1,] -0.2964323 16.79941
#> (anal-num)/(0.5*(abs(anal)+abs(num)))
#>                 mu        sigma
#> [1,] -7.229313e-08 8.435749e-11
#> Max relative difference: 7.229313e-08 
#> -------- END of compare derivatives -------- 

##
## An example with f returning a vector, t0 = a scalar
##
trig <- function(x)c(sin=sin(x), cos=cos(x))
Dtrig <- function(x)c(sin=cos(x), cos=-sin(x))
Dtrig. <- compareDerivatives(trig, Dtrig, t0=1)
#> -------- compare derivatives -------- 
#> Note: analytic gradient is vector.  Transforming into a matrix form
#> Function value:
#> [1] 0.8414710 0.5403023
#> Dim of analytic gradient: 2 1 
#>        numeric          : 2 1 
#>       
#> param  theta 0   analytic    numeric      rel.diff
#>   [1,]       1  0.5403023  0.5403023 -5.129895e-11
#>   [2,]       1 -0.8414710 -0.8414710  9.923691e-11
#> Max relative difference: 9.923691e-11 
#> -------- END of compare derivatives --------