Kernel Regression Significance Test with Mixed Data Types

npsigtest implements a consistent test of significance of an explanatory variable(s) in a nonparametric regression setting that is analogous to a simple \(t\)-test (\(F\)-test) in a parametric regression setting. The test is based on Racine, Hart, and Li (2006) and Racine (1997).

Usage

npsigtest(bws,
          ...,
          B = 399)

# S3 method for class 'formula'
npsigtest(bws, 
          data = NULL, 
          ...)

# S3 method for class 'npregression'
npsigtest(bws, ...)

# Default S3 method
npsigtest(bws, 
          xdat, 
          ydat, 
          ...)

# S3 method for class 'rbandwidth'
npsigtest(bws,
          xdat = stop("data xdat missing"),
          ydat = stop("data ydat missing"),
          B = 399,
          boot.method = c("iid","wild","wild-rademacher","pairwise"),
          boot.type = c("I","II"),
          pivot=TRUE,
          joint=FALSE,
          index = seq_len(ncol(xdat)),
          random.seed = 42,
          ...)

Arguments

Data, Bandwidth Inputs And Formula Interface

These arguments identify the bandwidth object, formula/data interface, and explicit data inputs.

bws

a bandwidth specification. This can be set as a rbandwidth object returned from a previous invocation, or as a vector of bandwidths, with each element \(i\) corresponding to the bandwidth for column \(i\) in xdat. In either case, the bandwidth supplied will serve as a starting point in the numerical search for optimal bandwidths when using boot.type="II". If specified as a vector, then additional arguments will need to be supplied as necessary to specify the bandwidth type, kernel types, selection methods, and so on.

data

This argument belongs to the formula interface (including supported formula-bearing bandwidth-object refits). Native calls must supply their vectors, matrices or data frames directly, or use with(data, ...). Supplying data, including explicit data = NULL, to a native route that does not consume it is an error; see np-data-interfaces.

an optional data frame, list or environment (or object coercible to a data frame by as.data.frame) containing the variables in the model. If not found in data, the variables are taken from environment(bws), typically the environment from which npregbw was called.

xdat

a \(p\)-variate data frame of explanatory data (training data) used to calculate the regression estimators.

ydat

a one (1) dimensional numeric or integer vector of dependent data, each element \(i\) corresponding to each observation (row) \(i\) of xdat.

Bootstrap And Test Controls

These arguments control bootstrap execution, tested effects, and reproducibility settings.

boot.method: a character string used to specify the bootstrap method for determining the null distribution. pairwise resamples pairwise, while the remaining methods use a residual bootstrap procedure. iid will generate independent identically distributed draws. wild will use a wild bootstrap. wild-rademacher will use a wild bootstrap with Rademacher variables. Defaults to iid.
B: an integer value specifying the number of bootstrap replications to use. Defaults to 399.
boot.type: a character string specifying whether to use a ‘Bootstrap I’ or ‘Bootstrap II’ method (see Racine, Hart, and Li (2006) for details). The ‘Bootstrap II’ method re-runs cross-validation for each bootstrap replication and uses the new cross-validated bandwidth for variable \(i\) and the original ones for the remaining variables. Defaults to boot.type="I".
index: a vector of indices for the columns of xdat for which the test of significance is to be conducted. Defaults to (1,2,...,\(p\)) where \(p\) is the number of columns in xdat.
joint: a logical value which specifies whether to conduct a joint test or individual test. This is to be used in conjunction with index where index contains two or more integers corresponding to the variables being tested, where the integers correspond to the variables in the order in which they appear among the set of explanatory variables in the function call to npreg/npregbw. Defaults to FALSE.
pivot: a logical value specifying whether to studentize the statistic. The default, TRUE, divides continuous derivatives and categorical finite differences by their asymptotic standard errors. This applies to individual and joint tests. FALSE requests the unstandardized statistic for every tested predictor. NULL is no longer accepted.
random.seed: an integer used to seed R's random number generator. This is to ensure replicability. Defaults to 42.

Additional Arguments

Further arguments are passed to the bandwidth-selection routines used by the test.

...: additional arguments supplied to specify the bandwidth type, kernel types, selection methods, and so on, detailed below.

Details

Documentation guide: see np.kernels for kernels, np.options for global options, and plot for plotting options.

npsigtest implements a variety of methods for computing the null distribution of the test statistic and allows the user to investigate the impact of a variety of settings including whether or not to pivot the statistic (pivot), whether pairwise or residual resampling is to be used (boot.method), and whether or not to recompute the bandwidths for the variables being tested (boot.type), among others.

Eligible individual tests with fixed bandwidths, IID residual resampling, and a supported local-polynomial or categorical-only local-constant engine evaluate bootstrap statistics in bounded native tiles. This is an internal computational optimization: it preserves bootstrap draws, random-number state, statistics, and P-values. Joint tests and configurations outside the qualified set continue through the incumbent bootstrap loop.

The categorical statistic of Racine, Hart, and Li (2006) is the unstandardized mean of squared finite-difference estimates, available with pivot=FALSE. The default extends this construction by studentizing categorical effects as well as continuous derivatives. For a joint test, the statistic is the mean of squared componentwise studentized effects over observations and tested predictors; it is not an inverse-covariance Wald statistic.

Studentized statistics use the residual-normalized gradient sandwich described in npreg, including its Ruppert et al. (1997) residual correction. Each bootstrap response supplies its own normalized training residuals; original-response errors are not held fixed across replications. Categorical errors use paired endpoint influence differences and retain endpoint covariance. Observed and bootstrap calculations use the same producer and information-availability rules. Eligible streamed calculations use the equivalent shared influence calculation without retaining a full influence matrix. This does not account for bandwidth or degree-selection uncertainty, or guarantee size calibration for every sample and configuration.

Studentization can reduce size distortion, but does not uniformly improve finite-sample calibration or power. In sparse designs it can be conservative and have lower power than the unstandardized statistic; rejection frequencies from an oversized unstandardized test are not size-adjusted power comparisons. The choice of pivot does not replace assessment of the fitted model and available local information.

Identical categorical endpoints contribute zero without changing the number of observations in the statistic's denominator. A native certificate of identically zero paired influence has the same interpretation; unavailable donor variance alone is not such a certificate. Unordered first-level reference rows have this property; ordered first-level rows instead use a forward difference. An entire finite observed effect vector that is exactly zero has statistic zero under the degenerate-zero convention. Because bootstrap statistics are nonnegative, the inclusive upper-tail P-value is one, without bootstrap fitting or bandwidth reselection. The bootstrap statistics need not also be zero. This applies to an individual test, or a joint test only when every tested observed effect vanishes. A zero component of a mixed joint test stays in the original denominator; it does not suppress the other components. The test has no separate bandwidth-based full-smoothing detector. However, the regression estimator's existing large-categorical-bandwidth approximation can make a categorical contrast exactly zero near the kernel's upper bandwidth bound, thereby triggering this convention. This can produce a discontinuity in the P-value at that approximation's cutoff; it is not evidence that the unapproximated contrast has vanished. The test respects the estimator option np.largelambda, including when the caller explicitly disables the approximation. Random draws are advanced to preserve later tests' random-number streams.

Otherwise a required zero, negative, non-finite or unavailable standard error does not define studentization. An isolated zero effect with zero standard error within a nonzero effect vector is not covered by the degenerate-zero convention unless its influence is separately certified as identically zero. The test stops with an informative diagnostic; it does not insert a variance floor, discard rows or failed replications, or substitute an unstandardized statistic. This rule applies equally to continuous and categorical effects and to observed and bootstrap responses.

When the supplied bandwidth object has regtype="lp", every tested continuous coordinate must have polynomial degree at least one. A degree-zero local polynomial does not identify the derivative used by this significance test, so the function stops before fitting or bootstrap work and names the affected coordinate(s). If the degree was selected by NOMAD, refit the bandwidth object with degree.min=1. This restriction does not apply to an ordinary regtype="lc" bandwidth object.

With nearest-neighbor bandwidths, repeated continuous values can give a required radius of zero. In this case the test stops with the shared nearest-neighbor diagnostic, identifying the affected variable and the failing stage. This can occur during the unrestricted gradient or null-model evaluation, before bootstrap replications. It is not a blanket restriction on repeated values. Refit using bwtype="fixed" (the default) to avoid this nearest-neighbor radius limitation; see np.kernels.

The empirical upper-tail P-value counts bootstrap statistics equal to the observed statistic, that is, \(P=B^{-1}\sum_{b=1}^B 1\{I_b^*\ge I\}\). Thus equality with the bootstrap critical value is non-rejection. In particular, when smoothing makes both the observed statistic and every bootstrap statistic exactly zero, the returned P-value is one rather than zero.

Defaults are chosen so as to provide reasonable behaviour in a broad range of settings and this involves a trade-off between computational expense and finite-sample performance. However, the default boot.type="I", though computationally expedient, can deliver a test that can be slightly over-sized in small sample settings (e.g. at the 5% level the test might reject 8% of the time for samples of size \(n=100\) for some data generating processes). If the default setting (boot.type="I") delivers a P-value that is in the neighborhood (i.e. slightly smaller) of any classical level (e.g. 0.05) and you only have a modest amount of data, it might be prudent to re-run the test using the more computationally intensive boot.type="II" setting to confirm the original result. Note also that boot.method="pairwise" is not recommended for the multivariate local linear estimator due to substantial size distortions that may arise in certain cases.

Bootstrap P-Values

The bootstrap upper-tail p-value is the proportion of bootstrap statistics greater than or equal to the observed statistic. Ties count as at least as extreme, giving the conservative tie convention for discrete statistics. This is the ordinary bootstrap proportion, without an add-one correction; a reported zero is possible with a finite number of replications.

Value

npsigtest returns an object of type sigtest. summary supports sigtest objects. It has the following components:

In: the vector of statistics In
P: the vector of empirical upper-tail P-values for each statistic in In; bootstrap values equal to the observed statistic are included in the upper tail.
In.bootstrap: contains a matrix of the bootstrap replications of the vector In, each column corresponding to replications associated with explanatory variables in xdat indexed by index (e.g., if you selected index = c(1,4) then In.bootstrap will have two columns, the first being the bootstrap replications of In associated with variable 1, the second with variable 4). An analytically skipped test has an NA column, not manufactured bootstrap draws.
pivot: the requested logical pivot mode, TRUE or FALSE.
pivot.effective: a named logical vector recording the statistic actually used for each tested predictor. For a joint test all entries are identical.
bootstrap.executed: the number of bootstrap statistics actually computed for each test (zero for analytic non-rejection).
bootstrap.reason: the structural reason for skipping bootstrap computation, or NA when it was performed. The requested number of replications remains in boot.num.

Book And Method Pointers

npsigtest assesses predictor relevance in nonparametric regression by comparing the fitted unrestricted regression with restrictions implied by excluding or smoothing over selected regressors. Bootstrap choices trade computational cost against finite-sample behavior.

For book-length background, see Li and Racine (2007), Chapter 12 Model Specification Tests, especially Sections 12.3 and 12.3.5, and Chapter 13 Nonsmoothing Tests, especially Section 13.3 where relevant. For the underlying conditional-mean setting, see Racine (2019), Chapter 6 Conditional Mean Function Estimation.

References

Aitchison, J. and C.G.G. Aitken (1976), “Multivariate binary discrimination by the kernel method,” Biometrika, 63, 413-420.

Li, Q. and J.S. Racine (2007), Nonparametric Econometrics: Theory and Practice, Princeton University Press.

Racine, J.S., J. Hart, and Q. Li (2006), “Testing the significance of categorical predictor variables in nonparametric regression models,” Econometric Reviews, 25, 523-544.

Racine, J.S. (1997), “Consistent significance testing for nonparametric regression,” Journal of Business and Economic Statistics 15, 369-379.

Wang, M.C. and J. van Ryzin (1981), “A class of smooth estimators for discrete distributions,” Biometrika, 68, 301-309.

Author

Tristen Hayfield tristen.hayfield@gmail.com, Jeffrey S. Racine racinej@mcmaster.ca

Usage Issues

If you are using data of mixed types, then it is advisable to use the data.frame function to construct your input data and not cbind, since cbind will typically not work as intended on mixed data types and will coerce the data to the same type.

For formula fits, character-valued missing-data policies are resolved through the stats model-frame owner; pass a caller-local policy as a function object rather than its name. With replacement training data, an explicit na.action, including NULL, takes precedence over the saved policy. Native fitting still requires complete rows, so a permissive policy does not guarantee the same retained sample as stats::model.frame.

When reusing a bandwidth object without explicit training or evaluation data, newly constructed bandwidth objects use retained training values. A fit with explicit replacement training data retains that sample in its returned bandwidth object; evaluation grids do not replace training data. Older objects without retained samples use their stored call environment when available and may require explicit training inputs after saveRDS() and readRDS(). Custom formula transformations used on new evaluation data must still be available.

Resampling can repeat continuous observations. With nearest-neighbor bandwidths, an inherited count can then give a zero literal radius, even when it was feasible in the preceding sample. Such a candidate is refused; the resulting error is propagated, not converted to a successful replicate or a different bootstrap method. Inspect ties and nearest-neighbor counts, or explicitly select fixed bandwidths (bwtype="fixed") when this occurs.

Caution: bootstrap methods are, by their nature, computationally intensive. This can be frustrating for users possessing large datasets. For exploratory purposes, you may wish to override the default number of bootstrap replications, say, setting them to B=99. A version of this package using the Rmpi wrapper is under development that allows one to deploy this software in a clustered computing environment to facilitate computation involving large datasets.

Examples

if (FALSE) { # \dontrun{
# EXAMPLE 1 (INTERFACE=FORMULA): For this example, we simulate 100 draws
# from a DGP in which z, the first column of X, is an irrelevant
# discrete variable

set.seed(12345)

n <- 100

z <- rbinom(n,1,.5)
x1 <- rnorm(n)
x2 <- runif(n,-2,2)

y <- x1 + x2 + rnorm(n)

# Next, we must compute bandwidths for our regression model. In this
# case we conduct local linear regression. Note - this may take a few
# minutes depending on the speed of your computer...

bw <- npregbw(formula=y~factor(z)+x1+x2,regtype="ll",bwmethod="cv.aic")

# We then compute a vector of tests corresponding to the columns of
# X. Note - this may take a few minutes depending on the speed of your
# computer... we have to generate the null distribution of the statistic
# for each variable whose significance is being tested using 399
# bootstrap replications for each...

npsigtest(bws=bw)

# If you wished, you could conduct the test for, say, variables 1 and 3
# only, as in

npsigtest(bws=bw,index=c(1,3))

# EXAMPLE 1 (INTERFACE=DATA FRAME): For this example, we simulate 100
# draws from a DGP in which z, the first column of X, is an irrelevant
# discrete variable

set.seed(12345)

n <- 100

z <- rbinom(n,1,.5)
x1 <- rnorm(n)
x2 <- runif(n,-2,2)

X <- data.frame(factor(z),x1,x2)

y <- x1 + x2 + rnorm(n)

# Next, we must compute bandwidths for our regression model. In this
# case we conduct local linear regression. Note - this may take a few
# minutes depending on the speed of your computer...

bw <- npregbw(xdat=X,ydat=y,regtype="ll",bwmethod="cv.aic")

# We then compute a vector of tests corresponding to the columns of
# X. Note - this may take a few minutes depending on the speed of your
# computer... we have to generate the null distribution of the statistic
# for each variable whose significance is being tested using 399
# bootstrap replications for each...

npsigtest(bws=bw)

# If you wished, you could conduct the test for, say, variables 1 and 3
# only, as in

npsigtest(bws=bw,index=c(1,3))
} # }