Improving Generalization Error Estimates

Alexander Nicholson
Learning Systems Group
California Institute of Technology
Pasadena, CA, 91125


Abstract:

We introduce an algorithm for estimating the out-of-sample
error for a learning problem without reserving a large
independent test set.  Existing theoretical results
are used for the estimates, but are not directly
applicable to a practical problem.
We show how to use a two stage learning process to
overcome the limitation of a random learning algorithm
and a required probability distribution is estimated from
the training data.  The process is illustrated with a
series of experiments and is shown to produce estimates
that can improve on those calculated on a small test set.