Problems with Estimating Accuracy

• Bias in the estimate - Observed accuracy of the learned hypothesis over the training examples is a optimistically biased estimate of hypothesis accuracy over future examples. Especially likely when the learner considers a very rich hypothesis space, enabling it to overfit the training examples. Typically we test the hypothesis on some set of test examples chosen independently of the training examples and the hypothesis.
• Variance in the Estimate - Even if the hypothesis accuracy is measured over an unbiased set of test examples, the measured accuracy can still vary from true accuracy, depending on the makeup of the particular set of test examples. The smaller the set of test examples, the greater the expected variance.