Statistical Significance in Machine Learning: Central Limit Theorem & Confidence Intervals, Lecture notes of Machine Learning

Download Statistical Significance in Machine Learning: Central Limit Theorem & Confidence Intervals and more Lecture notes Machine Learning in PDF only on Docsity!

Introduction to

Machine Learning

CH12: STATISTICAL SIGNIFICANCE

100 Randomly

Generated Binary

Numbers

The Central Limit

Theorem

We want to estimate the proportion, p , of 1s. The estimate is to be based on samples of size n. But: Each sample will result in a different estimate The Central Limit Theorem : ◦ (^) The distribution of these estimates roughly follows the Gaussian function ◦ (^) But only if the following two conditions are satisfied: ◦ (^) The distribution is characterized by the following two parameters ◦ (^) The average of the p’ s across different samples approaches that of the original population ◦ (^) The standard error (standard deviation) of this distribution is:

Standard Error of

Sample-Based

Estimates (Example)

Let the size of the testing set be Let the proportion of correct class labels, in this sample, be This is our estimate of classification accuracy Note: both and are satisfied The standard error of the estimate: Therefore: ◦ (^) Classification accuracy is estimated as

Confidence

For the given p and , calculate the confidence --the percentage of estimates that will fall into interval

Interpreting the

Confidence Level

99% of all values are in the interval 95% of all values are in the interval 68% of all values are in the interval

Margin of Error

The confidence interval has the form, Here, is called the margin of error The size of depends on the following: ◦ (^) Level of confidence, affecting ◦ (^) Size of the testing set, ◦ (^) Classification accuracy,

Statistical Evaluation of

a Classifier