Basic Probability Cheat Sheet
September 20, 2018
1 Probability and Expectation
1.1 Bayes Rule
Bayes rule:
p(θ|X) = p(X|θ)p(θ)
Rp(X|θ)p(θ)dθ=p(X|θ)p(θ)
p(X)
If Xrepresents data and θis an unknown quantity of interest, the Bayes rule
can be interpreted as making inference about θbased on the data X(Bayesian
inference) in the form of the posterior distribution p(θ|X).
Remark. In the machine learning course, you will encounter the words ’learning’
and ’inference’. From a Bayesian point of view, there’s no difference between
those two (because everything is expressed by posteriors) . But machine learning
people tend to use ’learning’ as tuning parameters of a model using data and
’inference’ as computing some quantity with the model (sometimes this includes
evaluating a posterior distribution). This distinction is not exhaustive but may
be good to know to avoid confusion.
1.2 Some Useful Formulas of Conditional Expectations
•E[X] = EYEX|Y[X|Y]
•Var[X] = VarYEX|Y[X|Y]+EYVarX|Y[X|Y]
2 Asymptotic Theory
Theorem. The Law of Large Numbers Let X1,X2. . . , be independent iden-
tically distributed (i.i.d.) real random variables. Let Sn=1
nPn
i=1 Xiand
µ=EX1. If E|X1|<∞, then Sn→µas n→ ∞1.
1To be precise, we need to define convergence of random variables.
1
