Bayes Theorem

Formal

Definition of conditional probability:

P(A∣B)≜P(A∩B)P(B)(1)P(A|B) \triangleq \frac{P(A \cap B)}{P(B)} \tag{1}

The Derivation:

  1. By the definition, we can write the joint probability P(A∩B)P(A \cap B) in two ways (swapping AA and BB):
P(A∩B)=P(A∣B)P(B)P(B∩A)=P(B∣A)P(A)\begin{align} P(A \cap B) = P (A|B)P(B) \\ P(B \cap A) = P(B|A)P(A) \end{align}
  1. Since P(A∩B)=P(B∩A)P(A\cap B) = P(B\cap A):
P(A∣B)P(B)=P(B∣A)P(A)P(A|B)P(B)=P(B|A)P(A)
  1. Divide by P(B)P(B):
P(A∣B)=P(B∣A)P(A)P(B).(2)P(A|B)=\frac{P(B|A)P(A)}{P(B)}. \tag{2}

ML Context

A→θA \rightarrow \theta (Model Parameters / Hypothesis) B→DB \rightarrow D (Observed Data / Evidence)

P(θ∣D)⏟Posterior=P(D∣θ)⏟Likelihood⋅P(θ)⏟PriorP(D)⏟Evidence(3)\underbrace{P(\theta | D)}_{\text{Posterior}} = \frac{\underbrace{P(D | \theta)}_{\text{Likelihood}} \cdot \underbrace{P(\theta)}_{\text{Prior}}}{\underbrace{P(D)}_{\text{Evidence}}} \tag{3}
Connections
  • #conditional-probability
  • #bayes-theorem