Foundations of Probability

Core Concepts: Sample Space and Events

The set of all possible outcomes of an experiment is called the sample space (S). Any subset of this space is an event. Probability assigns a number P(A) between 0 and 1 to each event A, with P(S) = 1 and P(∅) = 0 for the impossible event. The probability of the union of two events follows the addition rule: P(A ∪ B) = P(A) + P(B) − P(A ∩ B). When two events are mutually exclusive, P(A ∩ B) = 0 and the formula simplifies. Correctly identifying relationships among events is the first step toward correct calculation.

Conditional Probability, Independence, and the Multiplication Rule

When it is known that event B has occurred, the probability of event A is defined as the conditional probability: P(A|B) = P(A ∩ B) / P(B), provided P(B) > 0. The multiplication rule follows directly: P(A ∩ B) = P(A|B) × P(B). If A and B are independent, then P(A|B) = P(A), and the rule simplifies to P(A ∩ B) = P(A) × P(B). Independence is not the same as uncorrelatedness—confusing these two concepts, especially in causal analysis, leads to serious interpretive errors. The vast majority of statistical models rest on explicitly stated independence assumptions.

Bayes' Theorem and Belief Updating

Bayes' theorem provides a principled way to update prior beliefs in light of new evidence: P(A|B) = [P(B|A) × P(A)] / P(B). Here P(A) is the prior probability (existing knowledge), P(B|A) is the likelihood (how probable the evidence is if A holds), and P(A|B) is the posterior probability (the updated belief). A common misconception is treating P(A|B) and P(B|A) as interchangeable—an error with serious consequences in medicine, law, and the social sciences. Bayes' theorem is both the cornerstone of Bayesian inference beyond frequentist statistics and a fundamental building block of classifiers in machine learning.

Random Variables and Importance in Research Practice

A random variable is a function that assigns a real number to each outcome in a sample space. Discrete random variables take countable values; continuous random variables can take any value within an interval. Every statistical model—regression, ANOVA, logistic regression—is built on at least one random variable. A widespread misconception among researchers is interpreting probability solely as long-run frequency; the Bayesian framework instead treats probability as a degree of belief, and these two interpretations lead to different methodological decisions. Probability literacy is a prerequisite for correctly interpreting core inferential tools such as p-values, power analysis, and confidence intervals.

Sources