Science Wiki

Statistics is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. It provides methods for describing data, for drawing conclusions about a larger population from a sample, for quantifying uncertainty, and for designing studies so that the data collected can answer the question that was asked. Statistics is a branch of applied mathematics that is built on probability theory. It is also a field of its own, with concepts such as sampling, models, and inference that go beyond mathematics. Statistical methods are used in nearly every empirical field, including the natural and social sciences, medicine, engineering, economics, business, and government.

The word is used in several senses. In the singular, "statistics" names the field. In the plural, "statistics" can mean collections of numerical data, such as the unemployment statistics of a country. In technical usage, a statistic is a quantity that is computed from a sample, such as a sample mean, and an estimator or a test statistic is a statistic that is used for a particular purpose.

This page is an overview of the field. Each of the main branches and methods is described in more detail in its own article, which are linked in the relevant sections.

Early Use of Data

Counting people and goods for the purposes of the state is ancient, and censuses were conducted in Mesopotamia, Egypt, China, and Rome. The English word "statistics" is derived from the German Statistik, which was used in the eighteenth century by Gottfried Achenwall for the study of data about states. The systematic analysis of numerical records began in the seventeenth century. John Graunt analyzed the London bills of mortality in 1662 and estimated population sizes and death rates, and William Petty and Edmond Halley developed methods that led to life tables and to the mathematics of life insurance.

Probability and Error

The mathematical theory of probability began in the correspondence of Blaise Pascal and Pierre de Fermat in 1654 about games of chance, and it was developed by Christiaan Huygens, Jacob Bernoulli (whose Ars Conjectandi of 1713 contains the law of large numbers), Abraham de Moivre, and Thomas Bayes, whose posthumously published essay (1763) gave the first version of what is now called Bayes' theorem. Pierre-Simon Laplace extended the theory and used it for the analysis of observations, and he proved a form of the central limit theorem. In astronomy and geodesy, Adrien-Marie Legendre (1805) and Carl Friedrich Gauss (published 1809, with a claim of earlier use) developed the method of least squares for combining observations with errors, and Gauss connected it to the normal distribution of errors.

Statistics as a Discipline

In the nineteenth century, Adolphe Quetelet applied the normal distribution to human characteristics, and Francis Galton introduced regression and correlation in studies of heredity. Karl Pearson developed the correlation coefficient, the chi-squared test, and the method of moments, and he founded the journal Biometrika and the first university department of statistics at University College London. William Sealy Gosset, writing as "Student", derived the _t_-distribution for small samples.

Ronald Fisher created much of the framework of modern statistics in the 1920s and 1930s, including the analysis of variance, maximum likelihood estimation, the concept of sufficiency, and the principles of the design of experiments, including randomization. Jerzy Neyman and Egon Pearson developed the theory of hypothesis testing with error rates, and Neyman introduced confidence intervals and the theory of survey sampling. Andrey Kolmogorov gave probability theory its axiomatic basis in 1933. Abraham Wald developed sequential analysis and statistical decision theory, and Harold Jeffreys, Bruno de Finetti, and Leonard Savage developed the foundations of Bayesian statistics.

Computing and Recent Developments

John Tukey promoted exploratory data analysis in the 1960s and 1970s. The growth of computing made computationally intensive methods practical. They include the bootstrap (Bradley Efron, 1979), generalized linear models (Nelder and Wedderburn, 1972), the expectation-maximization algorithm (Dempster, Laird, and Rubin, 1977), Markov chain Monte Carlo (popularized for statistics by Gelfand and Smith, 1990), the false discovery rate (Benjamini and Hochberg, 1995), and regularized regression such as the lasso (Tibshirani, 1996). Since the 1990s, the field has overlapped increasingly with machine learning and data science, as data sets have grown in size and dimension.

Populations, Samples, and Variables

A population is the complete set of units (people, animals, cells, transactions, and so on) that are of interest. A sample is a subset that is observed. A parameter is a numerical characteristic of the population, such as its mean, and a statistic is the corresponding quantity that is computed from the sample. When every unit of the population is measured, the study is a census. Otherwise, a sample is used, and statistical inference is needed to say how the sample results relate to the population.

Variables are characteristics that vary among units. They are classified by their scale of measurement (Stevens, 1946):

• Nominal variables are categories without order, such as blood group.
• Ordinal variables have ordered categories, but the distances between categories are not defined, such as a pain rating of mild, moderate, or severe.
• Interval variables have meaningful differences but no true zero, such as temperature in degrees Celsius.
• Ratio variables have a true zero and meaningful ratios, such as mass or time.

Variables are also divided into discrete variables (counts) and continuous variables (measurements), and into categorical and quantitative variables. The scale determines which summaries and methods are appropriate.

Randomness and Variation

Statistical methods assume that data are affected by variation that can be modeled by probability. The source may be random sampling, random assignment in an experiment, measurement error, or natural variation among units. A random variable is a numerical quantity that depends on a random outcome. A probability distribution assigns probabilities to its possible values.

Kolmogorov Axioms and Bayes' Theorem

Probability theory supplies the mathematical language for statistics. In the axiomatic approach of Kolmogorov, a probability space consists of a set of outcomes, a collection of events, and a probability measure that is nonnegative, additive for disjoint events, and equal to 1 for the whole set. Basic results include conditional probability and independence, and Bayes' theorem:

Properties of random variables are summarized by the expected value _E_(X), the variance Var(X) = _E_[(X − _E_(X))²], and, for two variables, the covariance. Important distributions include the normal, binomial, Poisson, exponential, uniform, and the distributions derived from the normal that are used in tests (the chi-squared, t, and F distributions).

Law of Large Numbers and Central Limit Theorem

Two limit theorems are the basis of inference:

• The law of large numbers states that the sample mean of independent, identically distributed observations with a finite mean converges to the population mean as the sample size increases.
• The central limit theorem states that, under general conditions, the standardized sample mean converges in distribution to a standard normal distribution:

Numerical Summary Measures

Descriptive statistics summarize the features of a data set without drawing conclusions beyond it. The main types are:

• Measures of location. The arithmetic mean x̄ = (1/_n_) Σ x_i, the median (the middle value), and the mode (the most frequent value). The median is more robust to outliers than the mean.
• Measures of spread. The sample variance and standard deviation, the range, and the interquartile range. The sample variance uses a divisor of n − 1 so that it is an unbiased estimator of the population variance:

• Measures of shape. Skewness and kurtosis.
• Measures of association. The Pearson correlation coefficient for linear association between two variables:

Graphical Displays and Statistical Paradoxes

Data are also summarized with tables and graphs, such as histograms, box plots, and scatter plots. Summary numbers can hide important features. In Anscombe's quartet, four data sets have nearly identical means, variances, correlations, and regression lines, but very different patterns when plotted, which shows why data should be plotted. Simpson's paradox is the reversal of an association when data from several groups are combined, which shows the need to consider confounding variables.

Statistical inference uses a sample to draw conclusions about a population or process. A statistical model is a set of probability distributions that is assumed to contain the process that generated the data. A model is parametric if it is indexed by a finite number of parameters, nonparametric if it is not, and semiparametric if it has both parts. The main approaches to inference differ in how they interpret probability.

Point Estimation

Estimation is the use of data to assign values to unknown parameters. A point estimator θ̂ is judged by its properties:

• Bias, the difference between its expected value and the true value.
• Variance, the spread of the estimates from sample to sample.
• Mean squared error, which combines the two:

• Consistency (convergence to the true value as the sample grows), efficiency, and sufficiency.

The standard error of an estimator is its standard deviation across repeated samples. For the sample mean it is σ / √_n_, so that to halve the standard error the sample size must be quadrupled. An unbiased estimator is not always the best one, and a small bias can be accepted for a large reduction in variance, as in shrinkage and regularization.

Common methods of estimation include the method of moments, least squares, and maximum likelihood estimation (MLE). The likelihood function _L_(θ) = _p_(x | θ) is the probability (or density) of the observed data as a function of the parameter, and the MLE is the value that maximizes it. Under regularity conditions, the MLE is consistent and asymptotically normal, with variance that attains the Cramér-Rao bound.

Interval Estimation

A confidence interval is an interval computed from the data by a procedure that, over repeated samples, covers the true parameter with a stated probability (the confidence level). For a sample mean with unknown variance, a 95 percent interval is:

The confidence level refers to the procedure and not to a particular interval, which either contains the true value or not. Bayesian credible intervals have a different interpretation, as a probability statement about the parameter given the data and the prior.

Hypothesis Testing

In a hypothesis test, a null hypothesis H₀ (often of no effect) is compared with an alternative H₁ by a test statistic. The _p_-value is the probability, assuming that H₀ and the model assumptions are true, of a test statistic at least as extreme as that observed. The Neyman-Pearson framework specifies a significance level α (the probability of a type I error, rejecting a true null hypothesis) and considers the type II error rate β (failing to reject a false null) and the power 1 − β. The _p_-value is often misinterpreted. It is not the probability that the null hypothesis is true, and it does not measure the size or importance of an effect. A result can be statistically significant and unimportant in practice, particularly in large samples, and the reverse is true for small samples. Effect sizes and confidence intervals are therefore reported with tests. Common tests include the _t_-test, the chi-squared test, Fisher's exact test, the likelihood-ratio test, and the analysis of variance (ANOVA).

The multiple comparisons problem arises when many hypotheses are tested. If m independent tests are done at level α and all null hypotheses are true, the probability of at least one false positive is 1 − (1 − α)^m, which is about 64 percent for m = 20 and α = 0.05. Methods that control the family-wise error rate (such as the Bonferroni correction, which uses α / m) or the false discovery rate (the Benjamini-Hochberg procedure) are used in this case.

Bayesian Inference

In Bayesian inference, unknown quantities are given probability distributions that express uncertainty. A prior distribution _p_(θ) is combined with the likelihood by Bayes' theorem to give the posterior distribution:

Inferences are statements about the posterior, such as credible intervals and the probability of a hypothesis. For example, with a uniform Beta(1, 1) prior for a proportion and 7 successes in 20 trials, the posterior is Beta(8, 14), with mean 8/22 ≈ 0.36. Results depend on the prior, particularly for small samples, so sensitivity analysis is advised. Posterior distributions for realistic models are computed numerically, with Markov chain Monte Carlo, variational methods, or other approximations. The Bayesian and frequentist approaches often give similar answers with large samples and weak priors. Their differences are mainly in interpretation and in how prior information is used.

Other Approaches

Likelihood-based inference uses the likelihood function directly. Statistical decision theory evaluates procedures by the loss they cause, and fiducial inference was an approach of Fisher that is not widely used. Resampling methods, such as the bootstrap, permutation tests, and the jackknife, estimate the sampling distribution from the data themselves, and they are used when theoretical formulas are not available. Cross-validation estimates the predictive performance of a model by repeated splitting of the data.

The quality of the conclusions depends on how the data were collected. Statistics includes methods for the design of studies.

Sampling

Survey sampling studies how to select a sample that represents a population. A simple random sample gives each unit an equal chance of selection. Other designs include stratified sampling, cluster sampling, systematic sampling, and sampling with unequal probabilities. Estimators are weighted by the inverse of the probability of selection, as in the Horvitz-Thompson estimator. Nonresponse, selection bias, and coverage problems can cause bias that does not decrease as the sample grows, which is why a large sample of poor quality can be less informative than a small, well-designed one.

Experiments and Observational Studies

In a designed experiment, the investigator assigns treatments to units. The main principles are replication, randomization, and blocking, and more elaborate designs include factorial designs, split-plot designs, and crossover designs. Randomization balances known and unknown confounders on average, and it supports causal conclusions about the effect of the treatment. A randomized controlled trial is the standard design in medicine. In an observational study, the investigator does not assign the exposure, so that confounding and other biases have to be addressed by design (matching, restriction) and analysis (adjustment). The planning of a study includes a sample size calculation. For comparing two means with a two-sided test, the number needed per group is approximately:

where σ is the standard deviation, δ is the difference to be detected, and the z values are quantiles of the standard normal distribution. For α = 0.05, power of 80 percent, and δ / σ = 0.5, n is about 63 per group.

Regression and Linear Models

Regression analysis models the relationship between a response variable and predictors. In linear regression, y = _X_β + ε, and the least-squares estimate of the coefficients is:

By the Gauss-Markov theorem, this is the best linear unbiased estimator when the errors are uncorrelated with equal variance. Generalized linear models extend the framework to outcomes such as binary data (logistic regression) and counts (Poisson regression) through a link function. Mixed-effects models include random effects to account for clustered or repeated measurements. Generalized additive models and other nonparametric regression methods allow flexible relationships. Regression results describe associations, and they have a causal interpretation only under additional assumptions.

Multivariate Analysis

Multivariate statistics concerns data with several variables per unit. Methods include principal component analysis and factor analysis for reducing the number of dimensions, cluster analysis and discriminant analysis for grouping and classification, canonical correlation, and multivariate analysis of variance. The multivariate normal distribution and the covariance matrix are central.

Time Series

Time series analysis studies data that are ordered in time, in which successive observations are usually dependent. Models include autoregressive (AR), moving-average (MA), and ARIMA models, state-space models and the Kalman filter, GARCH models for changing volatility, and methods of spectral analysis. A basic AR(1) model is X_t = φ X_{_t_−1} + ε_t, which is stationary when |_φ_| < 1. Applications include forecasting, econometrics, and signal processing.

Survival and Event-History Analysis

Survival analysis concerns the time until an event, with censoring when the event has not been observed by the end of follow-up. The Kaplan-Meier estimator estimates the survival function, and the Cox proportional hazards model relates the hazard to predictors. Related areas include reliability analysis and competing risks.

Spatial Statistics

Spatial statistics analyzes data with locations. Areas include geostatistics (which uses the variogram and kriging for prediction), lattice and areal models, and point pattern analysis. A key issue is spatial autocorrelation, that nearby values tend to be more similar than distant ones.

Nonparametric and Robust Methods

Nonparametric statistics uses methods that do not assume a specific distribution, such as rank tests (the Wilcoxon and Kruskal-Wallis tests), kernel density estimation, and the bootstrap. Robust statistics aims for procedures that are not badly affected by outliers or by small departures from the model, such as the median, trimmed means, and M-estimators.

High-Dimensional Data and Statistical Learning

When the number of variables is large compared with the number of observations, methods that regularize the model are required. The lasso and ridge regression penalize the size of coefficients, and the lasso can set some of them to zero, which gives variable selection. The field of statistical learning studies methods for prediction, such as random forests, support vector machines, and neural networks, and it overlaps with machine learning. The bias-variance tradeoff is a central concept: flexible models have low bias and high variance, and simple models have the opposite, so that the best predictive accuracy comes from an intermediate complexity that is chosen by cross-validation or by information criteria such as the AIC and BIC.

Causal Inference

Causal inference is concerned with the effects of interventions. Two main frameworks are the potential outcomes framework (Neyman and Rubin), which defines the causal effect as the difference between the outcomes for a unit under different treatments, and the graphical framework of Judea Pearl, which uses causal diagrams and structural equation models. Methods for observational data include propensity scores, instrumental variables, regression discontinuity design, difference in differences, and Mendelian randomization. Their validity depends on assumptions that cannot be fully tested from the data. The statement that correlation does not imply causation reflects this.

The methods of statistics are used in a large number of fields, and several have developed their own traditions and specialized methods.

Modern statistics depends on computing. Widely used software includes R, Python (with libraries such as NumPy, SciPy, and statsmodels), SAS, Stata, SPSS, Julia, and Bayesian systems such as Stan. Computational methods include simulation, Monte Carlo integration, optimization algorithms for fitting models, and techniques for large data, such as stochastic gradient methods and parallel computing. Reproducible analysis, in which the code and data are shared, is promoted as good practice.

Common Statistical Biases and Pitfalls

Statistical methods can be misused, intentionally or not. Common problems include:

• Selection and bias. Samples that are not representative, survivorship bias, and publication bias, in which significant results are published more often than non-significant ones.
• Multiple testing and flexible analysis. Testing many hypotheses or choosing analyses after looking at the data inflates false positives ("_p_-hacking", the "garden of forking paths").
• Misinterpretation. Reading a _p_-value as the probability that the hypothesis is true, treating non-significance as proof of no effect, confusing statistical with practical significance, and drawing causal conclusions from associations.
• Ecological fallacy and Simpson's paradox. Drawing conclusions about individuals from group data, and ignoring confounders.
• Misleading graphics and summaries. Truncated axes and selective reporting.
• Overfitting. Models that fit noise in the sample and predict poorly on new data.

Reproducibility Reforms and Data Ethics

Concerns about the replication crisis, which refers to the failure to reproduce many published findings in fields such as psychology and medicine, led to reforms such as pre-registration of analyses, larger samples, reporting of effect sizes and intervals, and sharing of data and code. The American Statistical Association published a statement on _p_-values in 2016. Professional codes of ethics, such as that of the American Statistical Association and the Royal Statistical Society, require honest reporting, protection of the privacy of participants, and attention to the use of data. Privacy-preserving methods, such as differential privacy, and concerns about fairness and bias in algorithms are active areas.

Major professional organizations include the American Statistical Association, the Royal Statistical Society, the Institute of Mathematical Statistics, the International Statistical Institute, and the Bernoulli Society. Journals include the Journal of the American Statistical Association, Annals of Statistics, Journal of the Royal Statistical Society, and Biometrika. Statistics is taught as a component of many degrees, and as a major subject at the undergraduate and graduate levels. The International Prize in Statistics, first awarded in 2017, recognizes major contributions to the field.

• Probability theory provides the mathematical foundation.
• Data science and machine learning overlap with statistics, but with a larger emphasis on prediction, algorithms, and scale.
• Information theory has concepts such as entropy and the Kullback-Leibler divergence that are used in statistics.
• Operations research and optimization use statistical models for decisions.
• Econometrics, psychometrics, and biostatistics are statistical fields with their own methods.
• Measurement and metrology address the quality of data that statistics analyzes.

Why does sample variance divide by n − 1 (Bessel's correction) instead of n?

When estimating population variance σ² from a sample, deviations are measured around the sample mean x̄ rather than the unknown true population mean μ. Because x̄ is the arithmetic center of the observed sample, Σ(x_i − x̄)² is systematically smaller than Σ(x_i − μ)² by exactly _n_(x̄ − μ)² on average. Dividing by n − 1 degrees of freedom corrects this downward bias so _E_[s²] = σ².

What is the difference between controlling the Family-Wise Error Rate (FWER) and the False Discovery Rate (FDR)?

When testing m hypotheses simultaneously (such as 20,000 genes in an RNA-Seq experiment), FWER corrections like Bonferroni (α / m) bound the probability of making even a single false positive, which severely reduces statistical power. The Benjamini-Hochberg FDR procedure instead controls the expected proportion of false positives among all rejected null hypotheses, preserving sensitivity in high-dimensional omics.

Why does a _p_-value of 0.04 not mean there is a 4% probability that the null hypothesis is true?

A _p_-value is the conditional probability of observing a test statistic at least as extreme as the sample result given that the null hypothesis H₀ is true: _P_(Data | H₀). Equating _P_(Data | H₀) with the inverse posterior probability _P_(H₀ | Data) is the transpose conditional fallacy; computing _P_(H₀ | Data) via Bayes' theorem requires specifying the prior probability _P_(H₀) and statistical power (1 − β).

• Outline of statistics
• List of statistics articles
• Glossary of probability and statistics
• Probability
• Statistical inference
• Bayesian statistics
• Design of experiments
• Biostatistics
• Data science
• Machine learning