008_BD_[Peter_Bühlmann,_Petros_Drineas,_Michael_Kane,_Ma_470

234 Handbook of Big Data

ii. P (|Y | ≤ M ) = 1 for some M < ∞,

iii. P (supk,X |ηˆk(X)| ≤ M ) = 1,

iv. (log(Kn))/(n(R0(ηˆk˜) − R0(η0))) −P→ 0 as n → ∞, where −P→ denotes convergence
in probability,

we have that

R0 (ηˆ kˆ ) − R0(η0) −P→ 1 and E [R0 (ηˆ kˆ )] − R0(η0) → 1
R0 (ηˆ k˜ ) − R0(η0) E [R0 (ηˆ k˜ )] − R0(η0)

The proof of the result may be consulted in the original research article, along other useful

convergences and inequalities.
This is a very important result establishing the asymptotic optimality of cross-validation

model selection for prediction, in the sense that a selector based on cross-validation is

asymptotically equivalent to an optimal oracle selector based on the risk under the true,
unknown data generating distribution P0.

In practice, assumptions (ii) and (iii) imply that the outcome must be bounded (i.e., it
may not take on arbitrarily large values), and, perhaps most importantly, that each of the

predictors of the library must respect its bounds. In addition, assumption (iv) is satisfied
if the risk difference R0(ηˆk˜) − R0(η0) is bounded in probability and the ratio log(Kn)/n
converges to zero in probability.

We now illustrate the implication of these asymptotic results with a simulated example.
Simulation setup: Let X = (X1, X2, X3) and Y be drawn from the joint distribution
implied by the conditional distributions

X1 ∼ N (0, 1)

X2|X1 ∼ N (sin(X1), 1)

X3|X2, X1 ∼ N (X1X2, |X1|)
Y |X3, X2, X1 ∼ Ber{expit(cos(X1)|X2| + X32X1 − exp(X3))}

where Ber{p} is the Bernoulli distribution with probability p. Consider three of the
algorithms of Section 13.4: the GLM, GAM, and MARS, and the sample sizes n = 100,
500, 1,000, 5,000, 10,000. In addition, consider a validation sample of size n = 107,
used to approximate the true risks R0(ηˆk) (i.e., approximate integrals with respect to
the measure P0). For each sample size, we draw 500 samples. We then use each of these
samples to compute the selector kˆ and use the validation sample of size 107 to approximate
R0(ηˆk) : k = 1, 2, 3, which includes R0(ηˆkˆ). We proceed to compute R0(ηˆk˜) as mink R0(ηˆk),
and average the results across 500 samples to approximate E[R0(ηˆkˆ)] and E[R0(ηˆk˜)].

Simulation results: Figure 13.4 shows the ratios of the expected risk differences

E[R0(ηˆkˆ)] − R0(η0) (13.5)
E[R0(ηˆk˜)] − R0(η0)

which, as predicted by theory, can be seen converging to 1.
In addition, Table 13.4 shows an approximation to the joint distribution of k˜ and kˆ

for different sample sizes in the simulation described above. Each entry of the table is the
probability that a given algorithm is chosen as kˆ and k˜. These probabilities are approximated
by computing kˆ and k˜ for each of the 500 samples and counting the number of times that

each algorithm is selected. Note that both selectors converge to the MARS as the sample

size increases.

Estimator and Model Selection Using Cross-Validation 235

1.020

1.015

Risk quotient 1.010

1.005

1.000 2.5 3.0 3.5 4.0
2.0 Log 10 of sample size

FIGURE 13.4
Risk difference ratio in Equation 13.5 for different sample sizes

TABLE 13.4
Simulated joint distribution of kˆ and k˜ for
different sample sizes.

n kˆ MARS k˜ GLM
100
500 MARS 0.004 GAM 0.008
1,000 GAM 0.016 0.022
5,000 GLM 0.020 0.080 0.024
10,000 MARS 0.142 0.582 0.000
GAM 0.296 0.244 0.000
GLM 0.000 0.200 0.000
MARS 0.762 0.362 0.000
GAM 0.148 0.000 0.000
GLM 0.000 0.064 0.000
MARS 0.996 0.026 0.000
GAM 0.000 0.000 0.000
GLM 0.000 0.002 0.000
MARS 0.998 0.002 0.000
GAM 0.002 0.000 0.000
GLM 0.000 0.000 0.000
0.000
0.000

13.6.2 Asymptotic Inference for the Cross-Validated Risk Estimator

In addition to selecting an optimal algorithm, estimation of the risk may be an objective
in itself. We approach this problem as a statistical estimation problem and consider the
asymptotic distribution of the statistics involved so that standard errors and confidence
intervals for the risk estimate may be computed. For the cross-validated risk estimator
discussed, such standard errors and confidence intervals may be based on the Gaussian
asymptotic distribution of the estimator. For a general estimator ηˆ, Dudoit and van der

236 Handbook of Big Data

Laan [2] showed that

Rˆ(ηˆ) − R0(ηˆ) = 1 n√ (13.6)
n (L(Zi, η ) − R0(η )) + oP (1/ n)

i=1

where η is a fixed limiting function of the algorithm ηˆ, in the sense that (13.7)
√
n||L(z, ηˆ) − L(z, η )||P0 = oP (1)

assuming that this limit exists. Here we have denoted ||f ||P2 0 = f 2(z)dP0(z), for a function
f of z.

In particular, Equation 13.6 and the central limit theorem imply that

√n(Rˆ(ηˆ) − R0(ηˆ)) −→d N (0, σ2) (13.8)

where σ2 = var{L(Z, η ) − R0(η )}. This result may be used to construct a (1 − α)100%
confidence level interval for the conditional (on the sample) parameter R0(ηˆ) as

Rˆ(ηˆ) ± zα/2 √σˆn

where J
j=1
σˆ2 = 1 1 (L(Zi, ηˆTj ) − Rˆ(ηˆ))2
J |Vj |
i∈Vj

and zα is such that P (X > zα) = α, for a standard Gaussian random variable X. In general,
this cross-validated estimator of the variance should be preferred to its non-cross-validated

version, as the latter may not be appropriate to use√with data-adaptive estimators ηˆ. In our
working example, the estimated standard error σˆk/ n associated to the risks of Table 13.3
are given in Table 13.5.

The asymptotic linearity result given in Equation 13.6 has other additional useful
implications. For example, the joint distribution of the vector Rˆ = (Rˆ(ηˆ1), . . . , Rˆ(ηˆKn))
of estimated risks may be obtained as

√n(Rˆ − R0) −→d N (0, Σ)

where Rˆ0 = (R0(ηˆ1), . . . , R0(ηˆKn)) and Σ is a matrix with (k, l)th element equal to

σkl = cov{L(Z, ηk), L(Z, ηl )}

An important application of this result is testing whether two predictors have the same
conditional risk R0(ηˆ). For example, define θ = R0(ηˆSL) − R0(ηˆDSL) and assume that we
want to test the hypothesis

H0 : θ = 0 vs H1 : θ = 0

TABLE 13.5
Estimated risks and standard deviations in the chemical classification example.

SL DSL GLM GAM SVM RF MARS Mean
Rˆ(ηˆk) 0.0887 0.0931 0.0991 0.0976 0.1022 0.0931 0.1094 0.2239
σˆk 0.0108 0.0105 0.0114 0.0115 0.0107 0.0105 0.0126 0.0001

Estimator and Model Selection Using Cross-Validation 237

of whether the super learner and the discrete super learner provide the same conditional
risk. According to Equation 13.6, the statistic T = Rˆ(ηˆSL) − Rˆ(ηˆDSL) satisfies

√ − θ) −→d N (0, τ2)
n(T

where

τ2 = var{L(Z, ηSL) − L(Z, ηDL)}

Here ηSL and ηDSL are the limits of the super learner and the discrete super learner defined

as in Equation 13.7. √n|T |/τˆ
The hypothesis H0
may be rejected at the α level if > zα, where τˆ2 is the

cross-validated estimate of the variance τ2,

τˆ2 = 1 J 1 (L(Zi, ηˆSL,Tj ) − L(Zi, ηˆDSL,Tj ) − Rˆ(ηˆSL) + Rˆ(ηˆDSL))2
J j=1 |Vj |
i∈Vj

√
In our chemical classification example, the statistic n|T |/τˆ equals 2.1841, which yields

a rejection of the hypothesis of equality of the conditional risks at level 0.05. The

conditional (on the sample) 95% confidence interval for the risk difference θ is equal to

(−0.0083, −0.0004).

Appendix A: R Code

## read data
data <- read.table(’http://archive.ics.uci.edu/ml/

machine-learning-databases/00254/biodeg.csv’, sep=’;’)

## prepare outcome and covariates
X <- data[, -42]
Y <- data[, 42]
Y <- (Y==’RB’)*1

## load package and create library
require(’SuperLearner’)
L <- c(’SL.glm’, ’SL.gam’, ’SL.svm’, ’SL.randomForest’, ’SL.earth’, ’SL.mean’)

## super learner
SL <- SuperLearner(Y, X, SL.library = L, family=binomial(), cvControl=list(V=20))

## cross-validation of the super learner
CVSL <- CV.SuperLearner(Y, X, SL.library = L, family=binomial())

## estimate of the standard deviation of the risk of the discrete super learner
RDSL <- var(CVSL$discreteSL.predict - summary(CVSL)$Table[2,2])

## estimate of the standard deviation of the risk of the super learner
RSL <- var(CVSL$SL.predict - summary(CVSL)$Table[1,2])

## estimate of the standard deviation of the risk of the candidates in teh library
RS <- diag(var(t(t(CVSL$library.predict) - summary(CVSL)$Table[-c(1,2),2])))

## statistic for hypothesis test
T <- -diff(summary(CVSL)$Table[1:2,2])
tau <- sqrt(var(CVSL$SL.predict - CVSL$discreteSL.predict))

## test
abs(T) > qnorm(0.975) * tau / sqrt(dim(data)[1])

238 Handbook of Big Data

Appendix B: Description of the Variables of the Example in
Section 13.2

The 42 variables in the dataset are, in order of appearance:
1. SpMax L: Leading eigenvalue from Laplace matrix
2. J Dz(e): Balaban-like index from Barysz matrix weighted by Sanderson electronegativity
3. nHM: Number of heavy atoms
4. F01[N-N]: Frequency of N-N at topological distance 1
5. F04[C-N]: Frequency of C-N at topological distance 4
6. NssssC: Number of atoms of type ssssC
7. nCb-: Number of substituted benzene C(sp2)
8. C%: Percentage of C atoms
9. nCp: Number of terminal primary C(sp3)
10. nO: Number of oxygen atoms
11. F03[C-N]: Frequency of C-N at topological distance 3
12. SdssC: Sum of dssC E-states
13. HyWi B(m): Hyper–Wiener-like index (log function) from Burden matrix weighted by

mass
14. LOC: Lopping centric index
15. SM6 L: Spectral moment of order 6 from Laplace matrix
16. F03[C-O]: Frequency of C-O at topological distance 3
17. Me: Mean atomic Sanderson electronegativity (scaled on carbon atom)
18. Mi: Mean first ionization potential (scaled on carbon atom)
19. nN-N: Number of N hydrazines
20. nArNO2: Number of nitro groups (aromatic)
21. nCRX3: Number of CRX3
22. SpPosA B(p): Normalized spectral positive sum from Burden matrix weighted by

polarizability
23. nCIR: Number of circuits
24. B01[C-Br]: Presence/absence of C-Br at topological distance 1
25. B03[C-Cl]: Presence/absence of C-Cl at topological distance 3
26. N-073: Ar2NH/Ar3N/Ar2N-Al / R..N..R

Estimator and Model Selection Using Cross-Validation 239

27. SpMax A: Leading eigenvalue from adjacency matrix (Lovasz–Pelikan index)
28. Psi i 1d: Intrinsic state pseudoconnectivity index-type 1d
29. B04[C-Br]: Presence/absence of C-Br at topological distance 4
30. SdO: Sum of dO E-states
31. TI2 L: Second Mohar index from Laplace matrix
32. nCrt: Number of ring tertiary C(sp3)
33. C-026: R–CX–R
34. F02[C-N]: Frequency of C-N at topological distance 2
35. nHDon: Number of donor atoms for H-bonds (N and O)
36. SpMax B(m): Leading eigenvalue from Burden matrix weighted by mass
37. Psi i A: Intrinsic state pseudoconnectivity index-type S average
38. nN: Number of nitrogen atoms
39. SM6 B(m): Spectral moment of order 6 from Burden matrix weighted by mass
40. nArCOOR: Number of esters (aromatic)
41. nX: Number of halogen atoms
42. experimental class: ready biodegradable (RB) and not ready biodegradable (NRB)

References

1. L. Breiman. The little bootstrap and other methods for dimensionality selection in
regression: x-fixed prediction error. J. Am. Stat. Assoc., 87(419):738–754, 1992.

2. S. Dudoit and M. J. van der Laan. Asymptotics of cross-validated risk estimation in
estimator selection and performance assessment. Stat. Methodol., 2(2):131–154, 2005.

3. M. Lichman. UCI machine learning repository, School of Information and Computer
Science, University of California, Irvine, CA, http://archive.ics.uci.edu/ml, 2013.

4. K. Mansouri, T. Ringsted, D. Ballabio, R. Todeschini, and V. Consonni. Quantitative
structureactivity relationship models for ready biodegradability of chemicals. J. Chem.
Inf. Model., 53(4):867–878, 2013.

5. M. J. van der Laan, E. Polley, and A. Hubbard. Super learner. Stat. Appl. Genetics
Mol. Biol., 6:25, 2007.

This page intentionally left blank

14

Stochastic Gradient Methods for Principled
Estimation with Large Datasets

Panos Toulis and Edoardo M. Airoldi

CONTENTS 241
245
14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
14.1.1 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
247
14.2 Stochastic Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
14.2.1 Sakrison’s Recursive Estimation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
249
14.3 Estimation with Stochastic Gradient Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
14.3.1 Bias and Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
14.3.2 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
14.3.3 Choice of Learning Rate Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
14.3.4 Efficient Computation of Implicit Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
14.3.5 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
14.3.5.1 Second-Order Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
14.3.5.2 Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
14.3.5.3 Monte Carlo Stochastic Gradient Descent . . . . . . . . . . . . . . . . . 257
258
14.4 Other Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
14.4.1 Online EM Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
14.4.2 MCMC Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.4.3 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14.4.4 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14.5 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14.1 Introduction

Parameter estimation by optimization of an objective function, such as maximum-likelihood
and maximum a posteriori, is a fundamental idea in statistics and machine learning (Fisher,
1922, Lehmann and Casella, 2003, Hastie et al., 2011). However, widely used optimization-
based estimation algorithms, such as Fisher scoring, the Expectation-Maximization (EM)
algorithm, and iteratively reweighted least squares (Fisher, 1925, Dempster et al., 1977,
Green, 1984), are not scalable to modern datasets with hundreds of millions of data points
and hundreds or thousands of covariates (National Research Council, 2013).

To illustrate, let us consider the problem of estimating the true parameter value θ ∈
Rp from an i.i.d. sample D = {Xn, Yn}, for n = 1, 2, . . . , N ; Xn ∈ Rp is the covariate

241

242 Handbook of Big Data

vector and Yn ∈ Rd is the outcome distributed conditionally on Xn according to the known
distribution f and unknown model parameters θ ,

Yn|Xn ∼ f (·; Xn, θ )

We assume that the data points (Xn, Yn) are observed in sequence (streaming data). The

log-likelihood, log f (Y ; X, θ), as a function of the parameter value θ given a data point

(X, Y ), will be denoted by (θ; Y, X); for brevity, we define (θ; D) = N (θ; Xn, Yn) as
n=1

the complete data log-likelihood.

Traditional estimation methods are typically iterative and have a running time com-

plexity that ranges between O(N p3) and O(N p) in worst cases and best cases, respectively.

Newton–Raphson methods, for instance, update an estimate θn−1 of the parameters through

the recursion

θn = θn−1 − Hn−−11∇ (θn−1; D) (14.1)

where Hn = ∇∇ (θn; D) is the p × p Hessian matrix of the complete data log-likelihood.
The matrix inversion and the likelihood computation over the dataset D imply complexity
O(N p2+ ), which makes the algorithm unsuited for estimation with large datasets. Fisher

scoring replaces the Hessian matrix in Equation 14.1 with its expected value over a data

point (Xn, Yn), that is, it uses the Fisher information matrix I(θ) = −E (∇∇ (θ; Xn, Yn)).
The advantage of this method is that a steady increase in the likelihood is possible because

the difference

(θ + Δθ; D) − (θ; D) ≈ (θ; D) I(θ)−1 (θ; D) + O( 2)

can be made positive for an appropriately small value > 0, because I(θ) is positive definite.
However, Fisher scoring is computationally comparable to Newton–Raphson’s, and thus it is
also unsuited for estimation with large datasets. Other general estimation algorithms, such
as EM or iteratively reweighted least squares (Green, 1984), have similar computational
constraints.

Quasi-Newton methods are a powerful alternative that is widely used in practice. In
quasi-Newton methods, the Hessian in the Newton–Raphson algorithm is approximated by
a low-rank matrix that is updated at each iteration as new values of the gradient become
available. This yields an algorithm with complexity O(N p2) or O(N p) in certain favorable
cases (Hennig and Kiefel, 2013).

However, estimation with large datasets requires complexity that scales linearly with
N , the number of data points, but sublinearly with p, the number of parameters. The first
requirement on N seems hard to overcome, because each data point carries information
for θ by the i.i.d. data assumption. Therefore, gracious scaling with p is necessary.

Such computational requirements have recently sparked interest in procedures that
utilize only first-order information, that is, methods that utilize only the gradient function.
A prominent procedure that fits this description is stochastic gradient descent (SGD),
defined through the iteration

θsngd = θnsg−d1 + an∇ (θsng−d1; Xn, Yn) (14.2)

We will refer to Equation 14.2 as SGD with explicit updates or explicit SGD for short,
because the next iterate θnsgd can be computed immediately after the new data point
(Xn, Yn) is observed. The sequence an > 0 is known as the learning rate sequence, typically
defined such that nan → α > 0, as n → ∞. The parameter α > 0 is the learning rate
parameter, and it is crucial for the convergence and stability of explicit SGD.

From a computational perspective, the SGD procedure (Equation 14.2) is appealing

because the expensive inversion of p × p matrices, as in Newton–Raphson, is replaced by a

Stochastic Gradient Methods for Principled Estimation with Large Datasets 243

single sequence of scalars an > 0. Furthermore, the log-likelihood is evaluated at a single
data point (Xn, Yn), and not on the entire dataset D.

From a theoretical perspective, the explicit SGD procedure is justified because

Equation 14.2 is a special case of the stochastic approximation method of Robbins and

Monro (1951). By the theory of stochastic approximation, explicit SGD converges to a
point θ∞ that satisfies E (∇ (θ∞; X, Y)) = 0; under typical regularity conditions, θ∞ is
exactly the true parameter value θ . As a recursive statistical estimation method, explicit
SGD was first proposed by Sakrison (1965) in a simple second-order form, that is, using

the Fisher information matrix in iteration (Equation 14.2); the simplicity of SGD has also

made it very popular in optimization and machine learning with large datasets (Le Cun
and Bottou, 2004, Zhang, 2004, Spall, 2005).∗

However, the remarkable simplicity of explicit SGD comes at a price, as the SGD
procedure requires careful tuning of the learning rate parameter α. For small values of
α, the iterates θsngd will converge very slowly to θ (large bias), whereas for large values of
α, the iterates θnsgd will either have a large asymptotic variance (with respect to random
data D) or even diverge numerically. In large datasets with many parameters (large p),
the balance between bias, variance, and stability is very delicate, and nearly impossible to

achieve without appropriately modifying Equation 14.2.

Interestingly, the simple modification of explicit SGD defined through the iteration

θinm = θnim−1 + an∇ (θinm; Xn, Yn) (14.3)

can resolve its stability issue virtually at no cost. We will refer to Equation 14.3 as implicit

stochastic gradient descent or implicit SGD for short (Toulis et al., 2014, Toulis and Airoldi,
2015b). Equation 14.3 is implicit because the next iterate θnim appears on both sides of the
equation. This equation is a p-dimensional fixed-point equation, which is generally hard to
solve. However, for a large family of statistical models, it can be reduced to a one-dimensional

fixed-point equation. We discuss computational issues of implicit SGD in Section 14.3.4.

The first intuition for implicit SGD is obtained using a Taylor expansion of the implicit
update (Equation 14.3). In particular, assuming a common point θnsg−d1 = θnim−1 = θ, a Taylor
expansion of Equation 14.3 around θinm−1 implies

Δθinm = I + anIˆ(θ; Xn, Yn) −1 Δθsngd + O(a2n) (14.4)

where
Δθn = θn − θn−1 for both explicit and implicit methods
Iˆ(θ; Xn, Yn) = −∇∇ (θ; Xn, Yn) is the observed Fisher information matrix
I is the p × p identity matrix

Thus, implicit SGD uses updates that are a shrinked version of the explicit ones. The
shrinkage factor in Equation 14.4 depends on the observed information up to the nth data
point and is similar to shrinkage in ridge regression.

Naturally, implicit SGD has also a Bayesian interpretation. In particular, if the log-
likelihood is continuously differentiable, then the update in Equation 14.3 is equivalent to
the update

θnim = arg max an (θ; Xn, Yn) − 1 ||θ − θnim−1 ||2 (14.5)
2
θ∈Rp

∗Recursive estimation methods using stochastic approximation were originally developed for problems
with streaming data. However, these methods are more broadly applicable to estimation with a static
dataset. Asymptotically, these two scenarios are equivalent, with the estimates converging to the true
parameter value θ . Estimates with a static dataset (i.e., with a finite sample) converge instead to the point
that minimizes some predefined empirical loss, for example, based on the likelihood.

244 Handbook of Big Data

The iterate θnim from Equation 14.5 is the posterior mode of the following Bayesian model:

θ|θinm−1 ∼ N (θinm−1, anI) (14.6)
Yn|Xn, θ ∼ f (·; Xn, θ)

where N denotes the normal distribution. Therefore, the learning rate an relates to the
information received after n data points have been observed, and encodes our trust on
the current estimate θinm−1. The Bayesian formulation (Equation 14.6) demonstrates the
flexibility of implicit SGD. For example, depending on the parameter space of θ , the
Bayesian model in Equation 14.6 could be different; for instance, if θ was a scale parameter,
then the normal distribution could be replaced by an inverse chi-squared distribution.
Furthermore, instead of anI as the prior variance, it would be statistically efficient to use the
Fisher information matrix (1/n)I(θnim−1)−1, completely analogous to Sakrison’s method—we
discuss these ideas in Section 14.3.5.1.

There is also a tight connection of Equation 14.5 to proximal methods in optimization.
For example, if we replaced the stochastic component (θ; Xn, Yn) in Equation 14.5 with
the complete data log-likelihood (θ; D), then Equation 14.5 would be essentially the
proximal point algorithm of Rockafellar (1976) that applies to deterministic settings. This

algorithm is known for its numerical stability, and has been generalized through the idea

of splitting algorithms (Lions and Mercier, 1979); see Parikh and Boyd (2013) for a
comprehensive review. The convergence of proximal methods with a stochastic component,

as in Equation 14.5, has been analyzed recently—under various forms and assumptions—
by Bertsekas (2011), Ryu and Boyd (2014), and Rosasco et al. (2014). From a statistical
perspective, Toulis and Airoldi (2015a) derived the asymptotic variance of θnsgd and θnim as
estimators of θ , and provided an algorithm to efficiently compute Equation 14.5 for the
family of generalized linear models—we show a generalization of this result in Section 14.3.4.

In the online learning literature, regret analyses of implicit methods have been given by

Kivinen et al. (2006) and Kulis and Bartlett (2010). Further intuitions for proximal methods
(Equation 14.5) have been given by Krakowski et al. (2007) and Nemirovski et al. (2009),

who showed that proximal methods can fit better in the geometry of the parameter space.
Finally, (Toulis et al., 2015) derived a stochastic approximation framework that encompasses

stochastic proximal methods, and showed that it retains the asymptotic properties of the

framework of Robbins and Monro, but is significantly more robust in the non-asymptotic
regime.

Arguably, the normalized least mean squares (NLMS) filter (Nagumo and Noda, 1967)

was the first statistical model that used an implicit update as in Equation 14.3 and was
shown to be robust to input noise (Slock, 1993). Two other recent stochastic proximal meth-

ods are Prox-SVRG (Xiao and Zhang, 2014) and Prox-SAG (Schmidt et al., 2013, section 6).
The main idea in both methods is to replace the gradient in Equation 14.5 with an estimate

of the full gradient averaged over all data points that has the same expectation with the

gradient of Equation 14.3 but smaller variance. Because of their operational complexity, we
will not discuss these methods further. Instead, in Section 14.3.5.1, we will discuss a related

proximal method, namely AdaGrad (Duchi et al., 2011), that maintains one learning rate for

each parameter component and updates these learning rates as new data points are observed.

Example 14.1 Consider the linear normal model, Yn|Xn ∼ N (Xnθ , 1). The log-likelihood
1 )2.
for this model is (θ; Xn, Yn) = − 2 (Yn − Xn θ Therefore, the explicit SGD procedure

will be

θnsgd = θsng−d1 + an(Yn − Xn θnsg−d1)Xn = (I − anXnXn )θnsg−d1 + anYnXn (14.7)

Equation 14.7 is known as the least mean squares filter (LMS) in signal processing, or as the
Widrow–Hoff algorithm (Widrow and Hoff, 1960), and it is a special case of explicit SGD.

Stochastic Gradient Methods for Principled Estimation with Large Datasets 245

The stability problems of explicit SGD become apparent by inspection of Equation 14.7; a
misspecification of an can lead to a poor next iterate θsngd, for example, when I − anXnXn
has large negative eigenvalues—we discuss these issues in Section 14.3.2.

The implicit SGD procedure for the linear model is

θnim = θinm−1 + an(Yn − Xn θinm)Xn ⇒ (14.8)
θnim = (I + anXnXn)−1θinm−1 + an(I + anXnXn)−1YnXn

Equation 14.8 is known as the NLMS filter in signal processing (Nagumo and Noda, 1967).
In contrast to explicit SGD, the implicit iterate θinm is a weighted average between the
previous iterate θinm−1 and the new observation YnXn, which is now stable to misspecifications
of the learning rate an.

14.1.1 Outline

The structure of this chapter is as follows. In Section 14.2, we give an overview of the
Robbins–Monro procedure and Sakrison’s recursive estimation method, which provide the
theoretical basis for SGD methods. In Section 14.3, we introduce a simple generalization
of explicit and implicit SGD, and we analyze them as statistical estimation procedures
for the model parameters θ after n data points have been observed. In Section 14.3.1,
we give results on the frequentist statistical properties of SGD estimators, that is, their
asymptotic bias and variance across multiple realizations of the dataset D. We then leverage
these results to study optimal learning rate sequences an (Section 14.3.3), the loss of
statistical information in SGD, and numerical stability (Section 14.3.2). In Section 14.3.5,
we illustrate three extensions of the SGD methods, in particular: (1) second-order SGD
methods (Section 14.3.5.1), which adaptively approximate the Fisher information matrix;
(2) averaged SGD methods, which use larger learning rates together with averaging of the
iterates; and (3) Monte Carlo SGD methods, which can be applied when the log-likelihood
cannot be efficiently computed. In Section 14.4, we review applications of SGD in statistics
and machine learning, namely, online EM, Markov chain Monte Carlo (MCMC) posterior
sampling, reinforcement learning, and deep learning.

14.2 Stochastic Approximation

Consider a random variable H(θ) that depends on parameter θ; for simplicity, assume that

H(θ) and θ are real numbers. The regression function, h(θ) = E (H(θ)), is decreasing but

possibly unknown. Robbins and Monro (1951) considered the problem of finding the unique

point θ for which h(θ ) = 0. They devised a procedure, known as the Robbins–Monro

procedure, in which an estimate θn−1 of θ is utilized to sample one new data point H(θn−1);
by definition, E (H(θn−1)| θn−1) = h(θn−1). The estimate is then updated according to the

following rule:

θn = θn−1 + anH(θn−1) (14.9)

The scalar an > 0 is the learning rate and should decay to zero, but not too fast to guarantee
convergence. Robbins and Monro (1951) proved that E (θn − θ )2 → 0, if

a. E H(θ)2 θ) < ∞, for any θ, and

b. ∞ ai = ∞ and ∞ a2i < ∞.
i=1 i=1

246 Handbook of Big Data

Extensions to multiple dimensions were soon given by Blum (1954). The necessary
conditions for convergence in such cases are the negative definiteness of the Jacobian of
h, or that H is the stochastic gradient of a function with unique zero (Wei, 1987, Ruppert,
1988b, section 4).

The original proof of Robbins and Monro (1951) is technical but the main idea is
straightforward. Let bn E (θn − θ )2 denote the squared error of the iterates in
Equation 14.9; then from iteration (Equation 14.9), it follows:

bn = bn−1 + 2anE ((θn−1 − θ )h(θn−1)) + a2nE H(θn−1)2

In the neighborhood of θ , we assume that h(θn−1) ≈ h (θ )(θn−1 − θ ), and thus

bn = (1 + 2anh (θ ))bn−1 + a2nE H(θn−1)2 (14.10)

For a learning rate an = α/n, using typical techniques in stochastic approximation
(Chung, 1954), we can derive from Equation 14.10 that bn → 0. Furthermore, nbn →
α2σ2(2α|h (θ )| − 1)−1, where σ2 E H(θ )2 , as shown by several authors (Chung,
1954, Sacks, 1958, Fabian, 1968). Clearly, the learning rate parameter α is critical for
the performance of the Robbins–Monro procedure. Its optimal value is α = 1/h (θ ),
which requires knowledge of the true parameter value θ and the slope of h at that
point. This optimality result inspired an important line of research on adaptive stochastic

approximation methods, such as the Venter process (Venter, 1967), in which quantities that
are important for the convergence and efficiency of iterates θn (e.g., the quantity h (θ ))
are being estimated as the stochastic approximation proceeds.

14.2.1 Sakrison’s Recursive Estimation Method

Although initially motivated by sequential experiment design, the Robbins–Monro proce-
dure was soon modified for statistical estimation. Similar to the estimation setup in Section
14.1, Sakrison (1965) was interested in estimating the parameters θ of a model that
generated i.i.d. observations (Xn, Yn), in a way that is computationally and statistically
efficient. Sakrison first recognized that one could set H(θ) ∇ log (θ; Xn, Yn) in the
Robbins–Monro procedure (Equation 14.9), and use the identity E (∇ (θ ; Xn, Yn)) = 0
to show why the procedure will converge to the true parameter value θ . Sakrison’s
recursive estimation method was essentially the first explicit SGD method proposed in the
literature:

θn = θn−1 + anI(θn−1)−1∇ (θn−1; Xn, Yn) (14.11)

where an is a learning rate sequence that satisfies the Robbins–Monro conditions in Section
14.2. The SGD procedure (Equation 14.11) is second order, because it uses the Fisher infor-
mation matrix in addition to the log-likelihood gradient. By the theory of stochastic approxi-
mation, θn → θ , and thus I(θn) → I(θ ). Sakrison (1965) proved that nE ||θn − θ ||2 →
trace(I(θ )−1), which indicates that estimation of θ is asymptotically optimal, that is, it
achieves the minimum variance of the maximum-likelihood estimator. However, Sakrison’s
method is not computationally efficient, as it requires an expensive matrix inversion at every
iteration. Still, it reveals that the estimation of the Fisher information matrix is essential
for optimal SGD. Adaptive second-order methods leverage this insight to approximate the
Fisher information matrix and improve upon first-order SGD methods.

Stochastic Gradient Methods for Principled Estimation with Large Datasets 247

14.3 Estimation with Stochastic Gradient Methods

We slightly generalize the SGD methods in Section 14.1 through the definitions

θsngd = θnsg−d1 + anC∇ (θsng−d1; Xn, Yn) (14.12)

θinm = θinm−1 + anC∇ (θinm; Xn, Yn) (14.13)

where C is symmetric and positive definite, and commutes with I(θ ); adaptive second-order

methods where C is updated at every iteration are discussed in Section 14.3.5.1. The iterate
θnsgd is the explicit SGD estimator of θ after the nth data point has been observed; similarly,
θinm is the implicit SGD estimator of θ . The total number of data points, denoted by N ,

will be assumed to be practically infinite. We will then compare the asymptotic variance of

those estimators with the variance of the maximum-likelihood estimator on n data points,

which, under typical regularity conditions, has variance 1 I (θ )−1. The evaluation is done
n
from a frequentist perspective, that is, across multiple realizations of the dataset up to

n data points D = {(X1, Y1), (X2, Y2), . . . , (Xn, Yn)}, under the same model f and true
parameter value θ .∗

Typically, both SGD methods have two phases, namely the exploration phase and the

convergence phase (Amari, 1998, Bottou and Murata, 2002). In the exploration phase, the

iterates approach θ , whereas in the convergence phase, they jitter around θ within a

ball of slowly decreasing radius. We will overview a typical analysis of SGD in the final

convergence phase, where a Taylor approximation in the neighborhood of θ is assumed

accurate (Murata, 1998, Toulis et al., 2014). In particular, let μ(θ) = E (∇ (θ; Xn, Yn)),
and assume

μ(θn) = μ(θ ) + Jμ(θ )(θn − θ ) + o(an) (14.14)

where:
Jμ is the Jacobian of the function μ(·)
o(an) denotes a vector sequence with norms of order o(an)

Under typical regularity conditions, μ(θ ) = 0 and Jμ(θ ) = −I(θ ) (Lehmann and Casella,
1998).

14.3.1 Bias and Variance

Denote the biases of the two SGD methods with E(θnsgd − θ ) bnsgd and E(θinm − θ ) binm.
Then, by taking expectations in Equations 14.12 and 14.13, we obtain the recursions

bnsgd = (I − anCI(θ )) bnsg−d1 + o(an) (14.15)

bnim = (I + anCI(θ ))−1 binm−1 + o(an) (14.16)

We observe that convergence—the rate at which the two methods become unbiased in

the limit—differs in the two SGD methods. The explicit SGD method converges faster
than the implicit one because ||(I − anCI(θ ))|| < ||(I + anCI(θ ))−1||, for sufficiently
large n, but the rates become equal in the limit as an → 0. However, the implicit method
compensates by being more stable in the specification of the learning rate sequence and the

∗This is an important distinction because, traditionally, the focus in optimization has been to obtain fast
convergence to a parameter value that minimizes the empirical loss, for example, the maximum-likelihood.
From a statistical viewpoint, under variability of the data, there is a tradeoff between convergence to an
estimator and the estimator’s asymptotic variance (Le Cun and Bottou, 2004).

248 Handbook of Big Data

condition matrix C. Loosely speaking, the bias binm cannot be much worse than bnim−1 because
(I + anCI(θ ))−1 is a contraction matrix, for any choice of an > 0. Exact nonasymptotic
derivations for the bias of explicit SGD are given by Moulines and Bach (2011), and for the

bias of implicit SGD by Toulis and Airoldi (2015a).
Regarding statistical efficiency, Toulis et al. (2014) showed that, if (2CI(θ ) − I/α) is

positive definite, it holds that

nVar(θsngd) → α2(2αCI(θ ) − I)−1CI(θ )C (14.17)
nVar(θinm) → α2(2αCI(θ ) − I)−1CI(θ )C

where α = limn→∞ nan is the learning rate parameter of SGD, as defined in Section 14.1.
Therefore, both SGD methods have the same asymptotic efficiency, which depends on the
learning rate parameter α and the Fisher information matrix I(θ ). Intuitively, the term
(2αCI(θ ) − I)−1 in Equation 14.17 is a factor that shows how much information is lost by
the SGD methods. For example, setting C = I(θ )−1 and α = 1, implies (2αCI(θ )−I)−1 =
I, and the asymptotic variance for both estimators is (1/n)I(θ )−1, that is, it is the minimum

variance attainable by the maximum-likelihood estimator. This is exactly Sakrison’s result

presented in Section 14.2.1.

Asymptotic variance results similar to Equation 14.17, but not in the context of model

estimation, were first studied in the stochastic approximation literature by Chung (1954),

Sacks (1958), and followed by Fabian (1968) and several other authors (see also Ljung et al.,

1992, parts I, II), where more general formulas are possible using a Lyapunov equation.

14.3.2 Stability

Stability has been a well-known issue for explicit SGD (Gardner, 1984, Amari et al.,

1997). In practice, the main problem is that the learning rate sequence an needs to agree
with the eigenvalues of the Fisher information matrix I(θ ). To see this, let us simplify
Equations 14.15 and 14.16 by dropping the remainder terms o(an). It follows that

bnsgd = (I − anCI(θ ))bnsg−d1 = P1nb0

bnim = (I + anCI(θ ))−1binm−1 = Q1nb0 (14.18)

where P1n = in=1(I − aiCI(θ )), Q1n = n (I + aiC I (θ ))−1, and b0 denotes the initial
i=1
bias of the two procedures from a common starting point θ0. The matrices P1n and Q1n

describe how fast the initial bias b0 decays for both SGD methods. For small-to-moderate

n, the two matrices critically affect the stability of SGD methods. For simplicity, we compare

those matrices assuming rate an = α/n and a fixed condition matrix C = I.
Under such assumptions, the eigenvalues of P1n can be calculated as λi = jn=1(1 −

αλi/j) = O(n−αλi ), for 0 < αλi < 1, where λi are the eigenvalues of the Fisher information
matrix I(θ ). Thus, the magnitude of P1n will be dominated by λmax, the maximum
eigenvalue of I(θ ), and the rate of convergence to zero will be dominated by λmin, the

minimum eigenvalue of I(θ ). For stable eigenvalues λi, the terms in the aforementioned
product need to be less than 1; therefore, it is desirable that |1 − αλmax| ≤ 1 ⇒ α ≤ 2/λmax.

For statistical efficiency, it is desirable that (2αI(θ ) − I) is positive definite, as shown in

Equation 14.17, and so α > 1/(2λmin). In high-dimensional settings, the conditions for
stability and efficiency are hard to satisfy simultaneously, because λmax is usually much
larger than λmin. Thus, in explicit SGD, a small learning rate can guarantee stability, but
this comes at a price in convergence, which will be at the order of O(n−αλmin). On the other

hand, a large learning rate increases the convergence rate but it comes at a price in stability.





























































Learning Structured Distributions 279

15.8 Conclusions and Future Directions

In this chapter, we gave a biased survey of the distribution learning literature from a
computer science perspective, that is, with an explicit focus on the computational efficiency
of our estimators. We presented recent work in theoretical computer science on the design of
sample-optimal and computationally efficient estimators. Many important questions remain
open, and the interplay between algorithms and statistics is crucial to their resolution. We
conclude this chapter with two important research directions.

One of the most important challenges in statistical learning is handling data that are
corrupted by noise. In most cases, the difficulty is not information-theoretic but rather
computational. Many popular algorithms (e.g., MLE) are not tolerant to even a small
amount of noise in the data. A fundamental gap in our understanding concerns high-
dimensional problems.

Research Direction 15.1. Develop computationally efficient agnostic learning algorithms
for high-dimensional distribution learning problems.

A concrete problem is that of learning a binary product distribution with adversarial noise.
This problem is, of course, straightforward in the noiseless setting; however, it becomes
very challenging in the presence of even a small constant fraction of noisy observations. A
more challenging open problem is agnostically learning mixture models, for example, for
two high-dimensional Gaussians or even binary product distributions.

Overall, the body of work on statistical estimation has focused on worst-case instances
both in terms of algorithms and lower bounds. A natural goal is to go beyond worst-case
analysis and design algorithms that provably perform near optimally on every input.

Research Direction 15.2. Develop instance-by-instance optimal algorithms for distribu-
tion learning problems.

We believe that progress in this direction will lead to efficient algorithms that perform very
well in practice.

References

1. F. Balabdaoui, K. Rufibach, and J. A. Wellner. Limit distribution theory for maximum
likelihood estimation of a log-concave density. Annals of Statistics, 37(3):1299–1331,
2009.

2. F. Balabdaoui and J. A. Wellner. Estimation of a k-monotone density: Limit distribution
theory and the spline connection. Annals of Statistics, 35(6):2536–2564, 2007.

3. F. Balabdaoui and J. A. Wellner. Estimation of a k-monotone density: Characteriza-
tions, consistency and minimax lower bounds. Statistica Neerlandica, 64(1):45–70, 2010.

4. R. E. Barlow, D. J. Bartholomew, J. M. Bremner, and H. D. Brunk. Statistical Inference
under Order Restrictions. Wiley, New York, 1972.

5. L. Birg´e. On estimating a density using Hellinger distance and some other strange facts.
Probability Theory and Related Fields, 71(2):271–291, 1986.

280 Handbook of Big Data

6. L. Birg´e. Estimating a density under order restrictions: Nonasymptotic minimax risk.
Annals of Statistics, 15(3):995–1012, 1987.

7. L. Birg´e. On the risk of histograms for estimating decreasing densities. Annals of
Statistics, 15(3):1013–1022, 1987.

8. R. Blei, F. Gao, and W. V. Li. Metric entropy of high dimensional distributions.
Proceedings of the American Mathematical Society (AMS ), 135(12):4009–4018, 2007.

9. H. D. Brunk. On the estimation of parameters restricted by inequalities. The Annals of
Mathematical Statistics, 29(2):437–454, 1958.

10. L. Le Cam. Maximum likelihood: An introduction. International Statistical Review,
58:153–171, 1990.

11. B. Carl and I. Stephani. Entropy, Compactness and the Approximation of Operators,
volume 98 of Cambridge Tracts in Mathematics. Cambridge University Press, Cam-
bridge, 1990.

12. K. S. Chan and H. Tong. Testing for multimodality with dependent data. Biometrika,
91(1):113–123, 2004.

13. S. Chan, I. Diakonikolas, R. Servedio, and X. Sun. Learning mixtures of structured
distributions over discrete domains. In SODA, pp. 1380–1394, 2013.

14. S. Chan, I. Diakonikolas, R. Servedio, and X. Sun. Efficient density estimation via
piecewise polynomial approximation. In STOC, pp. 604–613, 2014.

15. S. Chan, I. Diakonikolas, R. Servedio, and X. Sun. Near-optimal density estimation in
near-linear time using variable-width histograms. In NIPS, pp. 1844–1852, 2014.

16. S. X. Chen and J. S. Liu. Statistical applications of the Poisson-Binomial and conditional
Bernoulli distributions. Statistica Sinica, 7:875–892, 1997.

17. M. Cule and R. Samworth. Maximum likelihood estimation of a multi-dimensional
log-concave density. Journal of the Royal Statistical Society: Series B (Statistical
Methodology), 72:545–607, 2010.

18. M. Cule and R. Samworth. Theoretical properties of the log-concave maximum
likelihood estimator of a multidimensional density. Electronic Journal of Statistics,
4:254–270, 2010.

19. S. Dasgupta. Learning mixtures of Gaussians. In Proceedings of the 40th Annual
Symposium on Foundations of Computer Science, pp. 634–644, 1999.

20. C. Daskalakis, I. Diakonikolas, R. O’Donnell, R. A. Servedio, and L. Tan. Learning
sums of independent integer random variables. In FOCS, pp. 217–226, 2013.

21. C. Daskalakis, I. Diakonikolas, and R. A. Servedio. Learning k-modal distributions via
testing. In SODA, pp. 1371–1385, 2012.

22. C. Daskalakis, I. Diakonikolas, and R. A. Servedio. Learning Poisson binomial distribu-
tions. In STOC, pp. 709–728, 2012.

23. C. Daskalakis, I. Diakonikolas, and R. A. Servedio. Learning k-modal distributions via
testing. Theory of Computing, 10(20):535–570, 2014.

Learning Structured Distributions 281

24. C. Daskalakis and G. Kamath. Faster and sample near-optimal algorithms for proper
learning mixtures of gaussians. In Proceedings of The 27th Conference on Learning
Theory, pp. 1183–1213, 2014.

25. A. De, I. Diakonikolas, and R. Servedio. Learning from satisfying assignments. In
Proceedings of the 26th Annual ACM-SIAM Symposium on Discrete Algorithms, pp.
478–497, 2015.

26. L. Devroye and L. Gy¨orfi. Nonparametric Density Estimation: The L1 View. Wiley,
1985.

27. L. Devroye and G. Lugosi. Combinatorial Methods in Density Estimation. Springer
Series in Statistics, Springer, New York, 2001.

28. D. L. Donoho and I. M. Johnstone. Minimax estimation via wavelet shrinkage. Annals
of Statistics, 26(3):879–921, 1998.

29. D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard. Wavelet shrinkage:
Asymptopia. Journal of the Royal Statistical Society, Series. B, 371–394, 1995.

30. D. L. Donoho, I. M. Johnstone, G. Kerkyacharian, and D. Picard. Density estimation
by wavelet thresholding. Annals of Statistics, 24(2):508–539, 1996.

31. R. M Dudley. Metric entropy of some classes of sets with differentiable boundaries.
Journal of Approximation Theory, 10(3):227–236, 1974.

32. L. Dumbgen and K. Rufibach. Maximum likelihood estimation of a log-concave density
and its distribution function: Basic properties and uniform consistency. Bernoulli,
15(1):40–68, 2009.

33. D. E. Edmunds and H. Triebel. Function Spaces, Entropy Numbers, Differential
Operators, volume 120 of Cambridge Tracts in Mathematics. Cambridge University
Press, Cambridge, 1996.

34. J. Feldman, R. O’Donnell, and R. A. Servedio. Learning mixtures of product distribu-
tions over discrete domains. SIAM Journal on Computing, 37(5):1536–1564, 2008.

35. A.-L. Foug`eres. Estimation de densit´es unimodales. Canadian Journal of Statistics,
25:375–387, 1997.

36. Y. Freund and Y. Mansour. Estimating a mixture of two product distributions.
In Proceedings of the 12th Annual Conference on Computational Learning Theory,
pp. 183–192, 1999.

37. F. Gao and J. A. Wellner. On the rate of convergence of the maximum likelihood
estimator of a k-monotone density. Science in China Series A: Mathematics, 52:
1525–1538, 2009.

38. U. Grenander. On the theory of mortality measurement. Skandinavisk Aktuarietidskrift,
39:125–153, 1956.

39. P. Groeneboom. Estimating a monotone density. In Proceedings of the Berkeley
Conference in Honor of Jerzy Neyman and Jack Kiefer, pp. 539–555, 1985.

40. P. Groeneboom and G. Jongbloed. Nonparametric Estimation under Shape Constraints:
Estimators, Algorithms and Asymptotics. Cambridge University Press, 2014.

282 Handbook of Big Data

41. D. L. Hanson and G. Pledger. Consistency in concave regression. Annals of Statistics,
4(6):1038–1050, 1976.

42. R. Hasminskii and I. Ibragimov. On density estimation in the view of Kolmogorov’s
ideas in approximation theory. Annals of Statistics, 18(3):999–1010, 1990.

43. D. Haussler and M. Opper. Mutual information, metric entropy and cumulative relative
entropy risk. Annals of Statistics, 25(6):2451–2492, 1997.

44. A. J. Izenman. Recent developments in nonparametric density estimation. Journal of
the American Statistical Association, 86(413):205–224, 1991.

45. H. K. Jankowski and J. A. Wellner. Estimation of a discrete monotone density.
Electronic Journal of Statistics, 3:1567–1605, 2009.

46. M. Kearns, Y. Mansour, D. Ron, R. Rubinfeld, R. Schapire, and L. Sellie. On the
learnability of discrete distributions. In Proceedings of the 26th STOC, pp. 273–282,
1994.

47. G. Kerkyacharian and D. Picard. Density estimation in Besov spaces. Statistics &
Probability Letters, 13(1):15–24, 1992.

48. G. Kerkyacharian, D. Picard, and K. Tribouley. Lp adaptive density estimation.
Bernoulli, 2(3):229–247, 1996.

49. R. Koenker and I. Mizera. Quasi-concave density estimation. Annals of Statistics,
38(5):2998–3027, 2010.

50. A. N. Kolmogorov and V. M. Tihomirov. ε-entropy and ε-capacity of sets in function
spaces. Uspekhi Matematicheskikh Nauk, 14:3–86, 1959.

51. B. Lindsay. Mixture Models: Theory, Geometry and Applications. Institute for Mathe-
matical Statistics, Hayward, CA, 1995.

52. G. G. Lorentz. Metric entropy and approximation. Bulletin of the American Mathemat-
ical Society, 72:903–937, 1966.

53. Y. Makovoz. On the Kolmogorov complexity of functions of finite smoothness. Journal
of Complexity, 2(2):121–130, 1986.

54. A. Moitra and G. Valiant. Settling the polynomial learnability of mixtures of Gaussians.
In FOCS, pp. 93–102, 2010.

55. K. Pearson. Contributions to the mathematical theory of evolution. ii. Skew variation
in homogeneous material. Philosophical Transactions of the Royal Society of London,
186:343–414, 1895.

56. B. L. S. Prakasa Rao. Estimation of a unimodal density. Sankhya Series A, 31:23–36,
1969.

57. R. A. Redner and H. F. Walker. Mixture densities, maximum likelihood and the EM
algorithm. SIAM Review, 26:195–202, 1984.

58. D. W. Scott. Multivariate Density Estimation: Theory, Practice and Visualization.
Wiley, New York, 1992.

59. D. W. Scott and S. R. Sain. Multidimensional Density Estimation. volume 24 of
Handbook of Statistics, pp. 229–261, 2005.

Learning Structured Distributions 283

60. B. W. Silverman. Density Estimation. Chapman & Hall, London, 1986.

61. C. J. Stone. The use of polynomial splines and their tensor products in multivariate
function estimation. Annals of Statistics, 22(1):118–171, 1994.

62. C. J. Stone, M. H. Hansen, C. Kooperberg, and Y. K. Truong. Polynomial splines and
their tensor products in extended linear modeling: 1994 Wald memorial lecture. Annals
of Statistics, 25(4):1371–1470, 1997.

63. A. T. Suresh, A. Orlitsky, J. Acharya, and A. Jafarpour. Near-optimal-sample esti-
mators for spherical gaussian mixtures. In Advances in Neural Information Processing
Systems, pp. 1395–1403, 2014.

64. D. M. Titterington, A. F. M. Smith, and U. E. Makov. Statistical Analysis of Finite
Mixture Distributions. Wiley, 1985.

65. A. B. Tsybakov. Introduction to Nonparametric Estimation. Springer, 2008.

66. L. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142,
1984.

67. L. G. Valiant. A theory of the learnable. In Proceedings of the 16th Annual ACM
Symposium on Theory of Computing, pp. 436–445. ACM Press, New York, 1984.

68. A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes.
Springer Series in Statistics. Springer-Verlag, New York, 1996. (With applications to
statistics.)

69. G. Walther. Inference and modeling with log-concave distributions. Statistical Science,
24(3):319–327, 2009.

70. E. J. Wegman. Maximum likelihood estimation of a unimodal density. I. and II. The
Annals of Mathematical Statistics, 41:457–471, 2169–2174, 1970.

71. E. J. Wegman and I. W. Wright. Splines in statistics. Journal of the American Statistical
Association, 78(382):351–365, 1983.

72. R. Willett and R. D. Nowak. Multiscale poisson intensity and density estimation. IEEE
Transactions on Information Theory, 53(9):3171–3187, 2007.

73. Y. Yang and A. Barron. Information-theoretic determination of minimax rates of
convergence. Annals of Statistics, 27(5):1564–1599, 1999.

74. Y. G. Yatracos. Rates of convergence of minimum distance estimators and Kolmogorov’s
entropy. Annals of Statistics, 13:768–774, 1985.