projection pursuit by gradient ascent on CIFAR-10 patches, live in the browser (WebGPU): type the objective G, watch the units appear, click one for its projection histogram, G and the integrand
runs on your GPU; type the objective; click a finished unit for its plots
Projection pursuit by gradient ascent (tab 01j of page 01) on 16 × 16 CIFAR-10 patches, live on your GPU. Each unit climbs
\(\mathbb E[G(w\cdot z)]\) on the unit sphere, orthogonal to the units already found, by plain gradient steps; type any \(G\) you like.
Click a finished unit to see where its objective comes from. Advanced settings has a raw switch (no whitening, variances kept).
how it works, in one paragraph
Patches are centred, projected onto their top \(M\) principal components and whitened there, so \(z\) has identity covariance and every
projection \(u = w\cdot z\) with \(|w| = 1\) has unit variance: a unit Gaussian is the null. Unit \(k\) starts from a random direction
orthogonal to the \(k-1\) found; each step takes the gradient of \(J(w) = \mathbb E[G(w\cdot z)]\) over all \(N\) patches, removes its components
along \(w\) and along the found units, moves by the learning rate times what is left, Gram-Schmidts against the found units and
renormalises. It stops when the remaining gradient is below the tolerance times the whole gradient, or at the step budget, and is
then frozen. On the GPU a step is three small kernels over the \(M \times N\) whitened data; nothing is read back except a convergence
number every ten steps, so a 64-unit run at \(N\) = 50,000 takes seconds. The pictures are the units mapped back to pixels.
advanced settings
16 × 16 patches sampled from 10,000 CIFAR-10 images; the whole set is used for every gradientthe 256 pixels are projected onto their top M principal components and whitened there; M is the most units there can behow many directions to find, one after another, each orthogonal to the ones beforestep along the gradient on the sphere; 0.3-1 converges in a few hundred steps for log cosha unit that has not converged by then is frozen as it isa unit has converged when the part of the gradient that can still move it (along the sphere, orthogonal to the found units) is below this fraction of the whole gradientwhitening divides each principal component by its standard deviation; eigenvalues below ε × the largest are raised to that floor so near-empty directions are not blown up. At 10-4 it never binds on CIFAR patches at M ≤ 192; 10-2 acts as a low-pass filterunchecked: whitened, every principal component scaled to unit variance, so a unit Gaussian is the null for every direction. Checked: the variances are kept (one common scale, so a random direction has unit variance on average); the plots then use u/σ so the Gaussian comparison is variance-matched, and σ is reported
edit the list of objectives
One expression per line, in u; every one is maximised, so write -log(cosh(u)) for the log cosh contrast. Functions: sin cos tan exp log cosh sinh tanh abs sqrt sign relu softplus max min; ** or ^; pi, e. Square brackets are indicators: [abs(u) > 3] is 1 where true, 0 elsewhere (its derivative is a Gaussian bump of width 0.1 at the edge, so it still pushes). The list is kept in this browser.
Units in the order found. Red frame: still climbing. Blue frame: shown on the right. Click a finished unit, even while the rest are running.
u = w · z, the projection of the patches on the unit (whitened: unit variance by construction; raw: the axis is u/σ); the dashed curve is the Gaussian of the same variance. Top: its density. Middle: G(u). Bottom: p(u) G(u), whose area is the objective E[G]: where it sits is where the cost lands.