01' - Sparse coding on CIFAR-10

Purpose

Doc 01 found localised, oriented Gabor filters with every classical method on 16 × 16 patches of Olshausen's ten whitened images. Doc 02 trains networks on CIFAR-10 and finds filters that are oriented but less localised. Before reading anything into that difference, this page asks the prior question (James, 2026-10-29): are the statistics that give Gabors present in CIFAR-10 at all? If the classical methods do not find Gabors here, a network cannot be expected to, and the doc 02 rerun must use data that has them. The whole doc 01 zoo is therefore run on 16 × 16 luminance patches of CIFAR-10, each method at the configuration its doc 01 tab shows, and each method both on whitened patches (PCA-whitened with the eigenvalue floor of doc 01; for Olshausen-Field, the same whitening applied in pixel space) and on raw patches (nothing but centring; the ICA family then only projects onto the top \(M\) principal components, keeping their variances). Whitened versus raw is the first menu on every tab. Every run saves 100 log-spaced checkpoints and the sliders show them all (about a factor 1.1 between frames).

How a filter is judged: doc 01's instrument. A 2-D Gabor is fitted to each basis function; oriented means fit \(R^2 \ge 0.8\) with at least 0.75 carrier cycles across the patch, localised means oriented with a Gaussian envelope at most a quarter of the patch. Each tab's info line quotes the same method's numbers on Olshausen's images beside this set's.

Every run against the same method on Olshausen's images

Classes of the fitted Gabor (fit \(R^2 \ge 0.8\)): Gabor = at least 0.75 carrier cycles across the patch (doc 01's "oriented"); edge = fewer cycles, odd carrier and elongated envelope, i.e. a derivative-of-Gaussian edge detector (the fit describes it as a ramp times a Gaussian, with a near-zero frequency, which is why the doc 01 "oriented" count misses it); blob = the remaining good fits; "localised" = envelope at most a quarter of the patch. The edge class was added for this page, after the pictures showed the whitened CIFAR filters to be oriented and compact while the oriented count said 0-6%.

Findings (2026-10-29, 15:01)

The statistics are there, and only sparse coding turns them into wavelets. On whitened 16 × 16 CIFAR patches, Olshausen-Field sparse coding gives 70% localised Gabor wavelets at \(M = 64\) (median fit \(R^2\) 0.87; doc 01 on Olshausen's images: 64%) and 75% at \(M = 256\) (\(R^2\) 0.92), though at 256 the units shrink to 2-3 pixel bars. They look like doc 01's.

Every other method finds edges, not wavelets. FastICA, projection pursuit, the Hebbian rule, Infomax and the AGF climbs on the same whitened patches give localised oriented edge and bar detectors: one or two lobes, 1-2 pixels wide, pixel-sharp with faint ripples. By the fit classes they are 50-70% edges and 0-11% Gabors at \(M = 64\), where the same methods on Olshausen's images gave 68% Gabors and 31% edges. At \(M \ge 192\) the ICA fits are mostly poor (55-67% of units), because a 16 × 16 patch of a 32-pixel image does not have that many independent directions; \(M = 64\) is the honest dimension here.

Whitening is necessary. On raw patches ICA gives noise (fit \(R^2\) 0.07-0.28, 94-99% poor), projection pursuit and the Hebbian rule give whole-patch gratings, the AGF climb finds the top principal component, and sparse coding gets 21-23% Gabors with the rest edges and blobs.

Why edges and not wavelets. Olshausen's images are whitened and low-pass filtered (\(R(f) = f\,e^{-(f/f_0)^4}\), \(f_0\) at 0.78 of Nyquist), so their finest structure is a few pixels wide and the ICA filters come out smooth. CIFAR images are downsampled photographs whose finest structure is the pixel; whitening flattens the spectrum all the way to Nyquist, and the most non-Gaussian projection of such patches is the sharpest edge, one pixel wide. Sparse coding is different: it is a generative model with a noise term, so it prefers a dictionary of smooth, reusable parts to pixel-sharp ones, and it recovers wavelets. A larger whitening floor (a crude low-pass; the \(\epsilon\) runs on tab c) does not rescue ICA: \(10^{-3}\) changes nothing, \(10^{-2}\) turns the edges into gratings and checkerboards, \(10^{-1}\) leaves noise.

What this means for doc 02. The oriented, often grating-like first-layer filters of the CNNs on CIFAR are what every projection-based method finds on this data; the Gabor wavelets of doc 01 come from the sparse generative objective, not from the images alone. Whitened versus raw is decisive for every method here, which is the knob the CNNs have not yet been given.

The images

The 50,000 cifar-10 training images, 32 × 32 pixels, converted to luminance and standardised (zero mean, unit variance over the set): 51 million pixels (Olshausen's ten images: 2.6 million). Shown: the first 48, at 96 px (luminance).

Patches. 144 random 16 × 16 patches, each scaled to its own range. A 16 × 16 patch is a quarter of a CIFAR image's side, a sixteenth of its area; on Olshausen's 512-pixel images it was a thousandth of the area.

Purpose. Does doc 01's 01b (olshausen-field sparse coding) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. Olshausen & Field (1996): a generative model \(x = A a + \text{noise}\) with a sparse prior on the coefficients. Per patch, infer \(a\) by minimising \(E(a) = \tfrac{1}{2\sigma_n^2}\|x - Aa\|^2 + \tfrac{\beta}{\sigma}\sum_i \log(1 + a_i^2/\sigma^2)\) (Cauchy prior; here 200 accelerated-gradient steps from \(a = 0\)); then take a Hebbian step on the basis, \(A \leftarrow A + \eta\,(x - Aa)\,a^{\top}/B\), and rescale each column so that its coefficient's running variance drifts towards a target (sparsenet.m's gain homeostasis). Knobs: \(\sigma_n^2 = 0.01\), \(\beta = 2.2\), \(\sigma = 0.316\), \(\eta = 1\), batch \(B = 100\), variance goal 0.1 with rates 0.001 and 0.02, plus the inference budget: eight, all at sparsenet.m's values. Time = learning steps (batches of 100 patches). Shown first is the simpler variant that drops the homeostasis and just keeps every column at unit norm (five knobs): at 20,000 steps it is the best Olshausen-Field run of the battery; the homeostatic runs are in the table.

patches whitenedrawbasis functions M 25664

time steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01c (ica (fastica, all units at once)) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. Hyvärinen's FastICA. Centre the patches and PCA-whiten them to \(M\) dimensions, \(z = V(x - \mu)\); look for an orthogonal \(W\) whose rows give maximally non-Gaussian projections \(s = Wz\), by the fixed-point iteration \(w \leftarrow \mathbb{E}[z\,g(w^{\top}z)] - \mathbb{E}[g'(w^{\top}z)]\,w\) on every row at once, followed by symmetric decorrelation \(W \leftarrow (WW^{\top})^{-1/2}W\). \(g = \tanh\) (the \(\log\cosh\) contrast), \(g(u) = u^3\) (kurtosis) or \(g(u) = u e^{-u^2/2}\). Shown: the basis functions, i.e. the columns of \(V^{-1}W^{\top}\) back in pixel space. Knobs: the number of PCA components \(M\) and the contrast function: two. No learning rate. Time = fixed-point iterations, each a pass over a fixed set of 100,000 patches. The images' low-pass filter leaves the last ~30 of 256 principal directions with \(< 10^{-4}\) of the top variance, so whitening to all 256 amplifies noise (a run at \(M = 256\) without a floor went NaN); the eigenvalues are floored at \(10^{-4}\) of the largest.

patches whitenedrawPCA dims M 19264256contrast logcoshexpkurtosis

time steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01d (projection pursuit (one unit at a time)) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. The same fixed-point update as 01c, but on a single direction \(w\) until it stops moving (\(1 - |w_{\text{new}}\cdot w| < 10^{-6}\)), orthogonalised against the directions already found; then the next one. This is projection pursuit (Friedman & Tukey) with FastICA's contrast as the projection index, and it is the literal version of "find one, let it solidify, find the next". The layout here is the order of discovery. Knobs: \(M\), the contrast, and a convergence tolerance: three. Time = cumulative one-unit iterations; the per-unit charts below say how many iterations each unit took.

patches whitenedrawPCA dims M 19264

time steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01e (nonlinear hebbian rule) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. On whitened patches: \(W \leftarrow W + \eta\,g(Wz)\,z^{\top}/B\) on a minibatch, then symmetric orthonormalisation, with \(g(u) = u^3\). That is stochastic gradient ascent on the fourth moment of every unit under an orthonormality constraint, the nonlinear Hebbian (Oja-type) rule of Hyvärinen & Oja (1998): a Hebb rule with a cubic output nonlinearity and nothing else. Knobs: \(M\) and one learning rate: two. Time = minibatch steps.

patches whitenedrawPCA dims M 19264

time steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01f (infomax ica (bell-sejnowski)) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. Bell & Sejnowski (1997): maximise the entropy of \(\tanh(Wz)\), which is maximum likelihood with a fixed super-Gaussian (logistic) prior on the sources. Natural-gradient minibatch updates \(W \leftarrow W + \eta\,(I - 2\tanh(u)\,u^{\top}/B)\,W\), \(u = Wz\), from \(W = I\). Shown: the columns of \(W^{-1}\) mapped back to pixel space. Knobs: \(M\) and the learning rate: two. Time = minibatch steps (B = 100).

patches whitenedrawPCA dims M 12864

time steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01g (agf utility maximisation) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. The utility-maximisation step of Alternating Gradient Flows (Kunin et al. 2025). In a two-layer network \(f(x) = \sum_i a_i\,\sigma(w_i\cdot x)\) trained from small initialisation, a dormant neuron's direction follows projected gradient flow on the unit sphere of its utility \(\mathcal U(w) = \mathbb E_x[(y - f(x))\,\sigma(w\cdot x)]\), the correlation of its output with the residual; the first neuron whose accumulated utility crosses threshold activates. Here the target is the constant \(y = 1\) and the network starts at \(f = 0\), so every dormant neuron climbs \(\mathcal U(w) = \mathbb E[\sigma(w\cdot z)]\) on PCA-whitened patches \(z\), where every direction has unit variance and only the shape of the projection's distribution can differ. With \(\sigma(z) = |z|^\alpha\) that is the \(\alpha\)-th absolute moment: \(\alpha = 4\) is kurtosis maximisation, \(\alpha < 2\) is maximised by sub-Gaussian directions. \(\sigma = \mathrm{relu}(z)^\alpha\) and \(\sigma = \mathrm{sign}(z)|z|^\alpha\) are the one-sided and odd versions. 256 neurons climb independently from random directions (no orthogonalisation, no deflation: what a wide network's hidden layer does); the questions are what the local maxima look like and how many distinct ones the neurons find. Knobs: \(\alpha\), the sign convention, the step angle (0.02 rad per step, gradient direction only): three. Time = sphere-ascent steps, each a pass over 50,000 patches. Runs named agfseq_* run the AGF sequence proper: the best-converged dormant neuron activates, its amplitude is fitted to the residual, and the rest climb the new residual's utility. The menus below select the run; \(\alpha = 2\) on whitened data is flat by construction (every direction has unit variance), so the whitened \(\alpha\) list skips it and the unwhitened runs show the principal-component limit instead.

patches whitenedraw\\(\\alpha\\) 2.53.0

time sphere-ascent steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01h (agf with the patch as target) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. As in 01g, but the network reconstructs its input: \(f(z) = \sum_i a_i\,\sigma(w_i\cdot z)\) with vector output weights \(a_i\) and target \(y = z\). A dormant neuron's utility, with \(a_i\) optimally aligned, is \(\mathcal U(w) = \|\mathbb E[r(z)\,\sigma(w\cdot z)]\|\), the norm of the cross-correlation between the residual \(r = z - f(z)\) and the neuron's activation. Even nonlinearities have zero utility on sign-symmetric data, so this tab uses \(\mathrm{sign}(z)|z|^\alpha\) and \(\mathrm{relu}(z)^\alpha\) (\(\alpha = 1\), odd, is PCA on the residual). All 256 dormant neurons climb; every 50 steps the one with the largest utility activates, all active output vectors are refitted to the target by least squares, and the residual shrinks, so the later neurons are climbing a different landscape: the constant-target version (01g) could not do this. Knobs: \(\alpha\), the sign convention, the step angle, the activation interval: four. Time = sphere-ascent steps, each a pass over 20,000 patches. The slider shows the input directions \(w_i\) (settle order); the chart tracks the residual.

patches whitenedraw

time sphere-ascent steps play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. Does doc 01's 01i (iterated agf, one neuron at a time) find on CIFAR-10 what it found on Olshausen's images, and does it need the patches whitened?

Method. James's iterated version of 01h. A single fresh neuron climbs the utility \(\mathcal U(w) = \|\mathbb E[r(z)\,\sigma(w\cdot z)]\|\) of the current residual from a random direction, with a decaying step (\(0.05 \times 0.95^k\) radians, 150 steps, so it has converged when its budget ends); it is then frozen, every frozen neuron's output vector is refitted to the target by least squares, the residual shrinks by what the new feature explains, and the next neuron starts. The residual is eaten one feature at a time, so this is projection pursuit (01d) with the reconstruction residual doing the deflation instead of orthogonalisation. Target \(y = z\): whitened patches, or ("unwhitened") the patches projected onto their top \(M\) principal components with the variances kept, where \(\alpha = 1\) odd must give the principal components in order. Knobs: \(\alpha\), the sign convention, the step schedule: three. Time = ascent steps, cumulative over neurons (150 per neuron); the layout is the order the neurons were found, which is the only order there is.

patches whitenedraw

time ascent steps, cumulative over neurons play

Basis functions in settle order (first learned first), every checkpoint of the run on the slider. Menus cascade.

Purpose. The doc 02 networks are scored by the excess kurtosis of their units' projections. Which directions of 16 × 16 CIFAR-10 patches are heavy-tailed at all? Three input distributions, as the doc 02 organisms see them: raw (per-pixel standardised), patch-whitened (ZCA in patch space, the MLP's input), image-whitened (ZCA of the 32 × 32 image, then 16 × 16 crops, the CNN's input). 100,000 crops each.

• PCA modes are not heavy-tailed, except the last ones. Whitened: PCs 1-190 sit at 1.0-1.6, about a random direction (1.1-1.2). The last 60 PCs (near-Nyquist checkerboards) climb to 3, and PC 256 reaches 8.5. Raw patches rise from 0.2 (PC 1) to 3 at the top of the spectrum.
• Kurtosis grows with spatial frequency. A plane-wave grating spanning the kernel scores 1.3 at 1 cycle, 2.0 at 4 cycles, 2.9 at 8 cycles (period 2 px), median over orientations; the best orientations reach 6-8. A Gabor (the same grating under a 2.5 px Gaussian) scores higher at every frequency: 4 at 4 cycles, 5-6 at 6-8 cycles, up to 10.
• The random-label network's units (02') are not PCA modes. Its top units (kurtosis 6-9) carry 50-98% of their energy beyond PC 64 (energy-weighted centre PC 110-165), and they are oriented: a single PC at those indices mixes orientations (near-degenerate eigenvalues) and scores 1-1.7, while an oriented plane wave at the same frequency scores 3-7. Kurtosis separates what PCA leaves mixed, which is ICA's job.
• Why the interactive page (01′′′) finds less. Its default keeps the top M = 64 PCs, which excludes exactly these directions; at M = 256 the kurtosis objective finds units at 25-50 (outlier-driven pixel patterns), log cosh 4-10, |u|³ 10-58. Set M = 256 in advanced settings to see them.

Purpose. The doc 02 score is the excess kurtosis of a unit's projection; its null is a Gaussian (0). But are the patches themselves heavy-tailed in a random direction? 64 random unit vectors \(w\) in pixel space, projected on the same 100,000 16 × 16 crops the networks are scored on, for the three input distributions of doc 02. Two kinds of direction: Gaussian (iid pixel weights) and a single pixel (the marginal of one input coordinate).

• Yes, a little: a random direction of whitened patches has excess kurtosis 1.2 (patch-whitened; 1.1 image-whitened), all 64 between 0.9 and 1.5, with 0.85% of the mass beyond 3σ against the Gaussian's 0.27%. Raw patches: 0.34, nearly Gaussian (a random mixture of 256 correlated pixels).
• A raw pixel is light-tailed (-0.64: intensities are bounded). A whitened pixel is as heavy-tailed as a sparse-coding unit (3.2 patch-whitened, 2.8 image-whitened; ICA and sparse coding score 2.4-3.4 on these patches): one coordinate of the whitened patch is a centre-surround filter applied to the image, a localised high-pass filter, and that is already most of the way to a Gabor's statistics.
• So the baselines of doc 02 are 0.3 raw and 1.1-1.2 whitened, not 0; a unit has learned something when it clears them, and clears the whitening filter itself when it passes 3.

input rawpatch-whitenedimage-whiteneddirection Gaussian weightsa single pixel

The 64 random directions as 16 x 16 pictures, sorted by excess kurtosis (highest first). There is nothing to slide: these are not trained. Click one: the histogram of its projection over the 100,000 crops, in units of its standard deviation, against the Gaussian of the same variance; the log y button shows the tails.