02 - Constant target

Networks trained by plain SGD (muP, no normalisation, no biases) on grayscale CIFAR-10 to predict the constant 1 (mean-squared error): no information about the images is in the target. Score: for each 16 × 16 first-layer unit \(w\), the excess kurtosis of \(u = w\cdot x\) over 100,000 crops of the input the network sees (whitened or raw). Gaussian = 0; the Gabor-like units of sparse coding and ICA score 2.4-3.4 on these patches. 485 of 637 runs finished.

• Verdict so far: a constant target is a weak but real kurtosis pressure. Best median 2.3-2.6 (three seeds) against a random-direction baseline of 1.1 (ICA on these patches 2.4), and only through one route: a pooled hidden layer with weight decay.
• Most networks do not train their first layer at all. On whitened input they end at the baseline; a one-layer readout (conv → relu → global average → linear) goes down to 0.5: outputting a constant through one rectified layer favours light-tailed directions.
• The head that works: conv → |·| or relu → pool → hidden 512 → 1, weight decay 10^-3. Pool 2×2: 2.26; pool 4×4: 1.65-1.80; pool 8×8: 1.94; global average then hidden: 1.33. The spatial layout of the pooled map matters.
• Weight decay is required: 10^-3; 10^-2 kills it (sub-Gaussian units); 0 or 10^-4: the filters hardly move.
• Longer training does not help: 160k steps give 1.5-1.6 where 40k gave 1.8. The pressure stops once the loss is small.
• Robust to the other knobs: hidden 128-2048, 16-256 channels, batch 32-512, three seeds: all 1.6-1.75.
• relu² diverges at η ≥ 0.3; extra conv layers with a global-average readout: ≤ 1.3. MLP on 16×16 patches: never beats a random direction (best 0.92 whitened, 1.04 raw).
• Raw input: ≤ 1.8 (baseline 0.4, but the high-frequency PCs of raw patches already score 1.4-3); the heavy-tailed units are the low-variance ones.
• The units are oriented gratings spanning the kernel, not localised Gabors. Why high frequencies score high, and why the PCs do not: 01' tab j.