02' - Random labels

Networks trained by plain SGD (muP, no normalisation, no biases) on grayscale CIFAR-10 to predict a fixed random label per image (10-way cross-entropy on 10,000 images): no information about the images is in the target. Score: for each 16 × 16 first-layer unit \(w\), the excess kurtosis of \(u = w\cdot x\) over 100,000 crops of the input the network sees (whitened or raw). Gaussian = 0; the Gabor-like units of sparse coding and ICA score 2.4-3.4 on these patches. 930 of 1093 runs finished.

• Yes: a tiny CNN learns localised Gabors from random labels. conv 16×16 (16 filters) → |·| → average-pool to 6×6 → hidden 512 → 10 random labels on 10,000 whitened images; plain SGD, batch 32, η 0.3, weight decay 10^-4, L1 10^-5 on the first layer, 100k steps. 81-100% of the filters are localised Gabors in nine of nine seeds, median excess kurtosis 3.7-4.7 (sparse coding on these patches: 69%, kurtosis 3.4). Without the L1 penalty 75-94% (eight runs); 32 or 64 filters 72-73%. Reference run (James, 2026-10-30): the same net on 50,000 images, batch 128, 160k steps, first-layer rate ×3 (nn7_r_cnn_combo_N50000_pool6_C16_abs_l1_lr1x3): 15 of 16 localised, kurtosis 4.71, the default on tab b. Note there is no relu anywhere: the hidden layer's nonlinearity is |·| too. Pictures on tab d.
• What is required (each removal tested): whitened input, |·| rather than relu (relu needs 200k steps and gets 81% at kurtosis 2.6), the hidden layer, a pooled map with 4-6 px cells, at least 10 classes, about 100k steps, weight decay. Without decay the filters become Gabors as the labels are fitted and then drift back to gratings.
• Coarser pooling gives gratings, not Gabors: pool 4×4 or a global average yields oriented filters spanning the kernel with kurtosis 3-4 and 0% localised. Finer than 8×8 and the hidden layer memorises without training the filters.
• It is the fitting, not the memorising: 50,000 images at 24-57% train accuracy still give 88-94% localised at kurtosis 4.5-4.7 (with the L1 penalty; without it, 0-38%).
• MLP on 16×16 patches (16 units): a random label per image with fresh random patches each step gives heavy-tailed oriented gratings (kurtosis 3-5, 0% localised). A random label per (image, 4×4 position cell) on 3,000 images localises some of the units: over 45 runs of that setup (weight decay 10^-4 to 10^-3, η 0.3, 100k-200k steps, hidden 512-2048, seeds) the median is 25% localised, the middle half 19-31%, the best 62% (kurtosis 3.4-6.4). Averaging the |·| features over a bag of 8 patches of the same cell with decay 10^-3 gives 25-56% over seven seeds (median 38%; 32 units: 41-50% over three); bags of 16-32, decay 10^-3 without a bag, 32-64 units without a bag: worse. A random label per patch gives nothing: it is memorised through random features.
• True labels with the same recipe: 81-100% localised. The constant target: never. The recipe transfers to ImageNet-64 (10,000 images, pool 12 for the same 5 px cells, η 0.1, whitening floor 10^-3, decay 3×10^-4): 69-94% localised over six runs with 16 filters (L1 on or off), kurtosis 5-7; 78% with 32 filters. Runs with fewer than 16 first-layer units are not shown (they are in results/).