← main page
2. Training at Large Width
02a · NTK param vs muP
02b · CNNs on MNIST
\[ f(\mathbf{x}) = W_2\,\mathrm{ReLU}(W_1 \mathbf{x}) \]
\(\mathbf{x}\)
\(\sigma\)
\(f(\mathbf{x})\)
\(\mathbf{x}\)
\(\sigma\)
\(f_*(\mathbf{x})\)
teacher
width
NTK param
max update param (μP)
start
random init
show plots
step
loss
\((W_1)_{i1}\)
\((W_1)_{i2}\)
channels
NTK param
max update param (μP)
REPLAYS
start
step
loss
step
NTK between the two images