How to make a double-sized net as good as SF NNUE in a few easy steps.

How to make a double-sized net as good as SF NNUE in a few easy steps.

Say we have a NNUE with topography 512 x 32 x 32 x 1 and we want to make a better (we hope) NNUE with topography 1024 x 32 x 32 x 1

Step 1. Initialise our new 1024 neuron network with all weights = 0.0

Step 2. Unload all SF NNUE weights (from topography 512) and use them to fill the left hand side of the 1024 topography net.

Step 3. Check the 1024-Net gives the same outputs as the 512 net. It should do, all the zero weighted half will contribute zero to the result.

Step 4. Start normal training at very low learning rate, and gradually the zero weighted side will fill and the SF-NNUE weighted side will adjust.

If you get it right, low enough LR, then there won't be any or few regressions, and you'll have a brand spanking new double size NNUE. Bingo.

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

Chris Whittington wrote:

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

Personally, I would start by calculating the covariance between the old weights and the new ones (the covar function in Google Sheets - or you can use an online calculator or whatever).

More difficult but potentially more effective - the top trick from cryptography - the very trick that, combined with some good luck, brought the first breakthrough against the Lorenz cipher in WWII (which was an order of magnitude more difficult than the more well known Enigma cipher, and of which no examples were seen by Station X until after the war): map the two sets of weights into multi-dimensional space and look for emergent patterns. They're not random numbers, so one would expect to see patterns. See how similar the patterns are in comparison with an NN from a completely different source.

Chris Whittington wrote:

How to make a double-sized net as good as SF NNUE in a few easy steps.

Say we have a NNUE with topography 512 x 32 x 32 x 1 and we want to make a better (we hope) NNUE with topography 1024 x 32 x 32 x 1

Step 1. Initialise our new 1024 neuron network with all weights = 0.0

Step 2. Unload all SF NNUE weights (from topography 512) and use them to fill the left hand side of the 1024 topography net.

Step 3. Check the 1024-Net gives the same outputs as the 512 net. It should do, all the zero weighted half will contribute zero to the result.

Step 4. Start normal training at very low learning rate, and gradually the zero weighted side will fill and the SF-NNUE weighted side will adjust.

If you get it right, low enough LR, then there won't be any or few regressions, and you'll have a brand spanking new double size NNUE. Bingo.

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

I think this is wishful thinking .. I believe (but I'm not the expert) that 1024 NET is not just "double" the size of the 512... The neurons of 512 cannot be used as a starting point for the 1024 net!

Damir Desevac wrote:

Chris Whittington wrote:

How to make a double-sized net as good as SF NNUE in a few easy steps.

Say we have a NNUE with topography 512 x 32 x 32 x 1 and we want to make a better (we hope) NNUE with topography 1024 x 32 x 32 x 1

Step 1. Initialise our new 1024 neuron network with all weights = 0.0

Step 2. Unload all SF NNUE weights (from topography 512) and use them to fill the left hand side of the 1024 topography net.

Step 3. Check the 1024-Net gives the same outputs as the 512 net. It should do, all the zero weighted half will contribute zero to the result.

Step 4. Start normal training at very low learning rate, and gradually the zero weighted side will fill and the SF-NNUE weighted side will adjust.

If you get it right, low enough LR, then there won't be any or few regressions, and you'll have a brand spanking new double size NNUE. Bingo.

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

I think this is wishful thinking .. I believe (but I'm not the expert) that 1024 NET is not just "double" the size of the 512... The neurons of 512 cannot be used as a starting point for the 1024 net!

Well, actually you can.

Can you come up with more detail regarding this ?

Damir Desevac wrote:

Can you come up with more detail regarding this ?

You need to do the work to understand the idea