ProDeo
Would you like to react to this message? Create an account in a few clicks or log in to continue.
ProDeo

Computer Chess
 
HomeHome  CalendarCalendar  Latest imagesLatest images  FAQFAQ  SearchSearch  MemberlistMemberlist  UsergroupsUsergroups  RegisterRegister  Log in  

 

 How to make a double-sized net as good as SF NNUE in a few easy steps.

Go down 
3 posters
AuthorMessage
Chris Whittington




Posts : 1254
Join date : 2020-11-17
Location : France

How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. EmptySun Feb 28, 2021 7:54 pm

How to make a double-sized net as good as SF NNUE in a few easy steps.

Say we have a NNUE with topography 512 x 32 x 32 x 1 and we want to make a better (we hope) NNUE with topography 1024 x 32 x 32 x 1

Step 1. Initialise our new 1024 neuron network with all weights = 0.0

Step 2. Unload all SF NNUE weights (from topography 512) and use them to fill the left hand side of the 1024 topography net.

Step 3. Check the 1024-Net gives the same outputs as the 512 net. It should do, all the zero weighted half will contribute zero to the result.

Step 4. Start normal training at very low learning rate, and gradually the zero weighted side will fill and the SF-NNUE weighted side will adjust.

If you get it right, low enough LR, then there won't be any or few regressions, and you'll have a brand spanking new double size NNUE. Bingo.

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

MikeGL likes this post

Back to top Go down
TheSelfImprover

TheSelfImprover


Posts : 2874
Join date : 2020-11-18

How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: Re: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. EmptySun Feb 28, 2021 10:53 pm

Chris Whittington wrote:
Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?


Personally, I would start by calculating the covariance between the old weights and the new ones (the covar function in Google Sheets - or you can use an online calculator or whatever).

More difficult but potentially more effective - the top trick from cryptography - the very trick that, combined with some good luck, brought the first breakthrough against the Lorenz cipher in WWII (which was an order of magnitude more difficult than the more well known Enigma cipher, and of which no examples were seen by Station X until after the war): map the two sets of weights into multi-dimensional space and look for emergent patterns. They're not random numbers, so one would expect to see patterns. See how similar the patterns are in comparison with an NN from a completely different source.
Back to top Go down
Damir Desevac

Damir Desevac


Posts : 316
Join date : 2020-11-27
Age : 42
Location : Denmark

How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: Re: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. EmptyMon Mar 01, 2021 7:16 pm

Chris Whittington wrote:
How to make a double-sized net as good as SF NNUE in a few easy steps.

Say we have a NNUE with topography 512 x 32 x 32 x 1 and we want to make a better (we hope) NNUE with topography 1024 x 32 x 32 x 1

Step 1. Initialise our new 1024 neuron network with all weights = 0.0

Step 2. Unload all SF NNUE weights (from topography 512) and use them to fill the left hand side of the 1024 topography net.

Step 3. Check the 1024-Net gives the same outputs as the 512 net. It should do, all the zero weighted half will contribute zero to the result.

Step 4. Start normal training at very low learning rate, and gradually the zero weighted side will fill and the SF-NNUE weighted side will adjust.

If you get it right, low enough LR, then there won't be any or few regressions, and you'll have a brand spanking new double size NNUE. Bingo.

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

I think this is wishful thinking .. I believe (but I'm not the expert) that 1024 NET is not just "double" the size of the 512... The neurons of 512 cannot be used as a starting point for the 1024 net!
Back to top Go down
Chris Whittington




Posts : 1254
Join date : 2020-11-17
Location : France

How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: Re: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. EmptyMon Mar 01, 2021 8:36 pm

Damir Desevac wrote:
Chris Whittington wrote:
How to make a double-sized net as good as SF NNUE in a few easy steps.

Say we have a NNUE with topography 512 x 32 x 32 x 1 and we want to make a better (we hope) NNUE with topography 1024 x 32 x 32 x 1

Step 1. Initialise our new 1024 neuron network with all weights = 0.0

Step 2. Unload all SF NNUE weights (from topography 512) and use them to fill the left hand side of the 1024 topography net.

Step 3. Check the 1024-Net gives the same outputs as the 512 net. It should do, all the zero weighted half will contribute zero to the result.

Step 4. Start normal training at very low learning rate, and gradually the zero weighted side will fill and the SF-NNUE weighted side will adjust.

If you get it right, low enough LR, then there won't be any or few regressions, and you'll have a brand spanking new double size NNUE. Bingo.

Next question. How do you hunt down correlations between weights from an original and one it has been doubled to? There must be some, right?

I think this is wishful thinking .. I believe (but I'm not the expert) that 1024 NET is not just "double" the size of the 512... The neurons of 512 cannot be used as a starting point for the 1024 net!

Well, actually you can.
Back to top Go down
Damir Desevac

Damir Desevac


Posts : 316
Join date : 2020-11-27
Age : 42
Location : Denmark

How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: Re: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. EmptyMon Mar 01, 2021 8:43 pm

Can you come up with more detail regarding this ?
Back to top Go down
Chris Whittington




Posts : 1254
Join date : 2020-11-17
Location : France

How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: Re: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. EmptyMon Mar 01, 2021 8:56 pm

Damir Desevac wrote:
Can you come up with more detail regarding this ?

You need to do the work to understand the idea
Back to top Go down
Sponsored content





How to make a double-sized net as good as SF NNUE in a few easy steps. Empty
PostSubject: Re: How to make a double-sized net as good as SF NNUE in a few easy steps.   How to make a double-sized net as good as SF NNUE in a few easy steps. Empty

Back to top Go down
 
How to make a double-sized net as good as SF NNUE in a few easy steps.
Back to top 
Page 1 of 1
 Similar topics
-
» SF double net
» How good was Deep Blue?
» Good luck with your new forum!
» Good to see a post from Prof. Hyatt again
» glasses for NNUE?

Permissions in this forum:You cannot reply to topics in this forum
ProDeo :: Computer Chess-
Jump to: