Fire 8 released

Admin wrote:

Did some overnight NNUE testing at 100ms with different SF nets and also included Nemerino 6.00 with its own net (presumably a SF net?) and special versions of Rubichess and Ethereal with SF NNUE support. You can see the impact of SF NNUE at - http://rebel13.nl/download/srl-1000.html



Conclusion, it's going nowhere.

I have a question. What answer are you trying to find out. Is Fat Fritz 2 just Stockfish 12 to Stockfish 13. If that is the question. Why not just take the NNUE file out of the equation. And run them as a normal A/B engine.

NN Structure
The neural network of Stockfish NNUE consists of four layers, W1 through W4. The input layer W1 is heavily overparametrized, feeding in the board representation for various king configurations. The efficiency of the net is due to incremental update of W1 in make and unmake move, where only a fraction of its neurons need to be recalculated. The remaining three layers with 32x2x256, 32x32 and 32x1 weights are computational less expensive, best calculated using appropriate SIMD instructions like AVX2 on x86-64, or if available, AVX-512.

Sim report at 1000ms

It's a bit hard to read as the 3 runs before are still in the same database. My bad, I should have told you to create a new database before a new run. For instance with this parameter:

set DATABASE=data\test.data

to for instance

set DATABASE=data\1000ms.data

------------

Anyway the results surprised me, 66 and 68% with FF2 but also sf12 vs sf13 68%. And so the question is, do sf12 vs sf13 without NNUE also produce such a high value of 68% and if not you maybe hit the jackpot. Else it's just a matter strength and not of similarity. I am running sf12 vs sf13 without nnue overnight. Can become interesting...

Admin wrote:

It's a bit hard to read as the 3 runs before are still in the same database. My bad, I should have told you to create a new database before a new run. For instance with this parameter:

set DATABASE=data\test.data

to for instance

set DATABASE=data\1000ms.data

------------

Anyway the results surprised me, 66 and 68% with FF2 but also sf12 vs sf13 68%. And so the question is, do sf12 vs sf13 without NNUE also produce such a high value of 68% and if not you maybe hit the jackpot. Else it's just a matter strength and not of similarity. I am running sf12 vs sf13 without nnue overnight. Can become interesting...

Running it again with the correct settings.

Admin wrote:

It's a bit hard to read as the 3 runs before are still in the same database. My bad, I should have told you to create a new database before a new run. For instance with this parameter:

set DATABASE=data\test.data

to for instance

set DATABASE=data\1000ms.data

------------

Anyway the results surprised me, 66 and 68% with FF2 but also sf12 vs sf13 68%. And so the question is, do sf12 vs sf13 without NNUE also produce such a high value of 68% and if not you maybe hit the jackpot. Else it's just a matter strength and not of similarity. I am running sf12 vs sf13 without nnue overnight. Can become interesting...

Actually, now I thought about it, and changing the focus somewhat, it is quite astonishing that using unchanged SF search with a differently trained set of weights would produce anything with semi-equivalent Elo to SF.
Reason? Well, SF search is stuffed full of parameter values that compare eval scores with all kinds of things, other eval scores, non-eval functions, all manner of stuff, and we know two things
1. SF searched is highly tuned
2. SF search does not use centipawn scale within, but it’s own variant of material scale, including the ratios Q:R:B:N:P for all game stages.

Which implies that, internally, FF provides the SF search with evals not only as “chessly good” as SF but also to the equivalent scaling.
That would be a mammoth (or desperately lucky) task to train a net to do. Unless you trained the net directly against and on SF.

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2

Admin wrote:

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2

Then the training problem reduces to:

Train against d=5 (just guessing at the 5), SF evaluated positions. Then you get d5 enhanced SF evaluations in centipawns. Convert the centipawns back to internal SF values (look the conversion up in the SF root output code), so train on those.
Then you get a net which is chessically a bit different but whose scoring range is pretty much the same.

Forensic requirement = get a load of output evals for a load of positions (kill off outlier mate scores), and see how they correlate. Maybe slope and intercept would be a good start.
Compare with correlations of other programs.

Results at 1000ms. Corrected.

Admin wrote:

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2

You know I can not say no. Get it ready!

Can you run all the simex timings at the same time. If you put them all in a different file, and run them at the same time? I have 16 locked cores.

mwyoung wrote:

Results at 1000ms. Corrected.

Great!

Now can you do the same for 500ms, 250ms and 100ms ?

set DATABASE=data\100ms.data
set DATABASE=data\250ms.data
set DATABASE=data\500ms.data

mwyoung wrote:

Admin wrote:

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2

You know I can not say no. Get it ready!

Can you run all the simex timings at the same time. If you put them all in a different file, and run them at the same time? I have 16 locked cores.

Yes, you can, just create multiple copies of the sim folder. Disadvantage is that you manually (and correctly) need to merge the *.data output into right time control *data files. But my request above is more simple, see for yourself.

mwyoung wrote:

Results at 1000ms. Corrected.

Apologies for pointing out the obvious and for being negative, but you've also got a high similarity between SF12 and SF13. Are you trying to confirm or disprove something?

Results at 500ms.

What CPU were you using?

Ozymandias wrote:

What CPU were you using?

My Chess testing setup is a CPU 2950x Threadripper, GPU RTX 2080 ti, 7man TB running on a Samsung 970 Plus 2 TB MVMe SSD.

Admin wrote:

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2

"then without NNUE and see what happens."

Hello Ed, can you post what settings I need to add to turn off NNUE for FF2, SF13, and SF12. So I get it correct.
I am running the 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE first so no rush.

Results at 250ms.

mwyoung wrote:

Admin wrote:

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2

"then without NNUE and see what happens."

Hello Ed, can you post what settings I need to add to turn off NNUE for FF2, SF13, and SF12. So I get it correct.
I am running the 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE first so no rush.

Change set OPTIONS this way:

set OPTIONS="Use NNUE=false,depth=77"

Results at 100ms. 2500ms results with NNUE will be next. The run time to complete will be 17+ hours.

Sofar I have: - http://rebel13.nl/text/simex-sf.txt

1000ms and 2500ms ready within 2 hours or so.

n = nnue
c = classic evaluation.

The question is what to do next -

1. Nemorino 6 which is in dispute in EO. If memory serves me well the author said -> own net.
2. I have versions of Ethereal and Rubichess that can handle SF nets.

Edit, never mind I run all 3, Ethereal and Rubichess with the SF12 net.

Update - finished 1000ms and 2500ms, start with Ethereal, Rubichess and Nemorino 100ms

http://rebel13.nl/text/simex-sf.txt

Final - http://rebel13.nl/text/simex-sf.txt

500ms and 1000ms in pictures.





With reference to Stockfish Handicap Matches IMO, what we see is not similarity but playing strength.

2500ms results.

Started same tests with NNUE=false. 2500ms started, run time needed 17+ hours.

For some reason, your 100ms results aren't comparable to mine, so do you think you could run the Github net? 250ms looks good enough.