ProDeo
Would you like to react to this message? Create an account in a few clicks or log in to continue.
ProDeo

Computer Chess
 
HomeHome  CalendarCalendar  Latest imagesLatest images  FAQFAQ  SearchSearch  MemberlistMemberlist  UsergroupsUsergroups  RegisterRegister  Log in  

 

 Fire 8 released

Go down 
+6
TheSelfImprover
Tibono
Nezhman
Mclane
Ray
matejst
10 posters
Go to page : Previous  1, 2, 3, 4, 5  Next
AuthorMessage
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyThu Mar 04, 2021 5:53 pm

Admin wrote:
Did some overnight NNUE testing at 100ms with different SF nets and also included Nemerino 6.00 with its own net (presumably a SF net?) and special versions of Rubichess and Ethereal with SF NNUE support. You can see the impact of SF NNUE at - http://rebel13.nl/download/srl-1000.html

Fire 8 released - Page 4 Simex2

Conclusion, it's going nowhere.

I have a question. What answer are you trying to find out. Is Fat Fritz 2 just Stockfish 12 to Stockfish 13. If that is the question. Why not just take the NNUE file out of the equation. And run them as a normal A/B engine.

NN Structure
The neural network of Stockfish NNUE consists of four layers, W1 through W4. The input layer W1 is heavily overparametrized, feeding in the board representation for various king configurations. The efficiency of the net is due to incremental update of W1 in make and unmake move, where only a fraction of its neurons need to be recalculated. The remaining three layers with 32x2x256, 32x32 and 32x1 weights are computational less expensive, best calculated using appropriate SIMD instructions like AVX2 on x86-64, or if available, AVX-512.

Fire 8 released - Page 4 Nnue10
Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 12:04 am

Sim report at 1000ms

Fire 8 released - Page 4 Ff2_re11

Admin and Ozymandias like this post

Back to top Go down
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 1:49 am

It's a bit hard to read as the 3 runs before are still in the same database. My bad, I should have told you to create a new database before a new run. For instance with this parameter:

set DATABASE=data\test.data

to for instance

set DATABASE=data\1000ms.data

------------

Anyway the results surprised me, 66 and 68% with FF2 but also sf12 vs sf13 68%. And so the question is, do sf12 vs sf13 without NNUE also produce such a high value of 68% and if not you maybe hit the jackpot. Else it's just a matter strength and not of similarity. I am running sf12 vs sf13 without nnue overnight. Can become interesting...
Back to top Go down
http://rebel13.nl/
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 2:20 am

Admin wrote:
It's a bit hard to read as the 3 runs before are still in the same database. My bad, I should have told you to create a new database before a new run. For instance with this parameter:

set DATABASE=data\test.data

to for instance

set DATABASE=data\1000ms.data

------------

Anyway the results surprised me, 66 and 68% with FF2 but also sf12 vs sf13 68%. And so the question is, do sf12 vs sf13 without NNUE also produce such a high value of 68% and if not you maybe hit the jackpot. Else it's just a matter strength and not of similarity. I am running sf12 vs sf13 without nnue overnight. Can become interesting...

Running it again with the correct settings.

Admin likes this post

Back to top Go down
Chris Whittington




Posts : 1254
Join date : 2020-11-17
Location : France

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 10:16 am

Admin wrote:
It's a bit hard to read as the 3 runs before are still in the same database. My bad, I should have told you to create a new database before a new run. For instance with this parameter:

set DATABASE=data\test.data

to for instance

set DATABASE=data\1000ms.data

------------

Anyway the results surprised me, 66 and 68% with FF2 but also sf12 vs sf13 68%. And so the question is, do sf12 vs sf13 without NNUE also produce such a high value of 68% and if not you maybe hit the jackpot. Else it's just a matter strength and not of similarity. I am running sf12 vs sf13 without nnue overnight. Can become interesting...

Actually, now I thought about it, and changing the focus somewhat, it is quite astonishing that using unchanged SF search with a differently trained set of weights would produce anything with semi-equivalent Elo to SF.
Reason? Well, SF search is stuffed full of parameter values that compare eval scores with all kinds of things, other eval scores, non-eval functions, all manner of stuff, and we know two things
1. SF searched is highly tuned
2. SF search does not use centipawn scale within, but it’s own variant of material scale, including the ratios Q:R:B:N:P for all game stages.

Which implies that, internally, FF provides the SF search with evals not only as “chessly good” as SF but also to the equivalent scaling.
That would be a mammoth (or desperately lucky) task to train a net to do. Unless you trained the net directly against and on SF.

mwyoung likes this post

Back to top Go down
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 10:27 am

I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2 Very Happy
Back to top Go down
http://rebel13.nl/
Chris Whittington




Posts : 1254
Join date : 2020-11-17
Location : France

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 10:58 am

Admin wrote:
I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2 Very Happy

Then the training problem reduces to:

Train against d=5 (just guessing at the 5), SF evaluated positions. Then you get d5 enhanced SF evaluations in centipawns. Convert the centipawns back to internal SF values (look the conversion up in the SF root output code), so train on those.
Then you get a net which is chessically a bit different but whose scoring range is pretty much the same.

Forensic requirement = get a load of output evals for a load of positions (kill off outlier mate scores), and see how they correlate. Maybe slope and intercept would be a good start.
Compare with correlations of other programs.

Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 11:16 am

Results at 1000ms. Corrected.

Fire 8 released - Page 4 Ff2_re12

Admin and Ozymandias like this post

Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 11:20 am

Admin wrote:
I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2 Very Happy

You know I can not say no. Get it ready!

Can you run all the simex timings at the same time. If you put them all in a different file, and run them at the same time? I have 16 locked cores.

Admin likes this post

Back to top Go down
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 11:25 am

mwyoung wrote:
Results at 1000ms. Corrected.

Fire 8 released - Page 4 Ff2_re12

Great!

Now can you do the same for 500ms, 250ms and 100ms ?

set DATABASE=data\100ms.data
set DATABASE=data\250ms.data
set DATABASE=data\500ms.data

Ozymandias and mwyoung like this post

Back to top Go down
http://rebel13.nl/
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 11:35 am

mwyoung wrote:
Admin wrote:
I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2 Very Happy

You know I can not say no. Get it ready!

Can you run all the simex timings at the same time. If you put them all in a different file, and run them at the same time? I have 16 locked cores.

Yes, you can, just create multiple copies of the sim folder. Disadvantage is that you manually (and correctly) need to merge the *.data output into right time control *data files. But my request above is more simple, see for yourself.

mwyoung likes this post

Back to top Go down
http://rebel13.nl/
TheSelfImprover

TheSelfImprover


Posts : 2874
Join date : 2020-11-18

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 12:08 pm

mwyoung wrote:
Results at 1000ms. Corrected.

Fire 8 released - Page 4 Ff2_re12


Apologies for pointing out the obvious and for being negative, but you've also got a high similarity between SF12 and SF13. Are you trying to confirm or disprove something?
Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 4:26 pm

Results at 500ms.

Fire 8 released - Page 4 Ff2_5010

Ozymandias likes this post

Back to top Go down
Ozymandias




Posts : 622
Join date : 2020-11-23

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 5:21 pm

What CPU were you using?
Back to top Go down
https://centaurchess.boards.net/
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 5:29 pm

Ozymandias wrote:
What CPU were you using?

My Chess testing setup is a CPU 2950x Threadripper, GPU RTX 2080 ti, 7man TB running on a Samsung 970 Plus 2 TB MVMe SSD.
Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 5:40 pm

Admin wrote:
I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2 Very Happy

"then without NNUE and see what happens."

Hello Ed, can you post what settings I need to add to turn off NNUE for FF2, SF13, and SF12. So I get it correct.
I am running the 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE first so no rush.
Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 6:20 pm

Results at 250ms.

Fire 8 released - Page 4 Ff2_2510

Ozymandias likes this post

Back to top Go down
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 7:02 pm

mwyoung wrote:
Admin wrote:
I was thinking along the same line, it explains why the depth=1 method fails. I am going to redo this simex stuff once again, 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE and then without NNUE and see what happens.

And perhaps Mark likes his new toy so much he is willing to do the same with sf13 sf12 and FF2 Very Happy

"then without NNUE and see what happens."

Hello Ed, can you post what settings I need to add to turn off NNUE for FF2, SF13, and SF12. So I get it correct.
I am running the 100ms, 250ms, 500ms, 1000ms and 2500ms with NNUE first so no rush.

Change set OPTIONS this way:

set OPTIONS="Use NNUE=false,depth=77"

mwyoung likes this post

Back to top Go down
http://rebel13.nl/
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 7:14 pm

Results at 100ms. 2500ms results with NNUE will be next. The run time to complete will be 17+ hours.
Fire 8 released - Page 4 Ff2_1010

Ozymandias likes this post

Back to top Go down
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 8:40 pm

Sofar I have: - http://rebel13.nl/text/simex-sf.txt

1000ms and 2500ms ready within 2 hours or so.

n = nnue
c = classic evaluation.

The question is what to do next -

1. Nemorino 6 which is in dispute in EO. If memory serves me well the author said -> own net.
2. I have versions of Ethereal and Rubichess that can handle SF nets.

Edit, never mind I run all 3, Ethereal and Rubichess with the SF12 net.

mwyoung likes this post

Back to top Go down
http://rebel13.nl/
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptyFri Mar 05, 2021 10:53 pm

Update - finished 1000ms and 2500ms, start with Ethereal, Rubichess and Nemorino 100ms

http://rebel13.nl/text/simex-sf.txt

mwyoung likes this post

Back to top Go down
http://rebel13.nl/
Admin
Admin
Admin


Posts : 2522
Join date : 2020-11-17
Location : Netherlands

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptySat Mar 06, 2021 7:55 am

Final - http://rebel13.nl/text/simex-sf.txt

500ms and 1000ms in pictures.

Fire 8 released - Page 4 Sim-500ms

Fire 8 released - Page 4 Sim-1000ms

With reference to Stockfish Handicap Matches IMO, what we see is not similarity but playing strength.
Back to top Go down
http://rebel13.nl/
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptySat Mar 06, 2021 12:34 pm

2500ms results.
Fire 8 released - Page 4 Ff2_2511

Admin and Ozymandias like this post

Back to top Go down
mwyoung

mwyoung


Posts : 880
Join date : 2020-11-25
Location : USA

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptySat Mar 06, 2021 12:44 pm

Started same tests with NNUE=false. 2500ms started, run time needed 17+ hours.
Back to top Go down
Ozymandias




Posts : 622
Join date : 2020-11-23

Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 EmptySat Mar 06, 2021 12:52 pm

For some reason, your 100ms results aren't comparable to mine, so do you think you could run the Github net? 250ms looks good enough.
Back to top Go down
https://centaurchess.boards.net/
Sponsored content





Fire 8 released - Page 4 Empty
PostSubject: Re: Fire 8 released   Fire 8 released - Page 4 Empty

Back to top Go down
 
Fire 8 released
Back to top 
Page 4 of 5Go to page : Previous  1, 2, 3, 4, 5  Next
 Similar topics
-
» Fire on Board {Chiron 5.01 vs Wasp 5.20}
» UHO 2024 released
» Eman 6.70 Released
» PGN Annotator 1.0 released
» Stockfish 15.1 Released.

Permissions in this forum:You cannot reply to topics in this forum
ProDeo :: Computer Chess-
Jump to: