Fire 8 released

mwyoung wrote:

Admin wrote:

First try - http://rebel13.nl/text/rating-1.txt

I am pretty good with that list. As that is what I test mostly by accident. I did not look at the sim test.

Mark, I read you have FF2 and I heard it is an UCI engine, is it really? If so are you willing to do me a favor and run a sim test? Takes not more than 30 minutes. I will prepare a download and you only have to click on a batch file.

Ed,

if I remember well there was some controversy about Schooner, and you retested it, with much lower similarity values. Nonetheless, you excluded it from the list. Is it just an oversight?

@matejst OT question - what's this? It looks like a VCR circa 1980 - apart from the two attachments.  

Admin wrote:

mwyoung wrote:

Admin wrote:

First try - http://rebel13.nl/text/rating-1.txt

I am pretty good with that list. As that is what I test mostly by accident. I did not look at the sim test.

Mark, I read you have FF2 and I heard it is an UCI engine, is it really?  If so are you willing to do me a favor and run a sim test? Takes not more than 30 minutes. I will prepare a download and you only have to click on a batch file.

Yes it is a UCI engine. I am also running Fat Fritz 2 in Cute chess gui. I am just now leaving for work. So get it ready, and I will try and run the sim tonight.

@TheSelfImprover:

I do not have a clue. I was searching for a profile photo, and found this. Thought it was an earlier computer with two cassettes.

@Ed, Mark:

Somebody already tested FF2 (with the available net) here:
http://talkchess.com/forum3/viewtopic.php?f=10&t=72250

matejst wrote:

Ed,

if I remember well there was some controversy about Schooner, and you retested it, with much lower similarity values. Nonetheless, you excluded it from the list. Is it just an oversight?

Yep, my bad.

TheSelfImprover wrote:

@matejst OT question - what's this?  It looks like a VCR circa 1980 - apart from the two attachments.  

matejst wrote:

@TheSelfImprover:

I do not have a clue. I was searching for a profile photo, and found this. Thought it was an earlier computer with two cassettes.

Hi guys, it is a NES (Nintendo Entertainment System).
I never had one nor played with, but my grown-up son just built a Lego reproduction; this is how I know about...

https://en.wikipedia.org/wiki/Nintendo_Entertainment_System
https://www.lego.com/en-us/product/nintendo-entertainment-system-71374

Cheers,
Tibono

Excellent! Thanks for clearing that up - https://en.wikipedia.org/wiki/Nintendo_Entertainment_System

Belongs on this forum - used a 6502 CPU.

One name you listed that comes as a surprise is Schooner. What did you find out about it?

I remember it had a great style since it was a 'lowly' 2500-2600 engine.

EDIT: never mind, I see it was an oversight.

mwyoung wrote:

Admin wrote:

mwyoung wrote:

Admin wrote:

First try - http://rebel13.nl/text/rating-1.txt

I am pretty good with that list. As that is what I test mostly by accident. I did not look at the sim test.

Mark, I read you have FF2 and I heard it is an UCI engine, is it really?  If so are you willing to do me a favor and run a sim test? Takes not more than 30 minutes. I will prepare a download and you only have to click on a batch file.

Yes it is a UCI engine. I am also running Fat Fritz 2 in Cute chess gui.  I am just now leaving for work. So get it ready, and I will try and run the sim tonight.

I don't have much hope, here is the trouble with NNUE networks:



1. You can see there is no similarity between sf12 and sf13. Before NNUE the similarity between new versions typically is 70-80%, NNUE changed everything.

2. sf13 is just a couple of weeks old, so I also tested with the latest network, hardly can't be improved a lot. But as you can see, no similarity. That's the trouble with NNUE networks, one tiny change and SIMEX has no clue.

Anyway, I will prepare the download for you.

Mark - http://rebel13.nl/dump/sim-ff.7z

Make sure FF2 is in the engine folder.

If that's a problem modify the run.bat file to the place on your hd.

Code:

set EXE=engines\fat-fritz2-x86-64-avx.exe

Double click run.bat, it will run FF2 first, then SF12 and as last one SF13.

Good luck.

Admin wrote:

Before NNUE the similarity between new versions typically is 70-80%, NNUE changed everything.

Maybe using depth=1, but with 100ms, I get consistent results. Almost 99% when running the same SFdev twice, over 66% vs. SF13. 56% for dark horse 3...

Ozymandias wrote:

Admin wrote:

Before NNUE the similarity between new versions typically is 70-80%, NNUE changed everything.

Maybe using depth=1, but with 100ms, I get consistent results. Almost 99% when running the same SFdev twice, over 66% vs. SF13. 56% for dark horse 3...

Time control seems to be the only option left. The disadvantage is that nowadays engines with 100ms already are above 3000 elo and thus similarity automatically increases. Tested this in the past, the more time the higher the similarity. See the 100/250/500/1000 and 2500ms examples at - http://rebel13.nl/misc/simex.html

That's unavoidable, either you choose test positions with many equally good solutions, or programs will agree more, the stronger they are. In any case, 100ms seems better than depth=1 from what you said/I tested. Good enough in any case to see if FF2 is more or less similar to SF13 than other dev versions or derivatives.

Admin wrote:

Mark - http://rebel13.nl/dump/sim-ff.7z

Make sure FF2 is in the engine folder.

If that's a problem modify the run.bat file to the place on your hd.

Code:

set EXE=engines\fat-fritz2-x86-64-avx.exe

Double click run.bat, it will run FF2 first, then SF12 and as last one SF13.

Good luck.

I got FF2 to run on the 2nd attempt. Here are the results for this run.

Admin wrote:

Ozymandias wrote:

Admin wrote:

Before NNUE the similarity between new versions typically is 70-80%, NNUE changed everything.

Maybe using depth=1, but with 100ms, I get consistent results. Almost 99% when running the same SFdev twice, over 66% vs. SF13. 56% for dark horse 3...

Time control seems to be the only option left. The disadvantage is that nowadays engines with 100ms already are above 3000 elo and thus similarity automatically increases. Tested this in the past, the more time the higher the similarity. See the 100/250/500/1000 and 2500ms examples at - http://rebel13.nl/misc/simex.html

"engines with 100ms already are above 3000 elo"

And I see that you understand.

Thanks! What I already suspected, non conclusive numbers. If you are in the mood would you like to do a second test? Now not with depth=1 but with 100ms.

All you have to do is to change the set OPTIONS= parameter and change depth=1 into depth=77

Admin wrote:

Thanks! What I already suspected, non conclusive numbers. If you are in the mood would you like to do a second test? Now not with depth=1 but with 100ms.

All you have to do is to change the set OPTIONS= parameter and change depth=1 into depth=77

D1 basically asks the NN direct? With some interference from QSearch() presumably.

How about asking for MultiPV, four(?) moves, and checking how those correlate? In fact you could maybe redo the entire SIM software based on best N moves correlations.

Another idea:
For SF compatible NNs, do a recompile hack of SF to kill QSearch(), then it really will return eval d=1, and you test all the nets within the SF framework.

Did some overnight NNUE testing at 100ms with different SF nets and also included Nemerino 6.00 with its own net (presumably a SF net?) and special versions of Rubichess and Ethereal with SF NNUE support. You can see the impact of SF NNUE at - http://rebel13.nl/download/srl-1000.html



Conclusion, it's going nowhere.

Chris Whittington wrote:

Admin wrote:

Thanks! What I already suspected, non conclusive numbers. If you are in the mood would you like to do a second test? Now not with depth=1 but with 100ms.

All you have to do is to change the set OPTIONS= parameter and change depth=1 into depth=77

D1 basically asks the NN direct? With some interference from QSearch() presumably.

How about asking for MultiPV, four(?) moves, and checking how those correlate? In fact you could maybe redo the entire SIM software based on best N moves correlations.

Another idea:
For SF compatible NNs, do a recompile hack of SF to kill QSearch(), then it really will return eval d=1, and you test all the nets within the SF framework.

Third option, compare 2 NNUE files and look for similarities (patterns). Do you know the NNUE data structure?

Admin wrote:

Chris Whittington wrote:

Admin wrote:

Thanks! What I already suspected, non conclusive numbers. If you are in the mood would you like to do a second test? Now not with depth=1 but with 100ms.

All you have to do is to change the set OPTIONS= parameter and change depth=1 into depth=77

D1 basically asks the NN direct? With some interference from QSearch() presumably.

How about asking for MultiPV, four(?) moves, and checking how those correlate? In fact you could maybe redo the entire SIM software based on best N moves correlations.

Another idea:
For SF compatible NNs, do a recompile hack of SF to kill QSearch(), then it really will return eval d=1, and you test all the nets within the SF framework.

Third option, compare 2 NNUE files and look for similarities (patterns). Do you know the NNUE data structure?

Connor said he’s done that and nothing stood out.

I’ll check pytorch docs, might be possible to get a visual weight inspection via heat maps.

Seems like nothing obvious apart from display image data.
Somewhere Gary Linscott was showing huge heat maps, but this probably means writing some software. I am to busy elsewhere to get motivated.

I concur, I looked at binaries, one big mistery. Heat maps are not convincible, I am looking for just that, something that is convincable, depth=1 was.

To kill QS from SF is worth a try. But I will do depth=2 first, who knows...

Admin wrote:

Thanks! What I already suspected, non conclusive numbers. If you are in the mood would you like to do a second test? Now not with depth=1 but with 100ms.

All you have to do is to change the set OPTIONS= parameter and change depth=1 into depth=77

I'm confused, the picture says "100ms".

Ozymandias wrote:

Admin wrote:

Thanks! What I already suspected, non conclusive numbers. If you are in the mood would you like to do a second test? Now not with depth=1 but with 100ms.

All you have to do is to change the set OPTIONS= parameter and change depth=1 into depth=77

I'm confused, the picture says "100ms".

Glitch.

Admin wrote:

Did some overnight NNUE testing at 100ms with different SF nets and also included Nemerino 6.00 with its own net (presumably a SF net?) and special versions of Rubichess and Ethereal with SF NNUE support. You can see the impact of SF NNUE at - http://rebel13.nl/download/srl-1000.html



Conclusion, it's going nowhere.

I did a 3rd run and the results were pretty much the same at 100ms. For fun I think I have the 4th run doing a 1000ms search. I do not know if this is meaningful. But I like playing with new toys. But this run will take some time.