CCRL Sanity Check Tournament. CCRL Top 4 Blitz Engines + Lc0 0.28.0 610032 (CCRL Blitz Conditions)

Posts : 880 Join date : 2020-11-25 Location : USA

CCRL Sanity Check Tournament. CCRL Top 4 Blitz Engines + Lc0 0.28.0 610032 (CCRL Blitz Conditions)

Hardware Threadripper 2950x, RTX 2080 64 Gb Ram, SSD 2 TB Evo 970 Plus.

Book Perfect book.
6 man table bases
TC = 1.5m +.75s
8 CPU
2 GB Hash
500 Rounds

Players

1 Fat Fritz 2 (in SF) 8CPU
2 Lc0 0.28.0 610032 RTX2080
3 Stockfish 14 8CPU
4 Dragon by Komodo 2 8CPU
5 Lc0 0.28.0 t40-1541 RTX2080

Posts : 880 Join date : 2020-11-25 Location : USA

I am not buying what CCRL is trying to sell me with the Lc0 0.28.0 t40-1541 results. The results shown were not rational. They were too good for that net and hardware. And why would CCRL not test with be best nets. But instead test with a very weak and obsolete net like t40-1541. I think I know why.

Will be interesting...

Current difference 120 elo Very Happy

Posts : 880 Join date : 2020-11-25 Location : USA

Admin wrote:: Will be interesting...

Current difference 120 elo

Code:: Rank Name Elo +/- Games Wins Losses Draws Points Score Draw
   1 Fat Fritz 2 (in SF) 8CPU 66 29 112 23 2 87 66.5 59.4% 77.7%
   2 Stockfish 14 8CPU 34 21 112 12 1 99 61.5 54.9% 88.4%
   3 Lc0 0.28.0 610032 RTX2080 25 22 112 11 3 98 60.0 53.6% 87.5%
   4 Dragon by Komodo 2 8CPU -12 30 112 10 14 88 54.0 48.2% 78.6%
   5 Lc0 0.28.0 t40-1541 RTX2080 -116 34 112 0 36 76 38.0 33.9% 67.9%

280 of 5000 games finished.

And it is becoming fascinating as more games are played.... And as our testing predicted Lc0 is ahead of Dragon 2. If using a modern net to test Lc0 and not the obsolete T40-1541. And Dragon 2 is just getting killed with CCRL testing standards.

Here is one thing I can say with 100% certainty. There is no way the neural net for Lc0 T40-1541 achieved a CCRL rating of "#4 Lc0 0.28.0 t40-1541 RTX2080 3669" This rating is total bull shit.

CCRL Sanity Check Tournament. CCRL Top 4 Blitz Engines + Lc0 0.28.0 610032 (CCRL Blitz Conditions) 2uia5210

Mark, I am running also a sanity check with 256x20-t40-1541.pb, size 44.289.015 bytes.

Is that the same net you are using?

Posts : 222 Join date : 2021-08-28

in the old spcc-list (Stefan Pohl) T40-1541 (2583 Elo) is weaker than LS 15 (2643 Elo), both are 256x20, in the current spcc-list T75 is 19 Elo stronger than LS 15. https://www.sp-cc.de/nn-vs-sf-testing.htm, old results here: https://www.sp-cc.de/files/old_nn_rating.txt

I will play 2 sanity matches.

Lc0-v28 for the GRL won with 102 - 98 against Komodo-Dragon 2

Match-1 : Lc0 T40-1541 vs Komodo-Dragon 2
Match-2 : Lc0 default net vs Komodo-Dragon 2

Will show some light on the issue, also for me.

Matches running here - http://rebel13.nl/a/grl.htm

Posts : 880 Join date : 2020-11-25 Location : USA

Admin wrote:: Mark, I am running also a sanity check with 256x20-t40-1541.pb, size 44.289.015 bytes.

Is that the same net you are using?

My size is 53,700 KB.

Either I have the wrong net, or you do. I grabbed mine off of Lc0. T40, and the 1541 iteration of the net.

Posts : 222 Join date : 2021-08-28

It is clearly the same net, I have checked it, the changed size only results after "unpacking" this net, however, this is not recommended and is not necessary...

Posts : 880 Join date : 2020-11-25 Location : USA

Admin wrote:: I will play 2 sanity matches.

Lc0-v28 for the GRL won with 102 - 98 against Komodo-Dragon 2

Match-1 : Lc0 T40-1541 vs Komodo-Dragon 2
Match-2 : Lc0 default net vs Komodo-Dragon 2

Will show some light on the issue, also for me.

Matches running here - http://rebel13.nl/a/grl.htm

What is your hardware used for the test. I am testing Dragon 2 with 8 CPU vs RTX 2080. At CCRL blitz standard settings. So the TC is 2m+1s is really 1.5m + .75s for my CPU. To equal ("Time control: From Jan 2020, equivalent to 2'+1" on an Intel i7-4770K. See the "about" page.")

Posts : 222 Join date : 2021-08-28

The hardware of this test should actually be completely irrelevant, it is about the comparison of the respective networks.

Posts : 880 Join date : 2020-11-25 Location : USA

Dio wrote:: It is clearly the same net, I have checked it, the changed size only results after "unpacking" this net, however, this is not recommended and is not necessary...

Thanks, and this NN came "unpacked" off of the Lc0 download.

Posts : 880 Join date : 2020-11-25 Location : USA

Dio wrote:: The hardware of this test should actually be completely irrelevant, it is about the comparison of the respective networks.

I was interested in how many threads Ed gave D2. I was assuming 1, but it was not clear. And what GPU.

It is always relevant with Lc0 since it is a GPU vs CPU.

Posts : 880 Join date : 2020-11-25 Location : USA

Current standings.

Code:: Rank Name Elo +/- Games Wins Losses Draws Points Score Draw
   1 Stockfish 14 8CPU 60 17 244 43 1 200 143.0 58.6% 82.0%
   2 Fat Fritz 2 (in SF) 8CPU 58 18 243 43 3 197 141.5 58.2% 81.1%
   3 Lc0 0.28.0 610032 RTX2080 24 16 244 25 8 211 130.5 53.5% 86.5%
   4 Dragon by Komodo 2 8CPU -20 20 243 18 32 193 114.5 47.1% 79.4%
   5 Lc0 0.28.0 t40-1541 RTX2080 -126 24 244 1 86 157 79.5 32.6% 64.3%

609 of 5000 games finished.

Posts : 880 Join date : 2020-11-25 Location : USA

Admin wrote:: Will be interesting...

Current difference 120 elo

Current Elo difference is now 150 Elo!. I smell a Rat.

mwyoung wrote:

Admin wrote:: Mark, I am running also a sanity check with 256x20-t40-1541.pb, size 44.289.015 bytes.

Is that the same net you are using?

My size is 53,700 KB.

Either I have the wrong net, or you do. I grabbed mine off of Lc0. T40, and the 1541 iteration of the net.

I got mine from : https://lczero.org/dev/wiki/best-nets-for-lc0/

Apparently there are more than one affraid

mwyoung wrote:

Dio wrote:: The hardware of this test should actually be completely irrelevant, it is about the comparison of the respective networks.

I was interested in how many threads Ed gave D2. I was assuming 1, but it was not clear. And what GPU.

It is always relevant with Lc0 since it is a GPU vs CPU.

GPU 1060, Dragon on one core. I am just replaying a previous match, same conditions.

mwyoung wrote:

Admin wrote:: Will be interesting...

Current difference 120 elo

Current Elo difference is now 150 Elo!. I smell a Rat.

I am half way, -55 elo.

Posts : 880 Join date : 2020-11-25 Location : USA

This was sent to me on my channel. Is this true? CCRL will now test with the T75 nets. And if so why not the better T60. CCRL Sanity Check Tournament. CCRL Top 4 Blitz Engines + Lc0 0.28.0 610032 (CCRL Blitz Conditions) 1f42d

CCRL Sanity Check Tournament. CCRL Top 4 Blitz Engines + Lc0 0.28.0 610032 (CCRL Blitz Conditions) 1f42d

Arun Sathyamurthy
@Mark Young They have now decided to go with default net T75

Graham said they from now on will use the net that comes with the download. That's why I will run a second match with that net. Will be interesting.

Posts : 880 Join date : 2020-11-25 Location : USA

Admin wrote:: Graham said they from now on will use the net that comes with the download. That's why I will run a second match with that net. Will be interesting.

Rebel - "Alright, I regret that I involved Mark, he can't defend himself."

And you are fine for involving me in the discussion. This needed to be said, and they always attack me, but CCRL always makes changes for the better. As I am correct.

And yes I have told CCRL about the Lc0 testing issue before many times, and I was just ignored. It is not until I start running live tests. That CCRL decides to change their ways.

This is the CCRL pattern, and no different then this thread. http://talkchess.com/forum3/viewtopic.php?f=6&t=71149&hilit=obsolete

Attack me, say I am wrong, but then change the CCRL testing standard as I suggested. lol!

Current results.

Code:: [Rank Name Elo +/- Games Wins Losses Draws Points Score Draw
   1 Stockfish 14 8CPU 58 15 315 53 1 261 183.5 58.3% 82.9%
   2 Fat Fritz 2 (in SF) 8CPU 54 16 314 53 5 256 181.0 57.6% 81.5%
   3 Lc0 0.28.0 610032 RTX2080 20 14 314 30 12 272 166.0 52.9% 86.6%
   4 Dragon by Komodo 2 8CPU -13 17 313 24 36 253 150.5 48.1% 80.8%
   5 Lc0 0.28.0 t40-1541 RTX2080 -122 21 314 1 107 206 104.0 33.1% 65.6%

785 of 5000 games finished.

Result T40-1541

Code:: No. Name Win Draw Loss Unf. Score Games %
-------------------------------------------------------------
1 Komodo-Dragon 2 +58 =115 -27 *0 115.5 200 57.8%
2 Lc0 v28-t40-1541 +27 =115 -58 *0 84.5 200 42.2%

Next the default net.

Posts : 880 Join date : 2020-11-25 Location : USA

Current results.

Code:: Rank Name Elo +/- Games Wins Losses Draws Points Score Draw
   1 Stockfish 14 8CPU 61 12 510 90 2 418 299.0 58.6% 82.0%
   2 Fat Fritz 2 (in SF) 8CPU 47 12 508 78 10 420 288.0 56.7% 82.7%
   3 Lc0 0.28.0 610032 RTX2080 23 11 508 52 19 437 270.5 53.2% 86.0%
   4 Dragon by Komodo 2 8CPU -13 13 509 41 60 408 245.0 48.1% 80.2%
   5 Lc0 0.28.0 t40-1541 RTX2080 -120 16 511 2 172 337 170.5 33.4% 65.9%

1273 of 5000 games finished.

Posts : 880 Join date : 2020-11-25 Location : USA

Admin wrote:

Result T40-1541

Code:: No. Name Win Draw Loss Unf. Score Games %
-------------------------------------------------------------
1 Komodo-Dragon 2 +58 =115 -27 *0 115.5 200 57.8%
2 Lc0 v28-t40-1541 +27 =115 -58 *0 84.5 200 42.2%

Next the default net.

On September 3 I ran this test that had the Lc0 default net as one of the players. This was my results.

Code:: Hardware Threadripper 2950x, RTX 2080 TI, 64 Gb Ram, SSD 2 TB Evo 970 Plus.

Book Perfect Book 2021.
7 man table bases.
TC = 3m+2s.

Est. Run time for Tournament 6.94 Days.

Players
   1 Lc0 0.28.0 - J98-1-16 Net
   2 Lc0 0.28.0 - 610025 Net
   3 Lc0 0.28.0 - J94-100 Net
   4 Lc0 0.28.0 - J92-300 Net
   5 Lc0 0.28.0 - Default Net
   6 Lc0 0.28.0 - CPU 16 Cores Default Net

Rank Name Elo +/- Games Wins Losses Draws Points Score Draw
   1 Lc0 0.28.0 - 610025 Net 47 17 200 28 1 171 113.5 56.8% 85.5%
   2 Lc0 0.28.0 - J94-100 Net 42 19 200 28 4 168 112.0 56.0% 84.0%
   3 Lc0 0.28.0 - J98-1-16 Net 38 17 200 25 3 172 111.0 55.5% 86.0%
   4 Lc0 0.28.0 - J92-300 Net 38 17 200 25 3 172 111.0 55.5% 86.0%
   5 Lc0 0.28.0 - Default Net 37 19 200 27 6 167 110.5 55.3% 83.5%
   6 Lc0 0.28.0 - CPU 16 Cores Default Net -230 36 200 0 116 84 42.0 21.0% 42.0%

And just a side note. As you can see from this test, a RTX GPU gives Lc0 a huge jump in performance over 16 CPU's, and this is equal to a lower end GPU.

Default net

Code:: No. Name Win Draw Loss Unf. Score Games %
------------------------------------------------------------
1 Komodo-Dragon 2 +53 =121 -26 *0 113.5 200 56.8%
2 Lc0 v28-752187 +26 =121 -53 *0 86.5 200 43.2%

Not great either.

» Top Engine Blitz Tournament (3300 Games) (12 Engines)
» Results of the top ten strongest engines tourney at CCRL
» CEGT BLITZ - version history engines
» June Blitz Tournament, today, Saturday 20:00 Berlin time
» Live Tournament NNUE Nets Vs Leela Nets, Blitz 5 min + 5s (12 Rounds)