SSDF Rating List 23-05-24

Posts : 11 Join date : 2023-05-24

Ozymandias wrote:: I know, I've been following the list for over 15 years, but there were no free notable programs without a book, back then. Now the two main contenders are left to play with unknown books.

I would hardly call Sedat Canbaz and Fauzi Dabat, unknown book makers and/or with unknown opening books. Sedat Canbaz has been actively developing and releasing his free and very strong Perfect-books for many many years now. Fauzi has also been the SF-team's choice of opening book maker for us in the SSDF since very many revisions of SF (I forgot for which SF-version it all started, but it was very long time ago now). He has been actively developing his dedicated (and free) SF-book for our testing during the years and has done some great work with the book in my opinion. It adheres to our policy and his SF-book is a free and smaller variant of his bigger opening book that he sells for money (and which we cannot use because of our policy which I mentioned in an earlier post).

Posts : 11 Join date : 2023-05-24

Chris Whittington wrote:

Ozymandias wrote:: I know, I've been following the list for over 15 years, but there were no free notable programs without a book, back then. Now the two main contenders are left to play with unknown books.

NN’s ought not to need a book, their moves should be better than human knowledge. Only real reason for book nowadays is variance

For me, opening books constitute:

* A source of variance which leads to fun. Fun is really important! Without variance and fun in my manual and automatic matches/meetings, I would hardly have been active in this field as long (33 years and counting) as I have been now. One of the key-factors behind that I still have been enjoying manual testplay for example, has been that the machines are able to vary their opening choices so that I don't have to play out the exact same opening each and every game. I prefer a varied book over an extremely selective book, but I don't mind how deep it is. We have had some automatically tested programs that have a too shallow selection of opening moves (and is not possible to change), and when paired with some special opponent, has been able to play out 40 (out of 40) automatic games in the exact same opening. Bone dry meeting and not the 40-game match which I will remember most fondly, that is for certain.
* A dedicated opening book has been part of the package almost since the start of the chess computer era. Without the knowledge of opening moves, chess computers would never have garnered the interest they have until today and earlier chess computer sales would have been affected badly. I believe that opening books, or the knowledge of some rudimentary opening moves, are important for us humans to be able to "humanize" the chess computers a bit. A not so unimportant feature I believe. Since its relatively human to at least have some basic knowledge of opening moves when you learn to play chess, I guess one finds that it is more human-like for a computer to have some knowledge in the first part of the game.
* A chance for the programmer to be able to "personalize" the chess computer a little bit and also a chance for the programmer to let the program come out of the opening book in favorable positions for the type of positions in which the programs excels in. This is of course not as important any more, since nearly all programs are so strong now and it's also maybe hard for a human to comprehend in which type of positions the specific program would play best. This isn't meaning that one should have a book full of only so called "dedicated killer-lines", since I believe it's rather pointless nowadays and also is not worth the effort.

I have some more points, but I will try to keep the reply shorter and exclude them from the above for the readability sake. That being said, an opening book is still an important part for me/us, as it is how we have been testing chess computers from the start. It's impossible for us in the SSDF to test otherwise, since it would nullify previous testing.

The day when a chess computer are able to make up it's own opening book/repertoire and being able to vary it accordingly just by playing, is probably coming, but we are not there yet to the best of my knowledge.

Posts : 11 Join date : 2023-05-24

Uri Blass wrote:

Lars Sandin wrote:

Code:: THE SSDF RATING LIST 2023-05-24* 162991 games played by 435 computers
   Rating + - Games Won Oppo
   ------ --- --- ----- --- ----
   1 Lc0 0.29.0 Cuda-808544 3060Ti 3610 45 -41 287 68% 3466
   2 Lc0 0.28.2 Cuda-611213 3060Ti 3571 26 -25 780 66% 3456
   3 Lc0 0.26.3 Cuda(67362) 3060Ti 3570 29 -27 680 68% 3441
   4 Stockfish 15 x64 1800X 3.6 GHz 3570 39 -36 367 66% 3456
   5 Stockfish 13 x64 1800X 3.6 GHz 3568 35 -32 480 69% 3432
   6 Dragon Komodo 3 x64 1800X 3.6 GHz 3567 40 -38 329 62% 3482
   7 Dragon Komodo 2.51 x64 1800X 3.6 GHz 3565 42 -39 320 65% 3454
   8 Stockfish 14 x64 1800X 3.6 GHz 3555 37 -35 400 65% 3452
   9 Stockfish 12 NNUE x64 1800X 3.6 GHz 3549 26 -25 760 60% 3479
  10 Rebel 16.2 x64 1800X 3.6 GHz 3544 48 -47 209 53% 3522
  11 Dragon by Komodo x64 1800X 3.6 GHz 3536 34 -32 460 63% 3443
  12 Stockfish 11 x64 1800X 3.6 GHz 3529 36 -34 450 70% 3386
  13 Arasan 23.4 x64 1800X 3.6 GHz 3524 54 -55 160 47% 3560
  14 Dragon Komodo 3.1 MCTS 1800X 3.6 GHz 3514 34 -33 440 60% 3447
  15 Stockfish 10 x64 1800X 3.6 GHz 3501 25 -23 920 67% 3378
  16 Booot 7 x64 1800X 3.6 GHz 3477 39 -40 300 46% 3507
  17 Dragon Komodo 2 MCTS x64 1800X 3.6 GHz 3473 45 -44 240 55% 3439
  18 Stockfish 9 x64 1800X 3.6 GHz 3464 24 -23 962 67% 3345
  19 Komodo 14 x64 1800X 3.6 GHz 3455 30 -30 520 52% 3441
  20 Arasan 23.01 x64 1800X 3.6 GHz 3449 33 -34 420 48% 3462
  21 Arasan 23.4 x64 Q6600 2.4 GHz 3439 40 -37 361 68% 3314
  22 Pedone 3.1 x64 1800X 3.6 GHz 3433 38 -39 320 45% 3468
  23 Wasp 6.5 x64 1800X 3.6 GHz 3433 44 -45 242 44% 3478
  24 Stockfish 9 x64 Q6600 2.4 GHz 3427 32 -31 480 56% 3384
  25 Booot 6.5 x64 1800X 3.6 GHz 3426 34 -34 400 48% 3436
  26 Wasp 5.5 x64 1800X 3.6 GHz 3378 37 -39 360 36% 3486
  27 Arasan 22.3 x64 1800X 3.6 GHz 3360 39 -42 320 35% 3468
  28 Marvin 6.0.0 x64 1800X 3.6 GHz 3358 47 -53 220 31% 3494
  29 Pedone 3.1 x64 Q6600 2.4 GHZ 3356 38 -37 340 58% 3303
  30 Deep Shredder 13 x64 1800X 3.6 GHz 3347 24 -23 920 62% 3261
  31 Chiron 5.01 x64 1800X 3.6 GHz 3344 33 -35 440 34% 3455
  32 Marvin 5.2.0 x64 1800X 3.6 GHz 3325 37 -40 360 32% 3456
  33 Chiron 5.01 x64 Q6600 2.4 GHz 3318 34 -34 400 52% 3301
  34 Vajolet2 2.8 x64 1800X 3.6 GHz 3279 27 -28 652 38% 3364
  35 Deep Hiarcs 14 1800X 3.6 GHz 3206 23 -24 880 38% 3289
  36 Revelation 2 A.E. Komodo 12.1 A7 1 GHz 3177 73 -68 100 59% 3113
  37 Revelation 2 Hiarcs 14.1 PXA320 800 MHz 2924 47 -45 228 56% 2881
  38 Chessmaster King 3.5 x64 Q6600 2.4 GHz 2857 24 -25 932 30% 3005
  39 Revelation Hiarcs 13.3 PXA255 500 MHz 2772 57 -52 177 66% 2660
  40 Revelation Shredder 12 PXA255 500 MHz 2706 60 -58 140 56% 2666
  41 Revelation Rybka 2.2 PXA255 500 MHz 2635 45 -43 260 63% 2546
  42 Revelation Deep Sjeng 3 PXA255 500 MHz 2600 62 -67 120 41% 2665
  43 Millennium The King Exclusive 300 MHz 2534 49 -48 200 55% 2501
  44 Revelation Ruffian 2.1 PXA255 500 MHz 2346 68 -71 100 45% 2385
  45 Millennium ChessGenius Excl. M7 300 MHz 2249 50 -48 204 58% 2193
  46 Mephisto London 68030 33 MHz 2195 31 -31 482 50% 2192
  47 Millennium ChessGenius Pro M4 120 MHz 2163 59 -54 160 63% 2067
  48 Mephisto London 68020 12 MHz 2089 60 -53 171 68% 1957
  49 Millennium ChessGenius ARM M4 48 MHz 2070 45 -43 251 58% 2015
  50 Mephisto Roma II 68000 10 MHz 1898 63 -59 132 58% 1843

1 Lc0 0.29.0 Cuda-808544 3060Ti, 3610
Stoc15 1800X 4-3 Reb162 1800X 21,5-18,5 Aras234 1800X 46-34
Stock9 1800X 29-11 Wasp65 1800X 28,5-11,5 Wasp55 1800X 66-14

10 Rebel 16.2 x64 1800X 3.6 GHz, 3544
Lc029 3060Ti 18,5-21,5 Dra3 1800X 4,5-4,5 Stoc14 1800X 19-21
Aras234 1800X 20,5-19,5 Boo7 1800X 23,5-16,5 Wasp65 1800X 24-16

13 Arasan 23.4 x64 1800X 3.6 GHz, 3524
Lc029 3060Ti 34-46 Reb162 1800X 19,5-20,5 Boo7 1800X 21-19

21 Arasan 23.4 x64 Q6600 2.4 GHz, 3439
Komo12 Q6600 21,5-18,5 Stock8 Q6600 21,5-18,5 Ped31 Q6600 22,5-17,5
Chi5 Q6600 85,5-34,5 DShre13 Q6600 24-16 Vaj28 Q6600 1-0
Mar51 Q6600 33,5-6,5 DRybka3 Q6600 34,5-5,5

23 Wasp 6.5 x64 1800X 3.6 GHz, 3433
Lc029 3060Ti 11,5-28,5 Stoc15 1800X 11,5-28,5 Reb162 1800X 16-24
Ped3 1800X 44,5-35,5 Komo11 1800X 21,5-18,5 Vaj28 1800X 2-0

36 Revelation 2 A.E. Komodo 12.1 A7 1 GHz, 3177
DShre13 Q6600 6-14 DHiar14 Q6600 9,5-10,5 DJuni12 Q6600 11,5-8,5
Baro343 Q6600 15,5-4,5 Naum 4 A1200 16,5-3,5

Since the last rating list we can now present six new entrants in the list.

First one out is our new leader in the list: Lc0 0.29.0, and also the first
program that has broken the 3600-barrier in our rating list. More games will
be needed to lower the error bars, but after the first 297 games it has placed
itself 39 points over the last version tested with a huge rating of 3610!
It will be interesting to see how it will hold up as more games are played
against the top programs.
The Lc0-version which we have tested this time is: Lc0 0.29.0, with the
19 filter + 512 block network, named 808544 (date: 2023-01-12). As with the
earlier Lc0-versions, we have used the free opening book "perfect2021.abk"
by Sedat Canbaz for the testing of Lc0.

Our next newcomer in the list are one really strong comeback from the team of
Ed Schroeder and Chris Wittington. Chris Wittington is formely known for
programs such as: Chess Player 2150/2175, Complete Chess System
and Chess System Tal (I and II).
Ed Schroeder is of course known for many of the old dedicated chess computers
like for instance: Mephisto Polgar, MM IV/V, RISC (I and II), Rebell
and also for versions of his program in the ChessMachine ISA-plugin card for PC.
On the PC-scene he has made the Gideon and Rebel-program famous and lately also
made several interesting and strong versions of his ProDeo-program.

Since last year they have utilized Neural Net-technology in their joint
effort and has seemingly improved fast. We have waited some time to test it,
since we wanted it to share most of the functions - like for instance Pondering
and Multi-Processor functionality, which the others programs that we test on our
latest hardware uses. The version which we have tested is named: Rebel 16.2 and
we have used its own opening book, made by the famous Jeroen Noomen, for the
testing. After the first 209 games, this strong combo of the program: Rebel 16.2,
and the opening book of Jeroen Noomen, has managed to get an impressive rating
of 3544 on our 1800X hardware. It is placed 10th in the list at the moment,
just 5 points shy of the Stockfish 12-program.
A very strong comeback by Ed, Chris and Jeroen!

Our next addition to the list are Jon Dart's strong and reliable Arasan-program.
The version that we have tested on our 1800X and Q6600 hardware are Arasan 23.4.
On our 1800X hardware it has reached a rating of 3524 after the initial 160
games played. That is at the moment 75 points ahead of the formerly tested 23.01
version and a very nice improvement indeed! More games will of course be needed
to lower the error bars a bit.
On our Q6600 hardware, Arasan 23.4 has reached a rating of 3439 after 361 games
played. We never tested the 23.01 version on the Q6600 hardware, but Arasan 23.4
are a whopping 202 points ahead of the 21.2 version at least!
We have, as always, used Arasan's own opening book for the testing.

We are also glad to be able to present a new strong Wasp version by John Stanback,
namely Wasp 6.5, on our latest rating list. After the first 242 games played, it
has reached a rating of 3433! This is 55 points stronger than the 5.5 version
which we tested before, so a nice improvement there also! We have used Wasp's own
opening book for the testing.

And we can now finally show the first rating of the manually tested:
Phoenix Revelation II Anniversary Edition chess computer by Ruud Martin/DGT. This
Anniversary Edition of the Revelation II, features an updated hardware relative
to the ordinary Revelation II. The hardware for the Anniversary Edition is an
Colibri iMX7D with 2 ARM Cortex A7 processors running at 1GHz. The software which
we have used for the testing is the program Komodo 12.1 by Mark Lefler and
Larry Kaufman.
The rating that has been reached after the first 100 games is 3177! This is a
rating that is 253 points ahead of Hiarcs 14.1, which we tested formerly
on the ordinary Revelation II. It is clearly the strongest dedicated chess
computer which we have tested so far and it has been able to perform admirably
against the strong Q6600 opposition which it mostly has faced in our testing
so far.

Aside from producing more games with the already mentioned newcomers in the
rating list, we are at the moment testing Richard Pijl's - The Baron 3.45,
and also Alex Morozov's - Booot 7.1. We also hope to have some more programs
ready for the next rating list.

* This rating list was produced and based on the games/results that were finished
on the first of May, but the publication of this rating list was unfortunately a
bit delayed this time.
The testing has of course not stood still since then, and the games/results
that has been played since this date will of course feature in the next rating
list, which we plan to have out sometime in august/september.

Lars Sandin

I see no games and I wonder if the leader only won or drew and did not lose a single game.
I expect that at some point the SSDF leaders are not going to lose a single game when the only difference between them is going to be in the result that they can beat weaker opponents.

You can always check out our SSDF homepage if you are interested in checking out the games played. We keep the database of test games there. The top is indeed more compact now with the advent of Neural Nets and as the playing-strength is progressing it would only be natural to find more and more drawn games between two closely spaced out programs. This wasn't such a huge factor when two programs was close to each other in the list before, as they both was of lower strength than today. But I hardly doubt that chess is solved and that todays best chess computers are playing perfect chess. There is still room for improvement IMHO.

Posts : 11 Join date : 2023-05-24

Uri Blass wrote:

Chris Whittington wrote:

Ozymandias wrote:: I know, I've been following the list for over 15 years, but there were no free notable programs without a book, back then. Now the two main contenders are left to play with unknown books.

NN’s ought not to need a book, their moves should be better than human knowledge. Only real reason for book nowadays is variance

Books can help to save time on the clock.
Also if you want to get first place in the SSDF rating list then
when you play many games against the same oponnent it is better to learn to repeat lines that you won games with them and avoid lines that you did not win with them.

Saving time is indeed a factor not to forget!

The function of book learning for many programs have been standard since long time ago. I doubt that you have to do something extremely special with the opening book to make it to the top, other than making a well-playing chess engine and having a good book. Lc0 is leading the rating list with Sedat Canbaz's: Perfect2021.abk book, and that opening book is hardly targeted for the SSDF-testing specifically. It is a well designed opening book that seem's to be working great for Lc0.

Subject: Re: SSDF Rating List 23-05-24

» New Rating List ?
» Speedy rating list
» First Gambit Rating List
» Rating List Experiment
» Gambit Rating List - May 20, 2021