Gambit Tournament, first results available

Subject: Gambit Tournament, first results available Fri Apr 23, 2021 10:48 am

Ratings of chess engines are well established on the known rating lists. Will anything change when we feed engines with only (playable) gambit positions?

We carefully created 50 gambit opening positions to find out.

With 6 top entries that means 500 games each. First run at 40/2 ccrl style as a warming up followed by the real thing 40/15 ccrl style.

In the 40/15 tournament Lc0 v27 (default net) will participate also but it will play at 40/2 to overcome the GPU hardware advantage.

....

http://rebel13.nl/misc/gambit-tournament.html

The mighty Dragon moved in. The gap between SF, Komodo and the rest of the pack is amazing.

Meanwhile I have given up on Lc0, it doesn't work well enough using multiple entries and running 700 games single threaded at 40/2 takes too long to my taste. Pity because I believe in these kind of positions Lc0 can beat the almighty Stockfish.

Subject: Re: Gambit Tournament, first results available Sat Apr 24, 2021 8:48 am

40 moves in 15 minutes
Tournament is expected to run for 5-7 days, there will be an update each day about its progress.
Update after 198 (of 1500) games are played

Code:: No. Engine 1 2 3 4 5 6 Score Games Perc Moves
------------------------------------------------------------------------------------
1 Stockfish 13 xxxx 6.5 10.0 10.5 10.5 12.0 49.5 / 66 (75.00%) 71.7
2 Komodo-Dragon 6.5 xxxx 9.0 10.0 10.5 10.5 46.5 / 66 (70.45%) 75.2
3 Ethereal_12.75 3.0 4.0 xxxx 7.0 5.0 7.5 26.5 / 66 (40.15%) 74.9
4 Igel_3.0.0 3.5 3.0 6.0 xxxx 7.5 5.5 25.5 / 66 (38.64%) 70.7
5 RubiChess_2.1 2.5 3.5 8.0 5.5 xxxx 6.0 25.5 / 66 (38.64%) 70.9
6 SlowChess_2.5 1.0 2.5 6.5 7.5 7.0 xxxx 24.5 / 66 (37.12%) 65.7

Code:: PLAYER : RATING POINTS PLAYED (%) W D L D(%) OppAvg
1 Stockfish 13 : 3657.6 483.0 600 81 375 216 9 36 3396.7
2 Komodo Dragon : 3572.2 421.5 600 70 294 255 51 43 3411.0
3 Komodo 14 : 3425.6 296.0 600 49 165 262 173 44 3435.4
4 RubiChess 2.1 : 3363.0 240.0 600 40 110 260 230 43 3445.8
5 SlowChess 2.5 : 3353.3 231.5 600 39 101 261 238 44 3447.4
6 Ethereal 12.75 : 3341.3 221.0 600 37 101 240 259 40 3449.5
7 Igel 3.0.0 : 3325.0 207.0 600 35 70 274 256 46 3452.2

In the 40/2 edition SF lost only 9 games!

An overall lesser draw rate also.

I have decided Lc0 plays the winner of the 40/2 tournament, thus sf13 -Lc0-v27, the 100 gambit games which is doable within an acceptable time, estimated 2 days.

I tested the speed of 3 nets:
128x10 - nps 20,000
256x20 - nps 4,000
384x30 - nps 1,200

I have chosen for the 256 net.

New set up -

Ratings of chess engines are well established on the known rating lists. Will anything change when we feed engines with only (playable) gambit positions? We carefully created 50 gambit opening positions to find out.

As first step following the established ratings on the CCRL rating list we divide (only active) engines into groups based on their strength and run a robin match. Thus a group of 6 engines will play 500 games each and 1500 games in total. Two time controls CCRL style, 40/2 and 40/15.

As second step with a reasonable number of engines and games we will create a rating list. Games will be available for download when we have created a second set of 50 gambit positions.

Excluded Engines
1. Inactive engines, those engines who haven't been updated for a long time.
2. Known derivatives.

http://rebel13.nl/misc/gambit-tournament.html

Posts : 13 Join date : 2021-04-28

Very interesting setup, the results without surprise to anyone had an expected outcome
Can you upload the 50 gambit book that was used during the tournament please?

polotoprak wrote:: Very interesting setup, the results without surprise to anyone had an expected outcome
Can you upload the 50 gambit book that was used during the tournament please?

I am in the process of creating a rating list from the all the games (pools) which is not a simple task. When finished the games will be available for download.

Meanwhile the new Igel 3.0.5 is running the gauntlet and Lc0 still running for the premier league. So far:

Code:: No. Engine 1 2 3 4 5 6 Score Games Perc Moves
------------------------------------------------------------------------------------
1 Lc0 v27 RTX-1060 xxxx 32.0 46.5 66.0 70.0 73.0 287.5 / 500 (57.50%) 83.9
2 Stockfish 13 68.0 xxxx 0.0 0.0 0.0 0.0 68.0 / 100 (68.00%) 65.1
3 Komodo Dragon 53.5 0.0 xxxx 0.0 0.0 0.0 53.5 / 100 (53.50%) 100.7
4 Komodo 14 34.0 0.0 0.0 xxxx 0.0 0.0 34.0 / 100 (34.00%) 88.9
5 SlowChess 2.5 30.0 0.0 0.0 0.0 xxxx 0.0 30.0 / 100 (30.00%) 79.5
6 Ethereal 12.75 27.0 0.0 0.0 0.0 0.0 xxxx 27.0 / 100 (27.00%) 85.5

It still must play Rubi and Igel and it's not unlikely Lc0 is going to win the tournament while it has lost badly against SF13. The others with the exception of the Dragon are crushed by Lc0. Point of attention (criticism) is the hardware advantage of Lc0. On the RTX-1600 playing with a 256 net Lc0 does ±4000 NPS.

Posts : 13 Join date : 2021-04-28

Admin wrote:

polotoprak wrote:: Very interesting setup, the results without surprise to anyone had an expected outcome
Can you upload the 50 gambit book that was used during the tournament please?

It still must play Rubi and Igel and it's not unlikely Lc0 is going to win the tournament while it has lost badly against SF13. The others with the exception of the Dragon are crushed by Lc0. Point of attention (criticism) is the hardware advantage of Lc0. On the RTX-1600 playing with a 256 net Lc0 does ±4000 NPS.

I honestly thought Lc0 will match SF13, gambit like positions are Lc0's best time to shine... kinda disappointed by the results, will wait for the final results

Subject: Re: Gambit Tournament, first results available Sat May 01, 2021 5:16 pm

Lc0 is a little stronger then latest stockfish.
If there is a huge gap in a tournament rating list between Stockfish and lc0,
There must be a major glitch.
Either the gpu is too old, or the net is wrong. Or any other influence.

I have a xeon 6 core with amd 540 (4 GB)
Amd phenom II 6 core with nvidia 1060
Acer laptop with i7 quad core with nvidia 1060 (6 GB)
Ryzen 5 3600 6 core with AMD 570 (8 GB)
AMD ryzen 16 core with nvidia 2070 (8GB)
AMD ryzen 3 2200G (built in gpu, appropriate for LC0)

On the ryzen 5 the gpu is a little too weak.
On the ryzen 9 3950x i am using 8 cores so far for stockfish and 2070 is ahead.
I could of course scale this up to see when the 2070 gets beaten with Stockfish getting more cores.

Dragon by komodo makers is mostly arround 2 ranks beneath Stockfish 13 but close.

I do test with permanent brain on and in arena. Arround 5 minutes per game. Normal time control, no blitz time controls. E.g. 40/5 etc.

Subject: Re: Gambit Tournament, first results available Sat May 01, 2021 5:23 pm

It's better to have a benchmark first before jumping into conclusions. I will start a new subject.

Posts : 13 Join date : 2021-04-28

Hello, today I found very similar Gambit tournament that was done in the early 2016 by Walter Eigenmann.
Using a large Fritz format gambit book the top 10 engines during that period of 2016 battled for the winning spot of the Blitz Tournament.

In the article he shares the Opening gambit book and the completed .pgn game database
if you are interested make sure to check out his article here: https://glarean-verlag.ch/schach/live-schachpartien/

Subject: Re: Gambit Tournament, first results available Sat May 01, 2021 8:45 pm

Ah, the famous Traxler.

Subject: Re: Gambit Tournament, first results available Tue May 04, 2021 9:10 pm

Added live support for 2 PC's.

http://rebel13.nl/misc/gambit-rating-list.html

Posts : 1254 Join date : 2020-11-17 Location : France

Admin wrote:: Added live support for 2 PC's.

http://rebel13.nl/misc/gambit-rating-list.html

If you like, there’s probably some time on my most powerful PC to make a strong hardware match.
Same PC, SF can have 64 cores, unlimited RAM, LC0 gets a 3090 GPU. Disable pondering.

Subject: Re: Gambit Tournament, first results available Tue May 04, 2021 9:50 pm

Interesting Chris, it would mean I first should make the software for live support available. I can do that. I suppose you don't have an own website? Else it must run on mine.

Fun, ProDeo 3.0 on CCRL is 78 elo higher than Benjamin (the gambit version of ProDeo), nevertheless Benjamin rates higher on the gambit list.

Secondly the draw rate is clearly lower than on CCRL, that was to be expected and hoped for.

Code::    Gambit Rating List : 2021-05-06
   Time Control : 40 moves in 2 minutes repeating
   Software : cutechess-cli 1.1.0.f.1.0
   Elo calculation : Ordo 1.2.6
   Games : 18.200

   # PLAYER : RATING ERROR POINTS PLAYED (%) LOS W D L DRAWS
   1 Stockfish 13 : 3572.7 18.2 724.5 900 81 100 563 323 14 36%
   2 Komodo Dragon : 3480.5 38.4 629.5 900 70 100 427 405 68 45%
   3 Lc0-v27 : 3429.1 23.0 501.0 800 63 100 307 388 105 49%
   4 Komodo 14 : 3331.3 27.8 443.5 900 49 98 235 417 248 46%
   5 Anchor Engine : 3300.0 14.3 827.0 1400 59 96 550 554 296 40%
   6 RubiChess 2.1 : 3271.0 32.7 365.0 900 41 52 162 406 332 45%
   7 Pedone 3.1 : 3270.3 26.7 529.0 800 66 67 386 286 128 36%
   8 SlowChess 2.5 : 3263.6 27.8 355.5 900 40 58 148 415 337 46%
   9 Igel 3.0.5 : 3260.7 26.6 307.5 800 38 74 108 399 293 50%
  10 Ethereal 12.75 : 3254.5 21.5 344.0 900 38 77 156 376 368 42%
  11 Igel 3.0.0 : 3242.2 28.1 328.5 900 37 98 111 435 354 48%
  12 Nemorino 6.00 : 3211.3 22.0 461.0 800 58 100 332 258 210 32%
  13 Pedone 3.0 : 3177.0 22.2 420.0 800 53 69 265 310 225 39%
  14 rofChade 2.3 : 3171.3 25.6 413.5 800 52 56 262 303 235 38%
  15 Booot 6.5 : 3169.7 31.9 411.5 800 51 100 242 339 219 42%
  16 Seer 2.0.1 : 3097.1 27.2 476.0 700 68 95 375 202 123 29%
  17 Anchor Engine : 3074.9 13.4 1245.5 2200 57 54 907 677 616 31%
  18 Halogen 10 : 3073.5 21.1 484.5 800 61 61 359 251 190 31%
  19 Danasah 8.8 : 3071.3 23.7 411.5 700 59 69 317 189 194 27%
  20 Wasp 4.50 : 3062.1 29.1 284.5 800 36 93 145 279 376 35%
  21 Seer 2.0.0 : 3029.8 28.9 552.0 900 61 52 425 254 221 28%
  22 Weiss 1.3 : 3028.7 26.5 592.0 1000 59 63 434 316 250 32%
  23 Minic 3.04 : 3023.3 19.3 381.0 700 54 71 272 218 210 31%
  24 Berserk 3.3.0 : 3012.8 26.3 491.5 700 70 53 407 169 124 24%
  25 Berserk 4.0.0 : 3011.5 22.6 446.5 800 56 54 298 297 205 37%
  26 Arasan 2.22 : 3009.4 37.5 227.5 800 28 63 117 221 462 28%
  27 Tucano 9.0 : 3001.5 18.5 644.0 1000 64 59 532 224 244 22%
  28 Marvin 5.0 : 2999.3 12.3 354.5 700 51 73 232 245 223 35%
  29 Mr Bob 1.0.0 : 2992.1 17.9 631.5 1000 63 90 522 219 259 22%
  30 Orion 0.8 : 2979.6 12.2 332.0 700 47 56 217 230 253 33%
  31 Stash 29.0 : 2977.8 23.1 330.0 700 47 72 217 226 257 32%
  32 Vajolet2 2.8 : 2968.7 20.9 320.0 700 46 87 214 212 274 30%
  33 Topple 0.8.0 : 2951.4 21.2 552.5 1200 46 84 364 377 459 31%
  34 Cheng 4.41 : 2939.2 29.7 469.5 1000 47 59 314 311 375 31%
  35 Counter 3.7 : 2935.8 26.6 525.0 1200 44 53 326 398 476 33%
  36 Seer 1.2.1 : 2934.4 20.5 430.0 1000 43 76 280 300 420 30%
  37 FabChess 1.16 : 2925.9 28.1 478.0 1100 43 91 310 336 454 31%
  38 Drofa 3.0.0 : 2885.6 36.6 357.0 500 71 61 303 108 89 22%
  39 Cheng 4.40 : 2878.4 23.7 369.5 1000 37 62 236 267 497 27%
  40 Amoeba 3.3 : 2872.3 30.8 218.5 700 31 88 122 193 385 28%
  41 Anchor Engine : 2850.0 14.3 482.5 1300 37 90 309 347 644 27%
  42 Cheese 2.2 : 2839.3 17.0 370.5 800 46 81 272 197 331 25%
  43 Olithink 5.9.3 : 2823.6 22.6 278.0 600 46 91 209 138 253 23%
  44 ProDeo 3.1 : 2804.1 16.7 329.5 800 41 79 238 183 379 23%
  45 Velvet 1.2.0 : 2781.7 44.2 342.5 500 69 99 296 93 111 19%
  46 Benjamin : 2754.3 45.2 720.0 1200 60 58 610 220 370 18%
  47 ProDeo 3.0 : 2749.0 23.5 267.5 800 33 71 186 163 451 20%
  48 k2 0.99 : 2735.1 36.1 377.5 700 54 90 294 167 239 24%
  49 Dumb 1.8 : 2718.9 43.5 396.5 800 50 72 308 177 315 22%
  50 Anchor Engine : 2710.3 26.1 485.5 1200 40 77 368 235 597 20%
  51 Supernova 2.3 : 2699.9 42.2 623.0 1200 52 68 501 244 455 20%
  52 Fridolin 3.10 : 2695.4 37.1 615.0 1200 51 100 495 240 465 20%
  53 Galjoen 0.41.1 : 2543.0 39.9 164.5 500 33 100 120 89 291 18%
  54 CT800 1.42 : 2438.1 41.9 97.0 500 19 96 65 64 371 13%
  55 Foxsee 7.8 : 2407.6 49.5 85.0 600 14 --- 44 82 474 14%

White advantage = -13.69 +/- 1.96
Draw rate (equal opponents) = 34.24 % +/- 0.27

Posts : 13 Join date : 2021-04-28

Admin wrote:: Fun, ProDeo 3.0 on CCRL is 78 elo higher than Benjamin (the gambit version of ProDeo), nevertheless Benjamin rates higher on the gambit list.

Secondly the draw rate is clearly lower than on CCRL, that was to be expected and hoped for.

True but Benjamin played 400 games more than ProDeo 3.0, maybe that has something to do with the 8 ELO difference in favor of Benjamin. I am happy to see Benjamin in front of ProDeo 3 when it comes to aggressive gambit postions, I've seen some really fun games with Benjamin

Got web-match (live broadcast eng-eng matches) working again, started Benjamin - ProDeo 3.0 to find out.

http://rebel13.nl/pgn4web-3.05/live-test.html

100 gambit games and Benjamin won the first one Very Happy

Subject: Re: Gambit Tournament, first results available Sat May 08, 2021 9:45 pm

Code:: Result Table

Name Gms : Win : Draw : Lose : Pts : S-B

ProDeo 3.0 100 : 44+ : 28= : 28- : 58.0 : 2436.00
Benjamin 100 : 28+ : 28= : 44- : 42.0 : 2436.00

Games played 100, games to go 0

Tournament finished

Sadly ProDeo 3.0 won convincingly.

» Top Chess Engine Tournament Results
» Gambit Openings
» Rebel 16.2 release
» First Gambit Rating List
» Gambit Rating List - May 31, 2021