I wonder what is the reason that so many cores work for stockfish

Stockfish is an engine that show no measurable improvement at long time control for a long time.

https://ccrl.chessdom.com/ccrl/4040/rating_list_all.html

Stockfish13,14,15 have almost the same elo.

I could expect in these conditions that people devote less computer time for stockfish but when I look at the stockfish framework I see a lot of machines works on tests for stockfish

https://tests.stockfishchess.org/tests

I understand from the page that 5287 cores from 222 machines work on stockfish.
Why so many cores for an engine that fail to improve in rating list at normal chess when you test long time control?

I could understand if they use the computer time to test long time control against weaker engines in order to get better results against the weaker engines and better results in the CCRL long time control rating list but they do not do it and test only the engine against previous version at bullet time control.

This method worked in the past also to get better results in long time control but does not work today.

Hmmmm...

My first thought is that getting best play from long time control and a large number of CPUs is a different optimisation problem than getting best play from a computer and time control that most people will use - indicating that an entirely new program would be needed.

I'll also offer: the closer SF gets to being unbeatable, the more difficult it becomes to improve. Just thinking out loud, so probably wrong, but maybe improvements that can lead to more wins at this level need to be "major breakthroughs", and these, where they still happen, tend to be in NN patterns (what we might call "insights") rather than greater CPU resource.

TheSelfImprover wrote:

Hmmmm...

My first thought is that getting best play from long time control and a large number of CPUs is a different optimisation problem than getting best play from a computer and time control that most people will use - indicating that an entirely new program would be needed.

I'll also offer: the closer SF gets to being unbeatable, the more difficult it becomes to improve. Just thinking out loud, so probably wrong, but maybe improvements that can lead to more wins at this level need to be "major breakthroughs", and these, where they still happen, tend to be in NN patterns (what we might call "insights") rather than greater CPU resource.

I think that even if stockfish is unbeatable it may still possible to get imrovement against weaker opponents by prefering moves that set traps that weaker engines can fall into them but the problem is that a change that set these traps may give no improvement against previous version that does not fall in the trap regardless of time control.

In other words stockfish need to test against 100-200 elo weaker engines in CCRL to prove that the change help and against previous version it is going to give 0 elo.

Assuming chess is a draw (which it is IMO), and "our" program (SF) is close to being unbeatable, then the question becomes, "Under what conditions do less strong programs make mistakes?"

In a recent thread in TC (link) I suggested that "complexity" might be a good way to get opponents to make mistakes, but this idea didn't attract much discussion.

It might be that there is nothing consistent to look for: that unexpected pitfalls occur randomly with no good pattern that would indicate their existence. If chess is like this, then rare opportunities to win will continue to exist for a long time to come.

More likely, IMO, is that "deep" (complex) patterns exist that NNs cannot find but which will evaluate a very high proportion of chess positions correctly. If they do exist, then they would be able to evaluate most positions accurately and quickly on cheap, low-power CPUs. However, the method of finding these patterns would need to be something different than training NNs, which tend to find a large number of surface (simple) patterns rather than a small number of deep ones.

Another good position type for inducing mistakes might be "featureless position with no pressure": when a computer is under attack, its direction forward becomes easy and obvious: pick the move that doesn't lose material. In a featureless position, the computer's search is not going to get good traction, and might not be as helpful.

For CCRL (emphasis added) I can think of a reason, the use of Stockfish derivatives that lower its rating.

If you look at the GRL that doesn't allow derivatives you can see clear elo progress by each SF version.

I don't have the computer time for it but I can imagine if I would involve Fat Fritz 2, Shashchess, Sugar (as CCRL does) it would lower the current rating of SF15.

Another reason could be the increasing number of draws, I wonder how a rating list would look like changing the game scores as for instance in soccer. Won game 3 points, draw 1 point.

CEGT has the same problem to see a significant progress for SF, we dont use SF derivatives (40/4).

There is something wrong with stockfish. I have no improvement on my pcs with stockfish

Mclane wrote:

There is something wrong with stockfish. I have no improvement on my pcs with stockfish

Bingo! And I have been saying, and showing this for a long long time.

Understand what this result is showing and what it means. It answers a lot except why.


https://prodeo.actieforum.com/t872-stockfish-what-is-really-better-faster-cores-more-cores-and-nps-vs-time-to-depth

Dio wrote:

CEGT has the same problem to see a significant progress for SF, we dont use SF derivatives (40/4).

I have said this before and it is a bit of an educated guess but maybe we are witnessing the end of the Bayeselo / Ordo system.

Consider this example:

Match SF vs Dragon, 10,000 games, result: won 100 | draw 9900 | loss 0 | 5050 - 4950 = 50.5% = 3 elo
With Bayeselo or Ordo both will be rated that way, right?

In reality SF is much stronger, never lost a game, but won 100.

Maybe we need a new formula.

I don’t think it has to do with draws. Its just the plein fact that Stockfish 15 is not capable to be better then old Stockfish versions.

Of course Stockfish made progress. Look at my full ratinglist. No doubt about it.

https://www.sp-cc.de/files/spcc_full_list.txt

Code:

     Program                    Elo    +    -  Games    Score   Av.Op. Draws

   1 Stockfish 15 220418      : 3802    5    5 16000    71.3%   3639   57.0%
   2 Stockfish 14.1 211028    : 3780    4    4 18000    74.2%   3588   51.0%
   3 Stockfish 14 210702      : 3761    4    4 26000    74.0%   3569   51.0%
   4 KomodoDragon 3 avx2      : 3761    5    5 15000    69.0%   3618   59.1%
   5 KomodoDragon 2.6 avx2    : 3745    5    5 13000    65.9%   3625   62.1%
   6 KomodoDragon 2.5 avx2    : 3730    5    5 13000    69.3%   3581   55.9%
   7 Stockfish 13 210218      : 3729    4    4 23000    77.7%   3492   42.8%
   8 Stockfish 12 200902      : 3693    5    5 24000    82.0%   3409   33.7%

In VLTC UHO Openings regression tests of Stockfish vs. Stockfish 15, of course progress, too. Mention, that UHO Openings spread Elo gaps by 2x up to 2.5x (if you mention this, the progress matches very well with the measured progress in my ratinglist.):
https://www.sp-cc.de/stockfish-regression.htm

pohl4711 wrote:

Of course Stockfish made progress. Look at my full ratinglist. No doubt about it.

https://www.sp-cc.de/files/spcc_full_list.txt

Code:

     Program                    Elo    +    -  Games    Score   Av.Op. Draws

   1 Stockfish 15 220418      : 3802    5    5 16000    71.3%   3639   57.0%
   2 Stockfish 14.1 211028    : 3780    4    4 18000    74.2%   3588   51.0%
   3 Stockfish 14 210702      : 3761    4    4 26000    74.0%   3569   51.0%
   4 KomodoDragon 3 avx2      : 3761    5    5 15000    69.0%   3618   59.1%
   5 KomodoDragon 2.6 avx2    : 3745    5    5 13000    65.9%   3625   62.1%
   6 KomodoDragon 2.5 avx2    : 3730    5    5 13000    69.3%   3581   55.9%
   7 Stockfish 13 210218      : 3729    4    4 23000    77.7%   3492   42.8%
   8 Stockfish 12 200902      : 3693    5    5 24000    82.0%   3409   33.7%

In VLTC UHO Openings regression tests of Stockfish vs. Stockfish 15, of course progress, too. Mention, that UHO Openings spread Elo gaps by 2x up to 2.5x (if you mention this, the progress matches very well with the measured progress in my ratinglist.):
https://www.sp-cc.de/stockfish-regression.htm

I talk about progress in normal chess at LTC
UHO opening is not normal chess.

Uri Blass wrote:

pohl4711 wrote:

Of course Stockfish made progress. Look at my full ratinglist. No doubt about it.

https://www.sp-cc.de/files/spcc_full_list.txt

Code:

     Program                    Elo    +    -  Games    Score   Av.Op. Draws

   1 Stockfish 15 220418      : 3802    5    5 16000    71.3%   3639   57.0%
   2 Stockfish 14.1 211028    : 3780    4    4 18000    74.2%   3588   51.0%
   3 Stockfish 14 210702      : 3761    4    4 26000    74.0%   3569   51.0%
   4 KomodoDragon 3 avx2      : 3761    5    5 15000    69.0%   3618   59.1%
   5 KomodoDragon 2.6 avx2    : 3745    5    5 13000    65.9%   3625   62.1%
   6 KomodoDragon 2.5 avx2    : 3730    5    5 13000    69.3%   3581   55.9%
   7 Stockfish 13 210218      : 3729    4    4 23000    77.7%   3492   42.8%
   8 Stockfish 12 200902      : 3693    5    5 24000    82.0%   3409   33.7%

In VLTC UHO Openings regression tests of Stockfish vs. Stockfish 15, of course progress, too. Mention, that UHO Openings spread Elo gaps by 2x up to 2.5x (if you mention this, the progress matches very well with the measured progress in my ratinglist.):
https://www.sp-cc.de/stockfish-regression.htm

I talk about progress in normal chess at LTC
UHO opening is not normal chess.

Exactly, no progress with longer time controls, or more cores. This is the exact reason Stockfish is oblivious to the problem. Along with some testers.

I also fined Stockfish makes progress at very fast time controls. But reality sets in as the time controls, and core counts increase. No Progress!

Even the hapless CCRL can see the truth.

I think that it might tie in with draws, and I think the issue with the elo rating system is a little different from how Ed described it.

Consider two extremes of play:

Level A: random move selection.

Level B: cannot be beaten.

Against level A, winning is very easy. Against level B, it's not possible.

As we get closer to level B, the elo system will still work - it just gets massively more difficult to get a higher score.

Assuming level B exists (and I believe it does), then if your opponents are also at this level, it's not actually possible to get any better. For me, this means that the elo system is working - but that it has an upper limit.

TheSelfImprover wrote:

I think that it might tie in with draws, and I think the issue with the elo rating system is a little different from how Ed described it.

Consider two extremes of play:

Level A: random move selection.

Level B: cannot be beaten.

Against level A, winning is very easy. Against level B, it's not possible.

As we get closer to level B, the elo system will still work - it just gets massively more difficult to get a higher score.

Assuming level B exists (and I believe it does), then if your opponents are also at this level, it's not actually possible to get any better. For me, this means that the elo system is working - but that it has an upper limit.

If you done some testing you would know B is totally false. So goes your theory.

I have shown that type B engines do not play perfect chess. And far from it. And Stockfish is a type B chess engines.

Why do you equate draws with perfect play?  

That would be also saying every win is a perfectly played game.

Admin wrote:

Match SF vs Dragon, 10,000 games, result: won 100 | draw 9900 | loss 0 | 5050 - 4950 = 50.5% = 3 elo
With Bayeselo or Ordo both will be rated that way, right?

In reality SF is much stronger, never lost a game, but won 100.

Look at that a slightly different way: out of every 100 games, 99 were draws.

Imagine you were a tournament promoter, and your job was to sell as many tickets as possible. I'm guessing your slogan wouldn't be, "99 out of every hundred games will be drawn". If it was, you wouldn't be going back to the printer to make more tickets!

I would go for the history angle: "Probably the last tournament at which a game will be won!"

I STILL wouldn't order a ticket for every seat on the first print run!

TheSelfImprover wrote:

Admin wrote:

Match SF vs Dragon, 10,000 games, result: won 100 | draw 9900 | loss 0 | 5050 - 4950 = 50.5% = 3 elo
With Bayeselo or Ordo both will be rated that way, right?

In reality SF is much stronger, never lost a game, but won 100.

Look at that a slightly different way: out of every 100 games, 99 were draws.

Imagine you were a tournament promoter, and your job was to sell as many tickets as possible. I'm guessing your slogan wouldn't be, "99 out of every hundred games will be drawn". If it was, you wouldn't be going back to the printer to make more tickets!

I would go for the history angle: "Probably the last tournament at which a game will be won!"

I STILL wouldn't order a ticket for every seat on the first print run!

Obviously you are not a chess fan. There is nothing wrong with draws.

And even funnier, what does this have to do with the topic.

mwyoung wrote:

TheSelfImprover wrote:

I think that it might tie in with draws, and I think the issue with the elo rating system is a little different from how Ed described it.

Consider two extremes of play:

Level A: random move selection.

Level B: cannot be beaten.

Against level A, winning is very easy. Against level B, it's not possible.

As we get closer to level B, the elo system will still work - it just gets massively more difficult to get a higher score.

Assuming level B exists (and I believe it does), then if your opponents are also at this level, it's not actually possible to get any better. For me, this means that the elo system is working - but that it has an upper limit.

If you done some testing you would know B is totally false. So goes your theory.

I have shown that type B engines do not play perfect chess. And far from it. And Stockfish is a type B chess engines.

Why do you equate draws with perfect play?  

I know that today's engines can still be beaten. However, the amount of "power" (quantity and quality of search and knowledge) it takes to beat them is steadily becoming prohibitive.

The correlation between standard of play and draw rate is holding up.

Despite the PHENOMENAL increase in knowledge and search (quantity AND quality), there is still no way to force the win of any material at all in the opening - or, indeed, any kind of decisive advantage.

It's looking very much as though a win in chess requires a mistake from the opponent, and that these mistakes are becoming uncommon as computers keep on improving.

mwyoung wrote:

TheSelfImprover wrote:

Admin wrote:

Match SF vs Dragon, 10,000 games, result: won 100 | draw 9900 | loss 0 | 5050 - 4950 = 50.5% = 3 elo
With Bayeselo or Ordo both will be rated that way, right?

In reality SF is much stronger, never lost a game, but won 100.

Look at that a slightly different way: out of every 100 games, 99 were draws.

Imagine you were a tournament promoter, and your job was to sell as many tickets as possible. I'm guessing your slogan wouldn't be, "99 out of every hundred games will be drawn". If it was, you wouldn't be going back to the printer to make more tickets!

I would go for the history angle: "Probably the last tournament at which a game will be won!"

I STILL wouldn't order a ticket for every seat on the first print run!

Obviously you are not a chess fan. There is nothing wrong with draws.

And even funnier, what does this have to do with the topic.

I'll concede it was off-topic, and you are right that there is nothing wrong with draws. However, it's a problem when wins become rare - hence the expression, "death by draw".

I also think that the games that sell books and get viewers on YouTube are generally wins.

TheSelfImprover wrote:

mwyoung wrote:

TheSelfImprover wrote:

Admin wrote:

Match SF vs Dragon, 10,000 games, result: won 100 | draw 9900 | loss 0 | 5050 - 4950 = 50.5% = 3 elo
With Bayeselo or Ordo both will be rated that way, right?

In reality SF is much stronger, never lost a game, but won 100.

Look at that a slightly different way: out of every 100 games, 99 were draws.

Imagine you were a tournament promoter, and your job was to sell as many tickets as possible. I'm guessing your slogan wouldn't be, "99 out of every hundred games will be drawn". If it was, you wouldn't be going back to the printer to make more tickets!

I would go for the history angle: "Probably the last tournament at which a game will be won!"

I STILL wouldn't order a ticket for every seat on the first print run!

Obviously you are not a chess fan. There is nothing wrong with draws.

And even funnier, what does this have to do with the topic.

I'll concede it was off-topic, and you are right that there is nothing wrong with draws. However, it's a problem when wins become rare - hence the expression, "death by draw".

I also think that the games that sell books and get viewers on YouTube are generally wins.

Again So What!

TheSelfImprover wrote:

mwyoung wrote:

TheSelfImprover wrote:

I think that it might tie in with draws, and I think the issue with the elo rating system is a little different from how Ed described it.

Consider two extremes of play:

Level A: random move selection.

Level B: cannot be beaten.

Against level A, winning is very easy. Against level B, it's not possible.

As we get closer to level B, the elo system will still work - it just gets massively more difficult to get a higher score.

Assuming level B exists (and I believe it does), then if your opponents are also at this level, it's not actually possible to get any better. For me, this means that the elo system is working - but that it has an upper limit.

If you done some testing you would know B is totally false. So goes your theory.

I have shown that type B engines do not play perfect chess. And far from it. And Stockfish is a type B chess engines.

Why do you equate draws with perfect play?  

I know that today's engines can still be beaten. However, the amount of "power" (quantity and quality of search and knowledge) it takes to beat them is steadily becoming prohibitive.

The correlation between standard of play and draw rate is holding up.

Despite the PHENOMENAL increase in knowledge and search (quantity AND quality), there is still no way to force the win of any material at all in the opening - or, indeed, any kind of decisive advantage.

It's looking very much as though a win in chess requires a mistake from the opponent, and that these mistakes are becoming uncommon as computers keep on improving.

This is unknowable and not true. Unless you have a 32 man tablebase. Please share your knowledge, or admit you have no clue what you  are talking about.

mwyoung wrote:

TheSelfImprover wrote:

mwyoung wrote:

TheSelfImprover wrote:

Admin wrote:

Match SF vs Dragon, 10,000 games, result: won 100 | draw 9900 | loss 0 | 5050 - 4950 = 50.5% = 3 elo
With Bayeselo or Ordo both will be rated that way, right?

In reality SF is much stronger, never lost a game, but won 100.

Look at that a slightly different way: out of every 100 games, 99 were draws.

Imagine you were a tournament promoter, and your job was to sell as many tickets as possible. I'm guessing your slogan wouldn't be, "99 out of every hundred games will be drawn". If it was, you wouldn't be going back to the printer to make more tickets!

I would go for the history angle: "Probably the last tournament at which a game will be won!"

I STILL wouldn't order a ticket for every seat on the first print run!

Obviously you are not a chess fan. There is nothing wrong with draws.

And even funnier, what does this have to do with the topic.

I'll concede it was off-topic, and you are right that there is nothing wrong with draws. However, it's a problem when wins become rare - hence the expression, "death by draw".

I also think that the games that sell books and get viewers on YouTube are generally wins.

Again So What!

Well... "death by draw" is a type of "death".

To be fair, Formula 1 is still a very popular sport, even though, each year, they have to add more constraints to the cars to:

1. enable the driver to survive (for example, engineers have long been able to get cars to go around corners fast enough to cause drivers to black out)

2. enable overtaking (without measures like DRS, a formula 1 race would be a procession)

For these reasons, while I agree that F1 is exciting, it's long ceased to be the reality of motor racing.

mwyoung wrote:

TheSelfImprover wrote:

It's looking very much as though a win in chess requires a mistake from the opponent, and that these mistakes are becoming uncommon as computers keep on improving.

This is unknowable and not true. Unless you have a 32 man tablebase. Please share your knowledge, or admit you have no clue what you  are talking about.

It's possible (even likely IMO) that there are ways to discern the outcome of a given position in chess without having to play out all the branches of the game tree from the given position.

When this was discussed previously, I was presented with a paper that showed that in a game that could transform to chess, the distance to checkmate can increase with the size of the board. However, this is not proof that it's impossible to come up with a way of determining which side will win (or that the position is drawn).

From a practical point of view, if nobody can find a way to win, then the game is a draw - and that's our direction of travel right now.

Uri Blass wrote:

I talk about progress in normal chess at LTC
UHO opening is not normal chess.

And this just false. My UHO openings are selected out of the Megabase and all openings were played by strong human players. So, no openings could be more "normal"/human chess than UHO. And I did not change anything in these openings, I just used the KomodoDragon eval for selecting them. Thats why I called them UHO: Unbalanced Human Openings. All Openings were played by strong humans (all players must have 2300+ Elo. All games played after WW2, so no crazy gambits from 19th century or so). All games out of the Megabase.

Just as example, here the first 5 lines out of the UHO_2022_6mvs_+120_+129 file, which I use for the VLTC Stockfish regression test.
(Shirov, Ivanchuk, Kamsky, Morozevich are the players, which played the lines with black, which ends with a clear white advantage. Are these players not "normal" or not strong enough for playing good chess? I am so tired of reading such nonsense about my work)

[Event "Novgorod"]
[Site "Novgorod"]
[Date "1994.??.??"]
[Round "1"]
[White "Kasparov, Garry"]
[Black "Shirov, Alexei"]
[Result "1/2-1/2"]
[ECO "A21"]
[WhiteElo "2805"]
[BlackElo "2740"]
[Annotator "depth=29 eval=+124"]
[PlyCount "12"]
[EventDate "1994.??.??"]

1. c4 e5 2. Nc3 Bb4 3. Nd5 Be7 4. d4 d6 5. e4 c6 6. Nxe7 Qxe7 1/2-1/2

[Event "Horgen"]
[Site "Horgen"]
[Date "1995.10.26"]
[Round "6"]
[White "Kasparov, Garry"]
[Black "Ivanchuk, Vassily"]
[Result "1/2-1/2"]
[ECO "C16"]
[WhiteElo "2795"]
[BlackElo "2740"]
[Annotator "depth=28 eval=+126"]
[PlyCount "12"]
[EventDate "1995.??.??"]

1. e4 e6 2. d4 d5 3. Nc3 Bb4 4. e5 b6 5. a3 Bf8 6. Nf3 Ne7 1/2-1/2

[Event "Amber-rapid 05th"]
[Site "Monte Carlo"]
[Date "1996.??.??"]
[Round "3"]
[White "Anand, Viswanathan"]
[Black "Ivanchuk, Vassily"]
[Result "1/2-1/2"]
[ECO "C41"]
[WhiteElo "2725"]
[BlackElo "2735"]
[Annotator "depth=30 eval=+126"]
[PlyCount "12"]
[EventDate "1996.??.??"]

1. e4 e5 2. Nf3 d6 3. Bc4 c6 4. d4 Nd7 5. O-O Be7 6. dxe5 dxe5 1/2-1/2

[Event "FIDE-Wch"]
[Site "Elista"]
[Date "1996.07.04"]
[Round "15"]
[White "Karpov, Anatoly"]
[Black "Kamsky, Gata"]
[Result "1/2-1/2"]
[ECO "A70"]
[WhiteElo "2770"]
[BlackElo "2735"]
[Annotator "depth=27 eval=+120"]
[PlyCount "12"]
[EventDate "1996.??.??"]

1. d4 Nf6 2. c4 c5 3. d5 e6 4. Nc3 exd5 5. cxd5 d6 6. e4 g6 1/2-1/2

[Event "Corus"]
[Site "Wijk aan Zee"]
[Date "2001.01.21"]
[Round "7"]
[White "Anand, Viswanathan"]
[Black "Morozevich, Alexander"]
[Result "1/2-1/2"]
[ECO "D07"]
[WhiteElo "2790"]
[BlackElo "2745"]
[Annotator "depth=29 eval=+129"]
[PlyCount "12"]
[EventDate "2001.??.??"]

1. d4 d5 2. c4 Nc6 3. Nc3 dxc4 4. Nf3 Nf6 5. e4 Bg4 6. Be3 Bxf3 1/2-1/2