News about Rebel or Chess System Tal ?

Dio wrote:

there were first tests by CEGT. The GUI used was the Banksia GUI. This GUI has no timemargin setting. In 100 games (TC: 4 Minutes + 2 Seconds) against Ethereal 13.75 there were 16 time losses.

How do we deal with this problem?

We could simply take the time controls values that the GUI sends to the engine and factor them down by 0.95 or something and work from those.

Or Ed suggested some time ago that the engine always insists on maintaining a time reservoir of 1000ms or similar.

Or we add a UserOption to do ether of the above. Nothing seems very satisfactory.

This is exactly what I would suggest. Rebel 16.0 did not have these problems, but it only used about 70 to 75 percent of the thinking time.

I would ideally use only 95 percent of the thinking time. Then you have a certain "security" not to lose by time, the "elo loss" due to these 5 percent should be max. 1 to 2 elo.

Just take a look at Stockfish code and add Move Overhead, Slow Mover and Minimum Thinking Time and Rebel will be fine..

Dio wrote:

This is exactly what I would suggest. Rebel 16.0 did not have these problems, but it only used about 70 to 75 percent of the thinking time.

I would ideally use only 95 percent of the thinking time. Then you have a certain "security" not to lose by time, the "elo loss" due to these 5 percent should be max. 1 to 2 elo.

That's not a bad suggestion at all. But it remains odd the engine is victimized by the GUI.

Damir Desevac wrote:

Just take a look at Stockfish code and add Move Overhead, Slow Mover and Minimum Thinking Time and Rebel will be fine..

Move overhead and minimum thinking time are fairly self explanatory but WTF is slow mover?

Slow Mover type spin default 100 min 10 max 1000
Lower values will make Stockfish take less time in games, higher values will make it think longer.

Admin wrote:

Dio wrote:

This is exactly what I would suggest. Rebel 16.0 did not have these problems, but it only used about 70 to 75 percent of the thinking time.

I would ideally use only 95 percent of the thinking time. Then you have a certain "security" not to lose by time, the "elo loss" due to these 5 percent should be max. 1 to 2 elo.

That's not a bad suggestion at all. But it remains odd the engine is victimized by the GUI.

Worse, victimised by lowest common denominator GUI.
Actually, any GUI without timemargin is badly designed and unsuitable for engine-engine tournament testing.
We should publicise which GUIs are well designed and which not.

Well-designed, suitable for engine testing:
Cutechess

Bad designed, unsuitable for engine testing:
Fritz, Banksia

Ipmanchess wrote:

Slow Mover type spin default 100 min 10 max 1000
Lower values will make Stockfish take less time in games, higher values will make it think longer.

User options are also a possible solution.

I’ve left my version on test while I’m out, but it begins to look like it will be very difficult to deal with odd rogue cases. For example, when engine fires up a timeout, it needs to unwind itself back to the root (possibly wait for other threads to unwind) and then send bestmove e2e4 or whatever. Unwinding takes time. If if if there’s high concurrency and/or windows performing itself elsewhere, then the unwinding time can find itself being stretched out by CPU time slicing between threads, won’t happen always, but can happen badly sometimes which is enough to give random spurious time forfeits. We don’t know when this might happen and if we want to safely stop it, then we need a higher than normally necessary overhead to apply to ALL moves. We’re into bad lowest common denominator situation.

I’m minded to stick with 250ms timemargin and give user options to modify various time control parameters as mentioned in another post (SF method).

It's just that programmers apparently as easy sheep have agreed to fix problems created by others (GUI, Windows, other stuff running in the background) at the cost of hard-fought elo, see my (and Chris) arguments at Talkchess. A chess GUI should be tolerant. It should either have a builtin tolerance or offer a move-overhead option at the cost of the GUI.

I only ever used the Shredder GUI for longer time testgames, 5 minutes per game or three minutes per game, with no increment. Never time losses because I think Shredder just gives a bonus that works like some increment. But you have to trust the GUI to do it fairly. Cutechess can test with very small values for Move Overhead (the Stockfish UCI option), because it reacts so quickly. Move Overhead as an UCI option in Stockfish is mainly intended to deal with time lag on servers. Or not so reliable GUIs like in TCEC tournaments. Stockfish lost games in TCEC, on time, because the Move Overhead was tiny as used in Fishtest. It is increased now whenever an official version is made, or a TCEC contender. But Move Overhead (in the engine) is not such a good parameter to use for the internal time management IMO. It is to compensate for the external situations that you have no control over as an engine. Like the clock used by the GUI that may differ from the internal clock, things like that.

Eelco wrote:

Move Overhead as an UCI option in Stockfish is mainly intended to deal with time lag on servers. Or not so reliable GUIs like in TCEC tournaments. Stockfish lost games in TCEC, on time, because the Move Overhead was tiny as used in Fishtest.

Oh really, what a bunch of amateurs.

I am not willing to give in to this kind amateurism, there will be no fix. Patches are for bugs, patches are not for well working engines that lower elo. Let each take care of their shortcomings.

Time losses in TCEC because of a difference of maybe 100 msecs was of course humiliating. I do not remember the exact difference, or when it was. I do think it happened. Elo increase at Cutechess millions of game testing by more aggressive timing, how much of that remains in CEGT? I have no idea but a few Elo points does not sound strange. 16 losses hurt.

(Edit: If those losses are counted of course, but I think they are not counted because it is not fair to other engines that do not play Rebel, they get no free points...? It was a long time ago from the time losses but if you want I can look it up Ed. Here is a Fishcooking (Marco's Google mailing list where most of the informal discussions were) discussion just as an example, between Ivan and Ronald de Man where Ronald suddenly had time losses too on 6 threads with Cfish, the suggestion was to increase slightly Move Overhead but Ivan also says here for TCEC Stockfish had to use Move Overhead 1000 miliseconds (this was also after the incident) and that this is no regression at those time controls. But of course it should be possible to prevent a loss on time that is caused this way (lagging GUI or network lag) by other means (than Move Overhead). Having a time reserve of one second will have maybe some impact if it is a large portion of total time but it becomes less of an impact in CEGT timecontrols. Buf if there is a possible time lag of a second every move, then you can move instantly but you will still lose a second per move, so then a time reserve of one second is not going to be enough. One second per move seems very large, but I'm not sure what the implementation of Move Overhead exactly was. I'm sure somebody here knows it and can tell you, but you guys maybe don't want to look at the code.) (Edit: I do think Move Overhead is meant as possible loss per move? So every move this is calculated in. As far as I can see, but you have to look at the code, it is not written up elsewhere, at least I could not find this, clearly. If the time lag is this large, the time losses also become more understandable.)

Quote :

Or Ed suggested some time ago that the engine always insists on maintaining a time reservoir of 1000ms or similar.

I do not remember what it was in Fruit but this seems basic if you do not have it yet and should take care of simple timepressure.

@Elco: a moveoverhead of 1000 (ms) refers to the whole game, not to a single move.

Quote :

@Elco: a moveoverhead of 1000 (ms) refers to the whole game, not to a single move

Thanks Dio! That sounds a lot more doable then, to implement something for that, without Elo loss. If the programmers want to, that is. I did think it was for the whole game originally, and then for Stockfish to lose on time in TCEC , if it had just reserved another second total, somewhere in its timemanagement, it would have been safe. I do think that was embarrassing.

A problem with time reservoirs and time margins is they are absolute values. Absolute values can’t work for both short and long time controls. How do you make a time reservoir for game in 1 second with 20 second increments? And so on. Of course, you can, by tweaking this and that and testing, but it’s not what an engine programmer needs or wants to spend his time on. It remains a problem of contradiction between gui time and engine time by interface and communication delays over which the engine has no control or knowledge. Thus it’s a tester and gui problem. Testers should establish that an engine is not deliberately or otherwise egregiously going over time and then use a realistic timemargin. If their GUI has no time margin, then they should ditch it and use one that has.

Overall objective is good chess, establish no cheating and then trust the engine programmer timings. Else we are in the realm of gui terrorism.

Chris Whittington wrote:

A problem with time reservoirs and time margins is they are absolute values. Absolute values can’t work for both short and long time controls. How do you make a time reservoir for game in 1 second with 20 second increments? And so on. Of course, you can, by tweaking this and that and testing, but it’s not what an engine programmer needs or wants to spend his time on. It remains a problem of contradiction between gui time and engine time by interface and communication delays over which the engine has no control or knowledge. Thus it’s a tester and gui problem. Testers should establish that an engine is not deliberately or otherwise egregiously going over time and then use a realistic timemargin. If their GUI has no time margin, then they should ditch it and use one that has.

Overall objective is good chess, establish no cheating and then trust the engine programmer timings. Else we are in the realm of gui terrorism.

That.

Has Rebel 16.1 been confirmed to be able to play from the 324 Chess324 positions ? The idea is that every engine that plays standard chess can do so, although I think there was at least one engine that required a fix. I guess I can try it and see.

Ray wrote:

Has Rebel 16.1 been confirmed to be able to play from the 324 Chess324 positions ?  The idea is that every engine that plays standard chess can do so, although I think there was at least one engine that required a fix. I guess I can try it and see.

There's no special code to account for any special positions. Nor any special positions in the train files. If it can read the EPD's or PGN's then it will be able to play.

Chris Whittington wrote:

Ray wrote:

Has Rebel 16.1 been confirmed to be able to play from the 324 Chess324 positions ?  The idea is that every engine that plays standard chess can do so, although I think there was at least one engine that required a fix. I guess I can try it and see.

There's no special code to account for any special positions. Nor any special positions in the train files.  If it can read the EPD's or PGN's then it will be able to play.

It plays but loses 1 game in 5 on time so is unsuitable for my tournament. CuteChessGUI, concurrency 10, 500ms margin. Blitz 2'+1". All the other engines have played over 5,000 games without a single time loss.

Time controls are broken

Ray wrote:

Chris Whittington wrote:

Ray wrote:

Has Rebel 16.1 been confirmed to be able to play from the 324 Chess324 positions ?  The idea is that every engine that plays standard chess can do so, although I think there was at least one engine that required a fix. I guess I can try it and see.

There's no special code to account for any special positions. Nor any special positions in the train files.  If it can read the EPD's or PGN's then it will be able to play.

It plays but loses 1 game in 5 on time so is unsuitable for my tournament. CuteChessGUI, concurrency 10, 500ms margin. Blitz 2'+1".  All the other engines have played over 5,000 games without a single time loss.

Weird.

Blitz 2'+1" is 120+1 in cutechess-ese?? (120 secs for game, 1000ms increment?)

How many cores?

If I test myself at those settings, do I need a special book for the 324?

Ah! Thought .... do you test play direct off the opening positions? My tests are mostly of book lines (UHO eg) starting 8 or more moves deep.

120 for the game +1 second per move, so Fischer time control. Plus 500ms margin. Yes, 324 plays direct from a starting position. No book moves, like FRC in that respect (only). Concurrency of 10 on a 10 core machine that supports 20 threads but I always limit myself to the number of real cores. So plenty of resource to handle any background windows processes. Nothing else running at the same time bar anything that Windows may be doing. Had it on my second machine also, same but 12 threads on a 12/24 machine. A pity because I think it was performing very strongly. I guess I could set the margin to 1000ms, but the point is no other engine that I use requires that.

Mclane wrote:

Time controls are broken

In Arena?

Please tell me more.

Ray wrote:

120 for the game +1 second per move, so Fischer time control. Plus 500ms margin.  Yes, 324 plays direct from a starting position. No book moves, like FRC in that respect (only). Concurrency of 10 on a 10 core machine that supports 20 threads but I always limit myself to the number of real cores. So plenty of resource to handle any background windows processes. Nothing else running at the same time bar anything that Windows may be doing. Had it on my second machine also, same but 12 threads on a 12/24 machine. A pity because I think it was performing very strongly.

I did a run with your 120+1 time control, timemargin=1000, using 5 threads. It ran on a busy PC, training a new net taking 6 threads, thus in total 11 of the 12 were in use. And no single forfeit.

Code:

Engine                 Depth       Time   Games     Moves  Average Forfeit  Book Depth     MIDG   EARLY    ENDG    LATE
rebel-16.1             24.89    4:30:57     101      6849    2.37     0      891  8.82    22.68 | 23.56 | 25.29 | 29.00
rebel-16a              23.27    4:10:34     101      6859    2.19     0      899  8.90    21.28 | 22.15 | 23.44 | 27.17

So to play in your tournament I have to give up my perfect working NNUE development system where I can train a net and test it at the same time and surrender to faulty system that somehow slipped in during the years. A time margin is NORMAL, see Stefan he even uses 2 seconds.  I am not going to bow down to a system that lowers my development time with 50%.

Ray wrote:

I guess I could set the margin to 1000ms, but the point is no other engine that I use requires that.

Let's put that into perspective, game in 120+1 with an average game length of (say) 80 moves means 200 seconds per game. Your timemargin is 500 (I am glad you use it) but what means half a second in comparison with a game that last 200 seconds?

I did the same 120+1 run on my other Ryzen, it has 24 threads, 12 threads in use for training a new net, 10 threads for cutechess, timemargin 1000ms. In total 22/24 threads in use, no single time forfeit.

Here is an advice, you have a 20 thread PC and only use 10 threads. Try 18 threads with a timemargin=1000 and see what happens, it might speed-up your testing tremendously.

OK so Pohl uses 2 seconds, that explains why no forfeits for him. I don't like that myself, but each to their own.

I'll increase to 1000ms if that is what is needed. It is a tiny amount of time as you say, and a sensible compromise.