Time forfeit contradiction

Following Ray's report of time forfeits, I did a little experiment (using Chess System Tal, but it should behave timewise similar to Rebel).

Ran 2000 game gauntlet at 120+1 with timemargin=2000. Result zero forfeits. Saved the PGNs.

Ran 2000 game gauntlet at 120+1 with timemargin=0. Result, forfeits. Saved the PGNs.

Now, bear in mind that the engine has no idea whether or not the GUI has timemargin set. So, in the PGN's saved from the 2000ms timemargin run, you'ld expect, if you add up the time taken per move (saved by cutechess for each move in the PGN) and compared to time limit allowable each move, you'ld expect to find time forfeit nodes. As many forfeits as in the timemargin=0 gauntlet.

Weird thing is, you don't find any at all. A human, looking at the PGN data and adding the times, won't find any time problems. So why does cutechess report forfeits? Maybe rounding errors at millisecond level? Whatever, there's quite a contradiction there.

Pls. have a look here: http://talkchess.com/forum3/viewtopic.php?f=6&t=81276
Andreas (RubiChess) gave some hints there concerning UCI specs, maybe helpfull  

Time for a slice of humble pie for Chris and Ed. Rebel is not a victim of GUI terrorism or a broken system after all.

I do find it incredible that anyone plays with a 2 second GUI overstep margin. That theoretically gives an advantage to engines that overstep the limit and get away with it. Is it an Elo gain that is measurable?  Probably not, but as a matter of principle I will never use a value that high. It just feels wrong to me. But as the saying goes, each to their own.

Wolfgang wrote:

Pls. have a look here: http://talkchess.com/forum3/viewtopic.php?f=6&t=81276
Andreas (RubiChess) gave some hints there concerning UCI specs, maybe helpfull  

Yup, thanks, I’m fixing my wrong assumption about when increment gets added in right now. Unfortunately a gruelling task, since all the tuning needs to be redone as well.

Does this mean there will be a new Rebel also..

Ipmanchess wrote:

Does this mean there will be a new Rebel also..

I think so. The current Rebel should be okay on zero increment (eg 40/10 or whatever), but we'll need to fix for large increment testing (where large = increment > timemargin, with resulting time forfeits)

Ray wrote:

Time for a slice of humble pie for Chris and Ed. Rebel is not a victim of GUI terrorism or a broken system after all.

You mix up 2 things.

1. Testing as it should.

2. A bug.

The latter is the case, the former still stands.

Well, in my opinion testing with an overstep margin of 2000ms is wrong. You do need some margin to account for lag caused by a GUI or some Windows process, but most people do leave some headroom on their PCs to cover that in terms of leaving a core or threads free. 500ms is ample for that in a good GUI, and almost an eternity in that context. Increasing it too much to accommodate engines that cut it too fine or aren't correctly programmed gives those engines an unfair advantage in my view compared to those that adhere to protocols and leave some internal margin. Theoretical advantage yes, but measurable in Elo - probably not. I can live with Increasing from 500ms to 1000ms if and when needed, after all it is only chess we are talking about, not anything that really matters in the world

It's IMO broken for x/y games.
At least on my machines I have
several time forfeits in arena

Ray wrote:

Well, in my opinion testing with an overstep margin of 2000ms is wrong. You do need some margin to account for lag caused by a GUI or some Windows process, but most people do leave some headroom on their PCs to cover that in terms of leaving a core or threads free. 500ms is ample for that in a good GUI, and almost an eternity in that context. Increasing it too much to accommodate engines that cut it too fine or aren't correctly programmed gives those engines an unfair advantage in my view compared to those that adhere to protocols and leave some internal margin. Theoretical advantage yes, but measurable in Elo - probably not. I can live with Increasing from 500ms to 1000ms if and when needed, after all it is only chess we are talking about, not anything that really matters in the world

I have not followed this very well but your system does not appear to be broken Ray? Nor is there evidence of GUI terrorism. That is an extraordinary claim and you know what they say about extraordinary claims. Putting an engine to its limits is also a purpose of testing. Without it bugs stay hidden. Now it was possible to find one. Thank you. So no, I would not change my testing.

Mclane wrote:

It's IMO broken for x/y games.
At least on my machines I have
several time forfeits in arena

Can you reproduce it with another GUI perhaps Thorsten? What is X and Y specifically? Maybe some pgns.

Eelco wrote:

Ray wrote:

Well, in my opinion testing with an overstep margin of 2000ms is wrong. You do need some margin to account for lag caused by a GUI or some Windows process, but most people do leave some headroom on their PCs to cover that in terms of leaving a core or threads free. 500ms is ample for that in a good GUI, and almost an eternity in that context. Increasing it too much to accommodate engines that cut it too fine or aren't correctly programmed gives those engines an unfair advantage in my view compared to those that adhere to protocols and leave some internal margin. Theoretical advantage yes, but measurable in Elo - probably not. I can live with Increasing from 500ms to 1000ms if and when needed, after all it is only chess we are talking about, not anything that really matters in the world

I have not followed this very well but your system does not appear to be broken Ray? Nor is there evidence of GUI terrorism. That is an extraordinary claim and you know what they say about extraordinary claims. Putting an engine to its limits is also a purpose of testing. Without it bugs stay hidden. Now it was possible to find one. Thank you. So no, I would not change my testing.

Well, when engine programmers, some of whom, at least, are actually human, are being ruled by an algorithm, which we know is imperfect and uses data errors, yet can't be argued with, I call that terrorism. Get with the AI-machine age and start arguing with it, comrade!

Eelco wrote:

Mclane wrote:

It's IMO broken for x/y games.
At least on my machines I have
several time forfeits in arena

Can you reproduce it with another GUI perhaps Thorsten? What is X and Y specifically? Maybe some pgns.

We know what it is.

Chris Whittington wrote:

Ipmanchess wrote:

Does this mean there will be a new Rebel also..

I think so. The current Rebel should be okay on zero increment (eg 40/10 or whatever), but we'll need to fix for large increment testing (where large = increment > timemargin, with resulting time forfeits)

Did Rebel 16a have the same misinterpretation of the UCI time control rules as 16.1 ?

Ray wrote:

Chris Whittington wrote:

Ipmanchess wrote:

Does this mean there will be a new Rebel also..

I think so. The current Rebel should be okay on zero increment (eg 40/10 or whatever), but we'll need to fix for large increment testing (where large = increment > timemargin, with resulting time forfeits)

Did Rebel 16a have the same misinterpretation of the UCI time control rules as 16.1 ?

The UCI spec does not make it clear whether the increment should be pre-added or post-added, so you can’t call it a misinterpretation of the UCI spec.
What’s happened is that the ambiguity has been resolved by “tradition”. If an engine interprets outside the tradition then it gets punished by the GUI.
Btw, I have Aspergers and I find very often that when something is open to two alternatives, I pick the one the neurotypicals (98% of the population) do not. As here. Often it’s a creative superpower, but sometimes you have to bend to the NT popular will.

* wtime
white has x msec left on the clock
* btime
black has x msec left on the clock
* winc
white increment per move in mseconds if x > 0
* binc
black increment per move in mseconds if x > 0
* movestogo
there are x moves to the next time control,
this will only be sent if x > 0,
if you don't get this and get the wtime and btime it's sudden death
* depth
search x plies only.
* nodes
search x nodes only,
* mate
search for a mate in x moves
* movetime
search exactly x mseconds
* infinite
search until the "stop" command. Do not exit the search without being told so in this mode!

Quote :

The UCI spec does not make it clear whether the increment should be pre-added or post-added, so you can’t call it a misinterpretation of the UCI spec.

OK understood, it wasn't meant as a criticism, just a question.

Ray wrote:

Chris Whittington wrote:

Ipmanchess wrote:

Does this mean there will be a new Rebel also..

I think so. The current Rebel should be okay on zero increment (eg 40/10 or whatever), but we'll need to fix for large increment testing (where large = increment > timemargin, with resulting time forfeits)

Did Rebel 16a have the same misinterpretation of the UCI time control rules as 16.1 ?

IIRC it’s quite likely the assumption about increment was the same but the general tendency to move faster than necessary resulted in no time forfeits.

Time control is really quite complex and the way it gets worked out is different for different TC possibilities. There are several points where an engine can decide to cut the search and the amount of time to cut at should also be a function of how the search is going (upwards, downwards, choosing obvious move etc etc). The amount of increment relative to the total time reservoir. How much reservoir to use relative to moves to time control (or unknown end of game). It’s a non-trivial function to try and optimise.

Quote :

It’s a non-trivial function to try and optimise.

Yes, and many engines have had issues with time control over the years.

I use ChessGUI only for my engine v engine testing.
I have the time overstep margin per game set to 5000 milliseconds.
Rebel 16.1 1CPU has had no losses on time at 40 moves in 11 minutes repeating, but Rebel 16.1 4CPU had 40% time losses.
I sent Ed the debug files for the time losses, so I hope those can help in fixing the issue.

Graham Banks wrote:

I use ChessGUI only for my engine v engine testing.
I have the time overstep margin per game set to 5000 milliseconds.
Rebel 16.1 1CPU has had no losses on time at 40 moves in 11 minutes repeating, but Rebel 16.1 4CPU had 40% time losses.
I sent Ed the debug files for the time losses, so I hope those can help in fixing the issue.

Thanks, that’s a good lead. Ed is already, after your kind send of CCRL debug files, testing at 4CPU to try and get a repeat …..

I’ld hazard a guess the reason TC bugs are fewer nowadays is that one source of fails was the one ply search explosion and most people both know about that and at 1Mnps or more an engine still manages to return a ply one move. The other reason is open source, possibly most engines are using similar TC’s that are proven. I have an annoying habit of writing stuff from scratch -> bugs.

I'm delighted to report that older Rebel 16a is running perfectly for me under Cutechess with a 500ms margin with no time losses. OK, my little tournament means nothing, but this is the first time I've ever run Rebel and it feels great to be running an engine with the long history and pedigree that Rebel has. Congrats Ed and Chris for your huge contributions over such a long period.

Graham Banks wrote:

I use ChessGUI only for my engine v engine testing.
I have the time overstep margin per game set to 5000 milliseconds.

5 seconds ??

Ray wrote:

Graham Banks wrote:

I use ChessGUI only for my engine v engine testing.
I have the time overstep margin per game set to 5000 milliseconds.

5 seconds ??

Yes.
When other things are running on the computer as well, I like to play it safe.

Graham Banks wrote:

Ray wrote:

Graham Banks wrote:

I use ChessGUI only for my engine v engine testing.
I have the time overstep margin per game set to 5000 milliseconds.

5 seconds ??

Yes.
When other things are running on the computer as well, I like to play it safe.

As long as programmers don't make use of that information it is okay. But I think 1000ms is more than enough for cutechess. But of course I don't know chessgui well enough, it seems to record anything as seen in the debug files you mailed me, that could have a negative effect on engines. IOW, when it comes to time control nothing is simple.

When a engine can play 1sec+0sec.games ,then i'm sure it can handle any time control in my tests!
And no matter what Gui you using..

last evening i run Stockfish with 0.6sec+0sec.games without any Time Forfeits.