EAS - for Stefan

I have my moments

I have a 22K human game collection between 2700+ players. If I have a PC free I will give it a try and make a polyglot book from it using the somu tool. Uploaded elo2700.pgn on the somu page, maybe it's a good testbed for the real 15M.

Over 30% of those games are either rapid or blitz, just so you know. You may want to check that.

Ozymandias wrote:

Over 30% of those games are either rapid or blitz, just so you know. You may want to check that.

And 1305 TCEC games ......

Do you have a tool that strip those games?

Anyway, SOMU using the [F8] key has such a function, executable SOMU-F8.EXE on github.

Pressing [F8] you can :

1. Strip draws

2. Make an elo selection.

3. Strip words

On [3], when you type "blitz" all games with "blitz" in the Event-tag or Site-tag will be stripped.

OUTPUT.PGN is the new created stripped file.

BTW, what is the rapid time control as used in human games, maybe they should be stripped as well.

Admin wrote:

Ozymandias wrote:

Over 30% of those games are either rapid or blitz, just so you know. You may want to check that.

And 1305 TCEC games ......

Do you have a tool that strip those games?

Engine games too? I didn't check, I was talking from experience. Once you've deleted all those games, you'll probably have less than 15K (which is what I have).

For those purposes, I use tagExtract and keep excludeT.pgn.

Admin wrote:

BTW, what is the rapid time control as used in human games, maybe they should be stripped as well.

15m to 25m, with increment of 3 to 5 seconds. And yes, I throw them out.

Years ago TWIC started to include TCEC games.

I found games of chess.com as well, 3350 (15%) to be precisely, yikes....

Code:

PGN pruning
PGN database      : pgn\elo2700.pgn
Read PGN game     : 22.315
Skip Online games y
To OUTPUT.PGN     : 12.456
Done...

This is what I get after stripping a pgn with the words :

"blitz" , "rapid" , "tcec" , "chess.com" , "comp"

In one run and case insensitive of course.

Other terms that might bring down the count: blind, simul, internet, active, knockout, tiebreak, K.O.

I've noticed that, quite often, play-offs between players are faster rounds for qualification purposes, even though there's usually no other indication other than several games being played on the same day between two players, which is quite telling.

Ozymandias wrote:

Other terms that might bring down the count: blind, simul, internet, active, knockout, tiebreak, K.O.

Can "tagExtract" handle all these terms in one run?

I can do it for SOMU by creating a simple text file with all the terms you want to exclude in one run

blind
simul
internet
active
knockout
tiebreak
K.O.
blitz
rapid
tcec
chess.com
comp

-------

or whatever.

Nope, tagExtract works one by one. if your tool doesn't face problems with big DBs, it holds a huge advantage.

Ozymandias wrote:

Nope, tagExtract works one by one. if your tool doesn't face problems with big DBs, it holds a huge advantage.

SOMU can handle any PGN size and is a lot faster than Norm's (great) utils.

I will do the text file idea, small change.

Code:

PGN pruning
PGN database      : pgn\elo2700.pgn
Read PGN game     : 22.315
Skip words (y/n)  y
To OUTPUT.PGN     : 11.051
Done...

Using "words.txt" another 1000 less.

SUMO.EXE and words.txt on github.

Run took 4 seconds.

----------

Added "lichess" and "chess24.com" to words.txt

Code:

PGN pruning
PGN database      : pgn\elo2700.pgn
Read PGN game     : 22.315
Skip words (y/n)  y
To OUTPUT.PGN     : 10.849
Done...

Well, that looks too good to overlook. I wasn't planning on another update, but since LSS games for the first half of the year are already online, cubail posted some PC and IC games for the past year, and an update of OTB games looks so streamlined with this tool, I guess I'll bite.

Only book testing games (via Sedat) won't be updated, since he's on vacation.

Wait till you get the invoice....

with 100% discount of course

You joke, but his weekend I was trying to explain to my nephew how some programmer was doing something for free, and he just couldn't understand it.

WARNING

In case you are making books with "polyglot make-book" get the fix for SUMO-F5 on github.

I tried to make a polyglot opening book from the remaining 10,000 games from elo2700.pgn and polyglot (with its extreme pgn checks) refused the pgn coming from SOMU. What gives? Well, when a PGN is truncated it misses the game result and Polyglot goes berserk.

Fixed it, makes a book. New SOMU.EXE on github.

Old output : 1. e4 e5 2. Nf3 Nc6 3. Bc4 Nf6 4. d3 Bc5
New output : 1. e4 e5 2. Nf3 Nc6 3. Bc4 Nf6 4. d3 Bc5  1-0

I thought it was nonsense to do, Fabien decided otherwise.

AS a PGN integrity check, polyglot has no equal.

Indeed.....

Book made, system seems to work, on github.

I forgot to ask. The last move saved is the one before the blunder, but when you establish an overall limit, is the position, where that value has been reached, saved to output? Or is the game truncated on the previous one? The later would be the consistent behavior.

PS: the latest exe doesn't work. "This app can't run on your PC".

Ozymandias wrote:

I forgot to ask. The last move saved is the one before the blunder, but when you establish an overall limit, is the position, where that value has been reached, saved to output? Or is the game truncated on the previous one? The later would be the consistent behavior.

Yes, consistent. The moves that fail the blunder margin or the max margin are stripped.

Quote :

PS: the latest exe doesn't work. "This app can't run on your PC".

Before I do another recompile that generates different code can you rename somu.exe to something strange like umos.exe? I think Windows remembers the false warning on somu.exe

Meanwhile we should discuss the book, there are issues, more in a next post.

I renamed it, changed it to another location: same message.

Sure, what issues?

Ozymandias wrote:

I renamed it, changed it to another location: same message.

Uploaded somu-new.exe

Ozymandias wrote:

Sure, what issues?

1. The use of analyzed books. Should be an advantage, but....

. Book made with blunder margin 50. Playing 1.e4 c5 2.Nf3 d6 and the only playable book move is Bb5, ridiculous.
. Book made with blunder margin 75. Playing 1.e4 c5 2.Nf3 d6 and now  3.d4 (97%) 3.Bb5 (3%), good. Disadvantage, the analysis takes ~3 x more time.

-------

2. Using normal polyglot books without analyzed data.

1. Using the small book "varied.bin".
2. Since book score is always 0 all moves in the book are accepted without analysis.
3. First move out of book (for instance using blunder margin 50) that gives a score of -51 is stripped which can be meaningless certainly with NNUE evaluation.

Both are undesirable, but that's just me.

ozzy-50.bin
ozzy-75.bin
ozzy-varied
on github

--------

Suggestions

1. Skip the book option, analyze every position. Not so nice but accurate.

2. Take a small book (for instance varied.bin) and analyze it with SF16 NNUE eval, depth=20 or so.