New function for Polyglot.

Hello, it's Massimiliano Goi from the site chess.massimilianogoi.com here. As someone of you may know I made a successful Polyglot book over the years nicknament as my family name 'Goi.bin', this book has a kickass strength. But now things are in a critical point, being the book now almost 800 Megabytes with my suppliers sending me tons of new games.

I'd like to ask to the great programmer maintaining Polyglot now: is it possible to add a particular new function? In detail I'm speaking of a function taking as input a Polyglot (.bin) book and making a copy truncated up to the Nth move, for example as output a book having all the variant braches at best uo to the 13th move, no more.

This function of truncating the games would be useful also for pgn databases: in this way we could reduce alot the size of the created opening books without reducing the numbers of branches.

Welcome in the first place.

It's an unfortunate fact existing Polyglot books once created hardly can be changed, such as your wish. Impossible.

The PGN is a different story, you can do anything. If you want your books truncated at move 13 then run Polyglot again as follows:

polyglot make-book -pgn book.pgn -bin book.bin -max-ply 26 -min-game 1

max-ply is the trick, 26/2=13 moves.

To truncate te pgn games you can use the command line tool truncate.
It is included in the package 40H-PGN-2021
You can also truncate pgn games with pgn-extract but truncate is much faster and the results looks good.

http://nk-qy.info/40h/

Jonathan003 wrote:

To truncate te pgn games you can use the command line tool truncate.
It is included in the package 40H-PGN-2021
You can also truncate pgn games with pgn-extract but truncate is much faster and the results looks good.

http://nk-qy.info/40h/

The truncate function of pgn-extract does not really truncate the games: it only leaves out the games under a said length... I've already tried that way.

Admin wrote:

Welcome in the first place.

It's an unfortunate fact existing Polyglot books once created hardly can be changed, such as your wish. Impossible

Strange, because there is the function merge-book (which I always use): in order to merge two books the program must have the ability to read the whole book. If it can read, it can make a copy of it for sure. And I'd say it can also make a copy of the book until a certain length.

From - https://github.com/flok99/polyglot

What you need to know about merge-book:
Input files are not symmetrical, "in1" has priority over "in2". In
other words when a position occurs both in "in1" and "in2" only the
moves and weights from "in1" will be retained in "out".
So it's really better to rebuild the book from the PGN to get the weights right.


What you need to know about Polyglot books:
Roughly speaking a PolyGlot opening book is a collection of triples
(position, move, weight). A "position" is represented by a 64-bit
Zobrist hash key. The weight is proportional to the probability the
move should be played.
There is no move number.

Admin wrote:

From - https://github.com/flok99/polyglot

What you need to know about Polyglot books:
Roughly speaking a PolyGlot opening book is a collection of triples
(position, move, weight). A "position" is represented by a 64-bit
Zobrist hash key. The weight is proportional to the probability the
move should be played.
There is no move number.

I don't know anything about programming.
If I use the option 'dump-book' with polyglot-tolerant, with a bin book created from the game of Magnus Carlsen.
And open this pgn with SCID (after normalizing the pgn with ChessX), the main lines are changed. 1.c4 and 1.Nc3 become the main lines. And it should be 1.e4 and 1.d4
I think this is because there are no move numbers in the bin books.
That's the same reason when creating opening lines in Lucas ChessR by importing the best moves from a bin book. And you try to export these opening lines to pgn. You get an explosion of number of games.
That's because of the many possible transpositions in the bin book that where not in the original pgn games.
I wonder if it is possible to ad move numbers to a bin book?
I understand bin books are not menth to be used this way. They are only menth to be used by engines. But this could be useful for creating repertoires for otb-games.

Jonathan003 wrote:

But this could be useful for creating repertoires for otb-games.

The word repertory is a returning ingredient in your posts. I surely want to help but you but then must explain in detail how the end result should look like.

1. Have a look at the end of this page - http://rebel13.nl/output.htm where it starts with Opening repertory for white. The page is the output of a 2500 games PGN match between Stockfish 11 and Lc0 analyzed by the MRI util. Meaning that creating repertoires should not be that hard.

2. There is also a new version of Annotator with a new function - 2. Create selected opening books - from a large PGN create a new PGN with the opening moves of your choice. For instance, create (or open) the textfile opening.txt and type: "e2e4 e7e5 g1f3 d7d6 ", save and press [F6] select the PGN and all the games of the Pirc-Defence are stored in opening.pgn

3. http://rebel13.nl/rebel13/pgn-annotator.html#updates - many openings are already analyzed.

Seems to me enough information to come up with a plan how a repertory book should look like.

Admin wrote:

Jonathan003 wrote:

But this could be useful for creating repertoires for otb-games.

The word repertory is a returning ingredient in your posts. I surely want to help but you but then must explain in detail how the end result should look like.

1. Have a look at the end of this page - http://rebel13.nl/output.htm where it starts with Opening repertory for white. The page is the output of a 2500 games PGN match between Stockfish 11 and Lc0 analyzed by the MRI util. Meaning that creating repertoires should not be that hard.

2. There is also a new version of Annotator with a new function - 2. Create selected opening books - from a large PGN create a new PGN with the opening moves of your choice. For instance, create (or open) the textfile opening.txt and type: "e2e4 e7e5 g1f3 d7d6 ", save and press [F6] select the PGN and all the games of the Pirc-Defence are stored in opening.pgn

3. http://rebel13.nl/rebel13/pgn-annotator.html#updates - many openings are already analyzed.

Seems to me enough information to come up with a plan how a repertory book should look like.

Thanks for the offer to help me with founding some method to easily build repertoires.
I will investigate the tools MRI and Annotator to see how they can help with building repertoires.
I can't tell you what I want exactly because I want flexibility. For example to choose to build a narrow repertoire or a more wider repertoire.
I can give you some description of what I would like to do. I want to build a repertoire that is wide the first few plies, and more narrow deeper in the game.
My purpose is that the variations ends when main openings theory ends. And to find the right balance between including to much opening lines, and not enough opening lines, (I have to be able to study it). I want the repertoire to follows the main lines Magnus Carlsen plays, (or some other example player). Just to fill in the gaps, (lines that should be in the repertoire) and to replace blunders (for the side of the repertoire), I want to use some other model games, (engine games and games from strong players). One problem I have noticed is that when using engine games to fill in the gaps, is that the opening lines or not always the best, (it depends on the opening book that was used). I want to blundercheck the games, (for the side of the repertoire) with Stockfish 13. For the other side I want to include all possible reply's except very rare played lines (for example al lines played more than 0,5 %). And I want the final resulting repertoire in pgn only to contain opening lines I can find in the original pgn games where the repertoire is created from (the games of Magnus Carlsen, and the games from engines and strong players to fill in the gaps). And not some added transposition lines, like you get when importing best moves from a bin book to opening lines in Lucas ChessR. I see this same problem when exporting repertoires to pgn in Chess Opening Wizard and Chess Position Trainer. Transposition lines that where not in the original pgn database are included in the exported pgn database. Especially with Chess Position Trainer you see this explosion of number of games because of added transposition lines.
I have created a 'How to document' whit a method I use now for creating repertoires. But I want to improve the method.
If you are interested I can send you this document and some tools I use in a PM or by mail.
I hope this explains my ideas what I'm looking for. English is not my first language, so let me know if I have to explain sommeting better.

I still don't understand. A repertory is something a competitive chess player needs. In my case (back in my youth) that would be playing 1.f4 as white and with the black pieces I favored 1.e4 c6 and 1.d4 f5. With that in mind you learn the theory about these openings by head.

With nowadays strong engines learning your favorite openings (a repertory) has become much easier. You can download the Dutch, do a blunder check with POLY, then with SCID zero the weight of every move you dislike.

Admin wrote:

I still don't understand. A repertory is something a competitive chess player needs. In my case (back in my youth) that would be playing 1.f4 as white and with the black pieces I favored 1.e4 c6 and 1.d4 f5. With that in mind you learn the theory about these openings by head.

With nowadays strong engines learning your favorite openings (a repertory) has become much easier. You can download the Dutch, do a blunder check with POLY, then with SCID zero the weight of every move you dislike.

I'm not saying the method I propose for building repertoires is the right method, it is just a method I want to try.
The Dutch is a bin book, I want my repertoires to be in pgn format. Most training tools like Chessable, Chesstempo, Listudy... have an option to import pgn files, but not to import bin books.
And I don't know a good way to convert bin books to pgn, (without changing the main lines and adding unnecessary transposition lines).

I hope for Chess Position Trainer version 6 to be released, but I doubt if it will be released anytime soon, if ever...
I like the function in Chess Position Trainer to jump to positions with more than one candidate moves for the side of the repertoire. To easily remove a candidate move.
It would be nice to have such an option to navigate bin books to.
One problem I have with Chess Position Trainer is that if I export a repertoire to pgn, there is an explosion of number of games. I think this is because of added transposition lines that were not in the original pgn games where the repertoire was created from.
To solve this I import and export back to pgn in Chess Opening Wizard. Then the number of games in the exported pgn is reasonable.
But I don't like to have to do all these steps because I don't know exactly what they do.
I would like the export to pgn in Chess Position Trainer (or any other repertoire making tool), would be exactly the same then the pgn imported to create the repertoire. Except the candidate moves that I deleted would not be in the exported pgn, and the lines I added to replace blunders would be included in the exported pgn (without changing the main lines and adding unnecessary transposition lines).