POLY 1.7 new features

1. Make Analyzed Book - Create a new and fully analyzed Polyglot opening book from a PGN collection.
   
2. Blunder Check - Opening books contain errors, the new Blunder Check option filters them out.
   
3. Browse Book - Quickly check the changes you made and if the weights / scores of critical positions are in harmony.
   
4. Stockfish Analysis - has been replaced with MEA analysis which allows you to use any UCI engine, even Lc0 and co.
   
5. From the ground up - Tour du Poly, a step by step introduction course how to use POLY effectively.
   

....

http://rebel13.nl/download/books.html

Hi Ed,
I tried the tool Poly 1.7 to analyze an existing smal bin book, but I run in to some troubles.


I follow the manual and it go's fine for step 1 and 2, but for step 3 I get some errors.

The bin book is called 'jonathan.bin' and if i import the analyzed epd 'jonathan.epd' I get 2806 errors.

I must be doing something wrong.
I also have made a video capture of it, to better explain what the problem is. What's the easiest way to attach a video here?

Here is a link to the video capture on my google drive:

https://drive.google.com/file/d/1-AS-G9gsHT6MX5y0kml8zc3ohQIgVyuy/view?usp=sharing

And here is a link to the bin book and pgn file I used:

https://drive.google.com/drive/folders/1QyNcDQeqjjdh81FQvyGyS6tMIYPqGJPy?usp=sharing

After the analysis is done the screen states:

Code:

Waiting till all 4 threads are finished

Creating file 'sf12.epd' in folder EPD

Done... run 'Import Analysis' now

So you need to use sf12.epd for Import Analysis.

It's a mistake on the documentation on the website, I will fix it later.

Admin wrote:

After the analysis is done the screen states:

Code:

Waiting till all 4 threads are finished

Creating file 'sf12.epd' in folder EPD

Done... run 'Import Analysis' now

So you need to use sf12.epd for Import Analysis.

It's a mistake on the documentation on the website, I will fix it later.

Thanks this works!
It looks like a promising tool to blunder check bin books, you made!
Is there some way to easy find al these positions, (where blunders are found), in the output bin book? To check them to see if I want to change anything.
If not, I would like to request this option for future updates. Maybe in some separate epd file where all these positions are listed.
I have sent you an email, with some request, but maybe it is better to discuss it here so others that want to use the tool can read it to.
Can I first use the option blunder check bin book, and afterwards use the option, tuning the weight using the WDL for the blunder checked book? Or will moves that became invisible because it are blunders, get a higher score again?

Jonathan003 wrote:

Admin wrote:

After the analysis is done the screen states:

Code:

Waiting till all 4 threads are finished

Creating file 'sf12.epd' in folder EPD

Done... run 'Import Analysis' now

So you need to use sf12.epd for Import Analysis.

It's a mistake on the documentation on the website, I will fix it later.

Thanks this works!
It looks like a promising tool to blunder check bin books, you made!
Is there some way to easy find al these positions, (where blunders are found), in the output bin book? To check them to see if I want to change anything.
If not, I would like to request this option for future updates. Maybe in some separate epd file where all these positions are listed.

Unfortunately that's impossible, a Polyglot book contains the hash key of a position, not the position itself.

Quote :

I have sent you an email, with some request, but maybe it is better to discuss it here so others that want to use the tool can read it to.
Can I first use the option blunder check bin book, and afterwards use the option, tuning the weight using the WDL for the blunder checked book? Or will moves that became invisible because it are blunders, get a higher score again?

The Blunder Check and WDL are unrelated. The WDL parameter is used to make a book more or less random.

Admin wrote:

Jonathan003 wrote:

Admin wrote:

After the analysis is done the screen states:

Code:

Waiting till all 4 threads are finished

Creating file 'sf12.epd' in folder EPD

Done... run 'Import Analysis' now

So you need to use sf12.epd for Import Analysis.

It's a mistake on the documentation on the website, I will fix it later.

Thanks this works!
It looks like a promising tool to blunder check bin books, you made!
Is there some way to easy find al these positions, (where blunders are found), in the output bin book? To check them to see if I want to change anything.
If not, I would like to request this option for future updates. Maybe in some separate epd file where all these positions are listed.

Unfortunately that's impossible, a Polyglot book contains the hash key of a position, not the position itself.

Quote :

I have sent you an email, with some request, but maybe it is better to discuss it here so others that want to use the tool can read it to.
Can I first use the option blunder check bin book, and afterwards use the option, tuning the weight using the WDL for the blunder checked book? Or will moves that became invisible because it are blunders, get a higher score again?

The Blunder Check and WDL are unrelated. The WDL parameter is used to make a book more or less random.

Does a book contain all the prior positions? If so, it should be possible to tree search for the hash.

I don't see how that can be done, example:

Code:

Hash Key    Moves     Score
0x11111111    e4       0.25
0x22222222    e5       0.00
0x22222222    g5      -0.60

Util says, 1..g5 is bad, throw it out. How do I get the EPD?

Admin wrote:

I don't see how that can be done, example:

Code:

Hash Key    Moves     Score
0x11111111    e4       0.25
0x22222222    e5       0.00
0x22222222    g5      -0.60

Util says, 1..g5 is bad, throw it out. How do I get the EPD?

Go back to the root and search the book tree until hash match

Which will take too long, even when the book is loaded in memory.

I think it's best (as Jonathan makes a fair point) to get the EPD from the analysis, that is, if the EPD is still present.

Code:

1b1q1rk1/1p1r1ppp/p1n1pn2/8/2PP4/P4N2/1BB1QPPP/3RR1K1 b - - bm Re8; ce -166; acd 23; po 1455232;

The "po" tag contains the offset in the book.

What happens if you blunder check a book, and poly finds a blunder, but there are no alternative moves in the bin book?
I think than this move just stays in the book. I think it would be better if the move got deleted and replaced by a better move. Also if there are alternative movers but they are not better than the find blunder, it would be better if they got replaced by a better move.
I have no Idea if it can be done.

1. The blunder check only tests cases when 2 or more moves are available for a position and in the worst case the best move is kept, the rest thrown out.

2. In ProDeo 3.1 (not released yet) I do the same blunder check but can do an extra check also, if the score if the previous move is clearly better than the current best move that book move might be a blunder and so I don't play that move and let the engine do the work.

I will implement option 2 in the next version of Poly that will have a (manual) book editor as well but for the moment I am busy with other things.

POLY 1.7a - bug fix version

http://rebel13.nl/download/books.html

I noted a couple of bugs while doing extreme things for book creation.

1. When using many threads (30) analyzing a book MEA doesn't always move the result to the proper folder.
2. Storing whole games in a book (instead of the first 20-30 moves) gave trouble in case a game ended in stalemate.
3. And to my shame I noticed a mean ep bug.

Is there any change to make poly work with the old book format obk?
The big advantage of obk is that it can easily be converted back to pgn. John Merlino made a obk making tool, where you can set different percentages for the book weights (the three dot's). When making a obk book and choose the option weight by frequently, you can now specify how much percentage a move have to be played to get one, two, three or no dot's assigned. Then you can export the obk book to pgn with obk2pgn and remove sidelines (moves with no dot's), by searching for games with a '?' in the annotation and delete them. That's a great approach for making repertoires for over the board games.

Jonathan003 wrote:

Is there any change to make poly work with the old book format obk?
The big advantage of obk is that it can easily be converted back to pgn. John Merlino made a obk making tool, where you can set different percentages for the book weights (the three dot's). When making a obk book and choose the option weight by frequently, you can now specify how much percentage a move have to be played to get one, two, three or no dot's assigned. Then you can export the obk book to pgn with obk2pgn and remove sidelines (moves with no dot's), by searching for games with a '?' in the annotation and delete them. That's a great approach for making repertoires for over the board games.

Can you post a sample of obk2pgn, or tell me where I can download the tool?

The 32 bit version of obk2pgn still can be find online here with the wayback machine:
https://web.archive.org/web/20181130013407/http://homepages.paradise.net.nz/npattins/ChessOpeningBookPage.html
The creator of that tool is a certain Nigel Pattinson, but he is not an active chess programmer anymore, and I don't know how I can contact him. I found a 64-bit version of it a long time ago on the internet. I can't find it anymore, luckily I still have a copy. I don't know if this 64-bit version was also created by Nigel Pattinson. I think it works exactly the same as the 32-bit version but only faster.
I have some troubles with obk2pgn, both with the 32bit version as with the 64bit version.
If I try to convert some obk books to pgn, it sometimes gets only partially to phase 1.
It has to do two phases, phase 1 and phase 2, but sometimes it doesn't even get to phase 2.
I think the problem is with underpromotion to a Knight or Bishop for example, to avoid stalemate.
Maybe it is something that easily can be fixed.

You can also use the obk2bin tool included in OpenBook, and use the option '-p' to also export a pgn with annotations according the book weights.
http://goneill.co.nz/chess.php
But with obk2bin there is a limitation of a maximum lengt of 100 ply for the obk books, to able to convert them.

Here is a link to my google drive with al the tools that I use:
https://drive.google.com/file/d/1Jl-Xh56B5nywBJvW9kpQsAreKFT-Rcq9/view?usp=sharing

I'm not sure what gives best results obk2pgn or obk2bin
I have also included MakeBook_v3.2 the latest version of the obk book making tool by John Merlino
He is still working on updates because the 'delete zero branches' option is not working like it should.

This is a small sample of how the output of obk2pgn looks like. It is a sample from the output I use for my black repertoire:

Code:

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "-"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
1. d4!! d5!! 2. Nc3? Nf6!! 3. Bf4!! a6!! 4. e3!! e6!! 5. Nf3!! c5!! 6. dxc5!!
Bxc5!! 7. Bd3!! b5?

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "-"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
1. d4!! d5!! 2. Nc3? Nf6!! 3. Bf4!! a6!! 4. e3!! e6!! 5. Nf3!! c5!! 6. dxc5!!
Bxc5!! 7. Bd3!! Nc6!!

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "-"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
1. d4!! d5!! 2. Nc3? Nf6!! 3. Bf4!! a6!! 4. e3!! e6!! 5. Nf3!! c5!! 6. a3!! b5!!

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "-"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
1. d4!! d5!! 2. Nc3? Nf6!! 3. Bf4!! a6!! 4. e3!! e6!! 5. Bd3!

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "-"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
1. d4!! d5!! 2. Bg5? f6!! 3. Bf4!! Nc6!! 4. Nf3!!

[Event "?"]
[Site "?"]
[Date "????.??.??"]
[Round "-"]
[White "?"]
[Black "?"]
[Result "1/2-1/2"]
1. d4!! d5!! 2. Bg5? f6!! 3. Bf4!! c5!! 4. Bxb8!! Rxb8!!

This is what the readme of obk2pgn says about the pgn output:

PGN Output

Each variation in the original OBK file is output as a separate game in the PGN file. Each move is annotated according to the book score: 3 = !!, 2 = !, 1 = , 0 = ? .

Often in a Chessmaster book it is possible to reach the same position via multiple different lines, often with different scores. This causes problems when converting to an opening book format which stores positions rather than moves. To avoid these problems, the program calculates which score is most common for each position and standardises them.

I get this error message when I analyzed a only-white book with poly 1.7a: 'Mea did not move 1 epds'.

So I looked in the epd_out folder and the file 'x2_sf12.epd' was missing. I find this file in the main folder of poly17. So I moved it to the epd_out folder.

I hope I handled this correctly?
I had to hit enter a couple of times in poly1.7a, after I moved 'x2_sf12.epd', to the epd_out folder, before I could continue. Maybe this was because I was recording a video with ZD SOFT Screen Recorder, at the same time.

What's the easiest way to attach a video here so you can see a thumbnail?  I see an option to insert a YouTube video, here is a video capture of it:

I see the weight of the moves get changed, after blunder checking a existing bin book, with poly 1.7a


The book 25_cent.bin is the blunder checked book, with the option 25 centi pawns.
I want an option to NOT change the weights, but only remove the lines, if they are worse than the 25 centi pawns.

[quote="Admin"][quote="Jonathan003"]

Admin wrote:

The Blunder Check and WDL are unrelated. The WDL parameter is used to make a book more or less random.

If I first do a blundercheck, with poly 1.7a, and worse moves are nullified, and afterwards I use the option Weights on WDL. Sometimes moves that where nullified by the blundercheck gets weight again.
That's what I was afraid of. I would like to have an option so that moves that where nullified (because they are blunders), don't get weight again after using the option Weights on WDL on a blunder checked book.




The left book is the book after the blunder check. The right book is the book after using the option Weight on WDL on this blunder checked book. Maybe I better do it the other way around? First use the option Weights on WDL, and then blundercheck this book.
There is some display bug with Scid vs PC for the book after the blunder check, in Scid you don't see this display bug, but you can load only one book in SCID.

The size of the pictures you posted made the thread a bit unreadable, can not answer in detail.

Jonathan003 wrote:

I get this error message when I analyzed a only-white book with poly 1.7a: 'Mea did not move 1 epds'.

So I looked in the epd_out folder and the file 'x2_sf12.epd' was missing. I find this file in the main folder of poly17. So I moved it to the epd_out folder.

I hope I handled this correctly?
I had to hit enter a couple of times in poly1.7a, after I moved 'x2_sf12.epd', to the epd_out folder, before I could continue. Maybe this was because I was recording a video with ZD SOFT Screen Recorder, at the same time.

Yes, you did well.

Out of curiosity, how many threads did you use?

One way or another I must find a way around this annoying problem.

Admin wrote:

The size of the pictures you posted made the thread a bit unreadable, can not answer in detail.

Sorry for this, I can see your point.
I don't want to make the posts looks messy. I would be better if it where small thumbnail links. I will fix it so it are thumbnail links.

Admin wrote:

Jonathan003 wrote:

I get this error message when I analyzed a only-white book with poly 1.7a: 'Mea did not move 1 epds'.

So I looked in the epd_out folder and the file 'x2_sf12.epd' was missing. I find this file in the main folder of poly17. So I moved it to the epd_out folder.

I hope I handled this correctly?
I had to hit enter a couple of times in poly1.7a, after I moved 'x2_sf12.epd', to the epd_out folder, before I could continue. Maybe this was because I was recording a video with ZD SOFT Screen Recorder, at the same time.

Yes, you did well.

Out of curiosity, how many threads did you use?

One way or another I must find a way around this annoying problem.

I have a windows 10 64 bit pc with 4 cores, (8 treads).
I used 4 treads, but I must have done something wrong. I tried it a second time and then I didn't get this error message. All 4 sf12 epd files where already in the right folder.
I still need to learn how everything works with poly 1.7a
But it seems like a really helpful tool.
I try to use it to make repertoires for over the board games. That's why I start with a only-white book for my white repertoire, and a only-black book for black repertoire.

Some questions about the options 'Weights on WDL' and 'Blunder check', to be used on existing bin books.

Say I have a bin book called 'reference.bin' made from a pgn database called 'reference.pgn'.

1. I use the option Weights on WDL, and I choose the value 250. I rename the resulting bin book, and the resulting wdl file, to 'reference_wdl_250.bin' and 'reference_wdl_250.wdl'

2. Afterwards I use the Blunder Check option on the book 'reference_wdl_250.bin', I let the analysis run overnight, and I use the option 25 centi pawns for the blunder check. I rename the new blunder checked book to 'reference_wdl_250_cent_25.bin'

Now my question is if I repeat step 1, but now I choose the value 100 for WDL. I again rename the resulting bin book, and the resulting wdl file, to 'reference_wdl_100.bin' and 'reference_wdl_100.wdl'. And now I want to blunder check this new bin book. Can I just import the analysis from sf12.epd, that I already have? Or will I have to do the overnight analysis again. Will the moves that where nullified in the bin book 'reference_wdl_250.bin', but not in the book 'reference_wdl_100.bin', get blunder checked this way?
Or is it better if I first blunder check the bin book 'reference.bin', and afterwards make the bin books 'reference_wdl_250.bin' and 'reference_wdl_100.bin'. And blunder check these books by importing the analysis from sf12.epd from when I blunder checked the bin book 'reference.bin'.

I have some request for future versions of poly.
Because, I use poly1.7a, to make repertoires. I would like to have options to only change weight for the white or for the black moves in a bin book. When using the options 'Weights on WDL' and 'Blunder check'.
I want this option because the obk2pgn tool I use (before using poly1.7a), sometimes fails to convert a obk book to pgn.
That's why I want to us obk2bin included in Open Book
http://goneill.co.nz/chess.php

And use the resulting bin books with poly 1.7a
The output bin books from obk2bin have weight for both colors.
A way around would be if it was possible somehow, to make a only-white, or only-black book from an existing bin book (with weights for both colors).

If I blundercheck a only-white bin book, will this go faster than if I blundercheck a bin book with moves both for white as for black? I mean if the bin book with moves both for both colors is made from the same pgn games as the only-white bin book is made of, of course.