STS re-re-re-re-re-visited

For STS lovers -

STS stands for Strategic Test Suite. From the CPW we read -

Strategic Test Suite, (STS) a series of themed test suites by Dann Corbit and Swaminathan Natarajan, designed to evaluate chess engine's long term understanding of strategical and positional concepts. More recently, the positions were revised and converted by Ferdinand Mosca to be used by an analysis tool, MEA.

......

We made an effort to give STS a major update by re-analyzing the 1500 positions with nowadays strongest engines, Stockfish 15 and Lc0. And with MEA you can produce your own alternative rating list. See [ SF15 ] [ LC0 ] or as text file [ SF15 ] [ LC0 ].

......

http://rebel13.nl/misc/sts.html

Hi!
Thanks a lot for that again.
Pity the batch- file, as I edited it, doesn't work for me.
That's how it looks like now:

Code:

set MT=100
set HASH=64
set THREADS=30
set PROTOCOL=uci
set EPD=epd\sts-sf15.epd
set MRL=mea_results

set EXE=engines\Blue_Marlin_15.3_avx2.exe
set NAME=Blue Marlin 15.3
mea.exe --engine %EXE% --name %NAME% --hash %HASH% --threads %THREADS% --protocol %PROTOCOL% --epd %EPD% --movetime %MT%
mrl %MRL% %MRL% %EPD% %MT%

set EXE=engines\stockfish_22091112_x64_avx2.exe
set NAME=Stockfish 220911
mea.exe --engine %EXE% --name %NAME% --hash %HASH% --threads %THREADS% --protocol %PROTOCOL% --epd %EPD% --movetime %MT%
mrl %MRL% %MRL% %EPD% %MT%

So I only changed names of engine- binaries (newly stored in folder engines, there they do work if started with double click) and the engine- names for output.

Can somebody help?

Thanks in advance!
Peter.

set Name="Blue Marlin 15.3"

Hey there!
Great, now it did work!

Code:

   EPD  : epd\sts-sf15.epd
    Time : 100ms
                                                      Solving    Max   Total   Time   Hash          
    Engine             Score   Used Time Found   Pos     Time     Score   Rate    ms    Mb  Cpu    CCRL
 1  Stockfish 220911  13942  00:04:10.0  1210  1500  00:00:00.0  15000  92.9%    100    64   30     0
 2  Blue Marlin 15.3  13834  00:04:10.0  1208  1500  00:00:00.0  15000  92.2%    100    64   30     0


                                    Created with MEA
                                          by
                                       Ferdinand
                                         Mosca

Thanks a lot for prompt help, regards

peter wrote:

Hey there!
Great, now it did work!

Code:

   EPD  : epd\sts-sf15.epd
    Time : 100ms
                                                      Solving    Max   Total   Time   Hash          
    Engine             Score   Used Time Found   Pos     Time     Score   Rate    ms    Mb  Cpu    CCRL
 1  Stockfish 220911  13942  00:04:10.0  1210  1500  00:00:00.0  15000  92.9%    100    64   30     0
 2  Blue Marlin 15.3  13834  00:04:10.0  1208  1500  00:00:00.0  15000  92.2%    100    64   30     0


                                    Created with MEA
                                          by
                                       Ferdinand
                                         Mosca

Thanks a lot for prompt help, regards

Peter, welcome.

Small patch.

http://rebel13.nl/dl/mrl.zip

Replace mrl.exe in the SF folder.

It fixes the text file output.

Code:

    EPD  : epd\sts-sf15.epd
    Time : 100ms
                                                Max   Total   Time   Hash          
    Engine           Score   Found  Pos   ELO  Score   Rate    ms     Mb  Cpu      
 1  Koivisto 8.0     12248   1031  1500  3676  15000  81.7%    100    64    1

Solving time out -> Elo rating in.

Admin wrote:

Small patch.

http://rebel13.nl/dl/mrl.zip

Replace mrl.exe in the SF folder.

It fixes the text file output.

Code:

    EPD  : epd\sts-sf15.epd
    Time : 100ms
                                                Max   Total   Time   Hash          
    Engine           Score   Found  Pos   ELO  Score   Rate    ms     Mb  Cpu      
 1  Koivisto 8.0     12248   1031  1500  3676  15000  81.7%    100    64    1

Solving time out -> Elo rating in.

Thanks for info and link,
regards

One more question, please.
How can I edit the list of results?
Having deleted some of the old runs, they get restored by next one new run again.
Even tried to rename the mea_reults.txt- file and create a new empty one, it doesn't matter, old entries appear in new file after next run again.
Guess it has something to do with directory epd_out, but even having renamed this one and stored a new empty one with old name too, yet in table of results always all old ones are shown again after next one new run.

Thanks in advance again, regards

From the website -

Delete engines - results are kept in the file mea-results.csv and here you can make changes or delete engines.

make-html.bat - double click for new html or text output, for instance if you have made changes to mea-results.csv.

Very prompt answer again, thank you very much!

I had tried editing of .csv- file already too but failed to store it in correct format then. Editor made a .txt- file out of it, so therefore it didn't work.

Having corrected that and having make-html.bat run then again everything's fine now.

Best thanks

The site is gone, doesn't show up on site news, and the page is blank.

exactly, blank page

Not here, try refresh page with [F5]

Unfortunately you are right, page is empty but the download links are still present.

http://rebel13.nl/dl/sts-lc0.7z
http://rebel13.nl/dl/sts-sf15.7z

Reason is my provider is overzealous adding new functions to the web editor and has a habit to introduce bugs with it. As a result the website recently was completely ruined showing the contents of the main page also on every page. Had to install an older backup and sts fell of it.

okay, thanks for addressing it.

How about that:
These 594 "best" (not too easy) positions out of 1500 of STS

https://www.dropbox.com/s/x4qmr9ucykk0r76/594.epd?dl=0

together with about 500 UHO (or of even less balanced Anti Draw)- opening postions out of Stefan Pohl's opening- sets?

https://www.sp-cc.de/uho_2022.htm

The 594 from first link can be used as single best move positions too, the most unbalaned Anti Draw Openings could serve such as well if evaluated with Frank Schubert's EloStatTS instead of MEA I guess, just for being not as eval- dependent of the single position and engine. Positions with more then single best move just with multiple solutions evaluated  "positive" and with (instead of STS- points) time- dependent mini- matches for each postion and engine- pairing, see description of program here:

https://glarean-magazin.ch/wp-content/uploads/2017/03/L%C3%B6sung-eines-alten-Problems-Frank-Schubert-1.pdf

Wouldn't that with about 1000 positions mixed up like that out of opening, midgame and endgame
make a nearer to game- playing positional test suite maybe?

Happy_2023_regards

I like the MEA tool fpr position testing. One thing I dont understand, how are the numbers on the html table calculated? I mean if I have 1000 püsitiond and the max points are 10, the max result should be 10.000. But I get a score of lets say 16.915. What does that mean and how is it calculated. Are ther any explanations in the net?

fsanders wrote:

I like the MEA tool fpr position testing. One thing I dont understand, how are the numbers on the html table calculated? I mean if I have 1000 püsitiond and the max points are 10, the max result should be 10.000. But I get a score of lets say 16.915. What does that mean and how is it calculated. Are ther any explanations in the net?

Can you post about 100 of your epd's ?

100 was just an example, but I can post a table that shows what I mean:


Positions 621, found 374 score 10.261

The points assigned to moves aren't always from 1 to 10, any values go.

18.360 / 621 = 30

So I think the epd file you tested all best moves are rewarded with 30 points and other moves less.

That's why I asked to post some of the epds.

The positions are a subset of STS, I think there is no best move more than 10.

1k1r2r1/1b4p1/p4n1p/1pq1pPn1/2p1P3/P1N2N2/1PB1Q1PP/3R1R1K b - - bm Nxf3; c0 "Bc6=10, Rge8=8, Rgf8=10, Rh8=9";
1k1r3r/pb1q2p1/B4p2/2p4p/Pp1bPPn1/7P/1P2Q1P1/R1BN1R1K b - - bm Bxa6; c0 "Rhe8=10, g6=6, Bxa6=3, Rde8=2";
1r4k1/p1rn1ppp/2p1p1b1/N1PpP3/3P1P2/1R4P1/P6P/R4BK1 w - - bm Rxb8+; c0 "Rxb8+=10, Rc1=0, Kf2=0, Be2=0";
1r4rk/2q1b2p/pN6/P1p1Pp2/3n1p2/5N2/5QPP/R4RK1 b - - bm Nxf3+; c0 "Nxf3+=10, Rbd8=0, Qb7=0, Qc6=0";
1r6/1R2bk2/P3p1pp/4Pp2/1r1P3P/1N2B1P1/5PK1/8 w - - bm Rxb4; c0 "Rxb4=10, Rxb8=9, Na5=7, Nc5=2";
1r6/3n4/p2b3p/2nN1kpP/1BP1p3/3rP3/3NKP2/1R1R4 w - - bm Bxc5; c0 "Bc3=10, Bxc5=8, Rdc1=0, Re1=0";
1r6/8/p2b3p/2nN1kpP/2P1p3/3rP3/3NKP2/1R1R4 w - - bm Rxb8; c0 "Rxb8=10, Ne7+=9, Rf1=0, Rg1=0";
2r1r1k1/1p1q1bbp/pn3pp1/n2p4/3P1B1P/2NB1N2/PPQ1RPP1/4R1K1 w - - bm Rxe8+; c0 "Rxe8+=10, Qb1=6, h5=0, Qd1=0";
2r2nk1/pp3p2/5qpb/4p1Np/P3P2P/1P2B1P1/2r2P2/2RRQ1K1 w - - bm Rxc2; c0 "Rxc2=10, Kg2=0, a5=0, Qf1=0";
2r3k1/4qpp1/4p2p/1P1bP3/4pPP1/4Q3/4B2P/2R3K1 b - - bm Rxc1+; c0 "Rxc1+=10, Ra8=0, Rb8=0, Rf8=0";
2rk1br1/4np1p/p2pb3/1p1Npp2/P3P3/2PB2Pq/1PN2P1P/R2Q1R1K w - - bm Nxe7; c0 "Ncb4=10, Rg1=10, Nxe7=6, Nb6=0";
2rq1rk1/1p2bpp1/4p3/pP1pP1np/P2P4/R3BP2/3Q2PP/1NR3K1 w - - bm Rxc8; c0 "Rac3=10, Rxc8=0, Ra2=0, Rb3=0";
2rr2k1/p4pp1/1p2pn1p/3b1P2/1qNP4/1P1B4/P2Q2PP/2RR2K1 b - - bm Qxd2; c0 "Qxd2=10, Qe7=5, Qf8=3, a5=0";
3r4/5pk1/4rqp1/p2p4/Pb1Q3p/1Pp1P2P/2R1BPP1/3R2K1 b - - bm Qxd4; c0 "Qxd4=10, Qxd4=4, g5=3, Red6=0";
3rr1k1/6p1/pb1p3p/1p1RnBqP/4PQN1/2P5/PP6/1K3R2 w - - bm Qxg5; c0 "Nxe5=10, a3=3, Qxg5=1, a4=0";
4r2k/p1r1b1pp/1p1pqn2/4p3/1PP4B/P4B1P/4QPP1/2RR2K1 w - - bm Bxf6; c0 "Bxf6=10, Re1=5, Rc2=4, Qd2=1";
5k2/4pp2/1N2n1pp/r3P3/P5PP/2rR1K2/P7/3R4 b - - bm Rxd3+; c0 "Rc6=10, Rxd3+=8, Rc2=0, Rc7=0";
5qk1/1p1rbrpp/pP3p2/P1pbp3/1n6/BN1P2P1/1Q2PPBP/R1R3K1 w - - bm Bxd5; c0 "Bxb4=10, Bxd5=9, Rab1=0, Qc3=0";
6r1/4pp1k/3p3p/2qP1Pb1/r3P1PB/1R5K/4Q3/1R6 b - - bm Bxh4; c0 "Bxh4=10, Qd4=3, Rga8=0, h5=0";
7r/2r1kp2/3pp2p/p3q1p1/4B3/P5P1/1Q1B1P1P/4R1K1 w - - bm Qxe5; c0 "Qb6=10, Qb3=7, Qb1=7, Qxe5=0";
8/1p1bkq2/p2pp1n1/P1b1p1P1/4P1QN/1KNB4/1PP5/8 b - - bm Nxh4; c0 "Nxh4=10, Bf2=9, Bd4=9, Be8=9";
8/3n1nk1/q2b4/3p2p1/2pPp3/2P1P3/1Q4PN/3BB1K1 b - - bm Bxh2+; c0 "Nh6=10, Nf6=8, Qa3=8, Qb6=8";
br2r1k1/5p1p/p7/nq4Pp/p3P3/P1PQ4/1P2R1B1/K1BR4 b - - bm Qxd3; c0 "Qxd3=10, Rbd8=3, Nb3+=2, Bc6=0";
br3rk1/p2q1pp1/3p1n1p/2p1p3/2PnP3/P1BP1NP1/1R1Q1PBP/1R4K1 b - - bm Rxb2; c0 "Rxb2=10, Qc7=5, Nxf3+=3, Nc6=3";
q3b3/4Bpkp/6p1/3Pp3/r3r3/1Q5P/R4PP1/R5K1 w - - bm Rxa4; c0 "Bd6=10, Kh2=5, Rxa4=0, Ra3=0";
qr4k1/3b1pp1/4p2p/n2pP3/n1pP4/Q1P2NP1/3N1P1P/1R3BK1 b - - bm Rxb1; c0 "Rxb1=10, Rb7=6, Be8=0, Bc6=0";
R1b2b2/2kp1p1p/B2npp2/2p5/4P1r1/6P1/3N1P1P/3K3R w - - bm Bxc8; c0 "f3=10, Bxc8=9, h3=6, Kc2=0";
r1r1bnk1/5pp1/p2pp3/2q1n1P1/3QP2P/1PN4B/2P1NR1K/3R4 b - - bm Qxd4; c0 "Qxd4=10, a5=0, Qa5=0, Bb5=0";
r2qr1k1/2p2pp1/3p1n2/3P1n1p/1Q3P2/1B5P/PP4PB/2R1R2K w - - bm Rxe8+; c0 "Rxe8+=10, Qc4=5, Ra1=0, a4=0";
r2r2k1/4qp2/p5p1/2p4p/5P2/3RP1K1/PPQ4P/3R4 b - - bm Rxd3; c0 "Rxd3=10, Re8=8, Rdb8=5, Rdc8=5";
r3r1k1/2qn3p/1p2ppp1/3b4/p1PP1P2/4QN2/R5PP/1R3BK1 b - - bm Bxf3; c0 "Bc6=10, Bb7=7, Bxf3=5, Bxc4=0";
r3r3/p5bk/2pq1ppp/1p1b4/3P4/P2N1N2/1P3PPP/2RQR1K1 b - - bm Rxe1+; c0 "a5=10, Rxe1+=8, Rac8=7, h5=3";
r4rk1/pp5p/2pp2pb/2n1nq2/2P2P2/2N1B3/PP2B1PP/2RQR1K1 w - - bm Bxc5; c0 "Kh1=10, Bxc5=6, Rf1=0, g4=0";
r5k1/5p2/7p/5qP1/1P1QR3/5P2/r4RK1/8 b - - bm Rxf2+; c0 "Rxf2+=10, h5=8, hxg5=5, R2a6=0";
1k2r3/1p1bP3/2p2p1Q/Ppb5/4Rp1P/2q2N1P/5PB1/6K1 b - - bm Kc7; c0 "Kc7=10, b4=8, Kc8=8, Qa1+=0";
1n5k/3r2p1/2p1qp1p/3p1N1P/1P1P1rP1/p1R5/P7/1KRQ4 w - - bm Ka1; c0 "Ka1=10, Rxa3=3, Qe1=0, Re3=0";
1r2rqk1/8/bn1p3p/B1p2p2/p1PnpP2/P3R2Q/1P3NPP/2R2BK1 b - - bm Kh7; c0 "Kh7=10, Re6=8, Re7=0, Qf6=0";
1R3bk1/7p/6p1/8/1pN3P1/8/r4P1P/6K1 b - - bm Kf7; c0 "Kf7=10, Kg7=7, Re2=7, Rc2=7";
...

I see....

I think it's better to use the latest version.


http://rebel13.nl/dl/sts-lc0.7z
http://rebel13.nl/dl/sts-sf15.7z

Here the output with the latest version

From the download I first ran the official 1500 positions, then the 38 you posted, I get :




As you can see 38 positions and maximum points = 380, as it should.

Thanks, I will download again, do a new install and try again.
(As you can see the header of the table is really different(used time, score rate,...), no idea why)