Fire 8 released

I already gave a qualifier, the SIM-test. The CSVN is doing it for 8 years. Several testers do it. It would make the many chess programmers with a competitive attitude happy, now they are not heard.

SIM tests are absolutely not definitive. At best they might raise suspicion that further investigation is needed. Excluding an engine solely on a SIM test is wrong. In my opinion.

On the other side of the coin, isn't it possible to copy an engine pretty much completely, and then alter the evaluation function so that it appears to be a different engine ?

If there was a quick, reliable and accurate way, everyone would do it. There isn't.

I am hearing the echo what Bob has preached for years without actually studying the subject.

I did, I have been doing this SIM stuff for about 10 years by now and when it accuses (say 65%) it is right and not a single false positive has been reported.

In the 2019 similarity report I did together with Chris, SIMEX rightly reported Fire 7.1, rightly reported Houdini as a mixture of Stockfish and Komodo, later in 2020 confirmed by the Houdini 6 source code hack and by the Komodo author.

Did you know SIMEX reported a 75% similarity between Rybka 1 and Fruit 2.1?

Don't get me wrong, CCRL and CEGT are great for a large audience but they are unsympathetic to competitive chess programmers who see their artwork go down one place every time a new strong derivative is added. An OERL list would be programmer sympathetic and easily can exist in peace next to the other ones.

Ray wrote:

SIM tests are absolutely not definitive. At best they might raise suspicion that further investigation is needed. Excluding an engine solely on a SIM test is wrong. In my opinion.

From what Ed wrote in response, it looks as though it's a "reliable indicator".

It looks as though the problem is that it's a "cheap test", and other tests are "expensive". On this basis, maybe there's a solution along the following lines:

* if an engine fails a SIM test it gets excluded

* the author of the excluded engine then has the right to appeal

* if the author can demonstrate that their work is original, they get reinstated

I recently had a quick look at the actual evidence from the Rybka 2011 controversy (link): some of the evidence is based on a comparison of assembly language (I'm guessing that the Rybka team didn't offer the source code? Or maybe some of the functionality was actually written in assembly language? I know that Franz Morsch and the original Fritz team have done this in the past). It looks to me as though this evidence must have taken a lot of time and effort to produce. The fact that Rybka didn't offer a defence is also quite damning: my opinion is that had Rybka been entirely original in origin, the Rybka team would have defended the case.

TheSelfImprover wrote:

Ray wrote:

SIM tests are absolutely not definitive. At best they might raise suspicion that further investigation is needed. Excluding an engine solely on a SIM test is wrong. In my opinion.

From what Ed wrote in response, it looks as though it's a "reliable indicator".

It looks as though the problem is that it's a "cheap test", and other tests are "expensive". On this basis, maybe there's a solution along the following lines:

* if an engine fails a SIM test it gets excluded

* the author of the excluded engine then has the right to appeal

* if the author can demonstrate that their work is original, they get reinstated

That's a great point.

Quote :

I recently had a quick look at the actual evidence from the Rybka 2011 controversy (link): some of the evidence is based on a comparison of assembly language (I'm guessing that the Rybka team didn't offer the source code? Or maybe some of the functionality was actually written in assembly language? I know that Franz Morsch and the original Fritz team have done this in the past). It looks to me as though this evidence must have taken a lot of time and effort to produce. The fact that Rybka didn't offer a defence is also quite damning: my opinion is that had Rybka been entirely original in origin, the Rybka team would have defended the case.

Don't get me started

Let me only quote the words of Marcel van Kervinck a Dutch software engineer and author of the chess program Rookie, a Panel Member who voted Rajlich guilty but had a change of heart afterwards:

Quote :

This is a manufactured justification.

A lot more people didn't follow the forum rules and that was just let go.

Chris was excluded because he would undermine the purpose of the panel: to provide a case against Rybka.

One member announced preliminary findings in a public forum.

One member leaked discussions on a public forum.

Half of the panel was discussing things behind the backs of the rest, despite the rules forbidding this.

In retrospect, and from my point of view, that panel was just setup to lend credibility to the desired outcome, and nothing else.

The whole story at - http://rebel13.nl/home/icga.html take a deep breath first.

Admin wrote:

The whole story at - http://rebel13.nl/home/icga.html take a deep breath first.

Wow!

A lot of very good points. In particular, an appeal should have been allowed, and maybe their definition of original code wasn't fit for purpose.

Sticking to the narrow question of whether Rybka contained code from Crafty and Fruit as alleged, for me the evidence (linked in my previous post) strongly suggests that it did, and if it didn't the Rybka team could easily have refuted that. They could also have refuted it in public afterwards even without an appeal - unless copies of their product which large numbers of people owned contained strong evidence that it did.

There's one more point from your link that I'd like to discuss - the idea that Rybka must have been original because it was so much stronger than other programs on the same hardware. I'm happy to be corrected, but putting together a few things I've read and jumping to a conclusion, it looks to me as though the source of Rybka's strength was mostly that it was the first to use automated tuning more than new ways of writing chess code.

What we need in computerchess are original chess engines, not clones.

Obviously fire is not original.
And many other engines are also not original.

Some other programmers in opposite continue to program their baby for 10-30 or more years.

This is what gets my full respect.

Chris w. e.g. did computerchess on z80a sinclair zx machines !!
And these engines were very similar to the engines he later did on
Atari st or pc.
Programmers like ed and marty hirsch and mark uniacke or
Frans morsch, johan de koning or richard lang who did the thing on their own
and have very original interpretations of a computerchess
Engine.

I would make an exception in the case that somebody has a breakthrough idea on one aspect of the program, or changes the program to play in a very interesting new way.

Think of it as a great new version of a song!

Admin wrote:

I have been doing this SIM stuff for about 10 years by now and when it accuses (say 65%) it is right and not a single false positive has been reported.
.

And yet, another chess engine author has said "these similarity testers are nearly worthless".  So, as with everything in the chess world, no-one agrees on anything.

So the approach of testing the controversial stuff, and letting the audience decide what is important to him or her, isn't a bad approach. The games are there, so anyone can download the database, delete any engines they don't want to see, and run bayeseo or Ordo against the filtered database they've just created. And be happy. Maybe. And those who want to see the controversial engines aren't denied either, and are happy as well.

Good point Ray, maybe I will give it a try.

Any idea about the parameters CCRL uses for Bayeseo or Ordo ?

No need to do it yourself. CCRL has enough mechanisms to make this list easily (they had a "pure list" a few years ago). I think that Graham will gladly do it himself if asked.

I made myself a custom selection (next post).

matejst wrote:

No need to do it yourself. CCRL has enough mechanisms to make this list easily (they had a "pure list" a few years ago). I think that Graham will gladly do it himself if asked.

I am doing it anyway, working on it right now

matejst wrote:

How did you make that ?

Besides, Critter, Chiron, Equinox are derived from Robbolito, not on my list, other than that looks great.

It is easy, Ed. You go under the list, and there is a custom engine selection. You simply select the engines you want in the list. It is easy to add or remove engines.

E.g.

First try - http://rebel13.nl/text/rating-1.txt

matejst wrote:

It is easy, Ed. You go under the list, and there is a custom engine selection. You simply select the engines you want in the list. It is easy to add or remove engines.

Wow, never knew, never tried, cool.

Admin wrote:

First try - http://rebel13.nl/text/rating-1.txt

I am pretty good with that list. As that is what I test mostly by accident. I did not look at the sim test.

I assume Lc0 is good, and I also did not see Fritz 17.?

http://rebel13.nl/text/rating-2.txt

This is a better list.

Interesting enough because of the excluded engines the CCRL-40-40 pgn did shrink with 250,000 games from 1.2 million to 970.000 games.

Code:

Excluded engines
Fat Fritz
Houdini
Fire
Deep Shredder 13
Chiron
Gull
Nirvana
Equinox
Critter
Bouquet
BlackMamba
Strelka
Onno
Loop
ShashChess
SugaR
asmFish
MateFinder
Schooner
Vitruvius
IvanHoe
RobboLito
DeepSaros

Now I will have to produce a best version list which requires some programming.

Admin wrote:

http://rebel13.nl/text/rating-2.txt

This is a better list.

Interesting enough because of the excluded engines the CCRL-40-40 pgn did shrink with 250,000 games from 1.2 million to 970.000 games.

Code:

Excluded engines
Fat Fritz
Houdini
Fire
Deep Shredder 13
Chiron
Gull
Nirvana
Equinox
Critter
Bouquet
BlackMamba
Strelka
Onno
Loop
ShashChess
SugaR
asmFish
MateFinder
Schooner
Vitruvius
IvanHoe
RobboLito
DeepSaros

Now I will have to produce a best version list which requires some programming.

I was thinking the list of excluded engines would be much bigger. My mistake. I thought Stockfish would be on that list. And I am willing to take your word on the Fire chess engines. As I am clearly not an expert in the programming side of chess engines. That would be you for sure.

It is funny that I have also excluded all those engines. In my current testing except Gull and Fire. As I was not sure.

Admin wrote:

matejst wrote:

How did you make that ?

Besides, Critter, Chiron, Equinox are derived from Robbolito, not on my list, other than that looks great.

What would the criteria be for exclusion? If it's just being a derivative, then Stockfish should be excluded, too, since it was derived from Glaurung. I'm definitely not advocating going that far, but this goes to show us how quickly such exclusions could get out of hand.

Is it the SIM test? Many of the excluded derivatives may well pass the SIM test, anyway. It seems all too arbitrary.
I think Graham/CCRL is right to err on the side of inclusion.

http://rebel13.nl/text/rating-best.txt

This is how a best version could look like.

Will answer some questions later.

Edit: oh well, Fritz is missing again....

Admin wrote:

http://rebel13.nl/text/rating-best.txt

This is how a best version could look like.

Will answer some questions later.

Edit: oh well, Fritz is missing again....

Please note that were issues about Alfil 15.07 and also the more recent Beef (being SF derivatives). Not that I agree or disagree.
I like how Alfil 15 plays, very good aggressive and risk-taking attacking play, far better in style than what SF from 2015 had to offer.
Also BlackMamba, not on your list, was supposed/suspected to be Ippolit-inspired, but played a much more attractive type of attacking chess.

I feel we're somehow poorer without such engines.

http://rebel13.nl/text/rating-best.txt

Fixed the bug, Fritz now on its rightful place

mwyoung wrote:

I thought Stockfish would be on that list.

Many engines came from TSCP and later became original. Many engines came from Fruit, TOGA, Fruit Reloaded and during the years added so much strength while mentioning the origin they no longer can be called a derivative. Likewise with Stockfish, they still name Tord as author. In the "Programmer Code" I use the term "inspired" instead of "derivative" for such engines.

Nezhman wrote:

What would the criteria be for exclusion? If it's just being a derivative, then Stockfish should be excluded, too, since it was derived from Glaurung. I'm definitely not advocating going that far, but this goes to show us how quickly such exclusions could get out of hand.

See above and no doubt this list may cause a lot of anger if I am going to give it my active support but I think a programmer poll (as the current rating lists are hostile to competitive programmers) could clear the sky. I am pretty sure most of them will interpret the term derivative with a different comprehension than just literal.

Quote :

Is it the SIM test? Many of the excluded derivatives may well pass the SIM test, anyway. It seems all too arbitrary.
I think Graham/CCRL is right to err on the side of inclusion.

The SIM test is part of the exclusion list. Some engines don't need a sim-test at all, FF1, FF2, Sugar, Shashchess, Houdini, Fire, Strelka, BlackMamba, Bouquet, Loop etc. In fact this is valid for the vast majority of the (current) exclusion list either by admission of the author or by research.

Nezhman wrote:

Please note that were issues about Alfil 15.07 and also the more recent Beef (being SF derivatives). Not that I agree or disagree. I like how Alfil 15 plays, very good aggressive and risk-taking attacking play, far better in style than what SF from 2015 had to offer. Also BlackMamba, not on your list, was supposed/suspected to be Ippolit-inspired, but played a much more attractive type of attacking chess. I feel we're somehow poorer without such engines.

I remember Alfil was mentioned at the time but never researched it. And indeed I wondered about Beef. Anyway, I found my inspiration doing this because of the rating lists happily adding a clone (FF2). It's hostile to direct competitors showing disrespect to the hands that feeding them.