Probabilities, Natural Justice, and Chess.com: Part 2
In attempting to find out more about what Chess.com does to detect cheating I found [5] that they had bought the company that produces the chess engine Komodo, currently ranked as the third strongest engine [6], and also an article on their web site which said this [7]:
‘Chess.com’s fair-play system is thorough, complex and rigorously verified by more than eight years of data from millions of games played by our own members online. Our system gathers and reviews different types of data and other information pulled automatically (and manually) from all member games.
We load these games into a tool that provides the probability that a given player is playing cleanly or with the assistance of a computer engine. Before any accounts are closed, all reports are thoroughly reviewed by a team of specialists who have reviewed and closed thousands of accounts in their roles as Chess.com statisticians.’
I also found an article by their Chief Chess Officer, IM Danny Rensch [8], which included this:
‘Chess.com’s system is based on the statistical measurement of how humans and engines compare in all aspects of the game. We can measure with statistical certainty how often a human can, over a series of games, match up with an engine and still be only human.’
So, it is a fair bet that they use Komodo to analyse games, and perhaps an enhanced version of it that is not publicly available. Whatever they do, it is clear statistics are part of the system; and as we all know there are lies, damned lies and statistics.
I was curious about what data concerning me they would have as input into their system. Their site says I joined in July 2011 and since then have played 56,128 games on it with the following results: 27,344 wins, 26,282 losses and 2,481 draws. So, you might think they know my play well and they can predict my moves in most positions. Maybe so, but I doubt it. All but six of these games were blitz (I expect most were 3-minute games, but I haven’t checked this). The game I think is the one in question was a 45-minute game (with a 15 second increment per move). I play very differently in blitz and classical games, as reflected in my much lower OTB rapid-play grade than my classical OTB grade, and the four games they have with the classical time allowance will not allow anything approaching a reasonable prediction of the probability of me playing a particular move.
Perhaps in addition to the games I have played on their site they have added games I played in OTB matches that are in public databases. Looking at the FIDE site [9], the chess tempo database [10] and the Mega2020 database [11], there is a total of 73 classical games involving me. These would be easy for them to add to their database, but even with these can they predict what I would do accurately in any given position? I assume they need to do this as it is not enough to say I played a move that an engine would play, they need to say you played a move an engine would play and it is not a move you are likely to play unaided. But even this approach is flawed because it takes no account of a player improving their play. Most of the chess players I know are like me, wanting to improve their game and using books, DVDs and on-line resources to do so, and, though I don’t do this, many local players have coaches or trainers to help them.
What else might Chess.com be doing? An American friend says they might be able to detect ‘task-switching’. This occurs when you switch from looking at the browser running the Chess.com site to another programme. If you do this, they might assume you are switching to an engine. So, did I switch between sites? I can’t be sure, but I expect I did. If my email programme pinged during a game, I would probably have looked at it quickly; and possibly I would have had other applications open, such as Word or Edge (I played Chess.com via the Brave browser) and may well have looked at those. What I wouldn’t do is have Fritz, Chessbase or any other chess programme open during a chess game, just as I wouldn’t consult one of the many opening books I have to make sure I played the opening right, or one of my many ending books to make sure I didn’t err as the game came to a close. However, from what Rensch writes [8], I doubt they do this.
‘Back on that 2010 fateful day that I like to call our “Fair Play Crossroads,” we realized some – thing was wrong with our approach: We were still trying to “track” what cheaters were doing, answer the “whys and hows,” and guess as to the method, program, etc. that was being used to do it. We were looking too much in the area of trying to prevent and “catch the hand in the cookie jar,” rather than investing into the area that would later be undeniable whether the person was caught in the act or not: a detailed review and comparison of the game itself to what the engine said about that game.’
As I came to terms with being labelled a cheat, as distinct from having my Chess.com account suspended, which didn’t unduly concern me beyond the label, I went onto the web to see what others in my position had said and done. And what a revelation that was!
There is a massive web literature about Chess.com and their so-called fair play policies and practices. This one, for example, from August 2011 [12] claiming Chess.com had banned 1,300 players from its site in a two-month period. A thread on the ECF web site was started in November 2013 and is still going strong [13]. I won’t give any quotes from it, but I found it very helpful, not only to see the comments of people I know about Chess.com, but also to see the extent to which others have fought the allegations. It helped me realise that no matter how badly I feel about this, I need to bring the matter to a close for myself and move on. It was after reading about the experiences of others and considering my own situation that the following thought struck me. Since probabilities are central to their method and since low probability events occur, they will condemn innocent people such as myself, and they must know this and be willing to inflict the hurt on such people that unwarranted condemnations provoke. What does this tell us about their company ethics?
While I was writing the above and researching Chess.com they considered my appeal. I was not optimistic about this as I had failed to find any evidence that they reversed a decision once it was made, and after a few days I got an email saying:
‘We’ve reviewed your case (meaning your full “Fair Play Report”, the “cheating statistics” that ultimately led to our decision, as well as many of your games) in great detail. – – The unfortunate truth is that the data associated with your account is convincing.’
The email did not address any of the points I had made in my appeal and went on to ask a question that makes me think they hadn’t even read my appeal! After setting out several scenarios where they say players had inadvertently broken their fair play policies they asked: ‘Is there anything along those lines that you can think of which might have, no matter how innocent, tripped our fair play sensors?’ They should have known the answer to that question was no given my classical Chess.com games were played in a tournament. Their email also said they had reviewed many of my games, but how many, which ones, and by what methods I would like to know. I am doubtful that a human played through the moves of any of the games.
Finally, their rejection email reaffirmed what their earlier emails had said:
‘ — we would be willing to sanction a second chance account on our site where we’d expect our fair play rules to be observed. For us to be able to do so, though, we do need a clear admission to using outside assistance.’
This insulting request made me think of Joseph K in Franz Kafka’s The Trial. I don’t want to overstate this though because although Joseph K is a fictional character, he met a terrible end while I have just been banned from playing chess on a website.
[to be continued]
(c) Geoffrey Moore 2020. Originally published by “En Passant” (vol 25: July 2020).