In their latest newsletter, the WBIF announced that they have implemented anti-cheating software called 5POINT.
From the WBIF newsletter:
By tracking various play statistics and player style markers, 5POINT provides a meticulous match analysis, ensuring a comprehensive approach to detecting irregularities and maintaining a fair and competitive environment for all participants.
While this seems cool and a dream to have software to help keep the game fair online. My gut tells me that the AI is likely not where it needs to be in terms of accuracy (false positives). In the writing sphere, AI is far from being definitive in helping teachers and professors prove cheating. I can't imagine that backgammon AI is solid yet. Beyond generalized cheating/inhuman play, this software claims to be personalized to determine a player's specific patterns. It throws up a lot of red flags for me.
So, of course, they recommend a manual review. But how many TDs are trained to do a manual review to make a determination?
Manual Review: We understand that allegations of rule breaches are a very serious and sensitive matter and so our analysis provides evidence to both incriminate the guilty, and to exonerate the innocent.
Notice that one of the goals of the analysis is to provide incriminating evidence.
I don't know, maybe the software is better than I am assuming, but my skepticism is high on this one. Mainly, I am concerned about overreliance on suspect "data" to accuse the innocent. That could do way more harm than good.
Long-term, we definitely need something like this. Chess has more mature anti-cheating software.
Does anyone have more information on this? Is it ready for prime time?
What will this anti cheating software consist of? Most probably somehow (manually or by an API) you feed XG with the matches.
You’ll catch only the stupid. Making only small errors or checking only on crucial moves or using another AI will go undetected or you get tons of false positive.
But maybe only announcing it is already helpful…
I’m skeptical too. Is this better than a TD simply being aware of a player’s average PR and flagging them if they are playing well below it? I could imagine this becoming a useful tool, but it’d just to be an aid to human judgement.
Some position types are harder to play than others, and 5POINT claims to be able to identify those. I could believe it, but it wouldn’t be easy to do accurately, so I’d have to see the evidence.
I’d love to see a sample report, but there’s no way to upload a match to try it out. It says it compares matches to an individual’s history, but does that mean it creates its own database, or uses an existing one like BG Studio?
In the meantime, I think requiring zoom for online tournaments is a decent measure to keep people honest.
Maybe just the announcement and the threat of doing this is going to stop the ones who are doing it now and others who may be thinking about it?
I might be off base but what about players who put up a 12 PR because they had two containment games in their match and the very next match played a 4-5 PR because they had easier moves?
My initial reaction was mixed, but then I started wondering how exactly they are doing it for chess.
It seems that Chess.com has (proprietary) algorithms written by Professor Kenneth Regan(eidt: not sure this is true after digging more). Here is a Twitch clip with some info: link.
Lichess uses an algorithm that is public and licensed under AGPL-3.0. It is called irwin (named after Steve Irwin, the Crocodile Hunter ) and is hosted on GitHub here.
Professor Kenneth Regan of the Department of Computer Science and Engineering, University at Buffalo has done work publicly on detecting cheaters in chess. Besides work on detecting cheating in chess, he has some cool work on other topics such as ratings based on move quality vs end game results. Here are his works via Google Scholar as well as his personal UB CSE page.
Specifically on cheating in chess, here are two relevant papers from his university webpage:
These works are a bit outside of my area of expertise, but I might give them a go when I get some time.
In general, I am biased towards first principle approaches over black box AI models, but will accept the latter in cases where first principles fail and/or are significantly outperformed (e.g., backgammon bots ). In either case, openness of the models and documentation is important when it comes to important decisions that affect safety, etc., or player integrity in this case. 5Point seems to offer neither yet.
I agree here and think there will/should be a human in the loop for a long time regarding cheating decisions. If this becomes a thing, I would like to see some guidelines from the various backgammon federations.
Yes, just the deterrent effect might make it worthwhile.
And I’d assume that a match or two aberration won’t be enough to say anything conclusive about a player. But if they play an entire online tournament averaging 4-5PR when they usually play a 12, now that would be worth looking into.
Not sure how I feel about this, but there certainly are some cool techniques that one could use here. It would be interesting to see an opensource cheat detection system based on Irwin. I took a quick look at the code and it seems to be doable to reuse (part of it) for backgammon. If someone wants to have a look at this I am more than willing to share some human-human games (or just give access to all games on OG) and some of my development time.
I am curious on how they would be able to identify 'hard' positions, because it seems that identifying those is a hard problem (at least way more complicated than in chess).
You’ll catch only the stupid.
This is always the case for anti-cheating, but if evading the anti-cheat is harder than just learning to play well, it is actually good enough. For backgammon the advantage of cheating is not as big as the advantage you get in chess, so we have to keep that in mind when looking at these systems from an adversarial point of view.
) and is hosted on GitHub 
). In either case, openness of the models and documentation is important when it comes to important decisions that affect safety, etc., or player integrity in this case. 5Point seems to offer neither yet.