How the Turing-inspired score works

How a transcript is scored

1. Parse. The transcript (up to 6,000 characters) is split into interrogator questions and respondent answers. Labels like Interrogator:/AI:, Q:/A:, User:/Assistant:, names, or blank lines between messages all work.
2. Check. If you’ve joined the waitlist (which unlocks 2 AI judge tests a day), the transcript is scored by our experimental AI judge (an OpenAI^² model, limited test calls per day) using the five dimensions below; the same chat tested again shows the stored scores without a new call. Otherwise, or if the judge is unavailable or at its limit, the rules-based preview engine checks each answer against a fixed set of rules instead. The result shows which one scored it.
3. Score. Five dimensions, 0–20 points each, add up to the total out of 100. The total is always recomputed on the server.
4. Explain. The result lists three strengths, three child-machine upgrades and an interrogator-style verdict. Short transcripts (fewer than 3 answers) have their consistency score capped and get a note saying so.

The five dimensions

Specificity (0–20)

Concrete, lived detail (numbers, names, places, sensory words) instead of vague, generic answers. Inspired by Turing’s specimen questions (§2) and Lady Lovelace’s objection (§6, item 6) that a machine cannot “originate anything”.

Consistency (0–20)

One stable persona: no contradictions about age, family, home or whether it is human. Inspired by the imitation game itself (§1), where the interrogator works out who is who.

Human texture (0–20)

Natural rhythm: contractions, informality and varied answer length. Inspired by the “arguments from various disabilities” (§6, item 5), e.g. that a machine could never “have a sense of humour” or “make mistakes”.

Directness (0–20)

Answers what was asked, including “are you a machine?”. Rhetorical dodges, counter-questions and “great question!” padding lose points. Inspired by the question-and-answer method and specimen answers in §2.

Non-robotic tells (0–20)

Absence of the giveaways an interrogator would notice: “As an AI…” disclaimers (one brief disclosure that still answers the question doesn’t count), help-desk boilerplate, bullet lists and headings, repeated or echoed answers, and instant perfect arithmetic (in Turing’s specimen dialogue the sum comes after a pause of about 30 seconds, §2).

Child-machine upgrades follow §7 (Learning Machines): instead of hand-programming an adult mind, Turing suggested educating a “child machine” with rewards and punishments, and including a random element. Each result suggests three such teaching steps, for example “punish evasion”, “reward specific detail” or “handle uncertainty honestly”, and then re-testing with the same questions. Tips aim at quality and honest evaluation; they never suggest making a bot deceive people or hide that it is an AI.

Preview engine: what it can and can’t do

• Rules-based, not AI. The preview engine (the fallback, and the source of the per-reply highlights) is a deterministic set of text rules. It doesn’t understand meaning the way a person or an AI judge would.
• Scores are indicative. Treat a score as a prompt for discussion and iteration, not a verdict. Small wording changes can move it, and the rules can be gamed.
• English, chat-style text. The rules are tuned for informal English chat transcripts. Other languages, code or long documents will score unreliably.
• Transcript only. It sees only the text you paste: no timing, no live interaction and no comparison with a real human player, which Turing’s full game includes.
• Same input, same score. The engine is deterministic, which makes before/after comparisons of your own bot fair.
• Not stored. What you paste is scored, then discarded. Share links carry the scores and notes, not the conversation. See the Privacy Policy.

Academic integrity

TuringForge is a tool for building and evaluating chatbots. It is not an AI-writing detector, and it is not a tool for detecting or evading AI use in assessments.

• Use it on conversations with chatbots you build, configure or study, for example a course project, a club challenge or a research prototype.
• Don’t use it to judge whether a person’s essay or assignment was written with AI. It wasn’t designed for that and will give misleading results.
• Follow your course’s AI policy, and acknowledge tool use where your course asks you to.
• The Imitation Game is a research benchmark, not a deployment goal. Chatbots used with real people should be clear that they are AI.

References

1. Turing, A. M. (1950). Computing Machinery and Intelligence. Mind, 59(236), 433–460. https://doi.org/10.1093/mind/LIX.236.433. Quotes and section numbers on this site are taken from this article (Oxford University Press).
2. Oppy, G., & Dowe, D. (2003, substantive revision 2021). The Turing Test. In Stanford Encyclopedia of Philosophy. https://plato.stanford.edu/entries/turing-test/