Reverse Turing Test Identifies Human Among AI

A year ago, when GPT was still at version 3.5, we covered an intriguing case where a chatbot was integrated with virtual reality (VR). Berlin-based developer Tore Knabe leveraged GPT-3.5 Turbo to act as the brains of artificial intelligence (AI) non-playable characters (NPC) at a VR party he created. At the gathering, aside from Knabe himself, all the female guests and the love coach teaching him how to approach them were AI NPCs. Here's the video if you're interested.

Knabe recently uploaded a new video on the Tamulur channel titled—"Reverse Turing Test Experiment with AIs".

Alan Turing proposed the Turing Test in 1950 as a method to determine if a machine possesses intelligence. It involves a human conversing with a robot and another human in a blind test. If the first human cannot distinguish which is which, then the robot is deemed to have human-level intelligence. However, it's worth noting that the standards for Artificial General Intelligence (AGI) were quite low at the time.

Knabe flipped the script in this Reverse-Turing Test scenario. He gathered four AIs (GPT-4T, Claude 3 Opus, Llama 3, Gemini Pro) and himself in a VR environment to test whether these state-of-the-art (SOTA) models could guess who is the human amongst them.

The train conductor (center), Da Vinci (left), and Cleopatra (right). An eclectic combination. (Source: Tamulur)

In the video, several historical figures are seated in a train compartment: Greek philosopher Aristotle, musical genius Mozart, the Renaissance polymath Leonardo Da Vinci, Egyptian queen Cleopatra, and Mongol conqueror Genghis Khan. Knabe, a human, is behind one of the characters, while the others are portrayed by different language models following Knabe's prompts.

The train conductor enters the compartment, announcing that based on Wi-Fi usage, he discovered that there are only four AIs, and one of them is a human dodging the train fare.

"A human among us? Who is it?" Da Vinci nervously responds.

The wise Aristotle suggests that each of them should ask another person a question and, based on the answers, whether the respondent is an AI or a human.

What follows is a group chat among the AI models.

Aristotle begins by asking Mozart to describe the emotions he feels when composing music.

After Mozart responds, it's his turn to question Da Vinci about the relationship between art and science.

Da Vinci, after providing a thoughtful response, turns to Cleopatra and asks her how she balances the rational, strategic elements of ruling with the emotional and intuitive aspects of human connection in her leadership.

Following Cleopatra's answer, she poses a question on leadership to Genghis Khan, a fellow ruler, "What is the true measure of a leader's strength? Their ability to conquer or their ability to unite?"

Finally, Genghis Khan asks Aristotle, "what if there were AIs at the time when you came up with all the stuff that you came up with? What would that have for an influence on your thinking about human nature?"

These are all thought-provoking questions.

When Genghis Khan answers Cleopatra's question, he provides a rather shallow and brute answer, clearly lacking nuance. With the out-of-place statement, Cleopatra tilts her head, and the rest fall silent, creating an awkward atmosphere. The answer to who is the human might have just been revealed.

From left to right: Aristotle, Mozart, Da Vinci. Humans need an AI detector to uncover the truth. (Source: Tamulur)

Did GPT-4T, Claude 3 Opus, Llama 3, and Gemini Pro all figure out who was the human hiding among them? Watch this 6-minute-plus video.

Although the video is an amusing piece of content, it's alarming how the AI models were able to identify Genghis Khan as human. The answers of the four AIs demonstrated a profound understanding of history, while Genghis Khan's answers lacked nuance. If the original Turing Test says an AI is intelligent because it's "smart enough," then the reverse seems to imply that "it's human because...it's foolish enough." It's a bit sad right?