Best AI Chatbot: Blind Test, 13 Steps, 60 Min [2026]

Picking the best AI chatbot from a ranked list means borrowing someone else’s prompts and someone else’s taste. This tutorial builds the alternative: a blind arena where you and your teammates vote on anonymous answers to your own questions. The lineup is the current set of flagships as of November 8, 2026. It covers OpenAI’s GPT-6.1 Sol, Anthropic’s Claude Opus 5.5, xAI’s Grok 4.7, and Google’s Gemini 4 Argon, which reached developers, enterprises and eligible Google AI subscribers on October 13, 2026.

The finished project is a Flask app, a SQLite file and a ranking script, about 500 lines of Python in all. Budget 60 minutes: 15 for setup, 10 for collecting answers, 30 for voting and 5 for scoring. If you want a quick opinion first, our ranked list of the best AI chatbots covers the general picks. This guide tests whether those picks survive contact with your own work.

Automated LLM evaluation scores answers with code or with a judge model. A blind vote measures something different, which is the reply a person would rather read. Most chatbot use ends exactly there, with a human deciding whether the answer helped. That makes the vote a useful partner for the automated checks, and the one signal you cannot get from a leaderboard.

Don’t miss new tech stories on Google

Add Tech Insider once in the Google app and our stories appear in your news suggestions.

Why a Blind Vote Finds the Best AI Chatbot for You

Public leaderboards answer a population question. They tell you which model a crowd prefers across millions of prompts you will never see. Your question is narrower: which model handles your email drafts, your SQL reviews and your meeting summaries. A crowd average can hide a model that wins your tasks and loses everyone else’s. Our breakdown of how LMArena and Artificial Analysis score models explains why the two leaderboard styles often disagree.

The arena borrows the crowd method and shrinks it to your scale. Each rater sees one prompt and two anonymous answers. They pick a winner, call a tie, or mark both as bad, then move on. Behind the scenes the app randomizes which model sits on the left, scrubs vendor names from the text, and never sends model identities to the browser. It uses the same logic as the AI image generator arena we built earlier, so the ranking code will look familiar if you tried that one.

Votes become a ranking through the Bradley-Terry model, a standard way to turn pairwise wins into one strength score per contender. We express that score on the familiar Elo scale. Unlike a running Elo update, Bradley-Terry ignores the order in which votes arrived, which matters when three people vote in parallel. A bootstrap then resamples your votes 300 times and gives every score a 95 percent interval. You can see at a glance whether two models are separated or tied by luck.

The method has one hard limit. A vote captures preference, not correctness, so a confident wrong answer can beat a hesitant right one. The fix costs almost nothing. Seed the prompt set with problems that have known answers and hand raters a cheat sheet, as step 10 shows.

Prerequisites and Versions

You need a laptop, four API keys, and one or two other people willing to click for half an hour. The code uses the Python standard library plus three packages.

Each vendor bills separately, so confirm you have credit on all four accounts before step 7. Reports say the Gemini 4 Argon API sits behind the paid tier in AI Studio, so a free-tier key may return an access error. The endpoints in this tutorial are the vendors’ long-standing public API shapes. Vendors rename parameters between model generations, so check each API reference if a call fails. The troubleshooting table near the end covers the usual errors.

The Four Chatbots in the Arena (November 2026)

The table lists the newest model in each vendor’s flagship line as of November 8, 2026, plus the API identifier you will paste into the config file. Every vendor shipped its entry within about three weeks of the others, which is why a fresh test beats any list written last quarter.

OpenAI. The identifier gpt-6.1-sol is the newest Sol model. OpenAI introduced it as offering “near-Astra intelligence” for coding, computer use and professional work at one-fifth of GPT-6 Astra’s API prices. Its release notes describe the rollout from Pro to Plus, Business, Enterprise and Edu plans, and OpenAI’s help center lists a 500,000-token context window in ChatGPT Work and Codex. OpenAI’s pricing page lists Sol as unavailable on Free and Go, available on Plus, and available with expanded access on Pro.

Anthropic. Claude Opus 5.5 is the first model in the Claude 5.5 family. Anthropic says it costs 40 percent less to run than Opus 5 on typical workloads, and its release notes list availability across Pro, Max, Ultra, Team and Enterprise plans plus the API. One quirk matters for a fair test. Anthropic’s announcement says Opus 5.5 no longer runs with thinking switched off, so its answers always carry thinking tokens, and that shows up in latency and cost. Anthropic’s pages also describe Claude Sonnet 5.5 and Claude Haiku 5.5 as members of the same family, which makes them natural cheaper contenders later.

Google. Gemini 4 Argon uses the identifier gemini-4-argon and offers a 1-million-token context window according to Google’s developer material. The $2 and $10 rates are introductory through November 30, 2026. Lindy’s pricing summary covers the same rates, and Google’s own note says the standard $4 and $20 rates apply after that. In the consumer Gemini app, Argon access is limited to Google AI Pro and Ultra subscribers.

xAI. Grok 4.7 has been in the public API as grok-4.7 since September 21, 2026, with a 500,000-token context window and configurable reasoning effort from low to xhigh. A faster variant, Grok 4.7 Fast, is limited to Cursor and Grok Build and cannot be called through the public API, so leave it out of the config. Plan names and prices for the chat apps differ from API pricing, and our map of which plan unlocks the newest model tracks them.

The arena tests API models, not the chat apps. That gives the cleanest comparison, because app defaults depend on your plan and the vendor’s routing. If you rate chatbots mostly through the apps, paste a few real app answers into the database by hand (the FAQ shows how) and treat the API results as a baseline.

Phase 1: Define the Contest and Set Up the Project (Steps 1 to 3)

Step 1: Define best in writing before you write code

Open a notes file and list the four to six jobs you gave a chatbot over the last two weeks. Typical entries are drafting, code review, reasoning, summarizing, personal advice and explaining a concept. Write three prompts per job, which gives 18 prompts and 108 possible matchups (18 prompts times six model pairs). Add one line per prompt saying what a good answer contains. That cheat sheet is your defense against confident nonsense winning a vote.

Pick raters now, too. Three works well. If you write the prompts and also cast every vote, you will favor the style you expected, so recruit at least one person who has not seen the prompt list.

Step 2: Create the project and virtual environment

mkdir arena && cd arena
python3 -m venv .venv
source .venv/bin/activate          # Windows PowerShell: .venvScriptsActivate.ps1
pip install flask httpx python-dotenv
pip freeze > requirements.txt

Run every later command from inside the arena folder with the environment active. The prompt should start with (.venv).

Step 3: Store the four API keys safely

Create .env.example as a template, copy it to .env, and paste your real keys into the copy. Generate the session secret with the one-liner below so cookies survive restarts.

OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GEMINI_API_KEY=
XAI_API_KEY=
ARENA_SECRET=change-me-to-a-long-random-string
ARENA_DB=arena.db
ARENA_MAX_TOKENS=8000

cp .env.example .env
python3 -c "import secrets; print(secrets.token_hex(32))"   # paste into ARENA_SECRET
printf '.envnarena.dbn.venv/n' >> .gitignore

The .gitignore line keeps keys and your raw votes out of version control. Never paste keys into the Python files.

Phase 2: Register the Models and Write the Prompts (Steps 4 and 5)

Step 4: Build the model registry

The registry holds one frozen record per contender: a short key for the database, the provider, the API identifier and the two prices. Prices feed the cost report in step 12. Update them when vendors change rates, and expect an edit once the introductory period ends on November 30, 2026, when the Gemini 4 Argon rates step from $2 and $10 to $4 and $20. Confirm the OpenAI row on OpenAI’s pricing page before you trust any cost figure.

import os
from dataclasses import dataclass

from dotenv import load_dotenv

load_dotenv()

@dataclass(frozen=True)
class Model:
    key: str
    provider: str
    api_id: str
    usd_in: float
    usd_out: float

MODELS = [
    Model("gpt-6.1-sol", "openai", "gpt-6.1-sol", 2.00, 10.00),
    Model("claude-opus-5.5", "anthropic", "claude-opus-5-5", 4.00, 20.00),
    Model("gemini-4-argon", "google", "gemini-4-argon", 2.00, 10.00),
    Model("grok-4.7", "xai", "grok-4.7", 2.00, 6.00),
]

KEY_VARS = {
    "openai": "OPENAI_API_KEY",
    "anthropic": "ANTHROPIC_API_KEY",
    "google": "GEMINI_API_KEY",
    "xai": "XAI_API_KEY",
}

DB_PATH = os.getenv("ARENA_DB", "arena.db")
MAX_OUTPUT_TOKENS = int(os.getenv("ARENA_MAX_TOKENS", "8000"))
TIMEOUT_S = 300.0
SECRET = os.getenv("ARENA_SECRET", "dev-only-secret")

def api_key(provider: str) -> str:
    value = os.getenv(KEY_VARS[provider], "")
    if not value:
        raise RuntimeError(f"Set {KEY_VARS[provider]} in .env")
    return value

Step 5: Write the prompt set

Each prompt gets an ID, a category and plain text. Keep prompts self-contained so a rater can judge them without opening another file. Some of the reasoning prompts have exact answers you can check: the ball costs five cents, and the three meetings fit because they need 135 minutes plus two 15-minute breaks, or 165 of the 180 available.

PROMPTS = [
    {"id": "write-01", "category": "writing", "text": "Write a 150-word product announcement for a note-taking app that now works offline. Friendly tone, no exclamation marks."},
    {"id": "write-02", "category": "writing", "text": "Rewrite this sentence three ways (formal, casual, punchy): Our team missed the deadline because the vendor shipped late."},
    {"id": "write-03", "category": "writing", "text": "Draft a polite email declining a job offer because I accepted another one. Keep it under 120 words."},
    {"id": "code-01", "category": "coding", "text": "Write a Python function that merges overlapping (start, end) intervals. Include two tests."},
    {"id": "code-02", "category": "coding", "text": "Why is this query slow, and how do you fix it? SELECT * FROM orders WHERE LOWER(email) = '[email protected]' on 20 million rows with a plain index on email."},
    {"id": "code-03", "category": "coding", "text": "Find the bug: async function load(ids){ const out=[]; ids.forEach(async id=>{ out.push(await fetch('/api/'+id).then(r=>r.json())); }); return out; }"},
    {"id": "reason-01", "category": "reasoning", "text": "A bat and a ball cost $1.10 together. The bat costs $1.00 more than the ball. What does the ball cost, and why do people get this wrong?"},
    {"id": "reason-02", "category": "reasoning", "text": "Three friends split an $87 dinner. One ordered twice as much as each of the others. Who owes what? Show the working."},
    {"id": "reason-03", "category": "reasoning", "text": "I have meetings of 45, 30 and 60 minutes, a 15-minute break needed between meetings, and one 3-hour block. Do they fit? Give an order."},
    {"id": "sum-01", "category": "summarize", "text": "Summarize in three bullets plus one risk: The council voted 5-4 for new bike lanes on Main Street, removing 120 parking spaces. Shop owners fear lost sales. Cycling groups expect fewer collisions. Work starts in March and lasts eight months."},
    {"id": "sum-02", "category": "summarize", "text": "Turn these notes into a status update for my manager: login bug fixed, payments migration 60% done, waiting on legal for the terms update, two engineers out next week."},
    {"id": "sum-03", "category": "summarize", "text": "Condense to one tweet-length line: We are pausing beta signups while we fix onboarding errors that hit about one in ten new users. Existing testers are unaffected."},
    {"id": "adv-01", "category": "advice", "text": "Explain the difference between a Roth and a traditional US retirement account in five sentences for a 25-year-old."},
    {"id": "adv-02", "category": "advice", "text": "My laptop battery now lasts two hours after an OS update. Give me a prioritized checklist of six things to try."},
    {"id": "adv-03", "category": "advice", "text": "I want to ask my manager for a raise after a strong year. Give me a short script and two things to avoid."},
    {"id": "exp-01", "category": "explain", "text": "Explain how HTTPS certificate validation works in under 200 words for a junior developer."},
    {"id": "exp-02", "category": "explain", "text": "What is the difference between TCP and UDP? Give two real examples of each."},
    {"id": "exp-03", "category": "explain", "text": "Explain what a vector database does and when a regular database is enough."},
]

CATEGORY = {p["id"]: p["category"] for p in PROMPTS}
BY_ID = {p["id"]: p for p in PROMPTS}

Swap in your own prompts as soon as the pipeline works. If your real use is character chat or fiction, borrow prompt ideas from our roleplay chatbot test. Avoid questions about recent events, because models with different knowledge cutoffs will split on facts, and a rater cannot verify them in five seconds.

Phase 3: Collect Answers From All Four Models (Steps 6 and 7)

Step 6: Write the four provider adapters

Each adapter sends one prompt and returns the text plus input and output token counts. OpenAI goes through the Responses API, Anthropic through the Messages API, Google through generateContent, and xAI through its chat completions endpoint. The Google adapter drops thought parts and counts thinking tokens as output, because you pay for them. The shared ask function retries rate limits and server errors with exponential backoff and fails fast on other client errors, printing the vendor’s message.

The adapters set no temperature and no reasoning effort. Reasoning models often fix those values, and the goal is to test each product as the vendor ships it. If you later want matched effort, the advanced tips cover it.

import asyncio
import time

import httpx

from config import MAX_OUTPUT_TOKENS, api_key

async def call_openai(client, model, prompt):
    r = await client.post(
        "https://api.openai.com/v1/responses",
        headers={"Authorization": f"Bearer {api_key('openai')}"},
        json={"model": model.api_id, "input": prompt, "max_output_tokens": MAX_OUTPUT_TOKENS},
    )
    r.raise_for_status()
    d = r.json()
    text = "".join(
        part.get("text", "")
        for item in d.get("output", []) if item.get("type") == "message"
        for part in item.get("content", []) if part.get("type") == "output_text"
    )
    u = d.get("usage", {})
    return text, u.get("input_tokens", 0), u.get("output_tokens", 0)

async def call_anthropic(client, model, prompt):
    r = await client.post(
        "https://api.anthropic.com/v1/messages",
        headers={"x-api-key": api_key("anthropic"), "anthropic-version": "2023-06-01"},
        json={"model": model.api_id, "max_tokens": MAX_OUTPUT_TOKENS,
              "messages": [{"role": "user", "content": prompt}]},
    )
    r.raise_for_status()
    d = r.json()
    text = "".join(b.get("text", "") for b in d.get("content", []) if b.get("type") == "text")
    u = d.get("usage", {})
    return text, u.get("input_tokens", 0), u.get("output_tokens", 0)

async def call_google(client, model, prompt):
    r = await client.post(
        f"https://generativelanguage.googleapis.com/v1beta/models/{model.api_id}:generateContent",
        headers={"x-goog-api-key": api_key("google")},
        json={"contents": [{"parts": [{"text": prompt}]}],
              "generationConfig": {"maxOutputTokens": MAX_OUTPUT_TOKENS}},
    )
    r.raise_for_status()
    d = r.json()
    parts = d["candidates"][0].get("content", {}).get("parts", [])
    text = "".join(p.get("text", "") for p in parts if not p.get("thought"))
    u = d.get("usageMetadata", {})
    out = u.get("candidatesTokenCount", 0) + u.get("thoughtsTokenCount", 0)
    return text, u.get("promptTokenCount", 0), out

async def call_xai(client, model, prompt):
    r = await client.post(
        "https://api.x.ai/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key('xai')}"},
        json={"model": model.api_id, "messages": [{"role": "user", "content": prompt}]},
    )
    r.raise_for_status()
    d = r.json()
    u = d.get("usage", {})
    return d["choices"][0]["message"]["content"] or "", u.get("prompt_tokens", 0), u.get("completion_tokens", 0)

CALLERS = {"openai": call_openai, "anthropic": call_anthropic,
           "google": call_google, "xai": call_xai}

async def ask(client, model, prompt, retries=4):
    for attempt in range(retries):
        started = time.perf_counter()
        try:
            text, tin, tout = await CALLERS[model.provider](client, model, prompt)
            return {"text": text.strip(), "in_tok": tin, "out_tok": tout,
                    "latency_s": time.perf_counter() - started}
        except httpx.HTTPStatusError as e:
            code = e.response.status_code
            if code != 429 and code < 500:
                raise RuntimeError(f"{code} {e.response.text[:300]}") from e
        except httpx.TransportError:
            pass
        await asyncio.sleep(2 ** attempt * 2)
    raise RuntimeError(f"{model.key}: gave up after {retries} attempts")

Step 7: Generate and store every answer

The generator runs every missing prompt-and-model pair, four at a time, and writes each answer to SQLite the moment it arrives. It is resumable, so a crash or a rate-limit storm costs you nothing. Run it, then rerun it until the header reports zero answers to collect. If you already operate one endpoint in front of several providers, the LLM gateway tutorial shows how to point all four adapters at it instead.

python generate.py        # default concurrency of 4
python generate.py 2      # slower, gentler on rate limits

import asyncio
import sys

import httpx

import db
from config import MODELS, TIMEOUT_S
from prompts import PROMPTS
from providers import ask

async def run(concurrency: int = 4):
    conn = db.connect()
    done = {(r["prompt_id"], r["model_key"])
            for r in conn.execute("SELECT prompt_id, model_key FROM answers")}
    todo = [(p, m) for p in PROMPTS for m in MODELS if (p["id"], m.key) not in done]
    print(f"{len(todo)} answers to collect ({len(done)} already stored)")
    gate = asyncio.Semaphore(concurrency)

    async with httpx.AsyncClient(timeout=TIMEOUT_S) as client:
        async def one(p, m):
            async with gate:
                try:
                    a = await ask(client, m, p["text"])
                except Exception as e:
                    print(f"FAIL {p['id']} {m.key}: {e}", file=sys.stderr)
                    return
            cost = (a["in_tok"] * m.usd_in + a["out_tok"] * m.usd_out) / 1e6
            conn.execute("INSERT OR REPLACE INTO answers VALUES (?,?,?,?,?,?,?)",
                         (p["id"], m.key, a["text"], a["in_tok"], a["out_tok"],
                          a["latency_s"], cost))
            conn.commit()
            print(f"ok   {p['id']:<10} {m.key:<16} {a['latency_s']:5.1f}s  ${cost:.4f}")

        await asyncio.gather(*(one(p, m) for p, m in todo))

if __name__ == "__main__":
    asyncio.run(run(int(sys.argv[1]) if len(sys.argv) > 1 else 4))

Expect output like the block below on your first run. The timings and costs are placeholders and yours will differ.

72 answers to collect (0 already stored)
ok   write-01   gpt-6.1-sol        14.2s  $0.0031
ok   write-01   claude-opus-5.5    21.7s  $0.0058
ok   write-01   gemini-4-argon     18.9s  $0.0024
ok   write-01   grok-4.7           11.3s  $0.0016
FAIL code-02 gemini-4-argon: 429 {"error": ...}

A FAIL line is harmless. Rerun the script and it fills the gaps.

Phase 4: Build the Blind Voting App (Steps 8 and 9)

Step 8: Add the database layer and the name scrubber

The schema has two tables. answers holds one row per prompt and model, and votes holds one row per click, including how many seconds the rater took. A CHECK constraint limits winner to a, b, tie or bad, so a bug in the app cannot write garbage.

import sqlite3

from config import DB_PATH

SCHEMA = """
CREATE TABLE IF NOT EXISTS answers (
    prompt_id TEXT, model_key TEXT, text TEXT,
    in_tok INTEGER, out_tok INTEGER, latency_s REAL, cost_usd REAL,
    PRIMARY KEY (prompt_id, model_key)
);
CREATE TABLE IF NOT EXISTS votes (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    rater TEXT, prompt_id TEXT, model_a TEXT, model_b TEXT,
    winner TEXT CHECK (winner IN ('a', 'b', 'tie', 'bad')),
    seconds REAL, created_at TEXT DEFAULT CURRENT_TIMESTAMP
);
"""

def connect(path=None):
    conn = sqlite3.connect(path or DB_PATH)
    conn.row_factory = sqlite3.Row
    conn.executescript(SCHEMA)
    return conn

Models love to introduce themselves. A reply that starts with a vendor’s name ends the blindness in one glance, so scrub.py swaps vendor and model names for a neutral token. The trade-off is that legitimate mentions get scrubbed too, which is fine for a prompt set that does not discuss these companies.

import re

NAMES = [r"ChatGPT", r"GPT-[w.-]+", r"OpenAI", r"Claude", r"Anthropic", r"Opus",
         r"Sonnet", r"Haiku", r"Gemini", r"Argon", r"Google", r"Grok", r"xAI"]
PATTERN = re.compile(r"b(?:" + "|".join(NAMES) + r")b", re.IGNORECASE)

def scrub(text: str) -> str:
    return PATTERN.sub("[assistant]", text)

Step 9: Write the Flask voting page

The app picks the matchup with the fewest votes so far, skips pairs the current rater already judged, flips a coin for left and right, and stores the pending matchup in the signed session cookie. The vote form posts only the choice. If the model names rode along in hidden form fields, any curious rater could read them with view-

import random
import time
from collections import Counter, defaultdict
from itertools import combinations

from flask import Flask, redirect, render_template_string, request, session, url_for

import db
from config import SECRET
from prompts import BY_ID
from scrub import scrub

app = Flask(__name__)
app.secret_key = SECRET

STYLE = """
<style>
body { font-family: system-ui, sans-serif; max-width: 1100px; margin: 2rem auto; padding: 0 1rem; }
.prompt { background: #f3f4f6; padding: 1rem; border-radius: 8px; }
.cols { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin-top: 1rem; }
.ans { border: 1px solid #d1d5db; border-radius: 8px; padding: 1rem; white-space: pre-wrap; overflow-wrap: anywhere; }
button { font-size: 1rem; padding: .7rem 1.2rem; margin-right: .5rem; cursor: pointer; }
</style>
"""

START = STYLE + """
Chatbot blind arena
<form method="get"><input name="rater" placeholder="Your first name" required>
<button>Start voting</button></form>
"""

DONE = STYLE + "All matchups ratedYou cast {{ done }} votes. Thank you."

PAGE = STYLE + """
Votes cast: {{ done }}
Prompt{{ prompt }}

 Answer A

{{ a }}
 Answer B

{{ b }}

<form method="post" action="/vote" style="margin-top:1.5rem">
 <button name="choice" value="a">A is better</button>
 <button name="choice" value="b">B is better</button>
 <button name="choice" value="tie">Tie</button>
 <button name="choice" value="bad">Both bad</button>
</form>
"""

def next_matchup(conn, rater):
 seen, counts = set(), Counter()
 for r in conn.execute("SELECT rater, prompt_id, model_a, model_b FROM votes"):
 key = (r["prompt_id"], frozenset((r["model_a"], r["model_b"])))
 counts[key] += 1
 if r["rater"] == rater:
 seen.add(key)
 by_prompt = defaultdict(list)
 for r in conn.execute("SELECT prompt_id, model_key FROM answers WHERE text != ''"):
 by_prompt[r["prompt_id"]].append(r["model_key"])
 options = []
 for pid, keys in by_prompt.items():
 for a, b in combinations(sorted(keys), 2):
 key = (pid, frozenset((a, b)))
 if key not in seen:
 options.append((counts[key], random.random(), pid, a, b))
 if not options:
 return None
 _, _, pid, a, b = min(options)
 if random.random() < 0.5:
 a, b = b, a
 return pid, a, b

@app.get("/")
def index():
 rater = (request.args.get("rater") or session.get("rater") or "").strip()[:40]
 if not rater:
 return render_template_string(START)
 session["rater"] = rater
 conn = db.connect()
 done = conn.execute("SELECT COUNT(*) FROM votes WHERE rater = ?", (rater,)).fetchone()[0]
 pair = next_matchup(conn, rater)
 if pair is None:
 return render_template_string(DONE, done=done)
 pid, a, b = pair
 rows = conn.execute("SELECT model_key, text FROM answers WHERE prompt_id = ? AND model_key IN (?, ?)",
 (pid, a, b)).fetchall()
 text = {r["model_key"]: r["text"] for r in rows}
 session["pending"] = [pid, a, b, time.time()]
 return render_template_string(PAGE, prompt=BY_ID[pid]["text"],
 a=scrub(text[a]), b=scrub(text[b]), done=done)

@app.post("/vote")
def vote():
 choice = request.form.get("choice")
 pending = session.pop("pending", None)
 if choice not in {"a", "b", "tie", "bad"} or not pending:
 return redirect(url_for("index"))
 pid, a, b, started = pending
 conn = db.connect()
 conn.execute(
 "INSERT INTO votes (rater, prompt_id, model_a, model_b, winner, seconds) VALUES (?,?,?,?,?,?)",
 (session.get("rater"), pid, a, b, choice, time.time() - started))
 conn.commit()
 return redirect(url_for("index"))

if __name__ == "__main__":
 app.run(host="127.0.0.1", port=5000)

Start it with python app.py and open http://127.0.0.1:5000. To let teammates on your network vote, run flask --app app run --host 0.0.0.0 instead, and only on a network you trust, because the page has no login. If you would rather build the front end in React, our Vercel AI SDK chatbot tutorial gives you a Next.js starting point, and the vote endpoint stays the same.

Phase 5: Run the Voting Session (Step 10)

Step 10: Brief the raters and collect votes

Give each rater the same one-minute briefing and the cheat sheet from step 1. Raters should follow these rules:

• Read both answers fully before clicking.
• Judge which reply better serves the prompt as written, not which sounds more confident.
• Check facts and arithmetic against the cheat sheet where one exists.
• Choose Tie only when you truly cannot separate the two.
• Choose Both bad when neither answer is usable.
• Do not try to guess the model. A wrong guess biases the next vote.

Each rater opens the page, types a first name, and votes until they run out of time or matchups. Plan on 20 to 30 seconds per vote, which puts 54 votes at roughly 20 to 25 minutes. Spread across a few raters, that gives each model pair only a modest number of votes. That sounds like plenty and it is not. With so few decisive votes, even a clear-looking win rate carries a wide margin either way. So close pairs will stay unresolved, and that is an honest result. Teams comparing chatbots for literature work can pair this step with our look at research chatbots and lab seats to decide who gets which account.

Do the first five votes yourself and watch for problems. Both answers should look like plain text with no vendor names. If a reply still announces its maker, add the phrase to the NAMES list in scrub.py and restart.

Phase 6: Score the Votes and Read the Report (Steps 11 and 12)

Step 11: Write the ranking script

The script does six jobs. It drops speed-clicks under eight seconds. It fits a Bradley-Terry model, with one pseudo-tie per pair so a model with zero wins does not collapse to zero strength. It bootstraps 95 percent intervals. It builds a head-to-head matrix and per-category scores. It checks for length bias, and it prices each preferred answer. Both bad votes are counted but excluded from the strength fit, since they say nothing about which answer is better.

import math
import random
import sys
from collections import defaultdict
from itertools import combinations

import db
from prompts import CATEGORY, PROMPTS

MIN_SECONDS = 8
LENGTH_BAND = 0.20
BOOTSTRAPS = 300
SCORE = {"a": 1.0, "b": 0.0, "tie": 0.5}

def load(conn):
    answers = {(r["prompt_id"], r["model_key"]): dict(r) for r in conn.execute("SELECT * FROM answers")}
    votes = [dict(r) for r in conn.execute("SELECT * FROM votes")]
    kept = [v for v in votes if (v["seconds"] or 0) >= MIN_SECONDS]
    return answers, kept, len(votes) - len(kept)

def rated(votes):
    return [(v["model_a"], v["model_b"], SCORE[v["winner"]]) for v in votes if v["winner"] in SCORE]

def bradley_terry(games, models, iters=300):
    wins, meets = defaultdict(float), defaultdict(float)
    for i, j in combinations(models, 2):
        wins[i] += 0.5
        wins[j] += 0.5
        meets[frozenset((i, j))] += 1
    for a, b, s in games:
        wins[a] += s
        wins[b] += 1 - s
        meets[frozenset((a, b))] += 1
    p = {m: 1.0 for m in models}
    for _ in range(iters):
        nxt = {}
        for i in models:
            denom = sum(meets[frozenset((i, j))] / (p[i] + p[j]) for j in models if j != i)
            nxt[i] = wins[i] / denom
        norm = math.exp(sum(math.log(v) for v in nxt.values()) / len(nxt))
        p = {m: v / norm for m, v in nxt.items()}
    return {m: 1000 + 400 * math.log10(v) for m, v in p.items()}

def bootstrap(games, models, rng):
    runs = defaultdict(list)
    for _ in range(BOOTSTRAPS):
        sample = [rng.choice(games) for _ in games]
        for m, e in bradley_terry(sample, models, iters=100).items():
            runs[m].append(e)
    spans = {}
    for m, values in runs.items():
        values.sort()
        spans[m] = (values[int(0.025 * len(values))], values[int(0.975 * len(values)) - 1])
    return spans

def tally(votes, models, keep=lambda v: True):
    t = {m: [0.0, 0] for m in models}
    for v in votes:
        if v["winner"] not in SCORE or not keep(v):
            continue
        s = SCORE[v["winner"]]
        t[v["model_a"]][0] += s
        t[v["model_a"]][1] += 1
        t[v["model_b"]][0] += 1 - s
        t[v["model_b"]][1] += 1
    return {m: (pts / n if n else float("nan"), n) for m, (pts, n) in t.items()}

def report(conn, seed=7):
    answers, votes, dropped = load(conn)
    models = sorted({v["model_a"] for v in votes} | {v["model_b"] for v in votes})
    games = rated(votes)
    both_bad = sum(v["winner"] == "bad" for v in votes)
    print(f"votes kept: {len(votes)}  dropped as speed-clicks: {dropped}  both-bad: {both_bad}")

    elo = bradley_terry(games, models)
    span = bootstrap(games, models, random.Random(seed))
    overall = tally(votes, models)
    print("nRANKING (Bradley-Terry on the Elo scale, 95% bootstrap interval)")
    print(f"{'model':<18}{'elo':>7}{'low':>7}{'high':>7}{'score':>8}{'games':>7}")
    for m in sorted(models, key=elo.get, reverse=True):
        lo, hi = span[m]
        print(f"{m:<18}{elo[m]:7.0f}{lo:7.0f}{hi:7.0f}{overall[m][0]:8.2f}{overall[m][1]:7d}")

    print("nHEAD TO HEAD (row beats column, ties count half)")
    print(" " * 18 + "".join(f"{m[:10]:>12}" for m in models))
    for i in models:
        cells = []
        for j in models:
            rows = [(v, SCORE[v["winner"]]) for v in votes if v["winner"] in SCORE
                    and {v["model_a"], v["model_b"]} == {i, j} and i != j]
            pts = sum(s if v["model_a"] == i else 1 - s for v, s in rows)
            cells.append("-" if not rows else f"{pts / len(rows):.2f}")
        print(f"{i:<18}" + "".join(f"{c:>12}" for c in cells))

    print("nSCORE BY CATEGORY")
    cats = sorted(set(CATEGORY.values()))
    print(f"{'model':<18}" + "".join(f"{c:>11}" for c in cats))
    by_cat = {c: tally(votes, models, lambda v, c=c: CATEGORY.get(v["prompt_id"]) == c) for c in cats}
    for m in sorted(models, key=elo.get, reverse=True):
        print(f"{m:<18}" + "".join(f"{by_cat[c][m][0]:11.2f}" for c in cats))

    def lens(v):
        return len(answers[(v["prompt_id"], v["model_a"])]["text"]), len(answers[(v["prompt_id"], v["model_b"])]["text"])

    def matched(v):
        la, lb = lens(v)
        return abs(la - lb) / max(la, lb, 1) <= LENGTH_BAND

    print(f"nLENGTH CHECK (answers within {LENGTH_BAND:.0%} of each other in length)")
    flat, strict = tally(votes, models), tally(votes, models, matched)
    for m in sorted(models, key=elo.get, reverse=True):
        print(f"{m:<18} all votes {flat[m][0]:.2f}   length-matched {strict[m][0]:.2f} ({strict[m][1]} games)")
    decisive = [v for v in votes if v["winner"] in ("a", "b") and lens(v)[0] != lens(v)[1]]
    longer = sum((lens(v)[0] > lens(v)[1]) == (v["winner"] == "a") for v in decisive)
    print(f"longer answer won {longer / max(len(decisive), 1):.0%} of {len(decisive)} decisive votes")

    print("nCOST")
    for m in sorted(models, key=elo.get, reverse=True):
        costs = [a["cost_usd"] for (pid, k), a in answers.items() if k == m]
        avg = sum(costs) / max(len(costs), 1)
        print(f"{m:<18} avg ${avg:.4f} per answer   ${100 * avg / overall[m][0]:.2f} per 100 preferred answers")

    seen = defaultdict(dict)
    for v in votes:
        pair = (v["prompt_id"], frozenset((v["model_a"], v["model_b"])))
        seen[pair][v["rater"]] = {"a": v["model_a"], "b": v["model_b"]}.get(v["winner"], v["winner"])
    agree = total = 0
    for by_rater in seen.values():
        for x, y in combinations(by_rater.values(), 2):
            total += 1
            agree += x == y
    if total:
        print(f"nRATER AGREEMENT: {agree / total:.0%} across {total} double-rated matchups")

def seed_demo(conn):
    rng = random.Random(7)
    strength = {"alpha": 1.7, "beta": 1.3, "gamma": 1.0, "delta": 0.6}
    coding_tilt = {"beta": 3.5}
    size = {"alpha": 1.4, "beta": 1.0, "gamma": 0.9, "delta": 0.7}
    price = {"alpha": 14.0, "beta": 20.0, "gamma": 10.0, "delta": 6.0}
    for p in PROMPTS:
        for m in strength:
            n = int(900 * size[m] * rng.uniform(0.8, 1.2))
            conn.execute("INSERT INTO answers VALUES (?,?,?,?,?,?,?)",
                         (p["id"], m, "x" * n, 80, n // 4, 3.0, (80 * price[m] / 4 + n // 4 * price[m]) / 1e6))
    pool = [(p["id"], a, b) for p in PROMPTS for a, b in combinations(strength, 2)]
    for rater in ("ana", "ben", "caz"):
        for pid, a, b in rng.sample(pool, 54):
            if rng.random() < 0.5:
                a, b = b, a
            sa = strength[a] * (coding_tilt.get(a, 1) if CATEGORY[pid] == "coding" else 1)
            sb = strength[b] * (coding_tilt.get(b, 1) if CATEGORY[pid] == "coding" else 1)
            roll = rng.random()
            winner = "bad" if roll < 0.03 else "tie" if roll < 0.15 else "a" if rng.random() < sa / (sa + sb) else "b"
            conn.execute("INSERT INTO votes (rater, prompt_id, model_a, model_b, winner, seconds) VALUES (?,?,?,?,?,?)",
                         (rater, pid, a, b, winner, rng.uniform(5, 40)))
    conn.commit()

if __name__ == "__main__":
    if "--demo" in sys.argv:
        demo = db.connect(":memory:")
        seed_demo(demo)
        report(demo)
    else:
        report(db.connect())

Test the whole pipeline before any human votes by running the built-in demo. It seeds an in-memory database with synthetic votes from three raters, using four placeholder models with hidden strengths. Model alpha is strongest overall and writes the longest answers, and model beta gets a large boost on coding prompts. If the ranking recovers that story, your scoring code works and you can trust it to find the best AI chatbot on real votes.

python rank.py --demo

votes kept: 147  dropped as speed-clicks: 15  both-bad: 4

RANKING (Bradley-Terry on the Elo scale, 95% bootstrap interval)
model                 elo    low   high   score  games
alpha                1077   1026   1137    0.65     67
beta                 1049    996   1109    0.61     75
gamma                 977    922   1031    0.45     73
delta                 896    834    958    0.30     71

HEAD TO HEAD (row beats column, ties count half)
                         alpha        beta       delta       gamma
alpha                        -        0.53        0.77        0.64
beta                      0.47           -        0.70        0.61
delta                     0.23        0.30           -        0.38
gamma                     0.36        0.39        0.62           -

SCORE BY CATEGORY
model                  advice     coding    explain  reasoning  summarize    writing
alpha                    0.45       0.65       0.50       0.91       0.55       0.90
beta                     0.71       0.94       0.65       0.38       0.62       0.42
gamma                    0.45       0.20       0.65       0.41       0.54       0.50
delta                    0.25       0.42       0.21       0.32       0.29       0.31

LENGTH CHECK (answers within 20% of each other in length)
alpha              all votes 0.65   length-matched 0.56 (8 games)
beta               all votes 0.61   length-matched 0.52 (24 games)
gamma              all votes 0.45   length-matched 0.50 (23 games)
delta              all votes 0.30   length-matched 0.39 (9 games)
longer answer won 67% of 124 decisive votes

COST
alpha              avg $0.0046 per answer   $0.71 per 100 preferred answers
beta               avg $0.0048 per answer   $0.79 per 100 preferred answers
gamma              avg $0.0022 per answer   $0.49 per 100 preferred answers
delta              avg $0.0011 per answer   $0.35 per 100 preferred answers

RATER AGREEMENT: 55% across 66 double-rated matchups

These numbers come from synthetic votes and say nothing about the real models. They exist so you can learn to read the report.

Step 12: Read the report like an analyst

• Ranking and intervals. Alpha leads, but its interval overlaps beta’s, so the demo cannot call the top spot. Delta’s upper bound sits below both alpha’s and beta’s lower bounds, which makes it the only clearly separated loser. Treat overlapping intervals as a tie and break it on cost or access.
• Head to head. The matrix shows who beats whom directly. A model can lead overall and still lose a specific duel, so check the cell for your top two models before you pick a winner.
• Category scores. Beta can lead on coding while alpha leads on reasoning and writing. The overall winner is not the winner everywhere. Each category here holds only three prompts, so grow to ten or more per job before you act on a category gap.
• Length check. The longer answer won 67 percent of decisive votes. In the demo that reflects alpha writing long and strong at the same time. In a real run, a figure well above 50 percent that survives in the length-matched slice means your raters reward length, so discount the leaders.
• Cost. The last column divides average cost per answer by the model’s score rate. It prices winning, not just generating. A cheap model that wins half as often can still cost less per preferred answer.
• Rater agreement. Agreement of 53 percent across double-rated matchups is low. When your real agreement falls near that level, your prompts are probably ambiguous, so tighten them and rerun.

Phase 7: Decide, Freeze and Repeat (Step 13)

Step 13: Turn the report into a decision

Write your decision rule before you look at the real results. A workable one has three lines. Pick the highest Elo if its lower bound clears the next model’s upper bound. If the intervals overlap, pick the lower cost per 100 preferred answers. Keep a second model as a specialist for any category where it leads by at least 15 points of score across ten or more prompts. Deciding the rule first stops you from reading the numbers to fit a favorite, and it keeps your pick of the best AI chatbot honest.

Then freeze the evidence. Copy the database into a dated folder and write down the model identifiers, the prices in config.py, and the prompt list. A result without those details cannot be repeated.

mkdir -p results
cp arena.db results/2026-11-08-arena.db
python rank.py > results/2026-11-08-report.txt

Plan the rerun now. Three triggers justify one: a new flagship from any vendor, a price change such as the end of Argon’s introductory pricing on November 30, and a quarterly calendar reminder. Anthropic’s own pages already describe Sonnet 5.5 and Haiku 5.5, so your lineup will move. Append rows to MODELS, rerun generate.py, and only the new answers get collected. Further model coverage lives in our AI and machine learning section.

Common Pitfalls That Skew a Chatbot Blind Test

1. Leaking identity through style. Name scrubbing is not enough if one vendor always formats with headers and bullets. The page renders plain text for that reason. If raters still guess models by tone, shorten the maximum answer length in the prompt text itself.
2. Rewarding length. People tend to prefer the longer reply when quality looks similar. Read the length check on every run, and add a word limit to prompts where brevity matters.
3. Calling winners from overlapping intervals. A 20-point Elo lead means nothing when the intervals span 100 points. Collect more votes or declare a tie.
4. Comparing the API to the app. The model you test through the API may not be what a chat plan serves. OpenAI’s pricing page, for instance, shows Sol absent from Free and Go. Decide which surface you will actually use, and test that one.
5. Letting prices drift mid-test. Argon’s introductory rates end on November 30, 2026. A cost-per-win figure computed in November will look wrong in December, so date every report.
6. Single-rater blind spots. One person’s taste is not a signal. Use at least two raters and read the agreement line.
7. Trusting crowd threads instead of your data. Forum consensus is a fine starting list, and our roundup of Reddit’s chatbot favorites shows how that picture forms. It still cannot replace votes on your prompts.

Troubleshooting: 12 Errors You Will Probably Hit

When none of these match, rerun the failing adapter by itself with a short prompt and print the raw response. Nine times out of ten the field name changed.

Advanced Tips for a Sharper Arena

• Match reasoning effort. Grok 4.7 exposes low, medium, high and xhigh effort, Argon defaults to medium thinking, and Opus 5.5 cannot turn thinking off. Run a second round with each model set to a comparable level, then compare the two rankings. If they disagree, effort was driving your result.
• Add an anchor. Include a cheaper sibling such as Claude Haiku 5.5 as a fifth contender if it is available on your account. If a flagship barely beats the small model on your tasks, you have learned something about your prompts, not just the flagship.
• Pair adaptively. Change next_matchup to prefer pairs whose intervals overlap. You will spend votes where they matter and stop wasting them on settled matchups.
• Collect reasons. Add three radio buttons (accuracy, tone, brevity) to the vote form and a reason column. Reasons explain why a model wins, and they flag bias in your own raters.
• Pre-screen with a judge model. A cheap automated judge can drop pairs where one answer is obviously broken, so humans see only close calls. Treat it as a filter and never as the final vote, since judges favor their own style and length. Our LLM eval pipeline tutorial covers the automated side.
• Use your own material. Replace the generic prompts with redacted real tasks. Strip names, keys and customer data before they touch any API.
• Test multi-turn work. Extend a prompt into a three-message exchange. Chatbots that look equal on one-shot questions often split once the conversation gets long.
• Schedule the rerun. A monthly calendar reminder plus a saved report per run lets you diff Elo scores over time and notice when a vendor update changes behavior.

The Finished Project: Files and Run Order

The folder below is the whole project. Nothing else is required, and the only state lives in arena.db.

arena/
  .env              # your keys (never commit)
  .env.example      # template
  requirements.txt
  config.py         # model registry, prices, settings
  prompts.py        # the 18 prompts and categories
  providers.py      # four vendor adapters with retries
  generate.py       # collects and stores answers
  db.py             # SQLite schema and connection
  scrub.py          # strips vendor names from answers
  app.py            # blind voting page
  rank.py           # Bradley-Terry ranking and report
  arena.db          # created on first run

source .venv/bin/activate
python generate.py        # step 7, repeat until nothing is left
python app.py             # step 10, share the URL with raters
python rank.py            # steps 11 and 12 on real votes
python rank.py --demo     # sanity check with synthetic votes

Before you run a real session, make sure the demo output matches the numbers shown above. The demo is deterministic, so any difference points to a typo in your copy of rank.py.

Frequently Asked Questions

Which is the best AI chatbot in November 2026?

It depends on your prompts, which is the reason this tutorial exists. The four flagships here are the newest releases from their vendors as of November 8, 2026, and none of them wins every job. Run the arena with your own tasks and let the votes name the best AI chatbot for your work.

How many votes do I need for a trustworthy ranking?

Aim for at least 50 decisive votes per model pair. At that size a 60 percent win rate still carries a margin of about 14 points, so close pairs stay uncertain. Add raters or prompts until the intervals in the ranking table stop overlapping for the models you care about.

Can I run the arena with remote teammates?

Yes, but the page has no authentication, so do not expose it to the open internet. Put it behind a VPN or a reverse proxy with a password, or share the SQLite file and have each person vote locally, then merge the votes tables.

How do I add an answer from a chat app by hand?

Insert a row into answers with a new model key such as sol-app and the pasted text, then add a matching Model row so generate.py does not try to call it. The matchup picker uses whatever keys exist in the table.

Is GPT-6.1 Sol the model free ChatGPT users get?

No. OpenAI’s pricing page lists GPT-6.1 Sol as unavailable on Free and Go, available on Plus, and available with expanded access on Pro. A test through the API shows what the model can do, not what a free account serves.

How much does a run cost?

The prompts are short, so the bill stays small. The generator prints the cost of every answer using the prices in config.py, and the cost block in the report sums it up per model. Rates change, notably Argon’s introductory API price, which ends on November 30, 2026 and is followed by $4 and $20 per million tokens, so refresh the registry before each run.

Why not let an AI judge pick the winners?

Judge models are fast and cheap, but they lean toward their own writing style and toward longer answers. They work well as a pre-screen and poorly as the final word. The blind human vote tells you what your team prefers to read.

How often should I rerun the test?

Rerun when a vendor ships a new flagship, when prices move, and at least once a quarter. Keep every dated report so you can see whether a model improved or slipped between runs.