How it works

In short. Three AI models predict what the Supreme Court will do in real cases before the Court acts. Each prediction is written to a public record with a timestamp, and scored later against what the Court actually did. Because the answer doesn’t exist yet when the prediction is made, the models can’t have memorised it.

FedCourtsAI asks a simple question: when a frontier AI model reads a case that is actually pending before the Supreme Court, and commits to a prediction before the Court acts, how good is that prediction? Not on a benchmark the model may have seen during training, and not on cases whose outcomes are already written down somewhere, but on the live docket, where the answer key does not exist yet.

The design follows from that question. Predictions are made before outcomes, written to a public repository with a timestamp, and scored later against what the Court actually did.

The events

A prediction is about one event: one question about one case, asked at one point in the case’s life. There are three stages, each with its own question.

Cert. Will the Court take the case? Asked when a petition is docketed, again when it is distributed for conference (and after each relist), and again if the Court asks for the Solicitor General’s views.
Interim. Will the Court grant an emergency application for a stay or an injunction? Asked when the application arrives, after the Court calls for a response, and once the response is filed.
Merits. Will the Court disturb the judgment below? Asked at the cert grant and again once both sides’ merits briefs are in.

These moments are declared in advance, in a closed table in the code. A case is predicted when it enters the pool and re-predicted only when something about it changes: a distribution, a relist, a call for the Solicitor General’s views, a new filing. Nothing runs on a fixed clock, and a maintainer cannot pick a case by hand: each run derives its own work from the committed record.

What a prediction contains

Each prediction is a small directory in the repository, written by one model about one event. The structured part is a JSON file carrying a probability (of a grant at cert and interim; of the judgment below being disturbed at merits), a single most-likely disposition, and, at merits, a predicted judgment and vote. Beside it are three prose files: why the number is what it is, what the model expects the Court to do and on what grounds, and what the model consulted beyond the materials it was given. That third file is the model’s own account of its retrieval; the harness also records the tool calls it actually made.

The reasoning matters as much as the number. A grant-or-deny call on the cert docket is close to a coin with a known bias; the harder and more interesting question is whether a model can anticipate which justices will be in the majority, who will write, and on what doctrinal ground.

Pre-registration is about time, not evidence

A model is not restricted in what it may read. It can search, pull commentary, and consult whatever it likes; a forecaster in the world would do the same. The commitment is temporal: the prediction lands in git, on a public branch, with a timestamp from the runner, before the Court’s order list is released. That timestamp, together with the workflow run that produced the commit, is what establishes that the prediction preceded the outcome.

This is a deliberate choice. A forecast that is only allowed to see a curated subset of the world tests something narrower than forecasting. What the design does instead is make retrieval visible, so that a prediction which leaned on something it should not have had (a leaked outcome, say) can be identified and excluded after the fact, rather than prevented by a gate that would also prevent legitimate research.

Who predicts, and who grades

Three model families currently compete, each running inside its own developer’s agentic coding harness: Claude, Codex, and Gemini. Each reads the same provisioned case record and answers independently. A prediction is one model’s answer; the tournament is the comparison between them.

After an event resolves, the same three models act as evaluators. Each grades every prediction on the event, and grading is blind: the prediction is staged under an opaque alias with the predictor’s identity masked, so a model does not know whose reasoning it is scoring, its own included. The quantitative score is arithmetic, computed by code from the outcome; the evaluator’s job is the qualitative part, judging the predicted reasoning against the opinion the Court actually issued, and checking the prediction’s retrieval for leakage.

What counts

Before the first term’s predictions could be scored, the process was frozen: the prompts, the model configurations, and the harness settings were digested into a tagged commit, and every prediction that counts toward a reported figure must carry a stamp matching those digests from after that instant. Predictions made while the pipeline was still being built are kept in the repository, with their timestamps, and labelled as shakedown. They exercised the machinery; nothing about them was pre-registered, and nothing is claimed from them.

When the process changes, the version changes, and results across the boundary are not pooled. Two processes on two information sets are two experiments.

How scoring works

Almost every cert petition is denied, so a model that answers “denied” to everything will be right most of the time. Accuracy alone therefore says nothing. Each prediction is scored for skill against a base rate matched to the kind of case it is: the historical grant rate for petitions in the same salience band, so that restating the base rate earns nothing and beating it earns something. The probability is also scored for calibration with the Brier score.

Two further scores are pre-registered alongside the forecast. A big-case score is each model’s read of how much a case matters, explicitly not its grant likelihood; it is what the front page ranks by, and it is graded later by an independent evaluator rather than by the outcome. At merits, the predicted vote is scored against the actual vote.

Which cases

The Court decides thousands of petitions a term. Predicting all of them equally would spend the whole budget on cases that resolve exactly as the base rate says. Instead, a deterministic score ranks the eligible petitions by how much each is worth forecasting, using features a docket acquires as it moves (relists, a call for the Solicitor General’s views, the originating court), and the models run on the top-ranked slice up to a fundable capacity, plus carve-outs the score cannot game: a random slice of new filings drawn from a pre-registered seed, every petition the federal government files, every case where the Court has asked the Solicitor General for its views. Once a case is in, it stays in; the record is never rewritten by a later capacity decision.

Excluded from prediction, though not from ingestion, are pro se and in forma pauperis petitions, original-jurisdiction and miscellaneous matters, and dockets whose outcome is not machine-readable. This is a budget line, not a claim that those cases do not matter.

Limitations

Some of these are temporary, some are inherent in the design, and it is worth being clear about which.

The numbers are small. The first frozen cert population is on the order of a hundred events, all distributed for one conference. A hundred events cannot separate three models with any confidence, and figures from it should be read as a first look, not a ranking.
The first population is not a sample of the docket. It is the residue of earlier funded rounds, weighted toward the higher salience bands. Read it band by band, against each band’s own base rate; a pooled number over it would flatter every model.
The timestamps rest on GitHub, not on a notary. Git author dates can be set to anything. What the design relies on is the runner clock the model cannot write, the public push to a protected branch, and the workflow run log that produced the commit. That is strong enough for the purpose; it is not cryptographic proof, and we do not describe it as such.
Retrieval is self-reported and graded, not prevented. A model that read an outcome it should not have had is caught by the evaluator reading its retrieval log, not by a wall. Leakage is an exclusion, not an impossibility.
Three models, one harness each. A result about “Claude” here is a result about a particular model running inside a particular coding harness with particular prompts. Changing any of those changes the process version.
Cert is the easy stage. The interesting predictions, about reasoning and votes, are only scorable at merits, and merits events resolve months after the forecast. The early results will be almost entirely grant-or-deny.
Scope follows the budget. Which cases are predicted is a function of a capacity number, and that number is a function of funding. A wider scope is a decision, not a discovery.
None of this is legal advice. The predictions are experimental model outputs about how a court may act. They are not statements of fact about any judge, and not a basis for any decision. See the terms.

Case summaries

Most cases on the board carry a short plain-language summary: what happened, what the Court is being asked, and where the case stands. They are there so a reader who doesn’t know the case can follow the predictions.

Each summary is written by Claude Sonnet 5, a model that is not one of the three being scored, from the case’s docket and filings on the Supreme Court’s own website as of the date shown, and nothing else. It has no web access and is told not to add anything it may know from elsewhere. It is instructed to describe both sides’ positions without predicting the outcome or judging how important the case is. A summary is rewritten when the case’s filings change, and each one is committed to the public repository with the model and date that produced it.

The summaries are AI-generated and can contain mistakes. They are not predictions and nothing is scored on them. For anything that matters, check the filings.

Case topics

Cases on the board carry a topic tag, such as Criminal law or First Amendment, saying what the petition asks about. Each tag comes from a fixed list of fifteen subjects and is assigned from the petition’s questions presented alone, by a Claude model working as a labeler, separate from the three models being scored. A few cases also carry a second topic.

The tags are added in batches, so newer petitions and emergency applications may not have one yet. Before a batch is published it is checked against a fixed reference set of cases labeled under the same written rules; the labels currently match that set about nine times in ten. The reference labels were also made by AI models, so that figure measures consistency rather than accuracy, and the boundaries between some subjects (civil procedure and constitutional rights, for instance) are genuinely blurry. Topics are descriptive only: no prediction depends on them and nothing is scored on them. The board can be filtered by topic; No topic collects the cases not yet labeled and the few the labeler could not place in any subject. The rules and every label are in the public repository.

Glossary

Cert (certiorari): The Supreme Court chooses almost all of its cases. A party asks it to take a case by filing a petition for certiorari; the Court “grants cert” to hear the case or “denies cert” to leave the lower court’s decision standing. It denies the great majority.
Docket: The Court’s list of pending cases, and each case’s public record of filings and orders. “Docketed” means the case has been entered on it.
Distribution and conference: The justices decide which petitions to take at private meetings called conferences. A petition is “distributed” for a particular conference once the papers are complete.
Relist: When the Court considers a petition at conference but neither grants nor denies it, and instead carries it over to a later conference. Repeated relists often signal that the justices are still debating a case.
CVSG (call for the views of the Solicitor General): An order asking the federal government’s top Supreme Court lawyer to say whether the Court should take a case in which the government is not a party. It raises the odds of a grant considerably.
GVR (grant, vacate, and remand): A short order in which the Court takes a case, wipes out the lower court’s decision, and sends it back to be reconsidered, usually in light of another recent Supreme Court ruling. No briefing or argument takes place. The ledger counts a GVR as a grant.
Summary reversal: A decision reversing the lower court on the petition papers alone, without full briefing or oral argument. Also counted as a grant.
Emergency application (the “interim” stage): A request to the Court to act quickly while a case is still moving through the lower courts, most often to pause (“stay”) a lower-court order or to block a government policy. These are sometimes called the “shadow docket”.
Merits: The stage after the Court has agreed to hear a case: full written briefs, oral argument, and a signed opinion deciding the legal question.
Reverse, vacate, affirm: What the Court can do with the lower court’s decision at the merits stage: reverse it (the other side wins), vacate it (set it aside and send the case back), or affirm it (leave it standing). The ledger’s merits question is whether the Court reverses or vacates.
Petitioner and respondent: The party asking the Court to take the case, and the party opposing. In a case name, the petitioner is listed first.

The full design, with the command behind every number that will appear on the results page, lives in the repository’s documentation.