Skip to the benchmark

ResearchBenchmark

GNOMON

Exact laws, far beyond the examples

GNOMON is AIDON’s benchmark for exact generalisation. It asks whether a system can learn a law from small examples and keep it, without a single wrong bit, far beyond anything it saw.

Example item · addition at 256 bits

affffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

b0000000000000000000000000000000000000000000000000000000000000001

Exact answer

a+b0000000000000000000000000000000000000000000000000000000000000000

The largest operand plus one. The carry runs through every bit and the exact answer is zero. A single wrong digit makes it wrong.
Status
In preparation
Tasks
Exact arithmetic, hidden laws
Participants
Language models, learned systems
Scoring
Exact answers only

At a glance

32bits

The widest operands a system may learn from.

4,096bits

The widest operands it is tested on.

6

Exact operations, from addition to the full product.

14%

Of the logic items hide no law at all. The right answer there is to abstain.

In plain words

Show someone a few sums with small numbers and they learn how adding works; after that they can add numbers of any length. GNOMON asks the same of a machine: learn the rule from small examples, then apply it to numbers far larger than any it has seen, with every digit right.

For engineers

Task A: six operations on unsigned operands, learned from 8, 16 and 32 bits and tested from 32 to 4,096 bits, with a structured split built on carries, borrows, leading zeros and one-bit differences. Task B: a sparse Boolean law hidden among up to 128 noisy inputs, judged by exact equality of its algebraic normal form, or an abstention. Every test item is regenerated from a public seed.

Why it matters

Nearly right is not enough when the answer is a number or a rule. GNOMON scores only exact answers and counts every false claim, so a result says plainly whether a system can be trusted with exact work: arithmetic, logic, verification.

The name

What grows without changing

In Greek geometry, a gnomon is the figure which, added to a figure, makes the whole similar to what it was. It was also the pointer of a sundial, the part that tells the time.

That is the test. Add bits to the problem, and a system that has learned the law answers exactly as it did on the small one.

What it measures

Two tasks with exact answers

Both tasks give small or noisy evidence and ask for an answer that is either exactly right or wrong. There is no partial credit.

Task A

Long exact arithmetic

Addition, subtraction, unsigned and signed comparison, and the low and full product of two operands. A system learns from small operands and must answer far wider ones, every bit right, in the same notation it was asked in.

Learns from
Operands of 8, 16 and 32 bits
Tested at
32, 64, 128, 256, 512, 1,024, 2,048 and 4,096 bits
Headline split
Structured: carries and borrows through every bit, thousands of leading zeros, repeated patterns, one-bit differences
Notation
Hexadecimal, binary or decimal

Task B

Hidden logical laws

Each item hides a law over a few of many input bits, seen only through noisy samples. A system must return the law itself, or abstain. Some items hide no law at all: claiming one there is an error.

Hidden law
At most 6 of 20, 64 or 128 input bits
Families
Parity, majority, choice, XOR of ANDs, a fresh random truth table per item, and no law
Evidence
From 128 to 8,192 samples, with 0, 1, 5 or 10% of the labels flipped
Answer
The law, judged by exact equality, or an abstention

Who takes part

Language models and learned systems

The same test, two populations, each ranked on its own. Reference lines sit outside the ranking.

Track 1

Language models

Any language model, named by its dated snapshot, answering with tools and code execution disabled; where a service cannot prove that, it is declared as such. Direct answers and answers with a scratchpad under a published token limit are two sub-tracks, never merged. Prompts and the rule that reads the answer are fixed and published; tokens, time and price per item are reported. Every submission of every team is counted and published.

Track 2

Learned systems

Any trained system that learns only from what GNOMON provides: the published small examples of task A, and the noisy samples inside each task B item. The evaluator trains it and runs it; at test time the system receives only the question, never the seed or the item’s identity. Every structure specified by hand is declared and shown beside the result, so pure learning and learning inside a declared template are never ranked as the same thing.

Outside the ranking

  • A language model allowed to run code, and a standard big-integer library. Both are exact; neither has learned anything.
  • AIDON’s own system, shown only as the author’s reference with its declared template. It is never ranked.

Rules

Built to be checked

Conflict of interest, declared. GNOMON is published by AIDON. The fingerprint of the specification is published with a public timestamp before AIDON’s own system runs on any official test.

  1. Registered before the test exists. A system is frozen by registering the SHA-256 fingerprint of its complete artefact: code, weights, prompt and model snapshot. The GNOMON registry timestamps it.

  2. A public draw. The test items come from a public randomness beacon: the first round emitted at least 600 seconds after registration, confirmed by relays of two independent operators. Nobody, the authors included, can know the fresh items before a system is frozen. The families that are fixed in advance are flagged and reported apart.

  3. Run by the evaluator. Once the round is public, anyone can regenerate the answers. That is why only executions run by the evaluator are official; self-reported runs on official tests never are.

  4. Every item counts. A missing or malformed answer is a wrong answer. Results are reported per operation, width and family, with exact 95% confidence intervals.

  5. False claims cost. In task B a false law costs nine times what an exact law earns, so answering pays only at 90% certainty or more. Abstaining scores zero. Systems that only predict answer 1,024 clean queries per item; those results are reported apart and never merged with proven laws.

Taking part

How to enter

Entries are run by the evaluator, never self-reported. Registration opens with the public release.

Language models

Name the model

Give the model and its dated snapshot. The evaluator runs it with its own access, through the published prompts, and reports every run.

Learned systems

Submit the system

Provide a training entry point, an inference entry point and the declaration card. The evaluator trains the system on the published data and runs it on the test.

Both tracks

Register first

Register the fingerprint of the frozen system. The registry fixes the test round; the result is official once the evaluator has run it.

Status

In preparation, on its way to public release

GNOMON is being prepared for its public release. The specification, the generator and the evaluator are written; the registry and the harnesses for each track come next. No baseline is run and no leaderboard opens before they exist, and only measured results will be shown.

Get in touch to hear when registration opens

Contact

Talk to the founders

Get in touch. We would be glad to share our ideas and walk you through the results, the proof and the measurements.