ata-validator

Two additions, both from the same measurement: when a model produces structured
output, the thing that decides whether it validates is how much of the schema
reached the model, and in what words.

describeSchema

describeSchema(schema, { name }) turns a schema into the description you put
in the prompt.

Everyone hand-writes a prose description of the shape they want and keeps it
next to a schema that enforces something slightly different. Nothing checks one
against the other, so they drift. This derives the first from the second.

Measured on one model, first attempt only, no retry, 30 documents per arm:

Three things in that table. Stating the constraints at all is the whole game. A
careful person matches the generator wherever the constraints can be put into
prose, which is worth saying plainly because the interesting claim would have
been that generation wins everywhere. And where the allowed values are an
internal vocabulary the person scores zero, in all 30 cases, on exactly those
fields: a person describes a field, and those values can only be listed.

toRetryMessage

toRetryMessage(errors, { limit }) builds the string to send back when the
output failed validation. ata already carried what that needs on every error,
in detail and received, but the obvious way to build the message is to join
message, and that throws both away.

Paired, one retry each, the same failing output to both arms:

The conventional message for enum is "must be equal to one of the allowed
values" and names neither the values nor what arrived, so the model guesses. It
guesses well and is always wrong: "HH" for a Hamburg office, "FR" for a Lyon
desk. Every one of those retries returned well formed JSON with the same two
fields still invalid, and more retries cannot fix it, because what was needed
was never in the loop.

Nothing about the default error shape changes, so anything pinned to the
conventional wording is untouched.

Phrasing is part of it

describeSchema first scored 27 of 30 on the first fixture, and all three
failures were multipleOf: 0.01. It said "a multiple of 0.01" where the
hand-written description said "rounded to two decimals". That one phrase was
the gap. Deriving a description is not transcription: each keyword has to be
worded for a reader that will act on it.

Limits

One model, two schema shapes, 30 documents per arm, no temperature control. The
harness is at github.com/mertcanaltin/retry-message-experiment if you want to
run it on your own.

Verified

Suite 1299/1299, 927/927, 1133/1133 across the three dialects, identical with
code generation blocked.