How to Judge AI Output Quality Like an Editor (2026)

You are reading Chapter 22 of 36 in the AI Bootcamp, AI Central's free course for professionals who want to use AI in their own work. The chapters build on each other, so if this is your first one, Chapter 1 is the place to start. All of them sit on the AI Bootcamp page, and three more arrive every week.

To judge AI output like an editor, write down what good looks like before you generate anything, then check the draft against those lines instead of against your impression of it. Read the draft first and you grade it against itself, because at that point the draft is the only standard you have.

This chapter separates the three questions you can ask about a piece of work, gives you a four-line bar to write before you generate, and names the four reasons to stop revising.

We publish AI systems like this for 300,000+ senior professionals at AI Central, and more of them live in the AI Central Library.

You have spent nine chapters getting better answers out of an assistant. Nobody has told you how to decide whether an answer is any good, so you have been deciding by feel, and by feel you accept whatever arrives looking finished.

The revision loop is the other half of the problem. It is where most people give back the time they saved: six rounds in, holding a version worse than round two, with no way back to it.

If you have not put your own thinking through a critique yet, Chapter 21 does that part. Bring one more thing with you: a draft the assistant produced that you were not happy with and could not say precisely why. That draft is the working material for this chapter.

You now have three ways to interrogate a piece of work, and they arrived in three different chapters. Running the wrong one is how a draft passes every check you know and is still unusable.

Is it true is a question about the world, and the answer is not in the draft. Is my thinking sound is a question about you, settled by making it argue the other side. Is this good is a question about the artefact, settled by comparing what arrived against what you needed it to be.

The true question belongs to Chapter 8, on fact-checking AI answers, and this chapter does not repeat it. A crisp, well-shaped, exactly-right-length brief can still be built on an invented statistic, and reading it as an editor will not save you, because it looks like what you asked for.

It runs the other way too. A summary where every figure checks out, built on an argument that survived a hostile reading, can still be 900 words when the person receiving it has 90 seconds and will read the first line only. Nothing in it is false. Nothing in it is muddled. It is unusable, and neither of the earlier checks would have caught that.

An editor does not ask whether they liked something. They ask whether it does the job it was commissioned to do, for the person who will receive it, in the space it has. Good here is not a feeling. It is a small set of conditions you can write down in advance and then check against.

Almost everyone works in the same order: generate, read, react. Reacting is comparative, and once you have read the draft the only thing available to compare it against is the draft. The standard has to exist before the output does, or the output becomes the standard.

There is a measurement of how strong that pull is, taken on machines rather than people. An August 2026 study fed a prior score to eight assistants as nothing more than context metadata and asked them to judge independently. Seven of the eight moved their ratings toward the number they had been shown, with an effect size up to 0.71. On one dataset the anchor blocked 48% of the corrections the model would otherwise have made. Telling it in the prompt to disregard the metadata did not reduce the overall effect.

The fix is four lines, written before anything exists to be influenced by.

Name the outcome and the reader. This is the line that stops you grading the piece on style.

Three items, no more. This is the line that turns a standard into a checkable list.

Say what makes it unusable, not only what would make it better. Rejects should be cheap and fast.

Length, format, and the screen it lands on. A miss on this line is visible at a glance.

For a customer update that might read: it has to stop them calling support; it must name the cause, the fix, and the date it is fixed by; it is dead if it blames the customer or if it uses the word unfortunately; 120 words, plain sentences, readable on a phone.

Write it down rather than hold it in your head. A bar you keep in your head bends to fit whatever turns up. Three of those four lines could be checked by a stranger who has never met you, which is what separates a bar from an opinion about a draft. Most of the bar belongs in the instruction anyway, so the same four lines get used twice, once as the ask and once as the check. For the instruction side in full, Chapter 13 sets out the five parts of a prompt.

Given an explicit, checkable list, the assistant is a decent inspector: does every claim carry a number, is this under 120 words, is there a named next step. Given the open question of whether the piece is any good, it is a poor arbiter, and the reasons are specific.

Its scores move with who it thinks wrote something: a June 2026 study found that labelling work as the judge's own raised its score and labelling it as someone else's lowered it, with the content unchanged. It also rewards confidence. In a July 2026 study, a judge choosing between two equally correct answers that differed only in tone picked the assertive one in 1,276 of 1,280 cases.

Strong assistants do agree with expert human verdicts between 78.8% and 85.1% of the time on a standard benchmark. But the same evaluation of 21 judge models from 9 providers, published in June 2026, found that this raw agreement overstates real discrimination by 33 to 41 percentage points once you correct for the agreement you would get by chance. Consistent is not the same as correct, and a machine that gives you the same wrong verdict twice will feel more trustworthy than one that hesitates.

So use it as an inspector. Hand it the bar as a numbered list and ask for a verdict on each item with the sentence that proves it quoted back, rather than a score out of 10.

You ask for a fix, get a version that fixes it and breaks something else, ask again, and six rounds later you are holding something worse than round two with no way back to it.

Two things are happening at once. The first is that a long thread is a worse place to work than a short one. In a study of over 200,000 simulated conversations across six generation tasks, published in May 2025, performance was on average 39% lower when the same work was spread over several turns than when it was asked in one go. The drop came mainly from a rise in unreliability, with only a small loss of raw capability. When it takes a wrong turn in a conversation, it gets lost and does not recover.

The second is that it will not reliably notice its own regression. A July 2025 study of 14 models found they were on average 64.5% less likely to fix an error when it sat in their own output than when the identical error came from the user. Revision pays when something outside the loop says what is wrong. In your work, that something is you.

Which gives you four rules.

If your message contains and also, you have asked for two things, and what comes back is a rewrite in which both requests are partly served and you cannot attribute anything. Ask for the single most important change. Read what happened. Then ask for the next one.

Keep the opening line and every number exactly as they are. Otherwise a fix in paragraph four edits paragraph one. That sentence costs you twelve words and saves an entire round.

A document, an email to yourself, a note. It takes four seconds and it is the rule people skip, because in the moment round three always looks like it is going to be an improvement on round two. Sometimes it is not, and by then round two is forty messages up a scroll, mixed in with three versions that look almost the same.

When round four is worse than round two, asking it to put back what it removed adds a fifth instruction on top of four rounds that have already drifted. Paste round two in as the starting text, say what you want changed about it, and let round five branch off the good version rather than patch the bad one. Undo is not an operation it can perform.

The loop has no natural end, so you have to supply one. There are four reasons to stop and only one of them means the piece is finished. Read down and stop at the first line that is true.

Making it yourself costs a keystroke. Describing it precisely enough for someone else to execute costs a sentence or two of careful writing. For a wrong word, a sentence that should go, or two paragraphs in the wrong order, you lose by asking. Handing it back wins when the change is systematic rather than local: every section needs a named owner and a date runs nine times off one instruction.

Send it. Anything past this point is taste, and taste never runs out: there is always a better verb. If you kept going past this point on the last three things you made, that was the absence of a stated finish line, and you now have one.

After each round, ask one question: is this closer to the bar, or just different? Different and not closer twice running means the loop has stopped converging, and more rounds will not restart it. Go back to the best version you kept and stop there. It will feel like giving up on an improvement that was one message away. It almost never was.

A piece that is 80% right sounds nearly finished, and that is only true when the missing 20% is sentences. The test is whether you can name the change as an edit, or only as a description of a different piece. If it is the second, open a new thread, take your four lines, add the one you now know was missing, and ask again from nothing. The failed draft showed you which line was missing.

The chapter closes with three exercises that take about twenty minutes. Pick one real piece of work you have not asked for yet, and write the bar before you open the assistant, so nothing exists to be influenced by.

Then generate it and score it against your bar without editing a word. Write down the line it failed, and only that line. If it passed all three boxes, your bar was too low, so write a harder one and try again.

Then run exactly one revision round on the line it failed and nothing else, and write one sentence: closer to the bar, sideways, or worse. If it is worse, paste the previous version back in and stop there.

Before: you accept the draft that arrives looking finished, revise it by feel until you run out of patience, and by round six you are holding something worse than round two with no copy of round two.

After: you write four lines before you generate, you check the draft against those lines instead of against your impression of it, and you stop at the first of four reasons that is true. Throwing a draft away at round three instead of round nine is the skill, and it gets easier once the bar is written down, because you can point at the line it will never meet.

Two things carry into Chapter 23: the four-line bar you wrote, and the draft you rejected along with the line of the bar it failed. Chapter 23 turns that standard into a catalogue of the specific ways a draft goes wrong. Every chapter is listed on the AI Bootcamp page, and more AI systems like this live in the AI Central Library.

Write the bar before you generate: what the piece has to accomplish and for whom, three things it must contain to be usable, two things that disqualify it whatever else is right, and its shape in length, form and where it will be read. Then check the draft against those four lines rather than against your impression of it.

Because a draft you have already read becomes the standard you grade it against. An August 2026 study fed a prior score to eight assistants as context metadata and seven of them moved their ratings toward it, with an effect size up to 0.71, and on one dataset the anchor blocked 48% of the corrections the model would otherwise have made. Instructing them to ignore it did not reduce the overall effect.

As an inspector against a checkable list you wrote, yes. As a critic on the open question of whether a piece is good, no. Scores shift when work is labelled as the judge's own, and in one study a judge almost always picked an assertive answer over an equally correct hedged one. An evaluation of 21 judge models from 9 providers, published in June 2026, found raw agreement overstates real discrimination by 33 to 41 percentage points.

Long threads are a worse place to work than short ones. Across over 200,000 simulated conversations in six generation tasks, published in May 2025, performance was on average 39% lower spread over several turns than asked in one go, mainly through a rise in unreliability. The fix is one change per round, naming what must not change, and keeping the best version outside the thread so you can go back to it.

When describing the change costs more than making it. Local, taste-level fixes are faster to type than to specify, so a wrong word or two paragraphs in the wrong order should be done by hand. Hand it back when the change is systematic, such as every section needing a named owner and a date.