Repository navigation
Conversation
…verlay efeb69f Clean-room lab generation (Opus 5.5) with the coding-agent-lab overlay at efeb69f (U6 #66, U3 #67/#68), then two independent audit and repair rounds. Repairs fixed consistency defects only; a script compared the generated and final spec and found no Ledger row changed and no rule or test added or removed. The one gate edit removed a hardcoded question count from the lab's [environment] paragraph. Checker 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Carr1005
commented
Oct 20, 2026
Comment on lines
87
to
88
| | # | Category | Decision | Invariant (must hold; blank unless the pattern imposes a capability requirement) | Default (course-derived) | Options | Trade-off | Owner | | ||
| |---|---|---|---|---|---|---|---| |
Collaborator
Author
There was a problem hiding this comment.
Decision Ledger for review: loop-engineering
A plain-language version of the spec's Decision Ledger, for review. Each heading links to its row in this PR's spec.coding-agent-lab.md.
How to read it. Before building, the build agent asks the learner one question per decision below. Each table lists the answers they can pick.
- ✅ Recommended: what the default build uses (chosen when the spec was written, not tailored to the learner).
- 📘 Course used: what the course itself did.
- If chosen: what the build leaves out or changes. "—" means nothing.
Some questions are asked only after a certain earlier answer; they say so.
Known open points (already flagged; tracked on #48)
- D2b (skills) breaks its own decision's rule that the instructions reach the agent at the start of every run.
- The list of what's left out marks "automatically applying a change that isn't an instruction edit" as something that can be built, although the course warned against it.
- The fake fetch failure has two values (2 for the demo, 0 normally), which breaks the rule of one value.
- "No grader" is recorded both as a failure and as an option (D3c); see N34.
- Which model each role uses: these settings name no rule for choosing between them; see N20.
Reviewers: reply in the thread on each decision's row (for example under D3c); use this thread for the review as a whole.
| | D2 | design-argued | **Verification stage present** — does a grader wrap the agent loop and send work back? | When present, a grader stage separate from the agent judges the finished file against the rubric after the agent declares completion, and can return it with per-criterion feedback (R10, R11). The learning loop's signal (first-attempt verdicts) exists only when the grader exists. | Grader on, with a revision budget — the course's Lesson 5–7 configuration. The budget *value* is a post-build lever pinned in R11, not part of this row. | (a) grader wraps the agent loop with a revision budget (course default); (b) agent loop alone — revision budget 0, exactly the Lesson 4 build (the course: "if you set this to 0, you will have the same basic agent loop you built in the last lesson"). Criteria that apply only under (a): AC13–AC17, AC19, AC20, AC28, AC31–AC35, AC39, AC46–AC48 and AC50 (every assertion of a grading pass, a `satisfied` status or a first-attempt pass rate, and AC50's Given, which presupposes AC46's `success`); AC3, AC10, AC11, AC22 and AC51 keep their trace and workspace assertions but their run status is `ungraded` and AC51's grading clause is dropped; AC12's and AC41's batch `v1` runs ungraded; AC18 exercises (b). | Lesson 5: "just because the agent says it is done, is the analysis actually good enough? In the agent loop, nothing is checking for this"; Lesson 2: verification "adds reliability" and "you don't need the verification loop per se" when composing loops. Litmus branch: a **row** — the course built the bare loop first (Lesson 4), showed it working at reduced scope, then wrapped it; a budget of 0 removes a component (the grader box), so this is a component choice, not a knob. A first draft the grader later rejects is the reason the grader is added, not the bare loop failing. Choosing (b) retires every grader AC and leaves the learning loop with no first-attempt pass-rate signal (CTX-C2). | course+learner | | ||
| | D3 | design-structural | **Grader kind** — what judges the file against the rubric. | Under D2 (a), the grader stage judges the *file the agent wrote*, never the conversation, criterion by criterion as pass/fail, returns a gap per failed criterion, and never edits the file (R10). | A model-driven judge that reads the file with a read tool and must quote line-numbered evidence for every pass (the course's review-agent grader, Lessons 5–7). | (a) model-as-judge with mandatory file read and verbatim quotes (course default); (b) a deterministic code grader that checks each criterion mechanically — the course's docs-writer example grader ("links resolve, CI passes", Lesson 2) and the narrated Lesson 2 slide: "graders can either be deterministic or agentic". Does not apply under (b): AC13 (a code grader makes no tool call and quotes no `L<n>:` evidence) and AC20's injected grader exception (the `grader_error_doc02` variant belongs to the scripted model's grader role; a code grader needs its own fault injection to reach `grader_error`); every other grader criterion asserts only the verdict shape, which (b) must produce identically. | Lesson 2 names both kinds; the course argues no trade-off between them, so this row enters on structure alone: the grader box's owner moves from the model to harness code. (a) can judge "every clause addressed" semantically but is prone to passing by resemblance — the failure the course's quote-with-line-number rule exists to stop (CTX-B4); (b) is reproducible and free but can only check what a label or regex can see. Redraw the diagram: the grader box changes owner. | course+learner | | ||
| | RR1 | realization | **Acceptance-suite model** — which model the offline acceptance suite runs the pipeline against. | The offline suite drives the real pipeline end to end — tool calls, grader (when D2 = (a)), event trigger, batch, analyst, apply, compare — against a stand-in whose per-role responses are declared in §5; criteria needing a real model are tagged `live`. | The **scripted model** stand-in declared in §5 (`scripted_model.py`) — an honestly-labelled fake standing in for the live model, so the suite runs with zero setup. Branch: substitution (the model is the keyed dependency); tier 1 — a module in the build itself, no package, no process. | (a) scripted stand-in (fake; see §5); (b) the course's live models through the platform proxy (course default — the lab exports the keys, so choosing it costs no setup here; it makes the suite non-deterministic and slow, and R22 governs which variables are read). Exact-value criteria that exercise only the scripted Default — under (b) each becomes a shape assertion like AC39's: AC1, AC3, AC4, AC7, AC8, AC10–AC20, AC22, AC28, AC30–AC35, AC45–AC47, AC50, AC51 (AC8's call count, AC15's and AC16's scripted grader verdicts, AC30's diff line, AC31's `satisfied` comment, AC34's graded count and AC50's Given all follow from scripted responses). | Lesson 4's notebook note: "LLMs do not always produce the same results … the agent behavior you experience may differ from the video." The scripted model is deterministic by construction; the live path is what the finished app uses. | course+learner | | ||
| | L1 | learner | **Scope boundary.** Present the §1 "Not Included" list inside this question — every item with its handling tag — and ask whether to keep every exclusion or bring one back. | | **L1a** — keep every exclusion; the full acceptance set is built. | L1a — Keep every exclusion and build the full acceptance set (Recommended) · L1b — Bring one excluded item back (name it; it is handled by its tag: `restorable` items are built, `buildable, no criteria` items are built with no parameters or criteria from this spec, `unavailable` items cannot be brought back in this lab) | Restoring an item adds scope that the acceptance suite does not cover. The course made no choice on this row. | learner | |
Collaborator
Author
There was a problem hiding this comment.
L1 · What this build leaves out
The learner is asked: here is everything this build does not include; keep it all out, or bring one item back? (The question shows the full list.) Why it's a choice: the course made no choice here; bringing an item back adds work the tests don't cover.
| Ref | Option | Setup | Labels | If chosen |
|---|---|---|---|---|
| L1a | Keep everything left out as listed | None | ✅ | — |
| L1b | Bring one item back, named by you; some items can't come back in this lab | Per item | The item is built as its note in the list says; no tests are added |
| | D3 | design-structural | **Grader kind** — what judges the file against the rubric. | Under D2 (a), the grader stage judges the *file the agent wrote*, never the conversation, criterion by criterion as pass/fail, returns a gap per failed criterion, and never edits the file (R10). | A model-driven judge that reads the file with a read tool and must quote line-numbered evidence for every pass (the course's review-agent grader, Lessons 5–7). | (a) model-as-judge with mandatory file read and verbatim quotes (course default); (b) a deterministic code grader that checks each criterion mechanically — the course's docs-writer example grader ("links resolve, CI passes", Lesson 2) and the narrated Lesson 2 slide: "graders can either be deterministic or agentic". Does not apply under (b): AC13 (a code grader makes no tool call and quotes no `L<n>:` evidence) and AC20's injected grader exception (the `grader_error_doc02` variant belongs to the scripted model's grader role; a code grader needs its own fault injection to reach `grader_error`); every other grader criterion asserts only the verdict shape, which (b) must produce identically. | Lesson 2 names both kinds; the course argues no trade-off between them, so this row enters on structure alone: the grader box's owner moves from the model to harness code. (a) can judge "every clause addressed" semantically but is prone to passing by resemblance — the failure the course's quote-with-line-number rule exists to stop (CTX-B4); (b) is reproducible and free but can only check what a label or regex can see. Redraw the diagram: the grader box changes owner. | course+learner | | ||
| | RR1 | realization | **Acceptance-suite model** — which model the offline acceptance suite runs the pipeline against. | The offline suite drives the real pipeline end to end — tool calls, grader (when D2 = (a)), event trigger, batch, analyst, apply, compare — against a stand-in whose per-role responses are declared in §5; criteria needing a real model are tagged `live`. | The **scripted model** stand-in declared in §5 (`scripted_model.py`) — an honestly-labelled fake standing in for the live model, so the suite runs with zero setup. Branch: substitution (the model is the keyed dependency); tier 1 — a module in the build itself, no package, no process. | (a) scripted stand-in (fake; see §5); (b) the course's live models through the platform proxy (course default — the lab exports the keys, so choosing it costs no setup here; it makes the suite non-deterministic and slow, and R22 governs which variables are read). Exact-value criteria that exercise only the scripted Default — under (b) each becomes a shape assertion like AC39's: AC1, AC3, AC4, AC7, AC8, AC10–AC20, AC22, AC28, AC30–AC35, AC45–AC47, AC50, AC51 (AC8's call count, AC15's and AC16's scripted grader verdicts, AC30's diff line, AC31's `satisfied` comment, AC34's graded count and AC50's Given all follow from scripted responses). | Lesson 4's notebook note: "LLMs do not always produce the same results … the agent behavior you experience may differ from the video." The scripted model is deterministic by construction; the live path is what the finished app uses. | course+learner | | ||
| | L1 | learner | **Scope boundary.** Present the §1 "Not Included" list inside this question — every item with its handling tag — and ask whether to keep every exclusion or bring one back. | | **L1a** — keep every exclusion; the full acceptance set is built. | L1a — Keep every exclusion and build the full acceptance set (Recommended) · L1b — Bring one excluded item back (name it; it is handled by its tag: `restorable` items are built, `buildable, no criteria` items are built with no parameters or criteria from this spec, `unavailable` items cannot be brought back in this lab) | Restoring an item adds scope that the acceptance suite does not cover. The course made no choice on this row. | learner | | ||
| | D1 | design-argued | **Trigger surface** — what starts a review run. | A run starts without anyone copying the requirement into the agent: the agent is told only the document's id and fetches the text itself, and a run started by a change analyzes the requirement's latest version. | **D1a** — the course's example realization of the trigger surface (a simulated tracker standing in for a real one); outside the lab, expect to swap it for your real tracker through the §1 seam. Branch: the course's own shipped realization, already keyless — kept as is. | D1a — A simulated ticket tracker inside the app: creating a ticket linked to a requirement page, or saving a new version of a page that an open ticket links to, starts a run; an open ticket also has a "re-run now" control (Recommended - course default) · D1b — Manual runs only: a page control starts a review of a chosen document, and nothing starts a run on its own (shown working in the course's Agent Loop, Verification Loop and Learning loop notebooks) | Lesson 6 argues for events: "If you have to open a separate agent, copy the requirement, paste it, and ask for a review, that's several extra steps. Even if the review is useful, you might not do it every time." Manual runs are shown producing correct reports (Lessons 4, 5, 7) and the course attributes no failure to them, only friction. Litmus: working (the alternative is shown working) → row. The course also showed the agent in real Slack, Jira and Confluence (screen recordings, no code shipped); those need service credentials this lab does not supply, so they are not offered here — see §1 Not Included. | course+learner | |
Collaborator
Author
There was a problem hiding this comment.
D1 · What starts a review
The learner is asked: should a review start on its own when a ticket or page changes, or only when someone starts it? Why it's a choice: Lesson 6 argues automatic starts mean reviews actually happen; Lessons 4, 5 and 7 show manual runs working, with only extra steps.
| Ref | Option | Setup | Labels | If chosen |
|---|---|---|---|---|
| D1a | A fake ticket tracker inside the app: a new ticket or an updated linked page starts a review, and a ticket can re-run it | None | ✅ 📘 | — |
| D1b | Manual runs only: you pick a document and start its review | None | Leaves out tickets and automatic runs; the report shows on the run's page; 6 tests no longer apply |
| | RR1 | realization | **Acceptance-suite model** — which model the offline acceptance suite runs the pipeline against. | The offline suite drives the real pipeline end to end — tool calls, grader (when D2 = (a)), event trigger, batch, analyst, apply, compare — against a stand-in whose per-role responses are declared in §5; criteria needing a real model are tagged `live`. | The **scripted model** stand-in declared in §5 (`scripted_model.py`) — an honestly-labelled fake standing in for the live model, so the suite runs with zero setup. Branch: substitution (the model is the keyed dependency); tier 1 — a module in the build itself, no package, no process. | (a) scripted stand-in (fake; see §5); (b) the course's live models through the platform proxy (course default — the lab exports the keys, so choosing it costs no setup here; it makes the suite non-deterministic and slow, and R22 governs which variables are read). Exact-value criteria that exercise only the scripted Default — under (b) each becomes a shape assertion like AC39's: AC1, AC3, AC4, AC7, AC8, AC10–AC20, AC22, AC28, AC30–AC35, AC45–AC47, AC50, AC51 (AC8's call count, AC15's and AC16's scripted grader verdicts, AC30's diff line, AC31's `satisfied` comment, AC34's graded count and AC50's Given all follow from scripted responses). | Lesson 4's notebook note: "LLMs do not always produce the same results … the agent behavior you experience may differ from the video." The scripted model is deterministic by construction; the live path is what the finished app uses. | course+learner | | ||
| | L1 | learner | **Scope boundary.** Present the §1 "Not Included" list inside this question — every item with its handling tag — and ask whether to keep every exclusion or bring one back. | | **L1a** — keep every exclusion; the full acceptance set is built. | L1a — Keep every exclusion and build the full acceptance set (Recommended) · L1b — Bring one excluded item back (name it; it is handled by its tag: `restorable` items are built, `buildable, no criteria` items are built with no parameters or criteria from this spec, `unavailable` items cannot be brought back in this lab) | Restoring an item adds scope that the acceptance suite does not cover. The course made no choice on this row. | learner | | ||
| | D1 | design-argued | **Trigger surface** — what starts a review run. | A run starts without anyone copying the requirement into the agent: the agent is told only the document's id and fetches the text itself, and a run started by a change analyzes the requirement's latest version. | **D1a** — the course's example realization of the trigger surface (a simulated tracker standing in for a real one); outside the lab, expect to swap it for your real tracker through the §1 seam. Branch: the course's own shipped realization, already keyless — kept as is. | D1a — A simulated ticket tracker inside the app: creating a ticket linked to a requirement page, or saving a new version of a page that an open ticket links to, starts a run; an open ticket also has a "re-run now" control (Recommended - course default) · D1b — Manual runs only: a page control starts a review of a chosen document, and nothing starts a run on its own (shown working in the course's Agent Loop, Verification Loop and Learning loop notebooks) | Lesson 6 argues for events: "If you have to open a separate agent, copy the requirement, paste it, and ask for a review, that's several extra steps. Even if the review is useful, you might not do it every time." Manual runs are shown producing correct reports (Lessons 4, 5, 7) and the course attributes no failure to them, only friction. Litmus: working (the alternative is shown working) → row. The course also showed the agent in real Slack, Jira and Confluence (screen recordings, no code shipped); those need service credentials this lab does not supply, so they are not offered here — see §1 Not Included. | course+learner | | ||
| | D2 | design-argued | **Instruction placement** — where the review agent's procedure and review guidance live, and when they load. | The instructions reach the agent's context at the start of every run, every version is kept outside the agent, and the agent cannot change them during a run. | **D2a**. Branch: the course's own realization, keyless. | D2a — One instruction file in the agent's own filesystem, loaded whole into its context at the start of every run (Recommended - course default) · D2b — The procedure stays in the instruction file; each section's review guidance becomes a skill whose one-line description loads up front and whose body the agent reads only when it reaches that section (shown working in the course's Deep Agent Primer on a poet agent; never built for the review agent) · D2c — The instructions are written into the system prompt fixed when the agent is created, with no instruction file in the agent's filesystem (shown in the course's base-agent example) | Lesson 3 argues the three placements: the system prompt "you specify when the agent is created, and you use that every time"; the always-loaded file "is loaded each time the agent's called, and you can modify it over time", and "costs context on every call, so it should stay short"; skills "work with progressive disclosure", so "twenty skills cost twenty lines of context, not twenty documents". Litmus: working — all three are shown working in Lesson 3 → row. The choice moves what an approved learning-loop change edits. | course+learner | |
Collaborator
Author
There was a problem hiding this comment.
D2 · Where the agent's instructions live
The learner is asked: where should the review agent's instructions live, and when are they loaded? Why it's a choice: Lesson 3 shows three places working: a file loaded on every run (easy to change, but it costs space each time), skills read only when needed, or a prompt fixed when the agent is created.
| Ref | Option | Setup | Labels | If chosen |
|---|---|---|---|---|
| D2a | One instruction file, loaded whole at the start of every run | None | ✅ 📘 | — |
| D2b | The procedure stays in one file; each section's guidance becomes a skill, a short description the agent sees up front and reads in full only when it reaches that section | None | never built by the course; the spec gives no tests for it | Guidance moves into skills the agent reads section by section; approved changes edit a skill |
| D2c | The instructions written into the system prompt, fixed when the agent is created | None | Leaves out the instruction file; an approved change applies only to agents built afterwards; 1 test no longer applies |
| | L1 | learner | **Scope boundary.** Present the §1 "Not Included" list inside this question — every item with its handling tag — and ask whether to keep every exclusion or bring one back. | | **L1a** — keep every exclusion; the full acceptance set is built. | L1a — Keep every exclusion and build the full acceptance set (Recommended) · L1b — Bring one excluded item back (name it; it is handled by its tag: `restorable` items are built, `buildable, no criteria` items are built with no parameters or criteria from this spec, `unavailable` items cannot be brought back in this lab) | Restoring an item adds scope that the acceptance suite does not cover. The course made no choice on this row. | learner | | ||
| | D1 | design-argued | **Trigger surface** — what starts a review run. | A run starts without anyone copying the requirement into the agent: the agent is told only the document's id and fetches the text itself, and a run started by a change analyzes the requirement's latest version. | **D1a** — the course's example realization of the trigger surface (a simulated tracker standing in for a real one); outside the lab, expect to swap it for your real tracker through the §1 seam. Branch: the course's own shipped realization, already keyless — kept as is. | D1a — A simulated ticket tracker inside the app: creating a ticket linked to a requirement page, or saving a new version of a page that an open ticket links to, starts a run; an open ticket also has a "re-run now" control (Recommended - course default) · D1b — Manual runs only: a page control starts a review of a chosen document, and nothing starts a run on its own (shown working in the course's Agent Loop, Verification Loop and Learning loop notebooks) | Lesson 6 argues for events: "If you have to open a separate agent, copy the requirement, paste it, and ask for a review, that's several extra steps. Even if the review is useful, you might not do it every time." Manual runs are shown producing correct reports (Lessons 4, 5, 7) and the course attributes no failure to them, only friction. Litmus: working (the alternative is shown working) → row. The course also showed the agent in real Slack, Jira and Confluence (screen recordings, no code shipped); those need service credentials this lab does not supply, so they are not offered here — see §1 Not Included. | course+learner | | ||
| | D2 | design-argued | **Instruction placement** — where the review agent's procedure and review guidance live, and when they load. | The instructions reach the agent's context at the start of every run, every version is kept outside the agent, and the agent cannot change them during a run. | **D2a**. Branch: the course's own realization, keyless. | D2a — One instruction file in the agent's own filesystem, loaded whole into its context at the start of every run (Recommended - course default) · D2b — The procedure stays in the instruction file; each section's review guidance becomes a skill whose one-line description loads up front and whose body the agent reads only when it reaches that section (shown working in the course's Deep Agent Primer on a poet agent; never built for the review agent) · D2c — The instructions are written into the system prompt fixed when the agent is created, with no instruction file in the agent's filesystem (shown in the course's base-agent example) | Lesson 3 argues the three placements: the system prompt "you specify when the agent is created, and you use that every time"; the always-loaded file "is loaded each time the agent's called, and you can modify it over time", and "costs context on every call, so it should stay short"; skills "work with progressive disclosure", so "twenty skills cost twenty lines of context, not twenty documents". Litmus: working — all three are shown working in Lesson 3 → row. The choice moves what an approved learning-loop change edits. | course+learner | | ||
| | D3 | design-argued | **Verification stage** — whether and how a finished report is graded before the run ends. | When a grader is present it judges the written report file (never the conversation) against the run's rubric, each criterion pass or fail; the run is done only when every criterion passes in one grading pass; a failing pass sends specific feedback back for another attempt within a bounded budget. | **D3a**. Branch: the course's own realization. | D3a — A model grader reads the report file, grades each rubric criterion pass or fail with quoted evidence, and sends the failing criteria back with their gaps, up to the revision budget (Recommended - course default) · D3b — A deterministic grader: code checks each criterion against the report file (shown in the course only as a narrated example on a different agent — links resolve, CI passes; never built for this agent, and the spec supplies no parameters and no acceptance criteria for it) · D3c — No grader: the run ends when the agent says it is done (the course's Agent Loop lesson, shown working) | Lesson 5: "just because the agent says it is done, is the analysis actually good enough? In the agent loop, nothing is checking for this", and "If you set this to 0, you will have the same basic agent loop you built in the last lesson." Lesson 2: "You don't need the verification loop per se", and verification "can make cheaper models viable for harder tasks". Litmus: a budget of 0 removes the grader — a component choice, not a knob — and the course shows the system working without it (Lesson 4) → row; the budget number itself stays a lever. Grader kind (model or code) is merged here as one subject: it moves the verdict's owner. The course also named a human acting as the grader (Lesson 2); here the offline suite would wait on a person, so it is not offered — see §1. | course+learner | |
Collaborator
Author
There was a problem hiding this comment.
D3 · Who checks the finished report
The learner is asked: should a grader check each report before the run ends, and should it be a model or code? Why it's a choice: Lesson 5 argues nothing else checks the agent's work; Lesson 4 shows the agent working without a grader.
| Ref | Option | Setup | Labels | If chosen |
|---|---|---|---|---|
| D3a | A model grader marks each rubric item pass or fail with quoted evidence and sends failures back for revision | None | ✅ 📘 | — |
| D3b | A code grader checks each rubric item against the report file; the spec gives it no settings | None | never built by the course; the spec gives no tests for it | Verdicts come from code checks, so several tests hold only where those checks agree; 2 tests no longer apply |
| D3c | No grader: the run ends when the agent says it is done | None | Leaves out grading and the learning loop; reports are marked not graded; 19 tests no longer apply. Skips 1 later question (D4) |
| | D1 | design-argued | **Trigger surface** — what starts a review run. | A run starts without anyone copying the requirement into the agent: the agent is told only the document's id and fetches the text itself, and a run started by a change analyzes the requirement's latest version. | **D1a** — the course's example realization of the trigger surface (a simulated tracker standing in for a real one); outside the lab, expect to swap it for your real tracker through the §1 seam. Branch: the course's own shipped realization, already keyless — kept as is. | D1a — A simulated ticket tracker inside the app: creating a ticket linked to a requirement page, or saving a new version of a page that an open ticket links to, starts a run; an open ticket also has a "re-run now" control (Recommended - course default) · D1b — Manual runs only: a page control starts a review of a chosen document, and nothing starts a run on its own (shown working in the course's Agent Loop, Verification Loop and Learning loop notebooks) | Lesson 6 argues for events: "If you have to open a separate agent, copy the requirement, paste it, and ask for a review, that's several extra steps. Even if the review is useful, you might not do it every time." Manual runs are shown producing correct reports (Lessons 4, 5, 7) and the course attributes no failure to them, only friction. Litmus: working (the alternative is shown working) → row. The course also showed the agent in real Slack, Jira and Confluence (screen recordings, no code shipped); those need service credentials this lab does not supply, so they are not offered here — see §1 Not Included. | course+learner | | ||
| | D2 | design-argued | **Instruction placement** — where the review agent's procedure and review guidance live, and when they load. | The instructions reach the agent's context at the start of every run, every version is kept outside the agent, and the agent cannot change them during a run. | **D2a**. Branch: the course's own realization, keyless. | D2a — One instruction file in the agent's own filesystem, loaded whole into its context at the start of every run (Recommended - course default) · D2b — The procedure stays in the instruction file; each section's review guidance becomes a skill whose one-line description loads up front and whose body the agent reads only when it reaches that section (shown working in the course's Deep Agent Primer on a poet agent; never built for the review agent) · D2c — The instructions are written into the system prompt fixed when the agent is created, with no instruction file in the agent's filesystem (shown in the course's base-agent example) | Lesson 3 argues the three placements: the system prompt "you specify when the agent is created, and you use that every time"; the always-loaded file "is loaded each time the agent's called, and you can modify it over time", and "costs context on every call, so it should stay short"; skills "work with progressive disclosure", so "twenty skills cost twenty lines of context, not twenty documents". Litmus: working — all three are shown working in Lesson 3 → row. The choice moves what an approved learning-loop change edits. | course+learner | | ||
| | D3 | design-argued | **Verification stage** — whether and how a finished report is graded before the run ends. | When a grader is present it judges the written report file (never the conversation) against the run's rubric, each criterion pass or fail; the run is done only when every criterion passes in one grading pass; a failing pass sends specific feedback back for another attempt within a bounded budget. | **D3a**. Branch: the course's own realization. | D3a — A model grader reads the report file, grades each rubric criterion pass or fail with quoted evidence, and sends the failing criteria back with their gaps, up to the revision budget (Recommended - course default) · D3b — A deterministic grader: code checks each criterion against the report file (shown in the course only as a narrated example on a different agent — links resolve, CI passes; never built for this agent, and the spec supplies no parameters and no acceptance criteria for it) · D3c — No grader: the run ends when the agent says it is done (the course's Agent Loop lesson, shown working) | Lesson 5: "just because the agent says it is done, is the analysis actually good enough? In the agent loop, nothing is checking for this", and "If you set this to 0, you will have the same basic agent loop you built in the last lesson." Lesson 2: "You don't need the verification loop per se", and verification "can make cheaper models viable for harder tasks". Litmus: a budget of 0 removes the grader — a component choice, not a knob — and the course shows the system working without it (Lesson 4) → row; the budget number itself stays a lever. Grader kind (model or code) is merged here as one subject: it moves the verdict's owner. The course also named a human acting as the grader (Lesson 2); here the offline suite would wait on a person, so it is not offered — see §1. | course+learner | | ||
| | D4 | design-argued | **Learning loop** — whether runs over many documents improve the agent's instructions. Applies only when `D3` is `D3a` or `D3b` | An improvement is drawn only from a failure that recurs across most runs of one batch, is proposed by an analyst that cannot alter the evidence it reads, is applied in plain code only after a human approves it, and is judged by re-running the same batch and comparing. | **D4a**. Branch: the course's own realization. | D4a — A batch learning loop: review a batch of documents under one instruction version, let an analyst propose one instruction change from the recurring first-attempt failures, apply it on human approval, re-run the batch and compare the two (Recommended - course default) · D4b — No learning loop: each run is corrected only by its own grading passes (shown working in the course's Event-Driven Loop lesson, whose app has no learning loop) | Lesson 7: "if we give that same agent a brand new document tomorrow, the agent will start from the same place, and if it makes the same mistake, it will correct it all over again." Lesson 2: "You don't necessarily have to do that for all four loops." Litmus: working — the event-driven app is shown working without it, and the course names a repeated correction cost, not a failure → row. | course+learner | |
Collaborator
Author
There was a problem hiding this comment.
D4 · Learning from many reviews
Asked only if D3 = D3a or D3b. The learner is asked: should the app learn from batches of reviews and improve the agent's instructions? Why it's a choice: Lesson 7 argues the agent otherwise repeats the same corrections on every new document; Lesson 2 says not every loop is needed.
| Ref | Option | Setup | Labels | If chosen |
|---|---|---|---|---|
| D4a | A learning loop: an analyst proposes one instruction change from failures that recur across a batch, a person approves it, and the batch is re-run to compare | None | ✅ 📘 | — |
| D4b | No learning loop: each run is corrected only by its own grading | None | Leaves out batches, the analyst and any instruction version after the first; 12 tests no longer apply |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the loop-engineering lab spec (
spec.coding-agent-lab.md, guide11b3673) with a fresh clean-room lab generation on guide5caba53and overlayefeb69f(U6 #66, U3 #67/#68; all onmain).How it was made (Opus 5.5)
The one gate edit removed a hardcoded question count ("here 6") from the lab's
[environment]paragraph.What this run checked (#48 U3, U6)
unavailable, each with its seam.For review (repair choices, new text within the rules)
harness/rubric.txt.harness/instructions/, and each test starts fromv1.md.Returns verdict: <result>.Known open points (decisions, not repaired; going to #48)
🤖 Generated with Claude Code