prompt.lab 4
← All labsLAB 4 · “Find the broken step” · WEEK 4 · LEVEL 3
Done live in the hands-on session · submitted on this page
COLL 100 · PROMPT ENGINEERING · HANDS-ON SESSION · IN CLASS, THEN ON YOUR OWN

“Find the broken step.”

Wednesday you watched one sentence turn a bare answer into a chain of steps you could check. Today you grade chains for an hour: the same problem run naive and step-by-step, a hunt for the exact broken step in a wrong run, and a false “always” killed with one counterexample.

HOUSEKEEPING, KINDLYLab 3 unlocks this one — if it isn’t finished, finish it on its page at the start of this session; see the TA if you’re stuck.
THE AUDIT RULE — READ FIRSTA run is judged per step, not per answer: Step 1 ✓/✗ · Step 2 ✓/✗ · … · then the answer. “The answer’s right” is not an audit.
RIGHT ≠ CHECKABLEModern models often get all three problems right even naive. You are not grading whether the model got it right — you’re grading which run you could check. The model does the arithmetic; YOU do the judging. (Calculators welcome — that’s judging, not cheating.)
POCKET CARD #4, SIDE B — THE REASONING TOOLKIT
Let's think step by step.
Show your working, then the answer.
Find a counterexample to this claim: "____."
Assume this is true: "____." List what would have to
follow — and what contradicts what we see.
USE IT — WHEN COT EARNS ITS KEEPMath & logic · planning · anything multi-step · anything you must defend.
SKIP ITSimple lookups · creative writing (it can flatten voice) · when speed beats certainty. Steps cost words — don’t paste the magic sentence onto everything all week.
KEY TERMSnaive run = the problem, nothing else · CoT (chain-of-thought) run = the problem + a toolkit sentence, so the model shows its steps · chain = the numbered steps a CoT run shows · audit = marking each step true/false · broken step = the first step where a wrong chain goes wrong · counterexample = one real exception that breaks an “always.”
EVERY RUN IS INDEPENDENTEach RUN on this page is a single shot — the model remembers nothing from your last one. That is exactly what this lab needs: a model that has just shown its steps would keep showing them, and your naive-vs-CoT comparison would die. Also from Wednesday: before every CoT run, write the first step you EXPECT — one line, in your log marked PREDICTED FIRST STEP; after the run, one word next to it: matched / didn’t. (An honest “didn’t” is worth more than a perfect record.)

1P1 together — two runs, one audit

GOALP1 answered both ways, with a step audit.
REQUIREMENTSNaive run logged verbatim, then your written prediction, then the CoT run in a fresh chat. Mark every step ✓ or ✗.
DONE WHENBoth runs and your audit marks in the log.
P1 — THE FUNDRAISER (run it as written)
A club fundraiser sells 84 tickets at $6 each. Costs: $12 per
table for 9 tables, and $95 for the band. How much profit is
left?
  1. Naive run — A fresh RUN, run P1 as written, nothing added. Log the output verbatim.
  2. Prediction — one line: what should a chain’s first step be?
  3. CoT run — A fresh RUN, P1 + Show your working, then the answer. Log it verbatim.
  4. Audit aloud, as a room — Step 1 ✓/✗? Step 2 ✓/✗? … then the answer. (Missed class? Same audit on paper: a ✓/✗ mark per step in your log.)
✓
You should now see the same problem twice — one bare answer, one chain with audit marks. If both said $301: compare CHECKABILITY, not correctness. Which run could you GRADE? That difference is the whole lab.

2P2 solo — the study-hours plan

GOALP2 with the same discipline, solo.
REQUIREMENTSWrite the prediction BEFORE the step-by-step run — that order is the whole point.
DONE WHENNaive run, prediction, CoT run and marks, all logged.
P2 — THE PLAN (run it as written)
A student has 11 study hours for 3 exams. Bio needs at least 4;
History at least 3; Stats at least 2. Stats deserves any spare
hour. How should the 11 hours split?

Same drill, on your own: naive run → prediction → CoT run (a separate chat) → per-step audit marks on the chain.

✓
You should now see a chain where each step is a decision you can argue with. “Why does Stats get the spare hour?” is answerable from the chain — invisible in the naive run. If your naive run mangled the split, even better: your audit shows exactly where.

3P3 — the trap, and the broken-step hunt

GOALCatch P3’s famous wrong answer.
REQUIREMENTSAudit the steps until you can name the exact broken one — “it treated printers as additive time” beats “it’s wrong”.
DONE WHENOne sentence that names the actual error.
P3 — THE RATE TRAP (run it as written)
If 3 printers print 3 posters in 3 minutes, how long do 9
printers take to print 9 posters?

Naive run first — this one has a famous wrong answer. Then prediction + CoT run + audit.

You do the hunt exactly once — your own wrong chain OR the provided transcript, not both. The say in your log which path you took.

✓
You should now see why the audit rule says per step: the wrong chain’s arithmetic is fine. The lie lives in one step — “tripling everything triples the time” — and only a step-by-step read catches it. Everyone leaves having found one broken step.

4Kill an “always” — the counterexample probe

GOALKill an “always” with a counterexample.
REQUIREMENTSRun the probe on the given claim, then once more on a sweeping claim of your own.
DONE WHENC1 and C2 logged, each with your favourite exception.
THE PROBE (from pocket card #4)
Find a counterexample to this claim: "____."
COUNTEREXAMPLE (today)Breaks an “always” with one real exception. “All devices work if plugged in” → a broken charger.
CONTRADICTION (Wednesday’s other probe)Grants the claim and chases consequences until something absurd falls out. Today needs only the first — the card carries both.
✓
You should now see: claim met exception; “always” died. Cheap, fast — and it works on your own claims too. Try “I always run out of time on essays.”

5The transfer — one problem from YOUR week

GOALThe full drill on a problem from YOUR week.
REQUIREMENTSA real multi-step question, nothing private. Naive run → your predicted first step → CoT run → did it match?
DONE WHENThe complete transfer log.

Pick one real multi-step question from your own life this week — splitting a bill, planning study hours, a homework problem (nothing private — no money details, grades, or health; if you’d say it in office hours, it’s fine). Then the full drill: naive run → prediction → CoT run → per-step audit → matched / didn’t.

✓
You should now see the toolkit working on a problem nobody assigned. If seeing the steps changed your answer — or made you TRUST an answer you’d have doubted — write that down; it’s your reflection.

✎Journal · reflection · submit

There is no file and nothing to upload: you do the lab right here in the hands-on session, and the submission panel below is where it all goes. Each piece is checked the moment you type it, and submitting records the lab and unlocks the next one.

HOW THIS IS GRADED — COMPLETION, IN ORDER
Completion, not polish — every item present and genuine, and the points are yours. Nobody grades your prose. The checklist: P1–P3 both-ways logs with per-step ✓/✗ marks · C1–C2 · the transfer, both ways · journal · reflection · the broken-step sentence names the actual error · statement in range.  Labs are 10% of the course, across six labs, and they unlock in order — one left undone blocks the next. No late window: finishing late beats not finishing — if you fall behind, tell your instructor rather than skipping ahead. AI use: Level 3 — submit your prompts and working results.

→Go further (optional)