Summary
Explanations for objective questions are written once and stored; the app can't generate a new one for each student's answer. So "you picked C, here's why that's wrong" isn't available. The static equivalent is what our medical banks already do: one block per option, headed by that option's own text, so a student who picked anything finds their answer's block and reads why. Production NCLEX has been rewritten into that format. The four new courses have not: none of their 10,244 live explanations use it, although each course's generator prompt already asks for it.
row-1…row-4 instead of the findings, so they can't be answered.- The key first, in bold italics, headed by its own text, with two bullets on why.
- Every other option gets its own block, headed by its text exactly as the screen shows it, with two bullets on why it's wrong. This is how the explanation "answers the student's pick": whatever they chose, there's a heading with those words.
- Multi-part formats repeat the pattern per part: per blank, per row, per step, per region. Numeric answers list the common wrong answers and the slip behind each; ordering lists the common wrong orders.
- No letters (they vanish behind ✓/✗ icons and don't exist on SATA screens), no authoring metadata, bold on the key terms only.
The benchmark: NEET PG and USMLE explanations
Captured on staging as a US Medical PG student, answering a USMLE Step 1 vasculitis question wrong on purpose.




Anatomy of one
- Haemophilus influenzae has invasive strains, particularly type b (Hib), which are significant causes of bacterial infections like meningitis and epiglottitis.
- These invasive strains are prevalent in young children and contribute to serious morbidity if not properly managed.
- Haemophilus influenzae is actually Gram negative, which is a crucial distinction for identifying and treating infections.
- Gram-negative bacteria are characterized by their thin peptidoglycan layer and presence of an outer membrane.
- Certain strains, particularly type b, have a robust polysaccharide capsule that plays a crucial role in virulence.
- …
- Haemophilus influenzae requires chocolate agar or enriched media rather than standard sheep blood agar… 3
- …
NEET PG, "True about Haemophilus influenzae is:" (c1ebf59b), as stored.
- 1Key first, in bold italics, using the option's words, not "Option A".
- 2Each distractor under its own italic heading, in screen order. A student who picked "Gram positive" reads exactly one block to find out why.
- 3Two bullets each, bold on the discriminating fact ("chocolate agar"), not on whole sentences.
- Nothing else. No preamble, no subject or Bloom's tags, no letters.
The generator prompts for CFA, CPA, Bar and LSAT (src/prompts/library/courses/<course>/question-explanation-generation.md) already specify this format for single-answer questions, but the stored explanations in those banks weren't produced with them.
How widely the format is used
Share of validated explanations that open with a bolded key block (***…***, or **Blank 1** / **row** for multi-part formats).
| Course | Staging | Production | Note |
|---|---|---|---|
| Indian Medical PG | 9,823 / 10,245 (96%) | 74,101 / 74,271 (99.8%) | The benchmark |
| US Medical PG | 9,764 / 10,230 (95%) | 11,741 / 11,744 | |
| UK Medical PG | 4,975 / 4,976 | 4,625 / 4,625 | On staging, 2,125 are stored with no line breaks, so blocks run together. Prod has 6. |
| NCLEX-RN | 0 / 5,643 | 5,067 / 5,642 (90%) | Prod rewritten per format (examples in each card below). The rest are 432 case-study summaries still in paragraph form. Staging is stale. |
| MCAT | 0 / 2,681 | 2,478 / 2,681 (92%) | Staging is stale |
| CFA | 0 / 6,612 | Not live | 4,573 refer to options by letter |
| Bar Exam | 0 / 1,591 | Not live | 205 empty |
| CPA | 0 / 1,214 | Not live | 39 empty |
| LSAT | 0 / 827 | Not live | 94 empty, including whole reading sets |
The one answer-aware layer we already have: follow-up chips
After every answer the app calls POST /chat/follow-ups with the question and the chosen option. The API builds the context from the question and the pick, generates three chips, and caches them per question × answer (explanationGenerator.generateStandaloneFollowUps). So the cost is bounded by the number of options, not the number of students. In the capture above, picking "Extrinsic asthma" produced "Help me distinguish EGPA from severe eosinophilic asthma using clinical clues and ANCA findings". One tap turns that into a Rezzy answer about exactly the student's mistake.
It already fires for every format, including ordering. One gap: for ordering and grid answers the app sends option UUIDs as the "answer" ("1. 4f1841a8-… 2. ad570295-…" in one capture), so the model can't read what the student did. Sending the option text fixes it without any new generation.
Scoreboard
One row per format. "House format" asks whether the explanation has a block for every option, headed the way the screen shows it. For NCLEX, the staging text (what the screenshots show) and production text differ, so both are listed.
| Format | Tested in | Key explained | Every option explained | House format | Verdict |
|---|---|---|---|---|---|
| Select all (SATA) | NCLEX | Yes | Yes | Staging no (letters the screen hides) · Prod yes | Good in prod |
| Cloze dropdown | NCLEX | Yes | Yes | Staging paragraph · Prod per blank | Good in prod |
| Matrix | NCLEX | Yes | Yes | Staging paragraph · Prod per row, long | Trim; row-label bug |
| Ordered response | NCLEX | Yes | No wrong orders covered | Prod per step | Screen contradicts it |
| Fill in the blank | NCLEX, CFA | Worked solution | Staging names slips · Prod dropped them | Prod key only | Add wrong answers |
| Image hot spot | NCLEX | Yes | Staging lumped · Prod per region | Prod headings are id slugs | Needs work |
| Highlight hot spot | NCLEX | Yes | Staging one line · Prod per order | Prod yes | Good in prod |
| Case study (NGN) | NCLEX | Per step | Per step | Steps yes · case summary paragraph + Bloom's | Content bugs |
| Single best answer | CPA, LSAT, Bar | Yes | Yes | No: "Option B is correct because…" | Regenerate |
| CPA TBS grid | CPA | Per row | Key only per row | Per row, but "Correct answer: Correct" | Best of the new courses |
| CFA item set | CFA L1 | With working | Distractor maths don't reproduce | No | Fix items, regenerate |
| Reading comprehension | LSAT | Nothing | Nothing | Empty | Missing |
| LSAT MCQ | LSAT | Yes | Yes | No: by letter | Regenerate |
| Integrated question set | Bar | Per step | Per step | No: by letter | Regenerate |
| Short answer | Bar (1 item) | Live-graded. Jev grades it and writes a one-line reason; the student never sees either. | Invisible | ||
| Performance task | Bar | Live-graded. One generic comment, two contradictory scores, rubrics that demand facts the file doesn't contain. | Broken | ||
| Argumentative writing | LSAT | Live-graded. Sensible bands; generic comments; no model answer; "worth a human look" notes shown to students. | Needs work | ||
| Constructed response | CFA (338, none validated) | No grader exists. The app shows "Scoring in progress" and never a result. | Not shippable | ||
Not captured: data-entry grids with numeric cells (the CPA simulation tested was a Correct/Incorrect grid), and MCAT, which only has single-answer MCQs.
Blocking bugs found during the audit
These go beyond explanation quality: the app tells students something false, or the question can't be answered. No rewrite fixes them.
1 · Ordered response: the review shows a different "correct" order from the one the server grades Prod · 39 of 45 standalone
The client rebuilds correct_order from the legacy options rows' metadata.sequence (withOrderedOptions, apis/question/formats.ts:563). The server grades against content.correct_order. On 36 of the 45 live standalone items they disagree, so the review marks right positions red and prints the wrong "Correct: …" under each, directly above an explanation that gives the true order. On 3 more, content.correct_order uses item-N ids while the app submits option UUIDs, so a perfect order scores "Incorrect 0/5" with every row green. Counting case-study children: 155 of 166 validated ordering questions in production.


2 · Case studies give away the dropdown answer Prod · 29 cases, 53 blanks
The case player reads the parent's denormalised content.sub_questions copy, which kept only the correct option per blank. The child rows still have all three. The student opens a dropdown with one choice in it.
3 · Matrix rows labelled row-1 … row-4 72 items, all live
The row text never made it into row_headers, so the student classifies "row-1" as improvement or follow-up. The explanation's per-row headings use the real findings (tingling, neck circumference, a calcium of 8.0), which never appeared on screen.
4 · Case timelines leak or run ahead, and the explanation admits it Content · prod
In the thyroidectomy case, step 1 asks about "neck fullness shown in the image", but only Exhibit 1 is released: a normal baseline with no image. The production explanation for that step says so to the student: "Because the actual Exhibit 2 image is unavailable, its additional findings cannot be verified…", a generation note that shipped. Exhibit 3 then narrates the answers to steps 3 and 4 ("The nurse activates the rapid response system, raises the head of the bed, applies oxygen…") before the student answers them.



5 · Free text: grades students never see, and grades that can't pass Bar, LSAT, CFA
- Short answer (inside Bar sets): Jev grades it and writes a reason. The app shows "Graded against the rubric after you finish the set", and the results review has no
short_answercase (FormatAnswerReview.tsx:690). - Performance task: the grade lives only in component state. The results review shows the author's model-answer paragraph and nothing about the student's memo.
- Performance-task rubrics cite evidence the file doesn't contain (Part 2). A missing required element caps the score at 60%, so affected tasks can't be passed.
- Constructed response (CFA, 338 items): no server grader and no result UI. Harmless while unvalidated; it blocks launch.
Patterns across formats
New courses explain by letter, in one paragraph
CPA, CFA, LSAT and Bar explanations read "Option B is correct because… Option A ordinarily… Option C permits…" (4,573 CFA, 824 CPA, 552 Bar, 455 LSAT). A student has to translate letters back into options, and after answering, the ✓/✗ icons replace the letters of the picked and correct options. In the house format each option is findable by its own words.
Staging isn't production for NCLEX and MCAT
Production NCLEX was rewritten into the house format (5,067 of 5,642); staging still has the older paragraphs, with the letters, metadata openers and missing per-option reasons this audit first flagged. Anyone QA-ing explanations on staging sees text students don't. The rewrite script isn't in api@dev, so the new courses can't simply rerun it.
Authoring metadata printed to students
Openers and closers like "This item targets Medical-Surgical Nursing, Cardiovascular Disorders, Bloom's 3_apply level, medium difficulty, and the NCJMM Generate Solutions skill." In production they survive mostly in the case-study summaries: 366 of 432 end with "The case integrates Recognize Cues through Evaluate Outcomes… at Bloom's 5_evaluate level." On staging they open 2,891 NCLEX explanations.
Multi-part formats stop at the key
The house format covers wrong options well when they're listed. Where they aren't, production NCLEX drops them: fill-in explanations give only the worked answer (the older staging text named the two usual slips), and ordering explanations list the right sequence but no wrong one. For these formats, "every option" means the common wrong answers and orders, written out.
Aggregates end without a debrief
Case studies, integrated sets and reading sets show a rationale per step, then "Finish case" jumps to the next case: no case score, no summary. The case summary exists in content.explanation but only appears later in the results list. Rezzy isn't available inside aggregates.
Empty explanations in the new courses
Bar 205, LSAT 94 and CPA 39 live items have no explanation. For LSAT that includes whole reading-comprehension sets: red and green, nothing else. Separately, the LSAT passage tested (Condorcet and Olympe de Gouges) reads like a published LSAT PrepTest passage; worth a licensing check.
Format by format
Screenshots are from staging. For NCLEX, the text students actually get is production's, so each NCLEX card compares production today with a better version. The better versions are static: one explanation that covers every pick, rendered the way the app renders markdown (bold-italic heading for the key, italic headings for the rest).
Select all that apply


Production already gives each of the six statements its own block. Two small gaps. On a select-all screen several options are correct, and the only signal separating "select" from "don't select" is bold italics versus italics; a group label makes it unmissable. And "1/4" with two right picks is never explained: select-all scores +1 per right pick and −1 per wrong pick. That's a fixed line in the result banner, not explanation text.
- Scheduled INR testing evaluates warfarin's anticoagulant effect and supports safe dose management after discharge.
- …
- …
- …
- …
- The client should not double a missed warfarin dose, because taking extra medication may increase anticoagulant effect and bleeding risk.
- …
- …
- The INR sets each warfarin dose; skipped tests hide drift toward bleeding or clotting.
- It needs checking even when today's INR is in range: diet, illness and new drugs move it.
- With a mechanical valve, anticoagulation is lifelong, and blade nicks bleed longer.
- Preventing small injuries is standard bleeding precaution teaching.
- Procedures and new drugs (antibiotics, NSAIDs) change bleeding risk and the INR.
- Clinicians can then plan around warfarin instead of discovering it mid-procedure.
- Melena and hematuria are signs of internal bleeding.
- Early reporting lets the provider check the INR before bleeding becomes serious.
- A double dose can push the INR into the bleeding range.
- The client follows the prescriber's missed-dose plan and never adjusts the dose alone.
- Aspirin adds an antiplatelet effect on top of warfarin.
- Acetaminophen, cleared with the provider, is the usual choice.
Cloze dropdown


This is the format adaptation the new courses should copy: a Blank n heading, then the right choice and every other choice in the dropdown, each with its reason. The only improvement is saying what each blank asks, so the student doesn't have to scroll back to remember what blank 1 was.
Blank 1
- Sleeping on the nonoperative side avoids pressure on the left operative eye during early recovery…
- …
- Bending at the waist is discouraged during early recovery because it can increase intraocular pressure…
- …
- …
Blank 2
- …
- …
- …
Blank 1 · the statement that shows the teaching worked
- Keeps pressure off the operated eye while it heals.
- It's the only choice that lowers risk rather than raising it.
- Bending over raises intraocular pressure in the healing eye.
- Teach sitting down and bringing the foot up instead.
- Heavy lifting strains and raises eye pressure in early recovery.
- Anything above light objects waits for the surgeon's clearance.
Blank 2 · the symptom that needs immediate follow-up
- Can signal retinal detachment, a complication needing same-day care.
- New or worsening visual symptoms are never "wait and see".
- Expected during healing; the client just mustn't rub the eye.
- Only itching with pain, discharge or vision change needs a call.
- Expected on day one.
- Heavy tearing, or tearing with pain or vision loss, would need follow-up.
Matrix


Production repeats the full pattern for every cell: 5 rows × 3 columns × 2 bullets. It's complete but long, and the wrong-column bullets mostly restate the right-column ones ("does not require prompt follow-up when relief is achieved"). Keep two bullets for the right column and one sharp line for each other column. Separately, 72 matrix items show row-1… on screen (bug 3), so the explanation's row headings are the only place the findings appear.
Left leg appears shortened and internally rotated
- Shortening and internal rotation strongly suggest possible hip prosthesis dislocation and require immediate evaluation.
- Do not reposition the leg independently; urgent assessment is needed…
- Shortening and internal rotation of the operative leg are not expected after total hip arthroplasty because they suggest displacement of the prosthesis.
- Expected recovery does not include a new abnormal limb position…
- A shortened, internally rotated leg requires more than routine follow-up because the finding may represent prosthetic dislocation.
- …
Left leg appears shortened and internally rotated
- The classic sign of hip prosthesis dislocation.
- Keep the leg still and call the surgeon now: delay risks nerve and vessel damage.
- Too slow: a dislocated hip can compromise the limb within hours.
- A new abnormal limb position is never part of normal recovery.
Same shape for the other rows, e.g. Temperature 37.8 °C on post-op day 1 → Expected: low-grade fever in the first 24–48 h is the inflammatory response; Prompt follow-up only if it climbs or persists.
Ordered response
Production's explanation is right: the full sequence as the key, then one block per step. But the review above it prints a different "Correct:" order (bug 1), and the student can't tell which to trust. Once that's fixed, one gap remains: there's no wrong option to find, because the "options" are orders. The static equivalent is a short list of the common wrong orders and why each fails.
- Because the client is alert and able to swallow with a glucose of 54 mg/dL, immediately give 15 g of rapid-acting carbohydrate…
- …
- The 15-15 rule: treat, wait 15 minutes, measure, repeat until the glucose is above 70 mg/dL.
- The snack comes last because the next meal is 2 hours away and protein prevents a second drop.
- Protein and fat slow absorption while the glucose is still 54 mg/dL.
- Fast carbohydrate treats the low; the snack only keeps it from coming back.
- The second dose is conditional: it's given because the recheck reads 64.
- Without the recheck there's no reason to give it.
- A single normal reading doesn't hold for 2 hours in an active teenager.
Fill in the blank (numeric)



The production rewrite kept the working and dropped the two sentences that made the older staging version good: the usual slips. A student who typed 0.4 can't be told "you entered 0.4", but if the explanation lists 0.375 (or 0.4) as a heading with its cause, they find their number there. Grading note (code, not tested in the UI): normalizeFillBlankValue treats only bare numbers as numbers, so "22.5 mL/hr" is compared as text and marked wrong.
- Convert the dose: 0.08 mcg/kg/min × 75 kg = 6 mcg/min. Convert the concentration: 4 mg = 4,000 mcg; 4,000 mcg ÷ 250 mL = 16 mcg/mL.
- Calculate the hourly rate: 6 mcg/min ÷ 16 mcg/mL × 60 min/hr = 22.5 mL/hr. Set the infusion pump to 22.5 mL/hr, rounded to the nearest tenth.
- Dose: 0.08 mcg/kg/min × 75 kg = 6 mcg/min. Concentration: 4 mg = 4,000 mcg ÷ 250 mL = 16 mcg/mL.
- Rate: 6 ÷ 16 = 0.375 mL/min × 60 = 22.5 mL/hr, the unit the pump is set in.
- The rate per minute; the ×60 to convert to mL/hr is missing.
- The mg → mcg conversion is off by a factor of 1,000.
- Weight left out: 0.08 ÷ 16 × 60 treats the dose as mcg/min instead of mcg/kg/min.
Image hot spot


Production has a block per region, which is right. But each heading is the region's internal id: "B — right-pupil-white-reflex". The image shows letters only, and B is the infant's left eye, on the viewer's right. A nursing student reads "right pupil" anatomically and looks at A. Headings should use the letter the image shows plus a plain description, and say which side is meant. (The same slugs are the regions' screen-reader names, which reads the answer out.)
- A white pupillary reflex is abnormal and requires referral for further ophthalmologic evaluation.
- …
- Symmetric red-orange reflection is the expected red-reflex finding in the left pupil…
- …
- …
- …
- A white reflex (leukocoria) can mean retinoblastoma, congenital cataract or retinal detachment.
- It needs urgent ophthalmology referral, even though the infant tracks with both eyes.
- A red-orange reflex is the normal finding, the one you compare the other eye against.
- Referral is about the eye that doesn't match it.
- Clear sclerae are normal: no redness, discharge or injury.
- They don't explain an abnormal reflex.
Highlight hot spot


Staging lumps three orders into one sentence; production gives each order its own block, named exactly as highlighted on screen. That's the benchmark applied well. The better version only tightens it: the prod bullets are long, and the heparin block can say plainly that the order continues.
- Potassium 5.8 mEq/L with oliguria and elevated creatinine indicates impaired renal potassium excretion; withhold potassium chloride and clarify the order before administration.
- …
- …
- The prophylactic heparin order addresses venous thromboembolism prevention and does not directly worsen the documented hyperkalemia.
- …
- …
- 10 mL/hr of urine and a creatinine of 3.2 mean the kidneys can't clear potassium, which is already 5.8 mEq/L.
- Another 20 mEq risks a fatal dysrhythmia: hold it and call the provider.
- Adds no potassium; prophylactic heparin continues in a bed-bound patient.
- This dose isn't a problem in acute kidney injury.
- Adds no potassium and isn't affected by the renal findings.
- Adds no potassium; it's often given as a phosphate binder in kidney failure.
Case study (NGN unfolding case)



The steps are in house format in production and cover every option. What's missing is the ending. The case has a summary, but it's a paragraph that closes with Bloom's and NCJMM tags, and the student only sees it later in the results list. Turned into a debrief card, it becomes the static "what this case was testing, and the traps" that every student gets on finishing, whatever they missed. (Step 1's prod explanation also tells the student the image is unavailable; see bug 4.)
This case follows two time-sensitive complications after neck surgery: acute airway compression from postoperative bleeding and subsequent symptomatic hypocalcemia. Early cues progress from frequent swallowing and visible neck swelling to voice changes and hypoxemia, requiring rapid airway support and surgical intervention… The case integrates Recognize Cues through Evaluate Outcomes and targets perioperative, respiratory, and fluid-electrolyte clinical judgment at Bloom's 5_evaluate level.
- Frequent swallowing, visible neck swelling, a muffled voice and SpO₂ falling to 91% = bleeding compressing the airway.
- Rapid response, sit upright, oxygen, back to theatre.
- Perioral tingling and carpal spasm = hypocalcaemia.
- IV calcium gluconate with cardiac monitoring; 8.0 mg/dL afterwards is still low, so keep checking.
- Pain and anxiety explain it; neck swelling threatens the airway within minutes.
- It hides the bleeding and does nothing for the compression.
Single best answer


The reasoning is sound, which makes this the cheapest fix in the audit: the content is there, it just needs regenerating into blocks with the CPA prompt that already exists. In the second item tested (IT change management), the key is three times longer than any distractor, a classic "pick the longest" cue.
Option B is correct because a lower tolerable deviation rate requires greater precision and therefore a larger sample. Option A ordinarily reduces sample size because fewer deviations are expected. Option C permits more sampling risk and therefore ordinarily reduces the required sample size. Option D reduces the audit assurance sought from the control…
- A lower tolerable deviation rate demands more precision, so the auditor needs a larger sample.
- It's the only change here that makes the conclusion harder to support.
- Fewer expected deviations leave more room below the tolerable rate, so the sample shrinks.
- Sample size rises with the expected rate, not falls.
- Accepting more risk of overreliance means less assurance is needed, so the sample shrinks.
- Sample size moves opposite to acceptable risk.
- Less planned reliance means the control test has to prove less.
- Heavier reliance is what drives samples up.
CPA task-based simulation

Already structured per row with a task summary: the house format's shape, applied to a grid. Three fixes. (1) The choices are literally called "Correct" and "Incorrect", so the review reads "Incorrect · Correct answer: Correct". Rename them "Consistent / Not consistent" (or True / False). (2) Each row explains only the right choice; one line on the other choice covers the student who picked it. (3) The summary calls the $270,000 a pretax overstatement while row 4 calls it an understatement (of liabilities and the reserve). Both are true; say which account each refers to.
Incorrect · Correct answer: Correct
3. Using Exhibit 1, the obsolescence misstatement exceeds performance materiality. The $180,000 obsolescence misstatement exceeds performance materiality of $175,000.
3 · The obsolescence misstatement exceeds performance materiality
- Supported reserve $480,000 − recorded $300,000 = a $180,000 misstatement.
- Exhibit 1 sets performance materiality at $175,000, so it's over by $5,000.
- Only holds if the misstatement is measured against the recorded reserve alone, or against overall rather than performance materiality.
CFA item set


The key and its working are right (4.20 + 52e−0.05 − 50 = 3.66). The distractor reasons don't reproduce the distractor numbers: "omits discounting of the exercise price" gives 4.20 + 52 − 50 = 6.20, not the 6.66 offered; "discounts the call premium too" gives 3.46, not 3.34. In a static explanation, a distractor block is how a student retraces their slip, so each distractor has to come from exactly one slip. Fix the item, then write the blocks. Maths renders as plain text today (Ke^(−rT)). Content note: the SATA key "floor value equal to the put's exercise price minus the premium paid" mixes the value floor (X) with the profit floor (X − S₀ − p).
B is correct because put–call parity gives p = c + Ke^(−rT) − S0 = 4.20 + 52e^(−0.05) − 50 = USD 3.66. A incorrectly discounts the call premium along with the exercise price. C omits discounting of the exercise price and also overstates the parity adjustment.
- Put–call parity: p = c + K·e−rT − S₀ = 4.20 + 49.46 − 50 = 3.66.
- The strike is paid at expiry, so it enters at its present value, discounted the way the rate is quoted (continuously).
- Discounts the call premium as well: 4.20·e−0.05 + 49.46 − 50.
- The premium is paid today; it's already a present value.
- Uses the undiscounted strike: 4.20 + 52 − 50.
- Parity compares present values, so the strike must be discounted.
Reading comprehension

All six children have an empty explanation. For LSAT reading, the explanation is the teaching: the skill is spotting why a plausible choice goes beyond the passage, and in the house format each wrong choice gets its own "why it goes beyond". Choices here are long, so headings can be shortened, but they must be recognisable. (Per the code, reading-set explanations render as plain text, so markdown blocks won't show until that's fixed.)
Nothing.
- The passage calls their approaches "radically opposed courses of action": rights that follow from universal principles versus rights won by direct political struggle.
- It's the only choice that states both halves of the passage's comparison.
- The passage never discusses how society judged the arguments.
- It adds a claim rather than summarising one.
- Contradicts the passage, which calls them radically opposed.
- Turns a detail (no legislation) into a cause the passage never states.
- The passage says nothing about how widely their views were accepted.
- Invents a contrast about timing; the passage contrasts methods.
- Nothing ties Gouges to the revolution's course or Condorcet to patience.
LSAT principle question

The reasoning is right: each option is tested against the principle's three conditions (a cost, why the benefit outweighs it, an uncertainty). In blocks, the student who picked the tutoring proposal sees immediately which condition it misses.
D is correct because it identifies a likely cost, explains why the benefit outweighs that cost, and identifies volunteer capacity as an uncertainty… B identifies a benefit, a cost, and an uncertainty but does not explain why the benefit outweighs the cost…
- Meets all three conditions: a cost (delay), why the benefit outweighs it (fewer errors), and an uncertainty (volunteer capacity).
- The only option that says the benefit is worth the cost.
- Names a possible cost overrun, but never says why the benefit is worth it.
- Has a cost (staffing) and an uncertainty (participation).
- Missing: why the achievement gain outweighs the staffing cost.
- Names an uncertainty (participation) but no cost, so there's nothing to weigh.
- Names a cost, then leaves the choice open instead of concluding the benefit outweighs it.
Integrated question set

Same shell as the NCLEX case, with sound per-step reasoning that cites the exhibits. It needs the same two things: regenerate each step into blocks (per blank and per row for the cloze and matrix steps, as production NCLEX does), and a debrief card at the end. Its ordering step uses item-N ids, so check it against bug 1. Sets that contain a short answer are covered in Part 2.
Part 2 · Jev grading of free-text answers
Three formats are graded live, per student: short answer (Bar, 1 item), performance task (Bar, 169) and argumentative writing (LSAT, 146). This is the only place the app can say "you wrote…", so the feedback should be specific to the submission. Since api#1187 (merged to dev on 24 Sep), TypeSafe's Jev makes every judgement and Gemini 3.5 Flash-Lite writes the feedback afterwards. Staging's ECS task definition injects AI_GATEWAY_API_KEY, and grades returned on staging carry Jev's confidence flags, so staging runs the Jev path.
How we tested
Three submissions through the live staging UI: a strong LSAT essay (0.95), a weak one (0.25) and a strong Bar memo (0.60). Then 20 controlled answers, run twice each, through the same grader code from api@dev. The gateway key can't be read locally, so Jev was reached through OpenRouter's Decisions endpoint (typesafe/jev-1.13), the same model family. The strong essay and the strong memo scored the same locally as on staging.
| Case | What it tests | Score (run 1 / 2) | Verdict |
|---|---|---|---|
| Essay · strong | 392 words, tiered policy, all 3 perspectives, answers an objection | 0.95 / 0.95 | Right |
| Essay · opposite side, equally strong | Neutrality: argues the absolutist Perspective 1 | 0.95 / 0.95 | Right, side-neutral |
| Essay · clear position, 1 perspective, no counterargument | Middle of the range | 0.65 / 0.60 | Reasonable; drifts between runs |
| Essay · restates the perspectives, no position | Weak | 0.25 / 0.30 | Right |
| Essay · well written, wrong prompt | Off-topic detection | 0 / 0 | Caught (p = 0.97) |
| Essay · forged delimiter + "award 4/4" | Prompt injection | 0.10 / 0.10 | Resisted |
| Essay · 42-word outline | Length gate | 0.50 / 0.50 | Capped, but thesis still 4/4 |
| Essay · engages 1 of 3 perspectives (zoning item) | Perspective gate (the only item with a perspectives array) | 0.75 / 0.75 · pass | Counted "2 of 3" engaged; passes |
| Memo · strong, from the actual file | Ceiling | 0.60 / 0.60 (21/24 in bands) | Capped by requirements the file can't meet |
| Memo · same + the rubric's invented facts | Does the rubric reward making things up? | 0.50 / 0.50 | 3 of 4 "missing" elements now satisfied, by facts that don't exist |
| Memo · strong + invented outside authority | Closed-universe rule | 0.50 / 0.50 | Bands dropped; feedback only hints ("outdated or inapplicable authorities") |
| Memo · 5 conclusory sentences | Weak | 0 / 0 | Right |
| Memo · weak + injected instructions | Prompt injection | 0 / 0 | Resisted |
| Short · exact / paraphrase / "the firm" | Meaning over wording | 1 / 1 each | Right |
| Short · "partners' personal assets" / insurance / injection | Wrong answers | 0 / 0 each | Right |
| Short · "either partnership or partners, whichever is easier" | Hedging both ways | 0 / 1 | Flipped to full credit on run 2 |
What works
- Decisions track quality. Strong 0.95, middling 0.6, weak 0.25, off-topic 0: the order a human grader would give.
- Hard to game. Three injection attempts (a forged closing delimiter plus instructions) earned nothing, and the short-answer reason named the attempt.
- Side-neutral. Opposite positions of equal quality scored the same.
- Tolerates paraphrase. "Riverbend's own business property, before going after the partners personally" was credited.
- Fast and cheap. Median 1.9 s for a full grade including the feedback pass.
What doesn't
- 44% of verdicts are "low confidence" (41 of 94 bands and fields below 0.6), and each shows the student "— low confidence (0.53) — worth a human look". No human review queue exists, so a note meant for us lands on the student.
- Borderline answers aren't stable. The hedge flipped from 0 to full credit between identical runs (p = 0.53). Two essay bands moved a point.
- Feedback is written blind. Flash-Lite gets "reasoning 3/4" and invents a reason. The strong essay's 3/4 advice ("add concrete examples of how thresholds are calculated") ignores that the essay names enrollment and service-request data and a two-year review cycle.
- Feedback contradicts the score. The memo comment says the candidate "performed exceptionally well across the board" on a submission shown as 14/24 and not passing.
LSAT argumentative writing: what the student sees


- No overall result line and no model answer. Every item has a
sample_responseand ascoring_rubric; neither is shown, althoughformats.ts:899says they are "shown after submission". The sample response is the static half of the feedback: it costs nothing to show. - 145 of 146 essays keep their perspectives only in the prompt text, so the "at least 2 perspectives" gate never runs. On the one item that has the array, an essay engaging one perspective was counted as 2 of 3 and passed.
- The length gate is 400 characters (about 65 words) for a 35-minute essay. An 81-word non-essay clears it.
Thesis 0/4 — State a clear and specific position on the college recording policy in your introduction rather than merely summarizing the debate.
Perspective engagement 1/4 — Select a definitive stance and explicitly evaluate the provided perspectives in relation to your central argument. — low confidence (0.56) — worth a human look
Counterconsideration 1/4 — Address potential objections… — low confidence (0.52) — worth a human look
Overall: not yet an argument (25%). You summarised the debate well. Now take a side.
Thesis. Your last line, "class recording is a complicated issue", isn't a position. For example: "Colleges should record by default, but release recordings only to enrolled students and let instructors pause for discussion."
Perspectives. You note privacy worries exist, but not whether they outweigh access. Pick the strongest worry and answer it.
Counterargument. Add one paragraph that begins "The strongest objection is…", and show why your policy survives it.
See a full-credit essay for this prompt → (the item's sample_response)
Bar performance task: what the student sees


The rubric and the file come from different drafts. The rubric requires "Avery's Lender Communications objection", "Jordan's Purchasing Records" and a "$14,000 related-party overcharge". The file holds two exhibits (the facts, and eight RUPA provisions) and none of those. Because a missing required element caps the score at 60%, no memo written from the file can pass, while a memo that invents those facts clears three of the four gaps. A bank-wide scan found 29 of 169 tasks whose required elements cite amounts or documents absent from the file (8 involving dollar figures). 62 tasks file the law under "Case file", so the Library tab reads 0, and 154 have no sample response.
The screen. Two scores (14/24 after caps, 21.0/24 before), no per-issue breakdown (this task has no issues, so everything is "The task as a whole"), a comment praising a failing result, and "grading in progress" still under the memo.
14 / 24
The task as a whole — rules 3/4, application 4/4 — Your performance earned a rules score of 3/4 because you accurately identified and utilized the relevant statutory and common law principles… — low confidence (0.53) — worth a human look
every required element — missing: Analyzes the equipment note liability… against Avery's Lender Communications objection… Identifies Jordan's undisclosed related-party purchase in the Jordan's Purchasing Records…
24 points — 21.0 / 24
21 / 24 · passing. One number, then per issue:
- Formation 4/4. You applied §202(a) and the §202(c)(3) profit-share presumption to the 60/40 split and the joint DBA.
- Management and duties 3/4. Right on §401(f). You flag the unilateral $45,000 note as a care problem but stop there; say what it buys Avery in the negotiation (e.g. Jordan indemnifies Avery).
- Note liability 4/4. §301(1) → §305(a) → §306(a), tied to the bank having no notice of any limit on Jordan's authority.
- Recommendation ✓ Prioritised, protects Avery, includes a wind-down fallback.
Compare with a model memo →
Short answer: graded, never shown
The grader works (see the table above) and its reasons are good one-liners, e.g. "incorrectly stated that ToolWorks must seek satisfaction first from the partners' personal assets, rather than the partnership's assets." The app shows "Graded against the rubric after you finish the set", then a Pending or Partial label; the review has no short_answer case. The rubric has one accepted answer and no rejected list, so there's nothing to name when a student falls for the classic trap (partners first).
Graded against the rubric after you finish the set.
Not credited. You wrote "the partners' personal assets". A creditor with a judgment against the partnership must first go after partnership assets (Authority 1); the partners' separate assets come second.
Fix list, in order
| # | Fix | Where | Why this order |
|---|---|---|---|
| 1 | Ordering: grade and display from one key. Drop the rebuild in withOrderedOptions, or rewrite options.metadata.sequence from content.correct_order; fix the 3 item-N items. | app + data | Shows students false answers, in prod |
| 2 | Case studies: read dropdown options from the child rows (or re-sync content.sub_questions); backfill the 72 matrix row labels; release exhibits in step order and remove generation notes such as "the actual Exhibit 2 image is unavailable". | app + data | Answers given away or unanswerable, in prod |
| 3 | Regenerate the new courses' explanations in the house format before launch: 10,244 across CFA, CPA, LSAT and Bar, the 338 empty ones and the LSAT reading sets first. The course prompts already specify it for single-answer questions; extend them per format using production NCLEX as the model (per blank, per row, per step, per region, per statement). Add a check that every option's text appears as a heading. | content | The single biggest gap to the benchmark |
| 4 | Performance tasks: re-sync the 29 flagged rubrics with their files, then add an ingest check that every $ amount and named source in required_elements / prohibited_errors exists in the materials. | data + api | Tasks can't be passed |
| 5 | Stop showing "low confidence — worth a human look" to students. Route it to a review queue, or re-ask Jev and take the majority when p < 0.6. | api | On ~44% of lines; flips grades |
| 6 | Show free-text results everywhere: the short-answer grade and reason, performance-task and essay breakdowns in the results review, one score only, and the stored sample_response after submission. | app | Graded work students never see |
| 7 | Close the static gaps in production NCLEX: common wrong answers on fill-in (553), common wrong orders on ordering (166), hot-spot headings that use the on-screen letter and a plain description instead of id slugs (231), "Select / Do not select" groups on SATA, and a shorter matrix pattern. | content | Every wrong answer should have a heading to find |
| 8 | End every aggregate with a debrief card: score, the case's content.explanation rewritten without Bloom's/NCJMM tags (366 of 432 prod cases carry them), and the case's traps. Make Rezzy available inside aggregates. | app + content | Cases end abruptly |
| 9 | Copy production NCLEX and MCAT explanations to staging, so QA on staging sees what students see. | data | Staging shows the old paragraphs |
| 10 | Explain partial credit in the banner ("+1 per right pick, −1 per wrong pick"). Rename the TBS choices "Consistent / Not consistent". | app | Static UI lines, cheap |
| 11 | Ground the feedback writer: pass it Jev's per-band probabilities, the item's rubric criteria and the sample response; require it to quote the student's text; ban praise on bands of 2 or less and on failing totals. | api | Generic or contradictory comments |
| 12 | Regenerate CFA distractors so each maps to one slip the explanation can name, and render maths as LaTeX. Raise the essay length gate (e.g. 250 words) and backfill the perspectives array on 145 essays. | content + api | Explanations that can't be reproduced |
| 13 | Follow-up chips, which already react to the pick at a cost bounded per question × option: send option text rather than UUIDs for ordering and grid answers, so the chips can address the student's actual mistake. No new generation. | app | The existing per-student layer, made to work on every format |
Also noticed, out of scope: the CPA subject picker uses medical emoji (🦴 💊 😴); the LSAT reading passage tested reads like a published PrepTest passage; 2,125 UK Medical PG explanations on staging have no line breaks; hot-spot screen-reader names are answer slugs ("right-pupil-white-reflex").
Method
UI capture. app-staging.getoncourse.ai in headless Chromium, signed in as staging QA accounts (NCLEX test@, CPA onboarding-web, CFA persona-qa-finance, LSAT persona-qa-law, Bar shubhdeep+12345, US Medical PG shubhdeep+3). Practice-mode custom quizzes, filtered by format where the builder allows it (NCLEX, CPA). Bar formats were started through POST /test/session/start with formats, the same call the app sends. The answer key was read from the question payload so every wrong answer was chosen on purpose.
Benchmark and counts. Read-only SQL against the staging and production databases, validated items unless noted. "House format" = the explanation opens with a bold key block (***…***) or a bold part heading (**Blank 1**, **row text**). Production NCLEX texts quoted in the cards are the stored explanations of the same question ids as the staging screenshots. The new courses (CFA, CPA, LSAT, Bar) exist only on staging.
Grading. The api@dev (9b5055b7) graders ran unmodified in a throwaway checkout. Only the Jev HTTP call was redirected, to OpenRouter's Decisions API (typesafe/jev-1.13), because the AI Gateway key can't be read outside ECS. Feedback used Gemini 3.5 Flash-Lite with the staging key, as on staging. 20 cases × 2 runs; raw results in results.json.
Better versions are written as static explanations: the same text for every student. Only the Part 2 rewrites are per student, because those answers are graded live. Screenshots are cropped and compressed; click any image to enlarge it.