Question explanation audit

Every question format students can reach on staging, answered right and wrong, judged against the explanations in our NEET PG and USMLE banks. Objective questions get one stored explanation that every student sees, so the bar is: does that one explanation answer, in advance, every student who picked any option? Part 2 audits the Jev grading of free-text answers, the one place where feedback is written for each student.

Tested 25–26 Sep 2026 on app-staging.getoncourse.ai (web)Courses: NCLEX-RN · CPA · CFA · LSAT · Bar, benchmarked against Indian and US Medical PGCounts from the staging DB, plus production where the course is live

Summary

Explanations for objective questions are written once and stored; the app can't generate a new one for each student's answer. So "you picked C, here's why that's wrong" isn't available. The static equivalent is what our medical banks already do: one block per option, headed by that option's own text, so a student who picked anything finds their answer's block and reads why. Production NCLEX has been rewritten into that format. The four new courses have not: none of their 10,244 live explanations use it, although each course's generator prompt already asks for it.

0 / 10,244live CFA, CPA, LSAT and Bar explanations use the house format, against 96–100% of NEET PG, USMLE and UK PG. 338 of them are empty (Bar 205, LSAT 94, CPA 39).
90%of production NCLEX explanations (5,067 of 5,642) already use it, adapted per format. Staging still carries the older paragraph text for all 5,643, so staging QA isn't seeing what students see. MCAT is the same (prod 92%, staging 0).
39 / 45live standalone NCLEX ordering questions show the wrong "Correct:" order in review (36) or score a perfect answer 0 (3). Production. The explanation is right; the screen around it contradicts it.
29live NCLEX case studies (53 dropdowns) offer only the correct choice. Production.
72matrix items label their rows row-1…row-4 instead of the findings, so they can't be answered.
29 / 169Bar performance tasks require facts their own file doesn't contain. On the one tested, a 21/24 memo is capped at 60% and fails.
What a good stored explanation does, given that it can't react to the student.
  1. The key first, in bold italics, headed by its own text, with two bullets on why.
  2. Every other option gets its own block, headed by its text exactly as the screen shows it, with two bullets on why it's wrong. This is how the explanation "answers the student's pick": whatever they chose, there's a heading with those words.
  3. Multi-part formats repeat the pattern per part: per blank, per row, per step, per region. Numeric answers list the common wrong answers and the slip behind each; ordering lists the common wrong orders.
  4. No letters (they vanish behind ✓/✗ icons and don't exist on SATA screens), no authoring metadata, bold on the key terms only.
The per-student layer already exists and needs no new generation: the Rezzy follow-up chips under each explanation are generated from the student's chosen option and cached per question and answer (below). Free-text answers are graded live by Jev, so that's where "you wrote…" feedback belongs (Part 2).

The benchmark: NEET PG and USMLE explanations

Captured on staging as a US Medical PG student, answering a USMLE Step 1 vasculitis question wrong on purpose.

Picked "Extrinsic asthma"; the key is EGPA
The key's block, then one block per distractor. "Extrinsic asthma" has its own heading
Every remaining option gets two bullets
Follow-up chips, generated from the wrong pick: "Help me distinguish EGPA from severe eosinophilic asthma…"

Anatomy of one

1Invasive strain is most common
  • Haemophilus influenzae has invasive strains, particularly type b (Hib), which are significant causes of bacterial infections like meningitis and epiglottitis.
  • These invasive strains are prevalent in young children and contribute to serious morbidity if not properly managed.
2Gram positive
  • Haemophilus influenzae is actually Gram negative, which is a crucial distinction for identifying and treating infections.
  • Gram-negative bacteria are characterized by their thin peptidoglycan layer and presence of an outer membrane.
It is not capsulated
  • Certain strains, particularly type b, have a robust polysaccharide capsule that plays a crucial role in virulence.
  • …
Grown on sheep blood agar and CO2
  • Haemophilus influenzae requires chocolate agar or enriched media rather than standard sheep blood agar… 3
  • …

NEET PG, "True about Haemophilus influenzae is:" (c1ebf59b), as stored.

  1. 1Key first, in bold italics, using the option's words, not "Option A".
  2. 2Each distractor under its own italic heading, in screen order. A student who picked "Gram positive" reads exactly one block to find out why.
  3. 3Two bullets each, bold on the discriminating fact ("chocolate agar"), not on whole sentences.
  4. Nothing else. No preamble, no subject or Bloom's tags, no letters.

The generator prompts for CFA, CPA, Bar and LSAT (src/prompts/library/courses/<course>/question-explanation-generation.md) already specify this format for single-answer questions, but the stored explanations in those banks weren't produced with them.

How widely the format is used

Share of validated explanations that open with a bolded key block (***…***, or **Blank 1** / **row** for multi-part formats).

CourseStagingProductionNote
Indian Medical PG9,823 / 10,245 (96%)74,101 / 74,271 (99.8%)The benchmark
US Medical PG9,764 / 10,230 (95%)11,741 / 11,744
UK Medical PG4,975 / 4,9764,625 / 4,625On staging, 2,125 are stored with no line breaks, so blocks run together. Prod has 6.
NCLEX-RN0 / 5,6435,067 / 5,642 (90%)Prod rewritten per format (examples in each card below). The rest are 432 case-study summaries still in paragraph form. Staging is stale.
MCAT0 / 2,6812,478 / 2,681 (92%)Staging is stale
CFA0 / 6,612Not live4,573 refer to options by letter
Bar Exam0 / 1,591Not live205 empty
CPA0 / 1,214Not live39 empty
LSAT0 / 827Not live94 empty, including whole reading sets

The one answer-aware layer we already have: follow-up chips

After every answer the app calls POST /chat/follow-ups with the question and the chosen option. The API builds the context from the question and the pick, generates three chips, and caches them per question × answer (explanationGenerator.generateStandaloneFollowUps). So the cost is bounded by the number of options, not the number of students. In the capture above, picking "Extrinsic asthma" produced "Help me distinguish EGPA from severe eosinophilic asthma using clinical clues and ANCA findings". One tap turns that into a Rezzy answer about exactly the student's mistake.

It already fires for every format, including ordering. One gap: for ordering and grid answers the app sends option UUIDs as the "answer" ("1. 4f1841a8-… 2. ad570295-…" in one capture), so the model can't read what the student did. Sending the option text fixes it without any new generation.

Scoreboard

One row per format. "House format" asks whether the explanation has a block for every option, headed the way the screen shows it. For NCLEX, the staging text (what the screenshots show) and production text differ, so both are listed.

FormatTested inKey explainedEvery option explainedHouse formatVerdict
Select all (SATA)NCLEXYesYesStaging no (letters the screen hides) · Prod yesGood in prod
Cloze dropdownNCLEXYesYesStaging paragraph · Prod per blankGood in prod
MatrixNCLEXYesYesStaging paragraph · Prod per row, longTrim; row-label bug
Ordered responseNCLEXYesNo wrong orders coveredProd per stepScreen contradicts it
Fill in the blankNCLEX, CFAWorked solutionStaging names slips · Prod dropped themProd key onlyAdd wrong answers
Image hot spotNCLEXYesStaging lumped · Prod per regionProd headings are id slugsNeeds work
Highlight hot spotNCLEXYesStaging one line · Prod per orderProd yesGood in prod
Case study (NGN)NCLEXPer stepPer stepSteps yes · case summary paragraph + Bloom'sContent bugs
Single best answerCPA, LSAT, BarYesYesNo: "Option B is correct because…"Regenerate
CPA TBS gridCPAPer rowKey only per rowPer row, but "Correct answer: Correct"Best of the new courses
CFA item setCFA L1With workingDistractor maths don't reproduceNoFix items, regenerate
Reading comprehensionLSATNothingNothingEmptyMissing
LSAT MCQLSATYesYesNo: by letterRegenerate
Integrated question setBarPer stepPer stepNo: by letterRegenerate
Short answerBar (1 item)Live-graded. Jev grades it and writes a one-line reason; the student never sees either.Invisible
Performance taskBarLive-graded. One generic comment, two contradictory scores, rubrics that demand facts the file doesn't contain.Broken
Argumentative writingLSATLive-graded. Sensible bands; generic comments; no model answer; "worth a human look" notes shown to students.Needs work
Constructed responseCFA (338, none validated)No grader exists. The app shows "Scoring in progress" and never a result.Not shippable

Not captured: data-entry grids with numeric cells (the CPA simulation tested was a Correct/Incorrect grid), and MCAT, which only has single-answer MCQs.

Blocking bugs found during the audit

These go beyond explanation quality: the app tells students something false, or the question can't be answered. No rewrite fixes them.

1 · Ordered response: the review shows a different "correct" order from the one the server grades Prod · 39 of 45 standalone

The client rebuilds correct_order from the legacy options rows' metadata.sequence (withOrderedOptions, apis/question/formats.ts:563). The server grades against content.correct_order. On 36 of the 45 live standalone items they disagree, so the review marks right positions red and prints the wrong "Correct: …" under each, directly above an explanation that gives the true order. On 3 more, content.correct_order uses item-N ids while the app submits option UUIDs, so a perfect order scores "Incorrect 0/5" with every row green. Counting case-study children: 155 of 166 validated ordering questions in production.

Ordering review with wrong key
Wrong order: pill 2/5, one green row, "Correct:" lines that aren't the key
Correct ordering marked incorrect
The exact right order: every row green, banner "Incorrect 0/5"

2 · Case studies give away the dropdown answer Prod · 29 cases, 53 blanks

The case player reads the parent's denormalised content.sub_questions copy, which kept only the correct option per blank. The child rows still have all three. The student opens a dropdown with one choice in it.

3 · Matrix rows labelled row-1 … row-4 72 items, all live

The row text never made it into row_headers, so the student classifies "row-1" as improvement or follow-up. The explanation's per-row headings use the real findings (tingling, neck circumference, a calcium of 8.0), which never appeared on screen.

4 · Case timelines leak or run ahead, and the explanation admits it Content · prod

In the thyroidectomy case, step 1 asks about "neck fullness shown in the image", but only Exhibit 1 is released: a normal baseline with no image. The production explanation for that step says so to the student: "Because the actual Exhibit 2 image is unavailable, its additional findings cannot be verified…", a generation note that shipped. Exhibit 3 then narrates the answers to steps 3 and 4 ("The nurse activates the rapid response system, raises the head of the bed, applies oxygen…") before the student answers them.

Step 1 cites Exhibit 2 and an image the student can't see yet
Step 3: one-choice dropdowns, and Exhibit 3 already states the action
Step 6: rows named row-1…row-4

5 · Free text: grades students never see, and grades that can't pass Bar, LSAT, CFA

  • Short answer (inside Bar sets): Jev grades it and writes a reason. The app shows "Graded against the rubric after you finish the set", and the results review has no short_answer case (FormatAnswerReview.tsx:690).
  • Performance task: the grade lives only in component state. The results review shows the author's model-answer paragraph and nothing about the student's memo.
  • Performance-task rubrics cite evidence the file doesn't contain (Part 2). A missing required element caps the score at 60%, so affected tasks can't be passed.
  • Constructed response (CFA, 338 items): no server grader and no result UI. Harmless while unvalidated; it blocks launch.

Patterns across formats

New courses explain by letter, in one paragraph

CPA, CFA, LSAT and Bar explanations read "Option B is correct because… Option A ordinarily… Option C permits…" (4,573 CFA, 824 CPA, 552 Bar, 455 LSAT). A student has to translate letters back into options, and after answering, the ✓/✗ icons replace the letters of the picked and correct options. In the house format each option is findable by its own words.

Staging isn't production for NCLEX and MCAT

Production NCLEX was rewritten into the house format (5,067 of 5,642); staging still has the older paragraphs, with the letters, metadata openers and missing per-option reasons this audit first flagged. Anyone QA-ing explanations on staging sees text students don't. The rewrite script isn't in api@dev, so the new courses can't simply rerun it.

Authoring metadata printed to students

Openers and closers like "This item targets Medical-Surgical Nursing, Cardiovascular Disorders, Bloom's 3_apply level, medium difficulty, and the NCJMM Generate Solutions skill." In production they survive mostly in the case-study summaries: 366 of 432 end with "The case integrates Recognize Cues through Evaluate Outcomes… at Bloom's 5_evaluate level." On staging they open 2,891 NCLEX explanations.

Multi-part formats stop at the key

The house format covers wrong options well when they're listed. Where they aren't, production NCLEX drops them: fill-in explanations give only the worked answer (the older staging text named the two usual slips), and ordering explanations list the right sequence but no wrong one. For these formats, "every option" means the common wrong answers and orders, written out.

Aggregates end without a debrief

Case studies, integrated sets and reading sets show a rationale per step, then "Finish case" jumps to the next case: no case score, no summary. The case summary exists in content.explanation but only appears later in the results list. Rezzy isn't available inside aggregates.

Empty explanations in the new courses

Bar 205, LSAT 94 and CPA 39 live items have no explanation. For LSAT that includes whole reading-comprehension sets: red and green, nothing else. Separately, the LSAT passage tested (Condorcet and Olympe de Gouges) reads like a published LSAT PrepTest passage; worth a licensing check.

Format by format

Screenshots are from staging. For NCLEX, the text students actually get is production's, so each NCLEX card compares production today with a better version. The better versions are static: one explanation that covers every pick, rendered the way the app renders markdown (bold-italic heading for the key, italic headings for the rest).

Select all that apply

NCLEX-RN · warfarin discharge teaching (4034f181)
Good in prod
Staging, wrong (picked A, B, E): "Partially correct 1/4", explanation in letters
Staging, correct: same kind of paragraph, opening with metadata
Key explainedYes, all 4
Every option explainedYes
House formatProd yes · staging no
Scoring explainedNo: "1/4" from two right picks

Production already gives each of the six statements its own block. Two small gaps. On a select-all screen several options are correct, and the only signal separating "select" from "don't select" is bold italics versus italics; a group label makes it unmissable. And "1/4" with two right picks is never explained: select-all scores +1 per right pick and −1 per wrong pick. That's a fixed line in the result banner, not explanation text.

Production today
Attend scheduled appointments for international normalized ratio testing
  • Scheduled INR testing evaluates warfarin's anticoagulant effect and supports safe dose management after discharge.
  • …
Use an electric razor when removing facial or body hair
  • …
Inform dentists and other healthcare professionals about the medication
  • …
Report black stools or red-colored urine to the provider promptly
  • …
Take twice the prescribed dose if one dose is missed
  • The client should not double a missed warfarin dose, because taking extra medication may increase anticoagulant effect and bleeding risk.
  • …
Use aspirin for minor headaches unless pain persists longer than 24 hours
  • …
Better version
Select
Attend scheduled appointments for international normalized ratio testing
  • The INR sets each warfarin dose; skipped tests hide drift toward bleeding or clotting.
  • It needs checking even when today's INR is in range: diet, illness and new drugs move it.
Use an electric razor when removing facial or body hair
  • With a mechanical valve, anticoagulation is lifelong, and blade nicks bleed longer.
  • Preventing small injuries is standard bleeding precaution teaching.
Inform dentists and other healthcare professionals about the medication
  • Procedures and new drugs (antibiotics, NSAIDs) change bleeding risk and the INR.
  • Clinicians can then plan around warfarin instead of discovering it mid-procedure.
Report black stools or red-colored urine to the provider promptly
  • Melena and hematuria are signs of internal bleeding.
  • Early reporting lets the provider check the INR before bleeding becomes serious.
Do not select
Take twice the prescribed dose if one dose is missed
  • A double dose can push the INR into the bleeding range.
  • The client follows the prescriber's missed-dose plan and never adjusts the dose alone.
Use aspirin for minor headaches unless pain persists longer than 24 hours
  • Aspirin adds an antiplatelet effect on top of warfarin.
  • Acetaminophen, cleared with the provider, is the usual choice.
Banner line (UI, all SATA): "Scored +1 for each right pick and −1 for each wrong pick."

Cloze dropdown

NCLEX-RN · cataract discharge teaching (8392b6bc)
Good in prod
Staging, wrong blank 1: red chip, key listed under the sentence
Staging, all three blanks right
Key explainedYes, per blank
Every option explainedYes, every dropdown choice
House formatProd per blank · staging paragraph
ReadableYes

This is the format adaptation the new courses should copy: a Blank n heading, then the right choice and every other choice in the dropdown, each with its reason. The only improvement is saying what each blank asks, so the student doesn't have to scroll back to remember what blank 1 was.

Production today

Blank 1

sleep on my nonoperative side
  • Sleeping on the nonoperative side avoids pressure on the left operative eye during early recovery…
  • …
bend at my waist to put on shoes
  • Bending at the waist is discouraged during early recovery because it can increase intraocular pressure…
  • …
lift a 9-kg bag of groceries
  • …

Blank 2

a sudden increase in flashes or floaters
  • …
mild itching around the covered eye
  • …
a small amount of tearing on the first day
  • …
Better version

Blank 1 · the statement that shows the teaching worked

sleep on my nonoperative side
  • Keeps pressure off the operated eye while it heals.
  • It's the only choice that lowers risk rather than raising it.
bend at my waist to put on shoes
  • Bending over raises intraocular pressure in the healing eye.
  • Teach sitting down and bringing the foot up instead.
lift a 9-kg bag of groceries
  • Heavy lifting strains and raises eye pressure in early recovery.
  • Anything above light objects waits for the surgeon's clearance.

Blank 2 · the symptom that needs immediate follow-up

a sudden increase in flashes or floaters
  • Can signal retinal detachment, a complication needing same-day care.
  • New or worsening visual symptoms are never "wait and see".
mild itching around the covered eye
  • Expected during healing; the client just mustn't rub the eye.
  • Only itching with pain, discharge or vision change needs a call.
a small amount of tearing on the first day
  • Expected on day one.
  • Heavy tearing, or tearing with pain or vision loss, would need follow-up.

Matrix

NCLEX-RN · post–hip arthroplasty findings (b38b4ed9)
Trim; row-label bug
Staging, 3/5: two rows red, the right column shown in green
Staging, all rows right
Key explainedYes, every row
Every option explainedYes, every column of every row
House formatProd per row
Length30 bullets for 5 rows

Production repeats the full pattern for every cell: 5 rows × 3 columns × 2 bullets. It's complete but long, and the wrong-column bullets mostly restate the right-column ones ("does not require prompt follow-up when relief is achieved"). Keep two bullets for the right column and one sharp line for each other column. Separately, 72 matrix items show row-1… on screen (bug 3), so the explanation's row headings are the only place the findings appear.

Production today (one of five rows)

Left leg appears shortened and internally rotated

Requires Immediate Action
  • Shortening and internal rotation strongly suggest possible hip prosthesis dislocation and require immediate evaluation.
  • Do not reposition the leg independently; urgent assessment is needed…
Expected
  • Shortening and internal rotation of the operative leg are not expected after total hip arthroplasty because they suggest displacement of the prosthesis.
  • Expected recovery does not include a new abnormal limb position…
Requires Prompt Follow-Up
  • A shortened, internally rotated leg requires more than routine follow-up because the finding may represent prosthetic dislocation.
  • …
Better version (same row)

Left leg appears shortened and internally rotated

Requires Immediate Action
  • The classic sign of hip prosthesis dislocation.
  • Keep the leg still and call the surgeon now: delay risks nerve and vessel damage.
Requires Prompt Follow-Up
  • Too slow: a dislocated hip can compromise the limb within hours.
Expected
  • A new abnormal limb position is never part of normal recovery.

Same shape for the other rows, e.g. Temperature 37.8 °C on post-op day 1 → Expected: low-grade fever in the first 24–48 h is the inflammatory response; Prompt follow-up only if it climbs or persists.

Ordered response

NCLEX-RN · hypoglycaemia in a teen athlete (1eb5ae6b)
Screen contradicts it

Production's explanation is right: the full sequence as the key, then one block per step. But the review above it prints a different "Correct:" order (bug 1), and the student can't tell which to trust. Once that's fixed, one gap remains: there's no wrong option to find, because the "options" are orders. The static equivalent is a short list of the common wrong orders and why each fails.

Production today
Give 15 g of a rapid-acting carbohydrate → Recheck the blood glucose level after 15 minutes → Give another 15 g … when the level is 64 mg/dL → Repeat the blood glucose measurement after another 15 minutes → Provide a snack containing carbohydrate and protein
1. Give 15 g of a rapid-acting carbohydrate
  • Because the client is alert and able to swallow with a glucose of 54 mg/dL, immediately give 15 g of rapid-acting carbohydrate…
2. Recheck the blood glucose level after 15 minutes
  • …
3 · 4 · 5 …
Better version
Give 15 g → Recheck in 15 min → Give another 15 g → Recheck again → Carbohydrate-and-protein snack
  • The 15-15 rule: treat, wait 15 minutes, measure, repeat until the glucose is above 70 mg/dL.
  • The snack comes last because the next meal is 2 hours away and protein prevents a second drop.
Common wrong orders
Starting with the snack
  • Protein and fat slow absorption while the glucose is still 54 mg/dL.
  • Fast carbohydrate treats the low; the snack only keeps it from coming back.
Giving the second 15 g before the first recheck
  • The second dose is conditional: it's given because the recheck reads 64.
  • Without the recheck there's no reason to give it.
Ending on a recheck, with no snack
  • A single normal reading doesn't hold for 2 hours in an active teenager.

Fill in the blank (numeric)

NCLEX-RN · norepinephrine drip rate (19e8d618) · CFA · present value of the strike (item set)
Add wrong answers
Staging: entered 0.4, "Incorrect, correct answer 22.5"
Staging: correct
CFA: 49.5 vs 49.46 marked wrong (the stem asks for two decimals)
Key explainedYes, full working
Wrong answers explainedStaging yes · prod no
House formatProd key only
ReadableYes

The production rewrite kept the working and dropped the two sentences that made the older staging version good: the usual slips. A student who typed 0.4 can't be told "you entered 0.4", but if the explanation lists 0.375 (or 0.4) as a heading with its cause, they find their number there. Grading note (code, not tested in the UI): normalizeFillBlankValue treats only bare numbers as numbers, so "22.5 mL/hr" is compared as text and marked wrong.

Production today
22.5
  • Convert the dose: 0.08 mcg/kg/min × 75 kg = 6 mcg/min. Convert the concentration: 4 mg = 4,000 mcg; 4,000 mcg ÷ 250 mL = 16 mcg/mL.
  • Calculate the hourly rate: 6 mcg/min ÷ 16 mcg/mL × 60 min/hr = 22.5 mL/hr. Set the infusion pump to 22.5 mL/hr, rounded to the nearest tenth.
Better version
22.5 mL/hr
  • Dose: 0.08 mcg/kg/min × 75 kg = 6 mcg/min. Concentration: 4 mg = 4,000 mcg ÷ 250 mL = 16 mcg/mL.
  • Rate: 6 ÷ 16 = 0.375 mL/min × 60 = 22.5 mL/hr, the unit the pump is set in.
Common wrong answers
0.375 or 0.4
  • The rate per minute; the ×60 to convert to mL/hr is missing.
0.0225 or 22,500
  • The mg → mcg conversion is off by a factor of 1,000.
0.3
  • Weight left out: 0.08 ÷ 16 × 60 treats the dose as mcg/min instead of mcg/kg/min.
Rule of thumb worth keeping in the stem or banner: enter the number only, in the unit asked for.

Image hot spot

NCLEX-RN · infant red-reflex screening (5378d6c6)
Needs work
Staging: picked A (red reflex); the key is B
Staging, another item: correct
Key explainedYes
Every region explainedProd yes · staging lumped
Named as on screenNo: id slugs
AnatomyConfusing: "right pupil"

Production has a block per region, which is right. But each heading is the region's internal id: "B — right-pupil-white-reflex". The image shows letters only, and B is the infant's left eye, on the viewer's right. A nursing student reads "right pupil" anatomically and looks at A. Headings should use the letter the image shows plus a plain description, and say which side is meant. (The same slugs are the regions' screen-reader names, which reads the answer out.)

Production today
B — right-pupil-white-reflex
  • A white pupillary reflex is abnormal and requires referral for further ophthalmologic evaluation.
  • …
A — left-pupil-red-reflex
  • Symmetric red-orange reflection is the expected red-reflex finding in the left pupil…
  • …
C — left-clear-sclera
  • …
D — right-clear-sclera
  • …
Better version
B · the white pupil (the infant's left eye, on your right)
  • A white reflex (leukocoria) can mean retinoblastoma, congenital cataract or retinal detachment.
  • It needs urgent ophthalmology referral, even though the infant tracks with both eyes.
A · the red-orange pupil
  • A red-orange reflex is the normal finding, the one you compare the other eye against.
  • Referral is about the eye that doesn't match it.
C and D · the clear whites of the eyes
  • Clear sclerae are normal: no redness, discharge or injury.
  • They don't explain an abnormal reflex.

Highlight hot spot

NCLEX-RN · withhold potassium in oliguria (6d5af6d7)
Good in prod
Staging: picked heparin. Explanation: "The other medication orders do not directly aggravate…". The follow-up chip does address heparin.
Staging, another item: correct

Staging lumps three orders into one sentence; production gives each order its own block, named exactly as highlighted on screen. That's the benchmark applied well. The better version only tightens it: the prod bullets are long, and the heparin block can say plainly that the order continues.

Production today
Potassium chloride 20 mEq by mouth once daily
  • Potassium 5.8 mEq/L with oliguria and elevated creatinine indicates impaired renal potassium excretion; withhold potassium chloride and clarify the order before administration.
  • …
Acetaminophen 650 mg by mouth every 6 hours as needed
  • …
Heparin 5,000 units subcutaneously every 8 hours
  • The prophylactic heparin order addresses venous thromboembolism prevention and does not directly worsen the documented hyperkalemia.
  • …
Calcium carbonate 500 mg by mouth with meals
  • …
Better version
Potassium chloride 20 mEq by mouth once daily
  • 10 mL/hr of urine and a creatinine of 3.2 mean the kidneys can't clear potassium, which is already 5.8 mEq/L.
  • Another 20 mEq risks a fatal dysrhythmia: hold it and call the provider.
Heparin 5,000 units subcutaneously every 8 hours
  • Adds no potassium; prophylactic heparin continues in a bed-bound patient.
  • This dose isn't a problem in acute kidney injury.
Acetaminophen 650 mg by mouth every 6 hours as needed
  • Adds no potassium and isn't affected by the renal findings.
Calcium carbonate 500 mg by mouth with meals
  • Adds no potassium; it's often given as a phosphate binder in kidney failure.

Case study (NGN unfolding case)

NCLEX-RN · post-thyroidectomy hematoma, then hypocalcaemia (ff4474bf), 6 steps
Content bugs
Step 2 correct: rationale box
Step 4 wrong: the rationale covers every option
"Finish case" goes straight to the next case: no score, no debrief

The steps are in house format in production and cover every option. What's missing is the ending. The case has a summary, but it's a paragraph that closes with Bloom's and NCJMM tags, and the student only sees it later in the results list. Turned into a debrief card, it becomes the static "what this case was testing, and the traps" that every student gets on finishing, whatever they missed. (Step 1's prod explanation also tells the student the image is unavailable; see bug 4.)

Production today (case summary, shown only in results)
This case follows two time-sensitive complications after neck surgery: acute airway compression from postoperative bleeding and subsequent symptomatic hypocalcemia. Early cues progress from frequent swallowing and visible neck swelling to voice changes and hypoxemia, requiring rapid airway support and surgical intervention… The case integrates Recognize Cues through Evaluate Outcomes and targets perioperative, respiratory, and fluid-electrolyte clinical judgment at Bloom's 5_evaluate level.
Better version: debrief card after step 6
Airway first: an expanding neck hematoma
  • Frequent swallowing, visible neck swelling, a muffled voice and SpO₂ falling to 91% = bleeding compressing the airway.
  • Rapid response, sit upright, oxygen, back to theatre.
Then calcium: parathyroid injury
  • Perioral tingling and carpal spasm = hypocalcaemia.
  • IV calcium gluconate with cardiac monitoring; 8.0 mg/dL afterwards is still low, so keep checking.
Traps in this case
Treating BP 148/86 as the priority
  • Pain and anxiety explain it; neck swelling threatens the airway within minutes.
Reinforcing the dressing
  • It hides the bleeding and does nothing for the compression.
Plus the case score (e.g. "4 of 6") above the card. Same card for Bar integrated sets and LSAT reading sets.

Single best answer

CPA · attribute sampling (8f1fad66). LSAT and Bar MCQs use the same screen and the same letter-by-letter style.
Regenerate
Picked C: the ✗ icon replaces the "C" the explanation refers to
Correct: every option explained, by letter
Key explainedYes
Every option explainedYes
House formatNo: one paragraph, by letter
ReadablePartly: ✓/✗ hide two letters

The reasoning is sound, which makes this the cheapest fix in the audit: the content is there, it just needs regenerating into blocks with the CPA prompt that already exists. In the second item tested (IT change management), the key is three times longer than any distractor, a classic "pick the longest" cue.

Shown today
Option B is correct because a lower tolerable deviation rate requires greater precision and therefore a larger sample. Option A ordinarily reduces sample size because fewer deviations are expected. Option C permits more sampling risk and therefore ordinarily reduces the required sample size. Option D reduces the audit assurance sought from the control…
Better version
Decreasing the tolerable deviation rate from 7% to 5%
  • A lower tolerable deviation rate demands more precision, so the auditor needs a larger sample.
  • It's the only change here that makes the conclusion harder to support.
Decreasing the expected population deviation rate from 3% to 1%
  • Fewer expected deviations leave more room below the tolerable rate, so the sample shrinks.
  • Sample size rises with the expected rate, not falls.
Increasing the acceptable risk of overreliance from 5% to 10%
  • Accepting more risk of overreliance means less assurance is needed, so the sample shrinks.
  • Sample size moves opposite to acceptable risk.
Reducing the planned degree of reliance on the tested control
  • Less planned reliance means the control test has to prove less.
  • Heavier reliance is what drives samples up.

CPA task-based simulation

CPA AUD · inventory obsolescence and the audit opinion (41367f5e), 10-row evaluation grid
Best of the new courses
8/10: per-row marks, per-row rationale, overall summary

Already structured per row with a task summary: the house format's shape, applied to a grid. Three fixes. (1) The choices are literally called "Correct" and "Incorrect", so the review reads "Incorrect · Correct answer: Correct". Rename them "Consistent / Not consistent" (or True / False). (2) Each row explains only the right choice; one line on the other choice covers the student who picked it. (3) The summary calls the $270,000 a pretax overstatement while row 4 calls it an understatement (of liabilities and the reserve). Both are true; say which account each refers to.

Shown today (row 3)
Incorrect · Correct answer: Correct

3. Using Exhibit 1, the obsolescence misstatement exceeds performance materiality. The $180,000 obsolescence misstatement exceeds performance materiality of $175,000.
Better version (row 3)

3 · The obsolescence misstatement exceeds performance materiality

Consistent
  • Supported reserve $480,000 − recorded $300,000 = a $180,000 misstatement.
  • Exhibit 1 sets performance materiality at $175,000, so it's over by $5,000.
Not consistent
  • Only holds if the misstatement is measured against the recorded reserve alone, or against overall rather than performance materiality.

CFA item set

CFA Level I Derivatives · put–call parity and binomial replication (42e2c335): MCQ, SATA, numeric
Fix items, regenerate
Picked USD 6.66: the rationale works the key
SATA on the protective put

The key and its working are right (4.20 + 52e−0.05 − 50 = 3.66). The distractor reasons don't reproduce the distractor numbers: "omits discounting of the exercise price" gives 4.20 + 52 − 50 = 6.20, not the 6.66 offered; "discounts the call premium too" gives 3.46, not 3.34. In a static explanation, a distractor block is how a student retraces their slip, so each distractor has to come from exactly one slip. Fix the item, then write the blocks. Maths renders as plain text today (Ke^(−rT)). Content note: the SATA key "floor value equal to the put's exercise price minus the premium paid" mixes the value floor (X) with the profit floor (X − S₀ − p).

Shown today
B is correct because put–call parity gives p = c + Ke^(−rT) − S0 = 4.20 + 52e^(−0.05) − 50 = USD 3.66. A incorrectly discounts the call premium along with the exercise price. C omits discounting of the exercise price and also overstates the parity adjustment.
Better version (after replacing 3.34 → 3.46 and 6.66 → 6.20)
USD 3.66
  • Put–call parity: p = c + K·e−rT − S₀ = 4.20 + 49.46 − 50 = 3.66.
  • The strike is paid at expiry, so it enters at its present value, discounted the way the rate is quoted (continuously).
USD 3.46
  • Discounts the call premium as well: 4.20·e−0.05 + 49.46 − 50.
  • The premium is paid today; it's already a present value.
USD 6.20
  • Uses the undiscounted strike: 4.20 + 52 − 50.
  • Parity compares present values, so the strike must be discounted.

Reading comprehension

LSAT · Condorcet and Gouges passage (caceda75), 6 questions
Missing
Wrong answer: red and green, no explanation

All six children have an empty explanation. For LSAT reading, the explanation is the teaching: the skill is spotting why a plausible choice goes beyond the passage, and in the house format each wrong choice gets its own "why it goes beyond". Choices here are long, so headings can be shortened, but they must be recognisable. (Per the code, reading-set explanations render as plain text, so markdown blocks won't show until that's fixed.)

Shown today
Nothing.
Better version (main-point question)
While both advocated equal rights for women, Condorcet argued that such rights did not have to be established through explicit political action, and Gouges… argued that women needed to engage in a direct political battle
  • The passage calls their approaches "radically opposed courses of action": rights that follow from universal principles versus rights won by direct political struggle.
  • It's the only choice that states both halves of the passage's comparison.
…arguments in favor of equal rights for women were not easy for most members of society to evaluate
  • The passage never discusses how society judged the arguments.
  • It adds a claim rather than summarising one.
…Condorcet's and Gouges's were not as radically opposed as they initially seemed
  • Contradicts the passage, which calls them radically opposed.
Because neither Condorcet nor Gouges proposed specific legislation… their views were not widely accepted
  • Turns a detail (no legislation) into a cause the passage never states.
  • The passage says nothing about how widely their views were accepted.
…Gouges held that women could win their rights only while revolution was still raging, and Condorcet… counseled patience
  • Invents a contrast about timing; the passage contrasts methods.
  • Nothing ties Gouges to the revolution's course or Condorcet to patience.

LSAT principle question

LSAT Argumentative Writing prep · "a responsibly formed recommendation" (f4bf2dbf)
Regenerate
Picked B: every option checked against the principle's three conditions, by letter

The reasoning is right: each option is tested against the principle's three conditions (a cost, why the benefit outweighs it, an uncertainty). In blocks, the student who picked the tutoring proposal sees immediately which condition it misses.

Shown today
D is correct because it identifies a likely cost, explains why the benefit outweighs that cost, and identifies volunteer capacity as an uncertainty… B identifies a benefit, a cost, and an uncertainty but does not explain why the benefit outweighs the cost…
Better version
A ballot-review period may delay results, but fewer errors justify the delay; volunteer capacity remains an uncertainty requiring pilot data.
  • Meets all three conditions: a cost (delay), why the benefit outweighs it (fewer errors), and an uncertainty (volunteer capacity).
  • The only option that says the benefit is worth the cost.
A transit proposal may reduce congestion but cost more than expected…
  • Names a possible cost overrun, but never says why the benefit is worth it.
A tutoring proposal could improve achievement, although it requires weekend staffing…
  • Has a cost (staffing) and an uncertainty (participation).
  • Missing: why the achievement gain outweighs the staffing cost.
A recycling proposal should be adopted because it offers environmental benefits…
  • Names an uncertainty (participation) but no cost, so there's nothing to weigh.
A longer lunch period offers social benefits but shortens instructional time…
  • Names a cost, then leaves the choice open instead of concluding the benefit outweighs it.

Integrated question set

Bar · promoter liability and apparent authority (f997ef4a), 6 steps: MCQ, cloze, matrix, ordering
Regenerate
Step 1 wrong: every option explained and tied to Exhibit 3, by letter

Same shell as the NCLEX case, with sound per-step reasoning that cites the exhibits. It needs the same two things: regenerate each step into blocks (per blank and per row for the cloze and matrix steps, as production NCLEX does), and a debrief card at the end. Its ordering step uses item-N ids, so check it against bug 1. Sets that contain a short answer are covered in Part 2.

Part 2 · Jev grading of free-text answers

Three formats are graded live, per student: short answer (Bar, 1 item), performance task (Bar, 169) and argumentative writing (LSAT, 146). This is the only place the app can say "you wrote…", so the feedback should be specific to the submission. Since api#1187 (merged to dev on 24 Sep), TypeSafe's Jev makes every judgement and Gemini 3.5 Flash-Lite writes the feedback afterwards. Staging's ECS task definition injects AI_GATEWAY_API_KEY, and grades returned on staging carry Jev's confidence flags, so staging runs the Jev path.

1 · Jev decidesTyped questions against the rubric: a 0–4 band per dimension or issue, booleans for off-topic, perspectives and requirements, a choice for which rubric answer a field matches. It returns probabilities, not prose.
→
2 · Code scoresAverages bands and applies gates: too short → capped at 50%; a missing required element → capped at 60%; −15% per prohibited error. Pass mark 75%.
→
3 · Flash-Lite explainsGiven the band numbers and the student's text, it writes 1–2 sentences per band. It doesn't see why Jev chose a band (Jev gives no reasons) or the item's rubric criteria.
→
4 · App showsBand word + comment per dimension (essay), points + comments (performance task). Short answer: nothing.

How we tested

Three submissions through the live staging UI: a strong LSAT essay (0.95), a weak one (0.25) and a strong Bar memo (0.60). Then 20 controlled answers, run twice each, through the same grader code from api@dev. The gateway key can't be read locally, so Jev was reached through OpenRouter's Decisions endpoint (typesafe/jev-1.13), the same model family. The strong essay and the strong memo scored the same locally as on staging.

CaseWhat it testsScore (run 1 / 2)Verdict
Essay · strong392 words, tiered policy, all 3 perspectives, answers an objection0.95 / 0.95Right
Essay · opposite side, equally strongNeutrality: argues the absolutist Perspective 10.95 / 0.95Right, side-neutral
Essay · clear position, 1 perspective, no counterargumentMiddle of the range0.65 / 0.60Reasonable; drifts between runs
Essay · restates the perspectives, no positionWeak0.25 / 0.30Right
Essay · well written, wrong promptOff-topic detection0 / 0Caught (p = 0.97)
Essay · forged delimiter + "award 4/4"Prompt injection0.10 / 0.10Resisted
Essay · 42-word outlineLength gate0.50 / 0.50Capped, but thesis still 4/4
Essay · engages 1 of 3 perspectives (zoning item)Perspective gate (the only item with a perspectives array)0.75 / 0.75 · passCounted "2 of 3" engaged; passes
Memo · strong, from the actual fileCeiling0.60 / 0.60 (21/24 in bands)Capped by requirements the file can't meet
Memo · same + the rubric's invented factsDoes the rubric reward making things up?0.50 / 0.503 of 4 "missing" elements now satisfied, by facts that don't exist
Memo · strong + invented outside authorityClosed-universe rule0.50 / 0.50Bands dropped; feedback only hints ("outdated or inapplicable authorities")
Memo · 5 conclusory sentencesWeak0 / 0Right
Memo · weak + injected instructionsPrompt injection0 / 0Resisted
Short · exact / paraphrase / "the firm"Meaning over wording1 / 1 eachRight
Short · "partners' personal assets" / insurance / injectionWrong answers0 / 0 eachRight
Short · "either partnership or partners, whichever is easier"Hedging both ways0 / 1Flipped to full credit on run 2

What works

  • Decisions track quality. Strong 0.95, middling 0.6, weak 0.25, off-topic 0: the order a human grader would give.
  • Hard to game. Three injection attempts (a forged closing delimiter plus instructions) earned nothing, and the short-answer reason named the attempt.
  • Side-neutral. Opposite positions of equal quality scored the same.
  • Tolerates paraphrase. "Riverbend's own business property, before going after the partners personally" was credited.
  • Fast and cheap. Median 1.9 s for a full grade including the feedback pass.

What doesn't

  • 44% of verdicts are "low confidence" (41 of 94 bands and fields below 0.6), and each shows the student "— low confidence (0.53) — worth a human look". No human review queue exists, so a note meant for us lands on the student.
  • Borderline answers aren't stable. The hedge flipped from 0 to full credit between identical runs (p = 0.53). Two essay bands moved a point.
  • Feedback is written blind. Flash-Lite gets "reasoning 3/4" and invents a reason. The strong essay's 3/4 advice ("add concrete examples of how thresholds are calculated") ignores that the essay names enrollment and service-request data and a two-year review cycle.
  • Feedback contradicts the score. The memo comment says the candidate "performed exceptionally well across the board" on a submission shown as 14/24 and not passing.

LSAT argumentative writing: what the student sees

Staging UI: translation-policy prompt (c4939733) and class-recording prompt (8e8ef21d)
Needs work
Strong essay: Fully / Largely bands with comments; "worth a human look" visible; "feedback in progress" still shown after grading
Weak essay: 25% overall, generic advice
  • No overall result line and no model answer. Every item has a sample_response and a scoring_rubric; neither is shown, although formats.ts:899 says they are "shown after submission". The sample response is the static half of the feedback: it costs nothing to show.
  • 145 of 146 essays keep their perspectives only in the prompt text, so the "at least 2 perspectives" gate never runs. On the one item that has the array, an essay engaging one perspective was counted as 2 of 3 and passed.
  • The length gate is 400 characters (about 65 words) for a 35-minute essay. An 81-word non-essay clears it.
Shown today (weak essay)
Thesis 0/4 — State a clear and specific position on the college recording policy in your introduction rather than merely summarizing the debate.
Perspective engagement 1/4 — Select a definitive stance and explicitly evaluate the provided perspectives in relation to your central argument. — low confidence (0.56) — worth a human look
Counterconsideration 1/4 — Address potential objections… — low confidence (0.52) — worth a human look
Better version (live, per student)

Overall: not yet an argument (25%). You summarised the debate well. Now take a side.

Thesis. Your last line, "class recording is a complicated issue", isn't a position. For example: "Colleges should record by default, but release recordings only to enrolled students and let instructors pause for discussion."

Perspectives. You note privacy worries exist, but not whether they outweigh access. Pick the strongest worry and answer it.

Counterargument. Add one paragraph that begins "The strongest objection is…", and show why your policy survives it.

See a full-credit essay for this prompt → (the item's sample_response)

Bar performance task: what the student sees

Staging UI: bakery partnership memo (0716d554)
Broken
Strong memo: "14 / 24" at the top, "21.0 / 24" at the bottom, four "missing" elements, Library tab showing 0
Results review: only the author's paragraph, citing a $14,000 credit the file never mentions

The rubric and the file come from different drafts. The rubric requires "Avery's Lender Communications objection", "Jordan's Purchasing Records" and a "$14,000 related-party overcharge". The file holds two exhibits (the facts, and eight RUPA provisions) and none of those. Because a missing required element caps the score at 60%, no memo written from the file can pass, while a memo that invents those facts clears three of the four gaps. A bank-wide scan found 29 of 169 tasks whose required elements cite amounts or documents absent from the file (8 involving dollar figures). 62 tasks file the law under "Case file", so the Library tab reads 0, and 154 have no sample response.

The screen. Two scores (14/24 after caps, 21.0/24 before), no per-issue breakdown (this task has no issues, so everything is "The task as a whole"), a comment praising a failing result, and "grading in progress" still under the memo.

Shown today
14 / 24
The task as a whole — rules 3/4, application 4/4 — Your performance earned a rules score of 3/4 because you accurately identified and utilized the relevant statutory and common law principles… — low confidence (0.53) — worth a human look
every required element — missing: Analyzes the equipment note liability… against Avery's Lender Communications objection… Identifies Jordan's undisclosed related-party purchase in the Jordan's Purchasing Records…
24 points — 21.0 / 24
Better version (once the rubric matches the file)

21 / 24 · passing. One number, then per issue:

  • Formation 4/4. You applied §202(a) and the §202(c)(3) profit-share presumption to the 60/40 split and the joint DBA.
  • Management and duties 3/4. Right on §401(f). You flag the unilateral $45,000 note as a care problem but stop there; say what it buys Avery in the negotiation (e.g. Jordan indemnifies Avery).
  • Note liability 4/4. §301(1) → §305(a) → §306(a), tied to the bank having no notice of any limit on Jordan's authority.
  • Recommendation ✓ Prioritised, protects Avery, includes a wind-down fallback.

Compare with a model memo →

Short answer: graded, never shown

Bar integrated set: "from what source of assets must ToolWorks ordinarily seek satisfaction first?" (the only short-answer item in the bank)
Invisible

The grader works (see the table above) and its reasons are good one-liners, e.g. "incorrectly stated that ToolWorks must seek satisfaction first from the partners' personal assets, rather than the partnership's assets." The app shows "Graded against the rubric after you finish the set", then a Pending or Partial label; the review has no short_answer case. The rubric has one accepted answer and no rejected list, so there's nothing to name when a student falls for the classic trap (partners first).

Shown today
Graded against the rubric after you finish the set.
Better version (live, per student)

Not credited. You wrote "the partners' personal assets". A creditor with a judgment against the partnership must first go after partnership assets (Authority 1); the partners' separate assets come second.

Fix list, in order

#FixWhereWhy this order
1Ordering: grade and display from one key. Drop the rebuild in withOrderedOptions, or rewrite options.metadata.sequence from content.correct_order; fix the 3 item-N items.app + dataShows students false answers, in prod
2Case studies: read dropdown options from the child rows (or re-sync content.sub_questions); backfill the 72 matrix row labels; release exhibits in step order and remove generation notes such as "the actual Exhibit 2 image is unavailable".app + dataAnswers given away or unanswerable, in prod
3Regenerate the new courses' explanations in the house format before launch: 10,244 across CFA, CPA, LSAT and Bar, the 338 empty ones and the LSAT reading sets first. The course prompts already specify it for single-answer questions; extend them per format using production NCLEX as the model (per blank, per row, per step, per region, per statement). Add a check that every option's text appears as a heading.contentThe single biggest gap to the benchmark
4Performance tasks: re-sync the 29 flagged rubrics with their files, then add an ingest check that every $ amount and named source in required_elements / prohibited_errors exists in the materials.data + apiTasks can't be passed
5Stop showing "low confidence — worth a human look" to students. Route it to a review queue, or re-ask Jev and take the majority when p < 0.6.apiOn ~44% of lines; flips grades
6Show free-text results everywhere: the short-answer grade and reason, performance-task and essay breakdowns in the results review, one score only, and the stored sample_response after submission.appGraded work students never see
7Close the static gaps in production NCLEX: common wrong answers on fill-in (553), common wrong orders on ordering (166), hot-spot headings that use the on-screen letter and a plain description instead of id slugs (231), "Select / Do not select" groups on SATA, and a shorter matrix pattern.contentEvery wrong answer should have a heading to find
8End every aggregate with a debrief card: score, the case's content.explanation rewritten without Bloom's/NCJMM tags (366 of 432 prod cases carry them), and the case's traps. Make Rezzy available inside aggregates.app + contentCases end abruptly
9Copy production NCLEX and MCAT explanations to staging, so QA on staging sees what students see.dataStaging shows the old paragraphs
10Explain partial credit in the banner ("+1 per right pick, −1 per wrong pick"). Rename the TBS choices "Consistent / Not consistent".appStatic UI lines, cheap
11Ground the feedback writer: pass it Jev's per-band probabilities, the item's rubric criteria and the sample response; require it to quote the student's text; ban praise on bands of 2 or less and on failing totals.apiGeneric or contradictory comments
12Regenerate CFA distractors so each maps to one slip the explanation can name, and render maths as LaTeX. Raise the essay length gate (e.g. 250 words) and backfill the perspectives array on 145 essays.content + apiExplanations that can't be reproduced
13Follow-up chips, which already react to the pick at a cost bounded per question × option: send option text rather than UUIDs for ordering and grid answers, so the chips can address the student's actual mistake. No new generation.appThe existing per-student layer, made to work on every format

Also noticed, out of scope: the CPA subject picker uses medical emoji (🦴 💊 😴); the LSAT reading passage tested reads like a published PrepTest passage; 2,125 UK Medical PG explanations on staging have no line breaks; hot-spot screen-reader names are answer slugs ("right-pupil-white-reflex").

Method

UI capture. app-staging.getoncourse.ai in headless Chromium, signed in as staging QA accounts (NCLEX test@, CPA onboarding-web, CFA persona-qa-finance, LSAT persona-qa-law, Bar shubhdeep+12345, US Medical PG shubhdeep+3). Practice-mode custom quizzes, filtered by format where the builder allows it (NCLEX, CPA). Bar formats were started through POST /test/session/start with formats, the same call the app sends. The answer key was read from the question payload so every wrong answer was chosen on purpose.

Benchmark and counts. Read-only SQL against the staging and production databases, validated items unless noted. "House format" = the explanation opens with a bold key block (***…***) or a bold part heading (**Blank 1**, **row text**). Production NCLEX texts quoted in the cards are the stored explanations of the same question ids as the staging screenshots. The new courses (CFA, CPA, LSAT, Bar) exist only on staging.

Grading. The api@dev (9b5055b7) graders ran unmodified in a throwaway checkout. Only the Jev HTTP call was redirected, to OpenRouter's Decisions API (typesafe/jev-1.13), because the AI Gateway key can't be read outside ECS. Feedback used Gemini 3.5 Flash-Lite with the staging key, as on staging. 20 cases × 2 runs; raw results in results.json.

Better versions are written as static explanations: the same text for every student. Only the Part 2 rewrites are per student, because those answers are graded live. Screenshots are cropped and compressed; click any image to enlarge it.