Summary
The authored explanations are mostly competent: most say why the key is right and mention each distractor. What lets students down is delivery, plus a handful of format bugs that show students a wrong answer as correct. No explanation responds to the option the student actually picked. For free-text answers, Jev's decisions are sound, but the feedback built on them is generic, sometimes contradicts the score, and in two places never reaches the student.
row-1…row-4 instead of the findings, so they can't be answered.rationale that the app never shows. The Rezzy follow-up chips do react to the wrong pick ("Compare why potassium chloride is withheld here while prophylactic heparin is generally continued"), so the plumbing is there.Scoreboard
One row per format tested. "Your pick" asks whether the student's specific mistake gets explained. Verdicts reflect what the student sees, not the raw content.
| Format | Tested in | Why the key is right | Why each other option is wrong | Your pick | Right vs wrong view | Verdict |
|---|---|---|---|---|---|---|
| Select all (SATA) | NCLEX | Yes | Yes, but by letters the screen doesn't show | No | Same text; colours differ | Needs work |
| Cloze dropdown | NCLEX | Yes | Yes | No | Same text; wrong blank shows the key | Good |
| Matrix | NCLEX | Yes | Yes | No | Same text; per-row marks | Good content |
| Ordered response | NCLEX | Yes | Partly | No | Review shows the wrong key | Broken |
| Fill in the blank | NCLEX, CFA | Worked solution | Names common slips | No (could, from the number) | Same text; shows key | Good |
| Image hot spot | NCLEX | Yes | Lumped together | No (stored rationale unused) | Same text; regions coloured | Needs work |
| Highlight hot spot | NCLEX | Yes | "The other orders…" | No (stored rationale unused) | Same text | Needs work |
| Case study (NGN) | NCLEX | Per step | Per step, lettered | No | Per-step rationale; no case debrief | Content bugs |
| Single best answer | CPA, LSAT, Bar | Yes | Yes | No | Same text | Good |
| CPA TBS grid | CPA | Per row | Per row | Marked, not explained | Per-row marks + overall | Best in app |
| CFA item set | CFA L1 | With working | Distractor maths don't reproduce | No | Per-step rationale | Needs work |
| Reading comprehension | LSAT | Nothing | Nothing | No | Colours only | Missing |
| Integrated question set | Bar | Per step | Per step | No | Per-step rationale | Good |
| Short answer | Bar (1 item) | Jev grades it and writes a one-line reason. The student never sees either. | Invisible | |||
| Performance task | Bar | Jev grades it. One generic comment, two contradictory scores, and rubrics that demand facts the file doesn't contain. | Broken | |||
| Argumentative writing | LSAT | Jev grades it. Bands are sensible; comments are generic; no model answer; internal "worth a human look" notes shown to students. | Needs work | |||
| Constructed response | CFA (338, none validated) | No grader exists. The app shows "Scoring in progress" and never a result. | Not shippable | |||
Not captured: data-entry grids with numeric cells (the CPA simulation tested was a Correct/Incorrect grid), and MCAT, which only has legacy single-answer MCQs.
Blocking bugs found during the audit
These go beyond explanation quality: the app tells students something false, or the question can't be answered. Each is sized against the live bank.
1 · Ordered response: the review shows a different "correct" order from the one the server grades Prod · 39 of 45 standalone
The client rebuilds correct_order from the legacy options rows' metadata.sequence (withOrderedOptions, apis/question/formats.ts:563). The server grades against content.correct_order. On 36 of the 45 live standalone items they disagree, so the review marks right positions red and prints the wrong "Correct: …" under each. In the first screenshot the pill says 2/5, only one row is green, and step 1 claims "Correct: Provide a snack…". On 3 more items, content.correct_order uses item-N ids while the app submits option UUIDs, so a perfect order scores "Incorrect 0/5" with every row green. Counting case-study children too: 155 of 166 validated ordering questions in production.
2 · Case studies give away the dropdown answer Prod · 29 cases, 53 blanks
The case player reads the parent's denormalised content.sub_questions copy, which kept only the correct option per blank. The child rows still have all three. The student opens a dropdown with one choice in it.
3 · Matrix rows labelled row-1 … row-4 72 items, all live
The row text never made it into row_headers, so the student classifies "row-1" as improvement or follow-up. The rationale then talks about tingling, neck circumference and a calcium of 8.0, none of which appeared as rows.
4 · Case timelines leak or run ahead Content · seen in 1 case, likely wider
In the thyroidectomy case, step 1 asks "Based on Exhibits 1 and 2 … neck fullness shown in the image", but only Exhibit 1 is released: a normal baseline with no image. Exhibit 3 then narrates the answers to steps 3 and 4 ("The nurse activates the rapid response system, raises the head of the bed, applies oxygen…") before the student answers them.
5 · Free text: grades students never see, and grades that can't pass Bar, LSAT, CFA
- Short answer (inside Bar sets): Jev grades it and writes a reason. The app shows "Graded against the rubric after you finish the set", and the results review has no
short_answercase (FormatAnswerReview.tsx:690). The student never learns whether they were credited. - Performance task: the grade lives only in component state. The results review shows the author's model-answer paragraph and nothing about the student's memo.
- Performance-task rubrics cite evidence the file doesn't contain (Part 2). A missing required element caps the score at 60%, so affected tasks can't be passed.
- Constructed response (CFA, 338 items): no server grader and no result UI. Harmless while unvalidated; it blocks launch.
Patterns across formats
Authoring metadata printed to students
Openers and closers like "This item targets Medical-Surgical Nursing, Cardiovascular Disorders, Bloom's 3_apply level, medium difficulty, and the NCJMM Generate Solutions skill." sit inside the explanation: 2,891 of 5,643 live NCLEX explanations on staging, 632 in production. Some contradict the quiz: a Med-Surg quiz showed "Subject: Nutrition and Diet Therapy". It's often the first sentence the student reads.
Letters the screen doesn't show
SATA explanations say "Option A is correct… option C…", but the NCLEX SATA screen shows no letters (only screen readers get "Option B."). The student maps six letters onto six sentences in their head. That's 354 of 573 live SATA items on staging (4 in prod today). The hot-spot explanation says "right pupil" while the image labels regions A–D. And after answering an MCQ, a ✓/✗ icon replaces the letter of the picked and correct options.
One explanation for right and wrong
Every format shows the same paragraph either way. A student who got it right rereads 900 characters; one who got it wrong has to find their own mistake inside it. Partial credit is never explained: "1/4" on SATA comes from +/− scoring (2 right − 1 wrong), and nothing on screen says so.
Rationale we store and hide
Hot-spot and highlight items carry a per-option rationale (for heparin: "addresses venous thromboembolism risk and is not contraindicated by the listed renal findings"). CPA grids carry per-row rationale, which is why the TBS is the best surface in the app. Single questions never render content.rationale: the screen uses only metadata.explanation || explanation (question/index.tsx:246).
Aggregates end without a debrief
Case studies, integrated sets and reading sets show a rationale per step, then "Finish case" jumps straight to the next case: no case score, no case-level explanation. The explanation exists in content.explanation ("This case follows two time-sensitive complications…") but only appears afterwards in the results list. Rezzy isn't available inside aggregates.
Empty explanations in the new courses
Bar 205, LSAT 94 and CPA 39 live items have no explanation. For LSAT that includes whole reading-comprehension sets: red and green, nothing else. Separately, the LSAT passage tested (Condorcet and Olympe de Gouges) reads like a published LSAT PrepTest passage; worth a licensing check.
Format by format
For each: what the student sees on a wrong and a right answer, whether the explanation does its job, and a rewrite of that same explanation. The rewrites follow one shape: your pick → why the key is right → one line per other option, named by its content → the rule to keep.
Select all that apply
The content is correct and complete; the student just has to decode it. They see six unlettered sentences and an explanation written in letters. "1/4" with two right picks is never explained. The first sentence is metadata.
This item targets Medical-Surgical Nursing, Cardiovascular Disorders, Bloom's 3_apply level, medium difficulty, and the NCJMM Generate Solutions skill. Option A is correct because regular international normalized ratio testing guides safe anticoagulant dosing, while option C reduces the risk of cuts and bleeding. Option D is correct because… Option B is incorrect because doubling a missed dose can produce excessive anticoagulation… Option F is incorrect because aspirin can increase bleeding risk…
You picked 3: two right, one trap. Select-all scores +1 for each right pick and −1 for each wrong pick, so this earned 1 of 4.
Include
- INR appointments. The INR sets each warfarin dose; missing tests hides drift toward bleeding or clotting.
- Electric razor. With a mechanical valve the anticoagulation is lifelong, and nicks bleed longer.
- Tell dentists and prescribers. Procedures and new drugs (antibiotics, NSAIDs) change bleeding risk and the INR.
- Report black stools or red urine. Melena and hematuria mean internal bleeding.
Leave out
- Double a missed dose (your pick). A double dose can push the INR into the bleeding range. Take it later the same day if remembered; otherwise skip it and call.
- Aspirin for headaches. It adds an antiplatelet effect on top of warfarin. Use acetaminophen and check first.
Rule: warfarin teaching = monitor the INR, prevent injury, disclose the drug, report bleeding, never adjust the dose yourself.
Cloze dropdown
The model the rest should copy: every dropdown option is named by its content and given a reason. The only gaps are the metadata opener and one paragraph where there should be one block per blank.
Subject: Medical-Surgical Nursing; topic: sensory disorders and postoperative eye care; NCJMM skill: Evaluate Outcomes. Sleeping on the nonoperative side demonstrates correct teaching because it avoids pressure on the operative eye. Bending at the waist and lifting a 9-kg bag can increase intraocular pressure… A sudden increase in flashes or floaters requires immediate follow-up… Mild itching… A small amount of tearing…
Blank 1: you chose "bend at my waist to put on shoes". Bending and heavy lifting raise pressure in the healing eye, so that statement shows the teaching didn't land. The right statement is "sleep on my nonoperative side": it keeps pressure off the operated eye.
Blank 2 ✓ A sudden increase in flashes or floaters can signal retinal detachment and needs same-day ophthalmology. Mild itching and a little tearing on day one are expected; the client just shouldn't rub the eye.
Rule: after cataract surgery, anything that raises eye pressure is out; new flashes, floaters, vision loss or pain are urgent.
Matrix
Good clinical reasoning, but the student holds five rows in mind while reading one paragraph. It belongs row by row. It also ends "This musculoskeletal and perioperative item is easy difficulty, targets Bloom's 5_evaluate…".
Pain rated 3 after analgesia is expected because… A shortened, internally rotated operative leg requires immediate action because it suggests prosthetic displacement… A temperature of 37.8° C during the first postoperative day can result from the inflammatory response and is expected… This musculoskeletal and perioperative item is easy difficulty, targets Bloom's 5_evaluate and the NCJMM Evaluate Outcomes skill…
- Pain 3/10 after analgesia → Expected. Controlled pain is the goal.
- Leg shortened and internally rotated → Immediate action (you chose Prompt follow-up). That's the classic sign of hip dislocation. Keep the leg still and call the surgeon now; waiting risks nerve and vessel damage.
- Pedal pulse 2+ → absent → Immediate action. A newly lost pulse means the limb isn't getting blood.
- 37.8 °C on post-op day 1 → Expected (you chose Prompt follow-up). Low-grade fever in the first 24–48 h is the inflammatory response. Watch the trend.
- New one-sided calf tenderness and swelling → Prompt follow-up. Suspect DVT: report it and expect a Doppler. It ranks below a pulseless or dislocated limb.
Rule: limb threatened now > possible clot > expected recovery.
Ordered response
The explanation itself is decent; the screen around it contradicts it (see bug 1 for both screenshots). The student reads "the nurse first gives 15 g of rapid-acting carbohydrate" while the review says "Correct: Provide a snack…" for step 1, and can't tell which to believe. The first sentence is metadata that, in this Med-Surg quiz, names another subject.
Subject: Nutrition and Diet Therapy; topic: Nutrition Across the Lifespan and Glycemic Safety; Bloom's level: Analyze; difficulty: Medium; NCJMM skill: Take Action and Evaluate Outcomes. Because the alert client can swallow, the nurse first gives 15 g of rapid-acting carbohydrate and rechecks the glucose after 15 minutes. A result of 64 mg/dL remains below the treatment goal, so another 15 g is given… Giving the mixed snack before correcting the low value would delay glucose absorption.
- Give 15 g of fast carbohydrate. Alert and able to swallow, so treat by mouth first.
- Recheck after 15 minutes. The 15-15 rule: treat, wait, measure.
- Give another 15 g, because 64 mg/dL is still below 70.
- Recheck after another 15 minutes to confirm recovery.
- Snack with carbohydrate and protein. The next meal is 2 hours away; protein prevents a second drop.
Where yours went off: you started with "give another 15 g", but there's no second dose until you've treated and rechecked once. A mixed snack first would slow absorption while the glucose is still 54.
Fill in the blank (numeric)
The strongest single-question explanations in the bank: a worked solution plus the two usual errors. What's missing is the easiest personalisation there is: the student typed 0.4, which is exactly the "forgot to convert to mL/hr" error the explanation already describes. One risk from the grading code (not tested in the UI): normalizeFillBlankValue treats only bare numbers as numbers, so "22.5 mL/hr" is compared as text and marked wrong. Students type units.
First calculate the prescribed dose: 0.08 mcg/kg/min × 75 kg = 6 mcg/min. The supplied concentration is 4,000 mcg ÷ 250 mL = 16 mcg/mL. The hourly rate is 6 mcg/min ÷ 16 mcg/mL × 60 min/hr = 22.5 mL/hr. A common error is failing to convert 4 mg to 4,000 mcg… Another common error is reporting 0.375 mL/min without converting the rate to mL/hr. This medium application item targets Pharmacology Principles…
You entered 0.4. That's 0.375 rounded: the rate in mL per minute. Pumps run in mL per hour, so 0.375 × 60 = 22.5 mL/hr.
Dose 0.08 mcg/kg/min × 75 kg = 6 mcg/min · Concentration 4 mg = 4,000 mcg ÷ 250 mL = 16 mcg/mL · Rate 6 ÷ 16 = 0.375 mL/min × 60 = 22.5 mL/hr.
Other slips: 0.0225 or 22,500 means the mg→mcg conversion is off by 1,000.
Rule: carry every unit through the calculation; if the answer isn't in mL/hr, you're not done.
Image hot spot
The image labels regions A–D, but the explanation says "the right pupil". The white reflex is at B, on the viewer's right, which is the infant's left eye. A nursing student reads "right pupil" anatomically and looks at A. Each region has its own rationale in content.rationale that never renders. The screen-reader names also give the answer away (radio "right-pupil-white-reflex").
The right pupil is the correct region because a white or yellow pupillary reflection is an abnormal screening result… The normal red-orange reflex in the left pupil is reassuring. Clear sclerae do not account for the abnormal reflection… This item targets Health Screening and Disease Prevention, Bloom's 4_analyze level…
The answer is B, the white pupil. A white reflex (leukocoria) can mean retinoblastoma, congenital cataract or retinal detachment, and needs urgent ophthalmology referral. B is the infant's left eye, on your right.
You chose A, the red-orange reflex: the normal finding you compare against.
C and D, the clear sclerae, are normal and don't explain the abnormal reflex. Tracking with both eyes doesn't rule out disease; a white reflex is referred regardless.
Rule: a red reflex that's present and symmetric is normal; white, absent or asymmetric means refer.
Highlight hot spot
The client's oliguria and elevated creatinine indicate severely impaired renal potassium excretion. Administering prescribed potassium could worsen hyperkalemia… The other medication orders do not directly aggravate the identified electrolyte threat. This medium Renal and Fluid-Electrolyte Disorders item targets Bloom's 3_apply…
Withhold potassium chloride. 120 mL of urine in 12 hours (10 mL/hr) and a creatinine of 3.2 mean the kidneys can't clear potassium, and it's already 5.8. Another 20 mEq risks a fatal dysrhythmia.
You chose heparin 5,000 units SC. Prophylactic heparin should continue: it prevents clots in a bed-bound patient and isn't dangerous at this dose in kidney injury.
Acetaminophen and calcium carbonate add no potassium; calcium carbonate is even used as a phosphate binder in kidney failure.
Rule: in oliguria, first stop what the kidneys must clear: potassium, and drugs like metformin after contrast.
Case study (NGN unfolding case)
The per-step rationales are among the best in the app: the case shell does letter its options, and every option is addressed. The failures are around them: the case bugs above, and no ending. The case's own summary exists but only shows up in the results list later.
Nothing. The player moves on to case 2.
Case 1 · 4 of 6. The story: neck swelling + frequent swallowing + muffled voice + falling SpO₂ = an expanding hematoma compressing the airway → rapid response, sit upright, oxygen, back to theatre. Later, tingling and carpal spasm = hypocalcaemia from parathyroid trauma → IV calcium gluconate, and keep monitoring (8.0 mg/dL is still low).
Where you slipped: Step 1, a BP of 148/86 is explained by pain and anxiety; neck swelling threatens the airway within minutes. Step 4, reinforcing the dressing hides bleeding and does nothing for the compression.
Rule: after neck surgery, airway first; after thyroid surgery, check calcium.
Single best answer
Solid: every option gets a direction and a reason. Two small things. The ✓/✗ icons replace the letters the explanation talks about. And in the second item the key is three times longer than any distractor, a classic "longest answer" cue.
Option B is correct because a lower tolerable deviation rate requires greater precision and therefore a larger sample. Option A ordinarily reduces sample size because fewer deviations are expected. Option C permits more sampling risk and therefore ordinarily reduces the required sample size. Option D reduces the audit assurance sought from the control…
You chose C, raising the acceptable risk of overreliance from 5% to 10%. Accepting more risk means you need less evidence, so the sample shrinks.
B ✓ Tightening the tolerable deviation rate (7% → 5%) needs more precision, so a bigger sample.
A expects fewer deviations (3% → 1%), which shrinks the sample. D relies on the control less, so it needs less evidence.
Rule: the sample grows when the tolerable rate goes down, acceptable risk goes down, the expected rate goes up, or planned reliance goes up.
CPA task-based simulation
The model for the other aggregates: every row gets a rationale, and the task gets a summary. Three fixes. (1) "Correct answer: Correct" and "Correct answer: Incorrect" read as nonsense because the choices themselves are called Correct and Incorrect; use "Consistent / Not consistent" or True/False. (2) The rationale list doesn't flag the rows the student missed; show those first. (3) The overall summary calls the $270,000 a pretax overstatement while row 4 calls it an understatement (of liabilities and the reserve). Both are true; a student sees a contradiction.
Incorrect · Correct answer: Correct
3. Using Exhibit 1, the obsolescence misstatement exceeds performance materiality. The $180,000 obsolescence misstatement exceeds performance materiality of $175,000.
Row 3: you said it doesn't exceed performance materiality. It does. The misstatement is $480,000 supported − $300,000 recorded = $180,000. Performance materiality in Exhibit 1 is $175,000, so it's over by $5,000.
CFA item set
The keys and their working are right (4.20 + 52e−0.05 − 50 = 3.66; hedge ratio 0.5, borrow 42.86 → 7.14). The distractor rationales are not: "C omits discounting of the exercise price" gives 4.20 + 52 − 50 = 6.20, not the 6.66 offered, and no single slip produces A's 3.34. A student trying to retrace their mistake from the explanation can't. The maths renders as plain text (Ke^(−rT)). Content note: the SATA key "floor value equal to the put's exercise price minus the premium paid" mixes the value floor (X) with the profit floor (X − S₀ − p).
B is correct because put–call parity gives p = c + Ke^(−rT) − S0 = 4.20 + 52e^(−0.05) − 50 = USD 3.66. A incorrectly discounts the call premium along with the exercise price. C omits discounting of the exercise price and also overstates the parity adjustment.
B, USD 3.66. p = c + K·e−rT − S₀ = 4.20 + 49.46 − 50 = 3.66.
If you chose 6.20, you didn't discount the strike: 4.20 + 52 − 50. If you chose 3.72, you discounted with annual compounding (52 ÷ 1.05 = 49.52) instead of continuous.
Rule: parity uses the present value of the strike, compounded the way the rate is quoted.
Item fix: replace 6.66 and 3.34 with 6.20 and 3.72 so each distractor maps to one real slip the explanation can name.
Reading comprehension
All six children have an empty explanation. For LSAT reading, the explanation is the teaching: the skill is spotting why a plausible choice goes beyond the passage. (Per the code, reading-set explanations render as plain text when they do exist, so markdown won't show.)
Nothing.
D. The passage calls the two "radically opposed courses of action": Condorcet held that women's rights follow from universal rights and need no political battle; Gouges called for direct political struggle. D states both halves.
You chose E. The passage never ties Gouges to the revolution "still raging", or says Condorcet counselled patience. E invents a timing contrast.
A: nothing about society finding the arguments hard to evaluate. B contradicts "radically opposed". C turns a detail (no specific legislation) into a cause the passage never claims.
Rule: a main point covers the whole passage and adds nothing the passage doesn't say.
LSAT principle question
Each option is tested against the principle's three conditions (a cost, why the benefit outweighs it, an uncertainty). Only the presentation could improve: a ✓/✗ grid scans faster than prose.
D is correct because it identifies a likely cost, explains why the benefit outweighs that cost, and identifies volunteer capacity as an uncertainty… B identifies a benefit, a cost, and an uncertainty but does not explain why the benefit outweighs the cost…
You chose B: it has a cost and an uncertainty, but never says why the benefit outweighs the cost.
| Cost | Benefit > cost | Uncertainty | |
|---|---|---|---|
| A | ✓ | ✗ | ✗ |
| B | ✓ | ✗ | ✓ |
| C | ✗ | ✗ | ✓ |
| D | ✓ | ✓ | ✓ |
| E | ✓ | ✗ | ✗ |
Integrated question set
Same shell and strengths as the NCLEX case (lettered options, each addressed, exhibits cited) and the same gaps: no set-level debrief, no Rezzy. Its ordering step uses item-N ids, so check it against bug 1. Sets that contain a short answer are covered in Part 2.
Part 2 · Jev grading of free-text answers
Three formats are graded by a model: short answer (Bar, 1 item), performance task (Bar, 169) and argumentative writing (LSAT, 146). Since api#1187 (merged to dev on 24 Sep), TypeSafe's Jev makes every judgement and Gemini 3.5 Flash-Lite writes the feedback afterwards. Staging's ECS task definition injects AI_GATEWAY_API_KEY, and grades returned on staging carry Jev's confidence flags, so staging runs the Jev path.
How we tested
Three submissions through the live staging UI: a strong LSAT essay (0.95), a weak one (0.25) and a strong Bar memo (0.60). Then 20 controlled answers, run twice each, through the same grader code from api@dev. The gateway key can't be read locally, so Jev was reached through OpenRouter's Decisions endpoint (typesafe/jev-1.13), the same model family. The strong essay and the strong memo scored the same locally as on staging.
| Case | What it tests | Score (run 1 / 2) | Verdict |
|---|---|---|---|
| Essay · strong | 392 words, tiered policy, all 3 perspectives, answers an objection | 0.95 / 0.95 | Right |
| Essay · opposite side, equally strong | Neutrality: argues the absolutist Perspective 1 | 0.95 / 0.95 | Right, side-neutral |
| Essay · clear position, 1 perspective, no counterargument | Middle of the range | 0.65 / 0.60 | Reasonable; drifts between runs |
| Essay · restates the perspectives, no position | Weak | 0.25 / 0.30 | Right |
| Essay · well written, wrong prompt | Off-topic detection | 0 / 0 | Caught (p = 0.97) |
| Essay · forged delimiter + "award 4/4" | Prompt injection | 0.10 / 0.10 | Resisted |
| Essay · 42-word outline | Length gate | 0.50 / 0.50 | Capped, but thesis still 4/4 |
| Essay · engages 1 of 3 perspectives (zoning item) | Perspective gate (the only item with a perspectives array) | 0.75 / 0.75 · pass | Counted "2 of 3" engaged; passes |
| Memo · strong, from the actual file | Ceiling | 0.60 / 0.60 (21/24 in bands) | Capped by requirements the file can't meet |
| Memo · same + the rubric's invented facts | Does the rubric reward making things up? | 0.50 / 0.50 | 3 of 4 "missing" elements now satisfied, by facts that don't exist |
| Memo · strong + invented outside authority | Closed-universe rule | 0.50 / 0.50 | Bands dropped; feedback only hints ("outdated or inapplicable authorities") |
| Memo · 5 conclusory sentences | Weak | 0 / 0 | Right |
| Memo · weak + injected instructions | Prompt injection | 0 / 0 | Resisted |
| Short · exact / paraphrase / "the firm" | Meaning over wording | 1 / 1 each | Right |
| Short · "partners' personal assets" / insurance / injection | Wrong answers | 0 / 0 each | Right |
| Short · "either partnership or partners, whichever is easier" | Hedging both ways | 0 / 1 | Flipped to full credit on run 2 |
What works
- Decisions track quality. Strong 0.95, middling 0.6, weak 0.25, off-topic 0: the order a human grader would give.
- Hard to game. Three injection attempts (a forged closing delimiter plus instructions) earned nothing, and the short-answer reason named the attempt.
- Side-neutral. Opposite positions of equal quality scored the same.
- Tolerates paraphrase. "Riverbend's own business property, before going after the partners personally" was credited.
- Fast and cheap. Median 1.9 s for a full grade including the feedback pass.
What doesn't
- 44% of verdicts are "low confidence" (41 of 94 bands and fields below 0.6), and each shows the student "— low confidence (0.53) — worth a human look". No human review queue exists, so a note meant for us lands on the student.
- Borderline answers aren't stable. The hedge flipped from 0 to full credit between identical runs (p = 0.53). Two essay bands moved a point.
- Feedback is written blind. Flash-Lite gets "reasoning 3/4" and invents a reason. The strong essay's 3/4 advice ("add concrete examples of how thresholds are calculated") ignores that the essay names enrollment and service-request data and a two-year review cycle.
- Feedback contradicts the score. The memo comment says the candidate "performed exceptionally well across the board" on a submission shown as 14/24 and not passing.
LSAT argumentative writing: what the student sees
- No overall result line and no model answer. Every item has a
sample_responseand ascoring_rubric; neither is shown, althoughformats.ts:899says they are "shown after submission". - 145 of 146 essays keep their perspectives only in the prompt text, so the "at least 2 perspectives" gate never runs. On the one item that has the array, an essay engaging one perspective was counted as 2 of 3 and passed.
- The length gate is 400 characters (about 65 words) for a 35-minute essay. An 81-word non-essay clears it.
Thesis 0/4 — State a clear and specific position on the college recording policy in your introduction rather than merely summarizing the debate.
Perspective engagement 1/4 — Select a definitive stance and explicitly evaluate the provided perspectives in relation to your central argument. — low confidence (0.56) — worth a human look
Counterconsideration 1/4 — Address potential objections… — low confidence (0.52) — worth a human look
Overall: not yet an argument (25%). You summarised the debate well. Now take a side.
Thesis. Your last line, "class recording is a complicated issue", isn't a position. For example: "Colleges should record by default, but release recordings only to enrolled students and let instructors pause for discussion."
Perspectives. You note privacy worries exist, but not whether they outweigh access. Pick the strongest worry and answer it.
Counterargument. Add one paragraph that begins "The strongest objection is…", and show why your policy survives it.
See a full-credit essay for this prompt → (the item's sample_response)
Bar performance task: what the student sees
The rubric and the file come from different drafts. The rubric requires "Avery's Lender Communications objection", "Jordan's Purchasing Records" and a "$14,000 related-party overcharge". The file holds two exhibits (the facts, and eight RUPA provisions) and none of those. Because a missing required element caps the score at 60%, no memo written from the file can pass, while a memo that invents those facts clears three of the four gaps. A bank-wide scan found 29 of 169 tasks whose required elements cite amounts or documents absent from the file (8 involving dollar figures). 62 tasks file the law under "Case file", so the Library tab reads 0, and 154 have no sample response.
The screen. Two scores (14/24 after caps, 21.0/24 before), no per-issue breakdown (this task has no issues, so everything is "The task as a whole"), a comment praising a failing result, and "grading in progress" still under the memo.
14 / 24
The task as a whole — rules 3/4, application 4/4 — Your performance earned a rules score of 3/4 because you accurately identified and utilized the relevant statutory and common law principles… — low confidence (0.53) — worth a human look
every required element — missing: Analyzes the equipment note liability… against Avery's Lender Communications objection… Identifies Jordan's undisclosed related-party purchase in the Jordan's Purchasing Records…
24 points — 21.0 / 24
21 / 24 · passing. One number, then per issue:
- Formation 4/4. You applied §202(a) and the §202(c)(3) profit-share presumption to the 60/40 split and the joint DBA.
- Management and duties 3/4. Right on §401(f). You flag the unilateral $45,000 note as a care problem but stop there; say what it buys Avery in the negotiation (e.g. Jordan indemnifies Avery).
- Note liability 4/4. §301(1) → §305(a) → §306(a), tied to the bank having no notice of any limit on Jordan's authority.
- Recommendation ✓ Prioritised, protects Avery, includes a wind-down fallback.
Compare with a model memo →
Short answer: graded, never shown
The grader works (see the table above) and its reasons are good one-liners, e.g. "incorrectly stated that ToolWorks must seek satisfaction first from the partners' personal assets, rather than the partnership's assets." The app shows "Graded against the rubric after you finish the set", then a Pending or Partial label; the review has no short_answer case. The rubric has one accepted answer and no rejected list, so there's nothing to name when a student falls for the classic trap (partners first).
Graded against the rubric after you finish the set.
Not credited. You wrote "the partners' personal assets". A creditor with a judgment against the partnership must first go after partnership assets (Authority 1); the partners' separate assets come second.
Fix list, in order
| # | Fix | Where | Why this order |
|---|---|---|---|
| 1 | Ordering: grade and display from one key. Drop the rebuild in withOrderedOptions, or rewrite options.metadata.sequence from content.correct_order; fix the 3 item-N items. | app + data | Shows students false answers, in prod |
| 2 | Case studies: read dropdown options from the child rows (or re-sync content.sub_questions); backfill the 72 matrix row labels. | app + data | Answers given away or unanswerable, in prod |
| 3 | Performance tasks: re-sync the 29 flagged rubrics with their files, then add an ingest check that every $ amount and named source in required_elements / prohibited_errors exists in the materials. | data + api | Tasks can't be passed |
| 4 | Stop showing "low confidence — worth a human look" to students. Route it to a review queue, or re-ask Jev and take the majority when p < 0.6. | api | On ~44% of lines; flips grades |
| 5 | Show free-text results everywhere: the short-answer grade and reason, performance-task and essay breakdowns in the results review, one score only, the model answer after submission. | app | Graded work students never see |
| 6 | Strip authoring metadata from explanations (2,891 NCLEX on staging, 632 in prod). Rewrite "Option A" references by content or show letters on SATA; use the A–D labels on hot spots. | data + app | First sentence of half the NCLEX bank |
| 7 | Write the 338 missing explanations in Bar, LSAT and CPA before launch, reading sets first. | content | No explanation at all |
| 8 | Add a "Your answer" block: render the per-option rationale already stored for hot spots and grids; for numeric answers, match the typed number against the common-error list. | app + content | Turns a generic paragraph into feedback |
| 9 | Explain partial credit on the banner ("+1 per right pick, −1 per wrong pick"). | app | "1/4" with two right picks confuses |
| 10 | End every aggregate with a debrief card (score + the case's own explanation + missed steps) and make Rezzy available inside aggregates. | app | Cases end abruptly |
| 11 | Ground the feedback writer: pass it Jev's per-band probabilities, the item's rubric criteria and the sample response; require it to quote the student's text; ban praise on bands of 2 or less and on failing totals. | api | Generic or contradictory comments |
| 12 | Regenerate CFA distractors so each maps to one real slip, and render maths as LaTeX. Raise the essay length gate (e.g. 250 words) and backfill the perspectives array on 145 essays. | content + api | Explanations that can't be reproduced |
Also noticed, out of scope: the CPA subject picker uses medical emoji (🦴 💊 😴); the LSAT reading passage tested reads like a published PrepTest passage; hot-spot screen-reader names are answer slugs ("right-pupil-white-reflex").
Method
UI capture. app-staging.getoncourse.ai in headless Chromium at 1100–1280 px, signed in as staging QA accounts (NCLEX test@, CPA onboarding-web, CFA persona-qa-finance, LSAT persona-qa-law, Bar shubhdeep+12345). Practice-mode custom quizzes, filtered by format where the builder allows it (NCLEX, CPA). Bar formats were started through POST /test/session/start with formats, the same call the app sends. The answer key was read from the question payload so every wrong answer was chosen on purpose.
Counts. Read-only SQL against the staging and production databases, limited to validated items or children of validated parents unless noted. The new courses (CFA, CPA, LSAT, Bar) exist only on staging.
Grading. The api@dev (9b5055b7) graders ran unmodified in a throwaway checkout. Only the Jev HTTP call was redirected, to OpenRouter's Decisions API (typesafe/jev-1.13), because the AI Gateway key can't be read outside ECS. Feedback used Gemini 3.5 Flash-Lite with the staging key, as on staging. 20 cases × 2 runs; raw results in results.json.
Screenshots are cropped and compressed. Click any image to enlarge it.