Generating 14,000+ exam-grade questions, and ensuring they are right
Examworthy is a certification study platform built solo. The hard part was never the app - it was a reliable LLM generation pipeline and a multi-vendor evaluation layer that does not rely on any single model.
- Built solo
- 2026
- Perth, WA
- certifications
- 58*certifications
- audited questions
- 14000+audited questions
- spaced-repetition flashcards
- 36000+spaced-repetition flashcards
* At the time of writing - the catalogue grows weekly.
The hard part is correct content at volume.
Certification candidates choose between cheap question dumps that are often wrong or stolen, and expensive official courseware. The hard problem behind a good product is not the interface.
It is producing thousands of original, technically-correct, exam-representative questions across dozens of certifications - cheaply enough that the unit cost does not sink the business. That is an AI engineering problem, not a content-writing one.
Defence in depth for content.
Every question moves through a deliberate state machine. Nothing reaches the product on a single model's say-so.
- 1 - Generate
- Each run is pinned to one certification, domain, and exam objective from the public blueprint. Provenance fields are overwritten after generation, so the model physically cannot drift them.
- 2 - Eight deterministic gates
- Structural checks, content-quality classifiers, and cross-field consistency checks - spanning schema integrity, distractor quality, bias detection, and citation integrity. A failure is a hard reject.
- 3 - Three-judge eval panel
- Each survivor is scored on five rubric dimensions by three independent LLM judges. Correctness and fabricated-reference are veto dimensions - one failing vote kills the item regardless of the majority.
- 4 - Human review
- A person is the last gate. Eval-passed questions are read by hand and explicitly approved, edited, or cut - nothing reaches the scored pool without that sign-off, and any edit is sent back through every gate before it can return.
An eval that does not trust one vendor.
A cheaper hybrid pipeline was benchmarked blind against the production bank, provenance stripped, scored by judges from two different model families. It reached production grade. It did not ship, because the one material gap was citation quality.
- Why two families
- An author and a judge from the same vendor share a blind spot, so the judge waves through the mistakes the author is prone to. Agreement between families is the only number worth defending, and here it was 91%.
- The decision
- A cheaper pipeline reaching production grade does not justify swapping a known-good bank for one with weaker citations. Keep the bank, stay on the model that is ahead, retest when the next capable lower-cost model lands.
- The method, in full
- The scoring tables, the four-model author comparison down to a local 14B, and what generalises beyond exam questions are written up separately as research.
A real study app behind the pipeline.
- Spaced repetition
- A from-scratch SM-2 scheduler, implemented as a pure function with no hidden clock - fully deterministic and unit-testable, with Anki-style four-button grading.
- Readiness, per domain
- A dashboard that derives mastery from practice history and predicts exam score against the pass line, with drill-down into the weak concepts costing the most marks.
- Built for production
- Custom auth, Stripe billing, self-hosted Postgres behind PgBouncer, strict security headers, and a CI gate that fails the build on banned characters or hardcoded colours.
Not prompting. Engineering.
- 0-to-1, solo
- A complete, deployed, billable SaaS - frontend, backend, auth, payments, content, infrastructure - built by one engineer in weeks, with the catalogue still growing 1-2 certifications per week off the same pipeline.
- A real LLM system
- Deterministic gates, a multi-judge eval panel, schema enforcement, and a measured approach to model selection, cost, and bias - not a single prompt behind a button.
- Judgement under tradeoffs
- Frontier-versus-local and cost-versus-quality decisions backed by evals, not opinions - including the discipline to not ship a change that was only churn.
- TypeScript
- Next.js 15
- React 19
- PostgreSQL + PgBouncer
- Custom auth + OAuth
- Stripe
- Claude Opus + Sonnet
- GLM 5.2
- Qwen (local)
- Vitest
- Playwright
Have something to build?
Tell us what you are working on. We reply within a day, in plain language, with a clear next step.
Perth, Western Australia