How we tested our grading engine

The front page of this site claims our grading engine agrees with human examiners. This page is the evidence behind that claim.

We publish nothing we have not measured. Every number below comes from a benchmark run we can re-execute, with raw per-essay result files on disk. If a number is not on this page, we are not claiming it.

We took two public corpora of essays that had already been scored by human raters, and graded them blind with RALE — the same production engine, the same code path, that grades a school submission. Then we measured how closely the engine’s ranking of the essays agreed with the human raters’ ranking.

Specifics:

• Blind. The engine never saw the human scores. It received each essay and its original writing prompt — the same context a real submission gets — and nothing else.

• Production engine, no tuning. We ran the engine exactly as deployed. We did not adjust prompts, models, or settings toward any of these test sets before measuring. (One calibration change came out of these results — we disclose it in section 4.)

• Stratified samples. From each corpus we drew 100 essays spread across the full human score range, so the test covers weak, middling, and strong writing rather than clustering in the middle.

• Two independent human-rater pools:

  ELLIPSE

  essays by English learners in US classrooms, scored by trained raters on six analytic dimensions. N = 100.

  ICNALE GRA

  essays by learners across Asia (including Korean learners), rated by a trained multi-rater panel on holistic and analytic dimensions. N = 100.

One production grader. Two separate sets of human professionals who have never heard of us. 200 scored essays in the comparison (300 result files on disk, counting a re-run described in section 4).

Agreement is reported as Spearman’s rank correlation (ρ).

* ICNALE’s exported summary metric is inverse-scaled (lower = better), so the raw correlation prints negative; we report the absolute value. See the footnote in section 5.

How to read these numbers. Spearman’s ρ runs on a 0-to-1 scale for agreement. 1.0 would mean the engine and the human raters put every essay in exactly the same order — which is not achievable in practice, because essay scoring always carries human judgement and noise. Across both corpora, our agreement with human raters falls between 0.83 and 0.85.

What we do not claim. We did not measure how often two human examiners agree with each other. Neither corpus gave us two independent raters per essay, so we never computed inter-rater reliability, and we make no claim about it. Our claim is exactly this: Agreement of 0.83–0.85 with human raters across two independent corpora (ELLIPSE 0.846; ICNALE 0.829, absolute). Nothing more.

The benchmark did not only confirm things. It found a defect.

• Writing only. These results cover written essays. Speaking assessment and IELTS-band validation have not yet been benchmarked this way. Until they are, we make no equivalent accuracy claim for them.

• Samples, not full corpora. Each benchmark used a stratified sample of 100 essays, not the full dataset. Larger runs with confidence intervals are planned, not done.

• Our protocol, not peer review. We designed and ran these benchmarks ourselves. The corpora and human scores are external and public, but no third party has audited our procedure.

• ICNALE scale direction. ICNALE’s summary export scores essays on an inverse scale (lower = better). We verified this against the corpus’s own CEFR bands before reporting: on their metric, A2 writers average 41.0 and B2 writers 32.2, while RALE’s scores rise 29.9 → 34.2 across the same bands. The raw correlation is therefore negative; the agreement in real terms is +0.829. We state this rather than silently flipping a sign.

• Population fit. ELLIPSE is drawn from US classroom English learners; some of the compression observed there may partly reflect population mismatch. This is why independent replication on a second, examiner-marked benchmark mattered before we shipped the calibration.

• A third benchmark, excluded here. A third benchmark on a Cambridge-examiner-marked research corpus is complete internally; its license does not permit citation in commercial material, so it is excluded here pending permission.

Every number on this page can be recomputed from raw files. The per-essay results are stored as line-delimited JSON:

• bench_results.jsonlELLIPSE run (N = 100)
• bench_results_v2.jsonlELLIPSE re-run after the top-end experiment
• gra_results.jsonlICNALE GRA run (N = 100), with gra_report.txt

alongside the runner and analysis scripts (bench.py, bench2.py, gra_bench.py, analyze.py). All correlations were re-verified from these raw files, most recently on 2026-07-27.

The benchmark is now a permanent regression suite: when the grading engine changes, we re-run the same essays and compare against these baselines. If agreement drops, the change does not ship.

• ELLIPSE. Crossley, S. A., Tian, Y., Baffour, P., Franklin, A., Kim, Y., Morris, W., Benner, B., Picou, A., & Boser, U. (2023). Measuring second language proficiency using the ELLIPSE Corpus. International Journal of Learner Corpus Research, 9(2), 248–269. The ELLIPSE corpus is released under the CC BY-NC-SA 4.0 license; this attribution is made in accordance with that license.
• ICNALE. Ishikawa, S. (2023). The ICNALE Guide. Routledge.