Automated Essay Scoring (AES)
Also known as: AES, automated writing evaluation, AWE, Otomatik Deneme Puanlaması
Automated Essay Scoring (AES) is a natural-language-processing task in which a computational model assigns scores to student-written essays across dimensions such as grammatical correctness, coherence, content richness, and organisation — replicating, at scale, what a human rater would do. The approach was formalised as a research field by Shermis and Burstein (2013) and has been transformed since 2019 by transformer language models, particularly BERT, which allow AES systems to leverage deep contextual representations of text.
Read the full method
Sign in with a free account to read this section.
Method map
The neighbourhood of related methods — select a node to explore.
When to use it
AES is appropriate when a labelled corpus of scored essays exists, the scoring rubric is clearly defined and stable, and the volume of essays to be scored is too large for manual scoring alone. The method suits large-scale standardised testing, intelligent tutoring systems that provide formative feedback, and educational research that requires consistent scoring across many participants. A minimum of around 30 scored essays is the bare threshold; in practice, several hundred to thousands of labelled essays are needed for reliable models. If no labelled scored data is available, or the rubric is ill-defined, AES cannot be applied.
Strengths & limitations
- Scales to arbitrarily large volumes of essays with consistent application of the scoring rubric.
- Eliminates inter-rater drift and fatigue that affect repeated human scoring.
- Transformer-based systems capture contextual, grammatical, and semantic signals simultaneously.
- Enables rapid formative feedback loops in intelligent tutoring contexts.
- Requires a labelled corpus of scored essays; the system cannot score without prior human-rated training data.
- Model performance is bounded by the quality and size of the labelled corpus.
- Systems can be gamed by surface features such as essay length or rare vocabulary without genuine quality.
- Bias toward dominant dialects or writing styles present in the training data is a documented risk.
Frequently asked
How many scored essays do I need to train an AES model?
The minimum threshold is around 30, but this is rarely sufficient for reliable predictions. In practice, several hundred labelled essays are needed for a traditional feature-based model, and several thousand for fine-tuning a transformer. Corpus size is the single strongest predictor of model quality.
Can AES replace human raters entirely?
AES is best used alongside human raters rather than as a complete replacement. High-stakes decisions — such as pass/fail in a standardised exam — should involve at least one human score, with AES serving as a second rater or as a filter for borderline cases. In low-stakes formative contexts, AES alone can provide useful feedback.
What metric should I use to evaluate an AES model?
Quadratic weighted kappa (QWK) is the standard metric because it accounts for ordinal agreement and penalises large disagreements more than small ones. Pearson correlation and mean squared error are reported alongside QWK. Accuracy alone is not appropriate for ordinal score scales.
Does BERT always outperform traditional feature-based methods?
On large, well-labelled corpora BERT-based models generally do outperform hand-crafted feature pipelines. On small corpora (under a few hundred essays) the advantage shrinks, and regularised regression on interpretable features can be competitive while offering greater transparency into which dimensions drive the score.
Sources
- Shermis, M.D. & Burstein, J. (2013). Handbook of Automated Essay Evaluation. Routledge. link ↗
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 4171-4186. DOI: 10.18653/v1/N19-1423 ↗
How to cite this page
ScholarGate. (2026, June 1). Automated Essay Scoring (AES). ScholarGate. https://scholargate.app/en/text-mining/automated-essay-scoring
Which method?
Set this method beside its closest kin and read them side by side — the library lays the books on the table; the choice is yours.
- BERT EmbeddingsText mining↔ compare
- Readability AnalysisText mining↔ compare
- Sentiment AnalysisText mining↔ compare
- Text ClassificationText mining↔ compare