Linguistic Acceptability Assessment — Grammaticality Judgment
Also known as: grammaticality judgment, acceptability judgment, CoLA task, Dilbilgisel Kabul Edilebilirlik Değerlendirme
Linguistic acceptability assessment is a natural-language-processing task that automatically estimates whether a sentence would be judged grammatically acceptable by a native speaker of the target language. Grounded in Chomsky's (1957) distinction between grammatical and ungrammatical utterances, the task was formalised as a neural benchmark by Warstadt, Singh and Bowman (2019) through the Corpus of Linguistic Acceptability (CoLA). It is used in language-learning research, linguistics studies, and quality auditing of natural-language-generation (NLG) systems.
Key highlights
- Automates native-speaker acceptability intuitions at scale, enabling evaluation of large sentence sets without manual annotation.
- Neural models trained on CoLA capture subtle grammatical contrasts that rule-based parsers miss.
- Output can be binary (acceptable/unacceptable) or graded, making it flexible for both classification and regression tasks.
- Applicable in language-learning feedback, NLG quality auditing, and computational linguistics without domain-specific rule writing.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Linguistic acceptability assessment fits when you need to automatically score the grammatical well-formedness of sentences in a target language and a model trained for that language is available. Typical applications include auditing NLG pipeline output, providing feedback in language-learning tools, and testing linguistic hypotheses computationally. The method requires a target-language model or annotated data; it cannot be applied to a language for which no trained model exists. The acceptability criterion must be measurable as binary or continuous, and the sentence sample should contain at least 20 items for meaningful evaluation.
Strengths & limitations
- Automates native-speaker acceptability intuitions at scale, enabling evaluation of large sentence sets without manual annotation.
- Neural models trained on CoLA capture subtle grammatical contrasts that rule-based parsers miss.
- Output can be binary (acceptable/unacceptable) or graded, making it flexible for both classification and regression tasks.
- Applicable in language-learning feedback, NLG quality auditing, and computational linguistics without domain-specific rule writing.
- A trained model for the target language must exist; coverage for low-resource languages is limited.
- Acceptability is gradient and context-dependent in natural language; a binary label may oversimplify borderline cases.
- Neural models reflect the biases of their training data, which may not match the dialect or register of the sentences under evaluation.
- A minimum of approximately 20 sentences is needed for stable evaluation; very small sets yield unreliable aggregate metrics.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
What is CoLA and why is MCC used instead of accuracy?
CoLA (Corpus of Linguistic Acceptability) is a benchmark of 10,657 English sentences from linguistics publications, labelled by expert judges as acceptable or unacceptable. Because roughly 69% of sentences are labelled acceptable, a model that always predicts 'acceptable' achieves 69% accuracy without learning anything. Matthews Correlation Coefficient (MCC) accounts for class imbalance and is therefore the standard metric on CoLA.
Can this method be applied to languages other than English?
Yes, but a model trained on acceptability judgments in the target language must exist. For languages beyond English, resources are limited; the analyst should verify that the model's training domain and register match the sentences under evaluation before drawing conclusions.
What is the difference between linguistic acceptability and grammaticality?
Grammaticality is a formal property of whether a sentence is generated by a given grammar. Acceptability is an empirical, gradient notion reflecting native-speaker intuitions, which can be influenced by frequency, processing difficulty, and context. The CoLA benchmark targets acceptability as judged by expert native speakers rather than formal grammaticality, making it a more realistic target for NLP systems.
How many sentences do I need to get meaningful results?
The method requires at least 20 sentences for stable aggregate metrics. With fewer sentences, individual label confidence matters more than aggregate MCC or accuracy. For robust evaluation — especially when comparing conditions — aim for several hundred sentences and report confidence intervals around aggregate metrics.
Sources
- 1.Warstadt, A., Singh, A. & Bowman, S. (2019). Neural Network Acceptability Judgments. Transactions of the Association for Computational Linguistics, 7, 625–641.
- 2.Chomsky, N. (1957). Syntactic Structures. Mouton, The Hague.ISBN 978-9027933249
You have read it. What now?
Cite this page
ScholarGate. (2026, June 1). Linguistic Acceptability Assessment. ScholarGate. https://scholargate.app/text-mining/linguistic-acceptability