UChicago CompLing Lab
Projects
Members of our lab are engaged in a range of projects aimed at better understanding the linguistic competence and nature of sensitivities
of pre-trained language models, studying and improving competence with compositional meaning understanding in NLP models, as well as
using computational models to test hypotheses about the mechanisms underlying language processing in humans.
Controlled evaluation of compositional meaning understanding in pre-trained LMs
In this body of work our priority is to establish reliable and robust evaluations for the compositional meaning
competence of pre-trained language models. This work employs controlled tests that tease apart compositional meaning
understanding from more superficial predictive behaviors, and often draws on methodology from psycholinguistics and cognitive neuroscience.
Example papers:
- Misra, K., Rayz, J., Ettinger, A. (2022). COMPS: Conceptual Minimal Pair Sentences for testing Property Knowledge and Inheritance in Pre-trained Language Models.
- Pandia, L., Ettinger, A. (2021). Sorting through the noise: Testing robustness of information processing in pre-trained language models. EMNLP 2021.
- Yu, L., Ettinger, A. (2021). On the Interplay Between Fine-tuning and Composition in Transformers. ACL Findings 2021.
- Yu, L., Ettinger, A. (2020). Assessing Phrasal Representation and Composition in Transformers. EMNLP 2020.
- Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. TACL 2020.
- Ettinger, A., Elgohary, A., Phillips, C., Resnik, P. (2018). Assessing Composition in Sentence Vector Representations. COLING 2018.
- Ettinger, A., Elgohary, A., Resnik, P. (2016). Probing for semantic evidence of composition by means of simple classification tasks. RepEval 2016.
~~~~~~~~~~~~~~~~~
Assessment of other linguistic competence and sensitivity in pre-trained LMs
Beyond assessment of compositional meaning understanding per se, our group has investigated a wide range of linguistic competences, behavioral sensitivities, and encoding patterns in pre-trained language models.
Topics of interest include conceptual property knowledge and induction, discourse and pragmatic competence, response to syntactic anomaly, sensitivity to presence of "prime words", and encoding in contextual embeddings
of different types of linguistic information from surrounding words.
Example papers:
Property knowledge and inductive reasoning
- Misra, K., Rayz, J., Ettinger, A. (2022). COMPS: Conceptual Minimal Pair Sentences for testing Property Knowledge and Inheritance in Pre-trained Language Models.
- Misra, K., Taylor Rayz, J., Ettinger, A. (2022). A Property Induction Framework for Neural Language Models. CogSci 2022.
- Misra, K., Ettinger, A., Taylor Rayz, J. (2021). Do language models learn typicality judgments from text? CogSci 2021.
Pragmatic competence
- Kim, S.J., Yu, L., Ettinger, A. (2022). “No, they did not”: Dialogue response dynamics in pre-trained language models. COLING 2021.
- Pandia, L., Cong, Y., Ettinger, A. (2021). Pragmatic competence of pre-trained language models through the lens of discourse connectives. CoNLL 2021.
Assorted dimensions of model linguistic sensitivity and representation
- Wu, Q., Ettinger, A. (2021). Variation and generality in encoding of syntactic anomaly information in sentence embeddings. BlackboxNLP 2021.
- Misra, K., Ettinger, A., Taylor Rayz, J. (2020). Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming. EMNLP Findings 2020.
- Klafka, J., Ettinger, A. (2020). Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words. ACL 2020.
~~~~~~~~~~~~~~~~~
Computational modeling of real-time language processing in humans
In this body of work we design computational models to test hypotheses about the cognitive mechanisms driving
the robust and rapid language processing in the human brain. We draw selectively on probabilistic and representational
measures from pre-trained language models to increase stimulus-level statistical sensitivities of our cognitive models.
Example papers:
- Li, J., Ettinger, A. Heuristic Interpretation as Rational Inference: A Computational Model of the N400 and P600 in Language Processing.
- Ettinger, A., Feldman, N.H., Resnik, P., Phillips, C. (2016). Modeling N400 amplitude using vector space models of word representation. CogSci 2016.
- Ettinger, A., Linzen, T. (2016). Evaluating vector space models using human semantic priming results. RepEval 2016.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~