Pretrained language models, neural models pretrained on massive amounts of data, have established the state of the art in a range of NLP tasks. They are based on a modern machine-learning technique, the Transformer which relates all items simultaneously to capture semantic relations in sequences. However, it differs from what humans do. Humans read sentences one-by-one, incrementally. Can neural models benefit by interpreting texts incrementally as humans do? We investigate this question in coherence modeling. We propose a coherence model which interprets sentences incrementally to capture lexical relations between them. We compare the state of the art in each task, simple neural models relying on a pretrained language model, and our model in two downstream tasks. Our findings suggest that interpreting texts incrementally as humans could be useful to design more advanced models.
SEEK ID: https://publications.h-its.org/publications/1156
Research Groups: Natural Language Processing
Publication type: InProceedings
Book Title: Proceedings of the 28th International Conference on Computational Linguistics
Publisher: International Committee on Computational Linguistics
Citation: In Proceedings of the 28th International Conference on Computational Linguistics (COLING), Online, December 2020, pp. 6752–6758
Date Published: 1st Dec 2020
URL: https://www.aclweb.org/anthology/2020.coling-main.594
Registered Mode: imported from a bibtex file
Views: 5588
Created: 4th Dec 2020 at 13:07
Last updated: 5th Mar 2024 at 21:24
This item has not yet been tagged.
None