Publications

What is a Publication?
1701 Publications visible to you, out of a total of 1701

Abstract (Expand)

Unlike classical lexical overlap metrics such as BLEU, most current evaluation metrics for machine translation (for example, COMET or BERTScore) are based on black-box large language models. They often achieve strong correlations with human judgments, but recent research indicates that the lower-quality classical metrics remain dominant, one of the potential reasons being that their decision processes are more transparent. To foster more widespread acceptance of novel high-quality metrics, explainability thus becomes crucial. In this concept paper, we identify key properties as well as key goals of explainable machine translation metrics and provide a comprehensive synthesis of recent techniques, relating them to our established goals and properties. In this context, we also discuss the latest state-of-the-art approaches to explainable metrics based on generative models such as ChatGPT and GPT4. Finally, we contribute a vision of next-generation approaches, including natural language explanations. We hope that our work can help catalyze and guide future research on explainable evaluation metrics and, mediately, also contribute to better and more transparent machine translation systems.

Authors: Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva, Wei Zhao, Yang Gao, Steffen Eger

Date Published: 1st Mar 2024

Publication Type: Journal

Abstract

Not specified

Authors: Steve Schulze, Claes Fransson, Alexandra Kozyreva, Ting-Wan Chen, Ofer Yaron, Anders Jerkstrand, Avishay Gal-Yam, Jesper Sollerman, Lin Yan, Tuomas Kangas, Giorgos Leloudas, Conor M. B. Omand, Stephen J. Smartt, Yi Yang, Matt Nicholl, Nikhil Sarin, Yuhan Yao, Thomas G. Brink, Amir Sharon, Andrea Rossi, Ping Chen, Zhihao Chen, Aleksandar Cikota, Kishalay De, Andrew J. Drake, Alexei V. Filippenko, Christoffer Fremling, Laurane Fréour, Johan P. U. Fynbo, Anna Y. Q. Ho, Cosimo Inserra, Ido Irani, Hanindyo Kuncarayakti, Ragnhild Lunnan, Paolo Mazzali, Eran O. Ofek, Eliana Palazzi, Daniel A. Perley, Miika Pursiainen, Barry Rothberg, Luke J. Shingles, Ken Smith, Kirsty Taggart, Leonardo Tartaglia, WeiKang Zheng, Joseph P. Anderson, Letizia Cassara, Eric Christensen, S. George Djorgovski, Lluı́s Galbany, Anamaria Gkini, Matthew J. Graham, Mariusz Gromadzki, Steven L. Groom, Daichi Hiramatsu, D. Andrew Howell, Mansi M. Kasliwal, Curtis McCully, Tomás E. Müller-Bravo, Simona Paiano, Emmanouela Paraskeva, Priscila J. Pessi, David Polishook, Arne Rau, Mickael Rigault, Ben Rusholme

Date Published: 1st Mar 2024

Publication Type: Journal

Abstract

Not specified

Authors: R. Andrassy, G. Leidi, J. Higl, P. V. F. Edelmann, F. R. N. Schneider, F. K. Röpke

Date Published: 1st Mar 2024

Publication Type: Journal

Abstract

Not specified

Authors: V. A. Bronner, F. R. N. Schneider, Ph. Podsiadlowski, F. K. Röpke

Date Published: 1st Mar 2024

Publication Type: Journal

Abstract

Not specified

Authors: Javier Morán-Fraile, Alexander Holas, Friedrich K. Röpke, Rüdiger Pakmor, Fabian R. N. Schneider

Date Published: 1st Mar 2024

Publication Type: Journal

Abstract

Not specified

Authors: V. A. Bronner, F. R. N. Schneider, Ph. Podsiadlowski, F. K. Röpke

Date Published: 1st Mar 2024

Publication Type: Journal

Abstract

Not specified

Authors: M. Golebiewski, G. Bader, P. Gleeson, T. E. Gorochowski, S. M. Keating, M. Konig, C. J. Myers, D. P. Nickerson, B. Sommer, D. Waltemath, F. Schreiber

Date Published: 1st Mar 2024

Publication Type: Journal

Powered by
(v.1.16.0)
Copyright © 2008 - 2024 The University of Manchester and HITS gGmbH