Evaluating Evaluation of AI Systems (Meta-Eval 2020)
Workshop at The thirty fourth AAAI Conference on Artificial Intelligence
Date: 8 Feb 2020
Location: Hilton New York Midtown (Conference Venue)
- Title: The questions that the current AI can't answer.
- Abstract: NLP is witnessing an explosion of question answering datasets, most of which get "solved" within months of publication. However, that does not necessarily mean that we are making fast progress towards machine language understanding. I discuss the current proposals for making the datasets more difficult, and the ways in which the current deep learning models "cheat", avoiding the complex verbal reasoning we expect them to perform.
- Bio: Anna Rogers is a post-doctoral associate at University of Massachusetts (Lowell). Her research focuses on representation learning, natural language understanding, evaluation methodology for NLP, and computational social science.
- Title: How can we know it is shoulders we stand on? - Reproducibility and Evaluation
- Title: People + AI = Magic
- Abstract: We will share new innovations in machine learning accuracy and the importance of maintaining people in the loop for AI applications. Many machine learning solutions require humans to review low confidence inference for accuracy, audit or compliance. We will demonstrate new mechanisms for bringing nuanced human inputs into the ML workflow which will enable sectors which had previously relied solely on humans or could not use ML at all.
- Bio: AWS A2I is an organization within the AI/ML division of AWS that is focused on accelerating ML deployment and improving model accuracy.
Contact Organizing Committee
Email: rigorous-evaluation AT googlegroups.com