Towards More Intentional LLMs

Summary: It is widely assumed in Pragmatics that when understanding and generating language, people analyze and formulate intentions,  namely what the speaker aims to do with their words. In this talk, I will present our initial investigation on  how to endow LLMs with the same ability. As a first step, we have explored ARR, an intuitive and effective zero-shot prompting method that explicitly incorporates three key steps in answering questions: Analyzing the intent of the question, Retrieving relevant information, and Reasoning step by step. In comprehensive experiments across diverse and challenging Question-Answering tasks, we demonstrate that ARR consistently outperforms the popular technique of Chain-Of-Thought, with intent analysis playing a vital role in the process. 

While ARR is about an LLM paying attention to the intentions behind a question, in a second line of work, we introduce the concept of Speaking with Intent (SWI), where the LLM is explicitly prompted to generate the intent behind every sentence it produces. Our hypothesis being that this provides high-level planning to guide subsequent analysis and communication.  Empirically, we show that SWI enhances the reasoning capabilities and generation quality of LLMs both on reasoning-intensive Question-Answering and Text Summarization benchmarks.   

Overall, ARR and SWI are just initial steps in making LLMs more intentional and therefore more rational, transparent and safe. 

Bio: Giuseppe Carenini is a Professor in Computer Science, an Amazon Scholar and the Director of the Master in Data Science at UBC (Vancouver, Canada). His work on natural language processing and information visualization to support decision making has been published in over 160 peer-reviewed papers (including best paper at UMAP-14, ACM-TiiS-14 and Sigdial-24). Dr. Carenini was the area chair for many conferences including for ACL'21 in “Natural language Generation”, as well as Senior Area Chair for NAACL'21 in “Discourse and Pragmatics”.  Dr. Carenini  was also the Program Co-Chair for IUI 2015 and  for SigDial 2016. In 2011, he published a co-authored book on “Methods for Mining and Summarizing Text Conversations”.  In his work, Dr. Carenini has also extensively collaborated with industrial partners, including Amazon, Microsoft, Google, Salesforce, ServiceNow, Huawei and IBM. He was awarded a Google Research Award in 2007 and a Yahoo Faculty Research Award in 2016.