publications

2023

1. RANLP

   Challenges of GPT‑3‑based Conversational Agents for Healthcare

   Fabian Lechner, Allison Lahnala, Charles Welch, and Lucie Flek

   In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2023), Sep 2023

   Abs Bib PDF

   The potential to provide patients with faster information access while allowing medical specialists to focus on urgent tasks makes medical domain dialog agents appealing. However, there can be dire consequences due to the limitations of large-language models (LLMs) built into such agents. This paper investigates the challenges and risks of using GPT-3-based models for medical question-answering (MedQA). We perform several evaluations contextualized in terms of standard medical principles. We provide a procedure for manually designing patient queries to stress-test high-risk limitations of LLMs in MedQA systems. Our analysis shows that the LLMs fail to respond safely to these queries, producing invalid medical information, dangerous recommendations, and offensive content.

   @inproceedings{lechner-lahnala-gpt3-healthcare,
     title = {Challenges of GPT‑3‑based Conversational Agents for Healthcare},
     author = {Lechner, Fabian and Lahnala, Allison and Welch, Charles and Flek, Lucie},
     booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2023)},
     month = sep,
     year = {2023},
     address = {},
     publisher = {},
     url = {},
     pages = {}
   }

2. WASSA

   Domain Transfer for Empathy, Distress, and Personality Prediction

   Fabio Gruschka, Allison Lahnala, Charles Welch, and Lucie Flek

   In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, Jul 2023

   Abs Bib PDF

   This research contributes to the task of predicting empathy and personality traits within dialogue, an important aspect of natural language processing, as part of our experimental work for the WASSA 2023 Empathy and Emotion Shared Task. For predicting empathy, emotion polarity, and emotion intensity on turns within a dialogue, we employ adapters trained on social media interactions labeled with empathy ratings in a stacked composition with the target task adapters. Furthermore, we embed demographic information to predict Interpersonal Reactivity Index (IRI) subscales and Big Five Personality Traits utilizing BERT-based models. The results from our study provide valuable insights, contributing to advancements in understanding human behavior and interaction through text. Our team ranked 2nd on the personality and empathy prediction tasks, 4th on the interpersonal reactivity index, and 6th on the conversational task.

   @inproceedings{gruschka-etal-2023-caisa,
     title = {Domain Transfer for Empathy, Distress, and Personality Prediction},
     author = {Gruschka, Fabio and Lahnala, Allison and Welch, Charles and Flek, Lucie},
     booktitle = {Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, {\&} Social Media Analysis},
     month = jul,
     year = {2023},
     address = {Toronto, Canada},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.wassa-1.50},
     doi = {10.18653/v1/2023.wassa-1.50},
     pages = {553--557}
   }

2022

1. EMNLP

   A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing

   Allison Lahnala, Charles Welch, David Jurgens, and Lucie Flek

   In Findings of the Association for Computational Linguistics: EMNLP 2022, Dec 2022

   Abs Bib PDF

   We review the state of research on empathy in natural language processing and identify the following issues: (1) empathy definitions are absent or abstract, which (2) leads to low construct validity and reproducibility. Moreover, (3) emotional empathy is overemphasized, skewing our focus to a narrow subset of simplified tasks. We believe these issues hinder research progress and argue that current directions will benefit from a clear conceptualization that includes operationalizing cognitive empathy components. Our main objectives are to provide insight and guidance on empathy conceptualization for NLP research objectives and to encourage researchers to pursue the overlooked opportunities in this area, highly relevant, e.g., for clinical and educational sectors.

   @inproceedings{lahnala-etal-2022-critical,
     title = {A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing},
     author = {Lahnala, Allison and Welch, Charles and Jurgens, David and Flek, Lucie},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
     month = dec,
     year = {2022},
     address = {Abu Dhabi, United Arab Emirates},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.findings-emnlp.157},
     pages = {2139--2158}
   }

2. WASSA

   CAISA at WASSA 2022: Adapter-Tuning for Empathy Prediction

   Allison Lahnala, Charles Welch, and Lucie Flek

   In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, May 2022

   Abs Bib PDF

   We build a system that leverages adapters, a lightweight and efficient method for leveraging large language models to perform the task Empathy and Distress prediction tasks for WASSA 2022. In our experiments, we find that stacking our empathy and distress adapters on a pre-trained emotion classification adapter performs best compared to full fine-tuning approaches and emotion feature concatenation. We make our experimental code publicly available at https://github.com/caisa-lab/wassa-empathy-adapters.

   @inproceedings{lahnala-etal-2022-caisa,
     title = {{CAISA} at {WASSA} 2022: Adapter-Tuning for Empathy Prediction},
     author = {Lahnala, Allison and Welch, Charles and Flek, Lucie},
     booktitle = {Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment {\&} Social Media Analysis},
     month = may,
     year = {2022},
     address = {Dublin, Ireland},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.wassa-1.31},
     doi = {10.18653/v1/2022.wassa-1.31},
     pages = {280--285}
   }

3. NAACL

   Mitigating Toxic Degeneration with Empathetic Data: Exploring the Relationship Between Toxicity and Empathy

   Allison Lahnala, Charles Welch, Béla Neuendorf, and Lucie Flek

   In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jul 2022

   Abs Bib PDF

   Large pre-trained neural language models have supported the effectiveness of many NLP tasks, yet are still prone to generating toxic language hindering the safety of their use. Using empathetic data, we improve over recent work on controllable text generation that aims to reduce the toxicity of generated text. We find we are able to dramatically reduce the size of fine-tuning data to 7.5-30k samples while at the same time making significant improvements over state-of-the-art toxicity mitigation of up to 3.4% absolute reduction (26% relative) from the original work on 2.3m samples, by strategically sampling data based on empathy scores. We observe that the degree of improvements is subject to specific communication components of empathy. In particular, the more cognitive components of empathy significantly beat the original dataset in almost all experiments, while emotional empathy was tied to less improvement and even underperforming random samples of the original data. This is a particularly implicative insight for NLP work concerning empathy as until recently the research and resources built for it have exclusively considered empathy as an emotional concept.

   @inproceedings{lahnala-etal-2022-mitigating,
     title = {Mitigating Toxic Degeneration with Empathetic Data: Exploring the Relationship Between Toxicity and Empathy},
     author = {Lahnala, Allison and Welch, Charles and Neuendorf, B{\'e}la and Flek, Lucie},
     booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
     month = jul,
     year = {2022},
     address = {Seattle, United States},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2022.naacl-main.363},
     doi = {10.18653/v1/2022.naacl-main.363},
     pages = {4926--4938}
   }

4. LREC

   Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion

   Flora Sakketou, Allison Lahnala, Liane Vogel, and Lucie Flek

   In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Jun 2022

   Abs Bib PDF

   There is an increasing need for the ability to model fine-grained opinion shifts of social media users, as concerns about the potential polarizing social effects increase. However, the lack of publicly available datasets that are suitable for the task presents a major challenge. In this paper, we introduce an innovative annotated dataset for modeling subtle opinion fluctuations and detecting fine-grained stances. The dataset includes a sufficient amount of stance polarity and intensity labels per user over time and within entire conversational threads, thus making subtle opinion fluctuations detectable both in long term and in short term. All posts are annotated by non-experts and a significant portion of the data is also annotated by experts. We provide a strategy for recruiting suitable non-experts. Our analysis of the inter-annotator agreements shows that the resulting annotations obtained from the majority vote of the non-experts are of comparable quality to the annotations of the experts. We provide analyses of the stance evolution in short term and long term levels, a comparison of language usage between users with vacillating and resolute attitudes, and fine-grained stance detection baselines.

   @inproceedings{sakketou-etal-2022-investigating,
     title = {Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion},
     author = {Sakketou, Flora and Lahnala, Allison and Vogel, Liane and Flek, Lucie},
     booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
     month = jun,
     year = {2022},
     address = {Marseille, France},
     publisher = {European Language Resources Association},
     url = {https://aclanthology.org/2022.lrec-1.405},
     pages = {3798--3808}
   }

2021

1. arXiv

   Modeling Proficiency with Implicit User Representations

   Kim Breitwieser, Allison Lahnala, Charles Welch, Lucie Flek, and Martin Potthast

   arXiv preprint arXiv:2110.08011, Jun 2021

   Abs Bib PDF

   We introduce the problem of proficiency modeling: Given a user’s posts on a social media platform, the task is to identify the subset of posts or topics for which the user has some level of proficiency. This enables the filtering and ranking of social media posts on a given topic as per user proficiency. Unlike experts on a given topic, proficient users may not have received formal training and possess years of practical experience, but may be autodidacts, hobbyists, and people with sustained interest, enabling them to make genuine and original contributions to discourse. While predicting whether a user is an expert on a given topic imposes strong constraints on who is a true positive, proficiency modeling implies a graded scoring, relaxing these constraints. Put another way, many active social media users can be assumed to possess, or eventually acquire, some level of proficiency on topics relevant to their community. We tackle proficiency modeling in an unsupervised manner by utilizing user embeddings to model engagement with a given topic, as indicated by a user’s preference for authoring related content. We investigate five alternative approaches to model proficiency, ranging from basic ones to an advanced, tailored user modeling approach, applied within two real-world benchmarks for evaluation.

   @article{breitwieser2021modeling,
     title = {Modeling Proficiency with Implicit User Representations},
     author = {Breitwieser, Kim and Lahnala, Allison and Welch, Charles and Flek, Lucie and Potthast, Martin},
     journal = {arXiv preprint arXiv:2110.08011},
     year = {2021},
   }

2. ACL

   Exploring Self-Identified Counseling Expertise in Online Support Forums

   Allison Lahnala, Yuntian Zhao, Charles Welch, Jonathan K. Kummerfeld, Lawrence C An, Kenneth Resnicow, Rada Mihalcea, and Verónica Pérez-Rosas

   In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Aug 2021

   Abs Bib PDF

   A growing number of people engage in online health forums, making it important to understand the quality of the advice they receive. In this paper, we explore the role of expertise in responses provided to help-seeking posts regarding mental health. We study the differences between (1) interactions with peers; and (2) interactions with self-identified mental health professionals. First, we show that a classifier can distinguish between these two groups, indicating that their language use does in fact differ. To understand this difference, we perform several analyses addressing engagement aspects, including whether their comments engage the support-seeker further as well as linguistic aspects, such as dominant language and linguistic style matching. Our work contributes toward the developing efforts of understanding how health experts engage with health information- and support-seekers in social networks. More broadly, it is a step toward a deeper understanding of the styles of interactions that cultivate supportive engagement in online communities.

   @inproceedings{lahnala-etal-2021-exploring,
     title = {Exploring Self-Identified Counseling Expertise in Online Support Forums},
     author = {Lahnala, Allison and Zhao, Yuntian and Welch, Charles and Kummerfeld, Jonathan K. and An, Lawrence C and Resnicow, Kenneth and Mihalcea, Rada and P{\'e}rez-Rosas, Ver{\'o}nica},
     booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
     month = aug,
     year = {2021},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2021.findings-acl.392},
     doi = {10.18653/v1/2021.findings-acl.392},
     pages = {4467--4480}
   }

3. EvoStar

   Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction

   Allison Lahnala, Gauri Kambhatla, Jiajun Peng, Matthew Whitehead, Gillian Minnehan, Eric Guldan, Jonathan K Kummerfeld, Anıl Çamcı, and Rada Mihalcea

   In Artificial Intelligence in Music, Sound, Art and Design: 10th International Conference, EvoMUSART 2021, Held as Part of EvoStar 2021, Virtual Event, April 7–9, 2021, Proceedings 10, Aug 2021

   Abs Bib PDF

   Natural language processing methods have been applied in a variety of music studies, drawing the connection between music and language. In this paper, we expand those approaches by investigating chord embeddings, which we apply in two case studies to address two key questions: (1) what musical information do chord embeddings capture?; and (2) how might musical applications benefit from them? In our analysis, we show that they capture similarities between chords that adhere to important relationships described in music theory. In the first case study, we demonstrate that using chord embeddings in a next chord prediction task yields predictions that more closely match those by experienced musicians. In the second case study, we show the potential benefits of using the representations in tasks related to musical stylometrics.

   @inproceedings{lahnala2021chord,
     title = {Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction},
     author = {Lahnala, Allison and Kambhatla, Gauri and Peng, Jiajun and Whitehead, Matthew and Minnehan, Gillian and Guldan, Eric and Kummerfeld, Jonathan K and {\c{C}}amc{\i}, An{\i}l and Mihalcea, Rada},
     booktitle = {Artificial Intelligence in Music, Sound, Art and Design: 10th International Conference, EvoMUSART 2021, Held as Part of EvoStar 2021, Virtual Event, April 7--9, 2021, Proceedings 10},
     pages = {171--186},
     year = {2021},
     organization = {Springer}
   }

2020

1. NLP COVID-19

   Expressive Interviewing: A Conversational System for Coping with COVID-19

   Charles Welch, Allison Lahnala, Veronica Perez-Rosas, Siqi Shen, Sarah Seraj, Larry An, Kenneth Resnicow, James Pennebaker, and Rada Mihalcea

   In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Dec 2020

   Abs Bib PDF

   The ongoing COVID-19 pandemic has raised concerns for many regarding personal and public health implications, financial security and economic stability. Alongside many other unprecedented challenges, there are increasing concerns over social isolation and mental health. We introduce Expressive Interviewing – an interview-style conversational system that draws on ideas from motivational interviewing and expressive writing. Expressive Interviewing seeks to encourage users to express their thoughts and feelings through writing by asking them questions about how COVID-19 has impacted their lives. We present relevant aspects of the system’s design and implementation as well as quantitative and qualitative analyses of user interactions with the system. In addition, we conduct a comparative evaluation with a general purpose dialogue system for mental health that shows our system potential in helping users to cope with COVID-19 issues.

   @inproceedings{welch-etal-2020-expressive,
     title = {Expressive Interviewing: A Conversational System for Coping with {COVID}-19},
     author = {Welch, Charles and Lahnala, Allison and Perez-Rosas, Veronica and Shen, Siqi and Seraj, Sarah and An, Larry and Resnicow, Kenneth and Pennebaker, James and Mihalcea, Rada},
     booktitle = {Proceedings of the 1st Workshop on {NLP} for {COVID}-19 (Part 2) at {EMNLP} 2020},
     month = dec,
     year = {2020},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2020.nlpcovid19-2.6},
     doi = {10.18653/v1/2020.nlpcovid19-2.6}
   }