publications | Allie Lahnala

I publish with my full first name Allison.

2024

arxiv
The Illusion of Empathy: How AI Chatbots Shape Conversation Perception

Tingting Liu, Salvatore Giorgi, Ankit Aich, Allison Lahnala, Brenda Curtis, Lyle Ungar, and João Sedoc

arXiv preprint arXiv:2411.12877, Nov 2024

Abs Bib PDF

As AI chatbots become more human-like by incorporating empathy, understanding user-centered perceptions of chatbot empathy and its impact on conversation quality remains essential yet under-explored. This study examines how chatbot identity and perceived empathy influence users’ overall conversation experience. Analyzing 155 conversations from two datasets, we found that while GPT-based chatbots were rated significantly higher in conversational quality, they were consistently perceived as less empathetic than human conversational partners. Empathy ratings from GPT-4o annotations aligned with users’ ratings, reinforcing the perception of lower empathy in chatbots. In contrast, 3 out of 5 empathy models trained on human-human conversations detected no significant differences in empathy language between chatbots and humans. Our findings underscore the critical role of perceived empathy in shaping conversation quality, revealing that achieving high-quality human-AI interactions requires more than simply embedding empathetic language; it necessitates addressing the nuanced ways users interpret and experience empathy in conversations with chatbots.
@article{liu-2024-bot-empathy, title = {The Illusion of Empathy: How AI Chatbots Shape Conversation Perception}, journal = {arXiv preprint arXiv:2411.12877}, author = {Liu, Tingting and Giorgi, Salvatore and Aich, Ankit and Lahnala, Allison and Curtis, Brenda and Ungar, Lyle and Sedoc, Jo{\~a}o}, month = nov, year = {2024}, }

EUtopia

An Examination of “Empathy” in Conversational Interfaces for LLMs

Alok Debnath, and Allison Lahnala

Oct 2024

@misc{debnath-2024-empath-convllm,
  address = {Gothenburg, Sweden},
  title = {An Examination of “Empathy” in Conversational Interfaces for LLMs},
  author = {Debnath, Alok and Lahnala, Allison},
  booktitle = {EUtopia: Human-centred AI Beyond Words workshop},
  month = oct,
  year = {2024},
}

LREC-COLING
Appraisal Framework for Clinical Empathy: A Novel Application to Breaking Bad News Conversations

Allison Lahnala, Béla Neuendorf, Alexander Thomin, Charles Welch, Tina Stibane, and Lucie Flek

In Proceedings of The 2024 Joint International Conference, on Computational Linguistics, Language Resources and Evaluation, May 2024

Abs Bib PDF

We introduce an innovative annotation approach that draws on well-established frameworks for clinical empathy and breaking bad news (BBN) conversations. Empathy is essential in healthcare communication and requires considering the interactive dynamics of discourse relations. We constructed Empathy in BBNs, a dataset of simulated BBN conversations in German, annotated with our novel annotation scheme in collaboration with a large medical school to support research on educational tools for medical didactics. The dataset contains conversations between medical students and standardized patients and fine-grained annotations of the components of empathic interactions. We provide a detailed description of our span and relation labeling annotation procedure, where two trained annotators obtained Krippendorff’s alpha agreement of ≥0.85. The annotation is based on 1) Pounds (2011)’s appraisal framework (AF) for clinical empathy, which is grounded in systemic functional linguistics (SFL), and 2) SPIKES, protocol for breaking bad news (Baile et al., 2000) commonly taught in medical didactics training. This approach presents novel opportunities to study clinical empathic behavior and enables the training of models to detect causal relations involving empathy, a highly desirable feature of systems that can provide feedback to medical professionals in training. We present illustrative examples and discuss applications of annotation scheme and insights we can draw from the framework.
@inproceedings{lahnala-2024-bbn-empathy, address = {Torino, Italia}, title = {Appraisal Framework for Clinical Empathy: A Novel Application to Breaking Bad News Conversations}, booktitle = {Proceedings of The 2024 Joint International Conference, on Computational Linguistics, Language Resources and Evaluation}, publisher = {European Language Resources Association}, author = {Lahnala, Allison and Neuendorf, Béla and Thomin, Alexander and Welch, Charles and Stibane, Tina and Flek, Lucie}, month = may, year = {2024}, }
LREC-COLING
LeadEmpathy: An Expert Annotated German Dataset of Empathy in Written Leadership Communication

Didem Sedefoglu, Allison Lahnala, Jasmin Wagner, Lucie Flek, and Sandra Ohly

In Proceedings of The 2024 Joint International Conference, on Computational Linguistics, Language Resources and Evaluation, May 2024

Abs Bib PDF

Empathetic leadership communication plays a pivotal role in modern workplaces as it is associated with a wide range of positive individual and organizational outcomes. This paper introduces \textscLeadEmpathy, an innovative expert-annotated German dataset for modeling empathy in written leadership communication. It features a novel theory-based coding scheme to model cognitive and affective empathy in asynchronous communication. The final dataset comprises 770 annotated emails from 385 participants who were allowed to rewrite their emails after receiving recommendations for increasing empathy in an online experiment. Two independent annotators achieved substantial inter-annotator agreement of ≥.79 for all categories, indicating that the annotation scheme can be applied to produce high-quality, multidimensional empathy ratings in current and future applications. Beyond outlining the dataset’s development procedures, we present a case study on automatic empathy detection, establishing baseline models for predicting empathy scores in a range of ten possible scores that achieve a Pearson correlation of 0.816 and a mean squared error of 0.883.
@inproceedings{sedefoglu-2024-lead-empathy, address = {Torino, Italia}, title = {LeadEmpathy: An Expert Annotated German Dataset of Empathy in Written Leadership Communication}, booktitle = {Proceedings of The 2024 Joint International Conference, on Computational Linguistics, Language Resources and Evaluation}, publisher = {European Language Resources Association}, author = {Sedefoglu, Didem and Lahnala, Allison and Wagner, Jasmin and Flek, Lucie and Ohly, Sandra}, month = may, year = {2024}, }
CLPSYCH
Archetypes and Entropy: Theory-Driven Extraction of Evidence for Suicide Risk

Vasudha Varadarajan, Allison Lahnala, Adithya V. Ganesan, Gourab Dey, Siddharth Mangalik, Ana-Maria Bucur, Nikita Soni, Rajath Rao, Kevin Lanning, Isabella Vallejo, Lucie Flek, H. Andrew Schwartz, Charles Welch, and Ryan L. Boyd

In Proceedings of the Tenth Workshop on Computational Linguistics and Clinical Psychology, Mar 2024

Abs Bib PDF

Psychological risk factors for suicide have been extensively studied for decades. However, combining explainable theory with modern data-driven language modeling approaches is non-trivial. Here, we propose and evaluate methods for identifying language patterns indicative of suicide risk by combining theory-driven suicidal *archetypes* with language model-based and *relative entropy*-based approaches. *Archetypes* are based on prototypical statements that evince risk of suicidality while *relative entropy* considers the difference between how probable the risk-familiar and risk-unfamiliar models find user language. Each approach performed well individually; combining the two strikingly improved performance, yielding our combined system submission with a BERTScore Recall of 0.906. Further, we find diagnostic language is distributed unevenly in posts, with titles containing substantial risk evidence. We conclude that a union between theory- and data-driven methods is beneficial, outperforming more modern prompt-based methods.
@inproceedings{varadarajan_archetypes_2024, address = {St. Julians, Malta}, title = {Archetypes and {Entropy}: {Theory}-{Driven} {Extraction} of {Evidence} for {Suicide} {Risk}}, booktitle = {Proceedings of the {Tenth} {Workshop} on {Computational} {Linguistics} and {Clinical} {Psychology}}, publisher = {Association for Computational Linguistics}, author = {Varadarajan, Vasudha and Lahnala, Allison and Ganesan, Adithya V. and Dey, Gourab and Mangalik, Siddharth and Bucur, Ana-Maria and Soni, Nikita and Rao, Rajath and Lanning, Kevin and Vallejo, Isabella and Flek, Lucie and Schwartz, H. Andrew and Welch, Charles and Boyd, Ryan L.}, month = mar, year = {2024}, }

2023

RANLP
Challenges of GPT‑3‑based Conversational Agents for Healthcare

Fabian Lechner, Allison Lahnala, Charles Welch, and Lucie Flek

In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2023), Sep 2023

Abs Bib PDF

The potential to provide patients with faster information access while allowing medical specialists to focus on urgent tasks makes medical domain dialog agents appealing. However, there can be dire consequences due to the limitations of large-language models (LLMs) built into such agents. This paper investigates the challenges and risks of using GPT-3-based models for medical question-answering (MedQA). We perform several evaluations contextualized in terms of standard medical principles. We provide a procedure for manually designing patient queries to stress-test high-risk limitations of LLMs in MedQA systems. Our analysis shows that the LLMs fail to respond safely to these queries, producing invalid medical information, dangerous recommendations, and offensive content.
@inproceedings{lechner-lahnala-gpt3-healthcare, title = {Challenges of GPT‑3‑based Conversational Agents for Healthcare}, author = {Lechner, Fabian and Lahnala, Allison and Welch, Charles and Flek, Lucie}, booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2023)}, month = sep, year = {2023}, address = {}, publisher = {}, url = {}, pages = {} }
WASSA
Domain Transfer for Empathy, Distress, and Personality Prediction

Fabio Gruschka, Allison Lahnala, Charles Welch, and Lucie Flek

In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, Jul 2023

Abs Bib PDF

This research contributes to the task of predicting empathy and personality traits within dialogue, an important aspect of natural language processing, as part of our experimental work for the WASSA 2023 Empathy and Emotion Shared Task. For predicting empathy, emotion polarity, and emotion intensity on turns within a dialogue, we employ adapters trained on social media interactions labeled with empathy ratings in a stacked composition with the target task adapters. Furthermore, we embed demographic information to predict Interpersonal Reactivity Index (IRI) subscales and Big Five Personality Traits utilizing BERT-based models. The results from our study provide valuable insights, contributing to advancements in understanding human behavior and interaction through text. Our team ranked 2nd on the personality and empathy prediction tasks, 4th on the interpersonal reactivity index, and 6th on the conversational task.
@inproceedings{gruschka-etal-2023-caisa, title = {Domain Transfer for Empathy, Distress, and Personality Prediction}, author = {Gruschka, Fabio and Lahnala, Allison and Welch, Charles and Flek, Lucie}, booktitle = {Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, {\&} Social Media Analysis}, month = jul, year = {2023}, address = {Toronto, Canada}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2023.wassa-1.50}, doi = {10.18653/v1/2023.wassa-1.50}, pages = {553--557} }

2022

EMNLP
A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing

Allison Lahnala, Charles Welch, David Jurgens, and Lucie Flek

In Findings of the Association for Computational Linguistics: EMNLP 2022, Dec 2022

Abs Bib PDF

We review the state of research on empathy in natural language processing and identify the following issues: (1) empathy definitions are absent or abstract, which (2) leads to low construct validity and reproducibility. Moreover, (3) emotional empathy is overemphasized, skewing our focus to a narrow subset of simplified tasks. We believe these issues hinder research progress and argue that current directions will benefit from a clear conceptualization that includes operationalizing cognitive empathy components. Our main objectives are to provide insight and guidance on empathy conceptualization for NLP research objectives and to encourage researchers to pursue the overlooked opportunities in this area, highly relevant, e.g., for clinical and educational sectors.
@inproceedings{lahnala-etal-2022-critical, title = {A Critical Reflection and Forward Perspective on Empathy and Natural Language Processing}, author = {Lahnala, Allison and Welch, Charles and Jurgens, David and Flek, Lucie}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022}, month = dec, year = {2022}, address = {Abu Dhabi, United Arab Emirates}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.findings-emnlp.157}, pages = {2139--2158} }
WASSA
CAISA at WASSA 2022: Adapter-Tuning for Empathy Prediction

Allison Lahnala, Charles Welch, and Lucie Flek

In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, May 2022

Abs Bib PDF

We build a system that leverages adapters, a lightweight and efficient method for leveraging large language models to perform the task Empathy and Distress prediction tasks for WASSA 2022. In our experiments, we find that stacking our empathy and distress adapters on a pre-trained emotion classification adapter performs best compared to full fine-tuning approaches and emotion feature concatenation. We make our experimental code publicly available at https://github.com/caisa-lab/wassa-empathy-adapters.
@inproceedings{lahnala-etal-2022-caisa, title = {{CAISA} at {WASSA} 2022: Adapter-Tuning for Empathy Prediction}, author = {Lahnala, Allison and Welch, Charles and Flek, Lucie}, booktitle = {Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment {\&} Social Media Analysis}, month = may, year = {2022}, address = {Dublin, Ireland}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.wassa-1.31}, doi = {10.18653/v1/2022.wassa-1.31}, pages = {280--285} }
NAACL
Mitigating Toxic Degeneration with Empathetic Data: Exploring the Relationship Between Toxicity and Empathy

Allison Lahnala, Charles Welch, Béla Neuendorf, and Lucie Flek

In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jul 2022

Abs Bib PDF

Large pre-trained neural language models have supported the effectiveness of many NLP tasks, yet are still prone to generating toxic language hindering the safety of their use. Using empathetic data, we improve over recent work on controllable text generation that aims to reduce the toxicity of generated text. We find we are able to dramatically reduce the size of fine-tuning data to 7.5-30k samples while at the same time making significant improvements over state-of-the-art toxicity mitigation of up to 3.4% absolute reduction (26% relative) from the original work on 2.3m samples, by strategically sampling data based on empathy scores. We observe that the degree of improvements is subject to specific communication components of empathy. In particular, the more cognitive components of empathy significantly beat the original dataset in almost all experiments, while emotional empathy was tied to less improvement and even underperforming random samples of the original data. This is a particularly implicative insight for NLP work concerning empathy as until recently the research and resources built for it have exclusively considered empathy as an emotional concept.
@inproceedings{lahnala-etal-2022-mitigating, title = {Mitigating Toxic Degeneration with Empathetic Data: Exploring the Relationship Between Toxicity and Empathy}, author = {Lahnala, Allison and Welch, Charles and Neuendorf, B{\'e}la and Flek, Lucie}, booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies}, month = jul, year = {2022}, address = {Seattle, United States}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2022.naacl-main.363}, doi = {10.18653/v1/2022.naacl-main.363}, pages = {4926--4938} }
LREC
Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion

Flora Sakketou, Allison Lahnala, Liane Vogel, and Lucie Flek

In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Jun 2022

Abs Bib PDF

There is an increasing need for the ability to model fine-grained opinion shifts of social media users, as concerns about the potential polarizing social effects increase. However, the lack of publicly available datasets that are suitable for the task presents a major challenge. In this paper, we introduce an innovative annotated dataset for modeling subtle opinion fluctuations and detecting fine-grained stances. The dataset includes a sufficient amount of stance polarity and intensity labels per user over time and within entire conversational threads, thus making subtle opinion fluctuations detectable both in long term and in short term. All posts are annotated by non-experts and a significant portion of the data is also annotated by experts. We provide a strategy for recruiting suitable non-experts. Our analysis of the inter-annotator agreements shows that the resulting annotations obtained from the majority vote of the non-experts are of comparable quality to the annotations of the experts. We provide analyses of the stance evolution in short term and long term levels, a comparison of language usage between users with vacillating and resolute attitudes, and fine-grained stance detection baselines.
@inproceedings{sakketou-etal-2022-investigating, title = {Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion}, author = {Sakketou, Flora and Lahnala, Allison and Vogel, Liane and Flek, Lucie}, booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference}, month = jun, year = {2022}, address = {Marseille, France}, publisher = {European Language Resources Association}, url = {https://aclanthology.org/2022.lrec-1.405}, pages = {3798--3808} }

2021

arXiv
Modeling Proficiency with Implicit User Representations

Kim Breitwieser, Allison Lahnala, Charles Welch, Lucie Flek, and Martin Potthast

arXiv preprint arXiv:2110.08011, Jun 2021

Abs Bib PDF

We introduce the problem of proficiency modeling: Given a user’s posts on a social media platform, the task is to identify the subset of posts or topics for which the user has some level of proficiency. This enables the filtering and ranking of social media posts on a given topic as per user proficiency. Unlike experts on a given topic, proficient users may not have received formal training and possess years of practical experience, but may be autodidacts, hobbyists, and people with sustained interest, enabling them to make genuine and original contributions to discourse. While predicting whether a user is an expert on a given topic imposes strong constraints on who is a true positive, proficiency modeling implies a graded scoring, relaxing these constraints. Put another way, many active social media users can be assumed to possess, or eventually acquire, some level of proficiency on topics relevant to their community. We tackle proficiency modeling in an unsupervised manner by utilizing user embeddings to model engagement with a given topic, as indicated by a user’s preference for authoring related content. We investigate five alternative approaches to model proficiency, ranging from basic ones to an advanced, tailored user modeling approach, applied within two real-world benchmarks for evaluation.
@article{breitwieser2021modeling, title = {Modeling Proficiency with Implicit User Representations}, author = {Breitwieser, Kim and Lahnala, Allison and Welch, Charles and Flek, Lucie and Potthast, Martin}, journal = {arXiv preprint arXiv:2110.08011}, year = {2021}, }
ACL
Exploring Self-Identified Counseling Expertise in Online Support Forums

Allison Lahnala, Yuntian Zhao, Charles Welch, Jonathan K. Kummerfeld, Lawrence C An, Kenneth Resnicow, Rada Mihalcea, and Verónica Pérez-Rosas

In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Aug 2021

Abs Bib PDF

A growing number of people engage in online health forums, making it important to understand the quality of the advice they receive. In this paper, we explore the role of expertise in responses provided to help-seeking posts regarding mental health. We study the differences between (1) interactions with peers; and (2) interactions with self-identified mental health professionals. First, we show that a classifier can distinguish between these two groups, indicating that their language use does in fact differ. To understand this difference, we perform several analyses addressing engagement aspects, including whether their comments engage the support-seeker further as well as linguistic aspects, such as dominant language and linguistic style matching. Our work contributes toward the developing efforts of understanding how health experts engage with health information- and support-seekers in social networks. More broadly, it is a step toward a deeper understanding of the styles of interactions that cultivate supportive engagement in online communities.
@inproceedings{lahnala-etal-2021-exploring, title = {Exploring Self-Identified Counseling Expertise in Online Support Forums}, author = {Lahnala, Allison and Zhao, Yuntian and Welch, Charles and Kummerfeld, Jonathan K. and An, Lawrence C and Resnicow, Kenneth and Mihalcea, Rada and P{\'e}rez-Rosas, Ver{\'o}nica}, booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021}, month = aug, year = {2021}, address = {Online}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.findings-acl.392}, doi = {10.18653/v1/2021.findings-acl.392}, pages = {4467--4480} }
EvoStar
Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction

Allison Lahnala, Gauri Kambhatla, Jiajun Peng, Matthew Whitehead, Gillian Minnehan, Eric Guldan, Jonathan K Kummerfeld, Anıl Çamcı, and Rada Mihalcea

In Artificial Intelligence in Music, Sound, Art and Design: 10th International Conference, EvoMUSART 2021, Held as Part of EvoStar 2021, Virtual Event, April 7–9, 2021, Proceedings 10, Aug 2021

Abs Bib PDF

Natural language processing methods have been applied in a variety of music studies, drawing the connection between music and language. In this paper, we expand those approaches by investigating chord embeddings, which we apply in two case studies to address two key questions: (1) what musical information do chord embeddings capture?; and (2) how might musical applications benefit from them? In our analysis, we show that they capture similarities between chords that adhere to important relationships described in music theory. In the first case study, we demonstrate that using chord embeddings in a next chord prediction task yields predictions that more closely match those by experienced musicians. In the second case study, we show the potential benefits of using the representations in tasks related to musical stylometrics.
@inproceedings{lahnala2021chord, title = {Chord Embeddings: Analyzing What They Capture and Their Role for Next Chord Prediction and Artist Attribute Prediction}, author = {Lahnala, Allison and Kambhatla, Gauri and Peng, Jiajun and Whitehead, Matthew and Minnehan, Gillian and Guldan, Eric and Kummerfeld, Jonathan K and {\c{C}}amc{\i}, An{\i}l and Mihalcea, Rada}, booktitle = {Artificial Intelligence in Music, Sound, Art and Design: 10th International Conference, EvoMUSART 2021, Held as Part of EvoStar 2021, Virtual Event, April 7--9, 2021, Proceedings 10}, pages = {171--186}, year = {2021}, organization = {Springer} }

2020

NLP COVID-19
Expressive Interviewing: A Conversational System for Coping with COVID-19

Charles Welch, Allison Lahnala, Veronica Perez-Rosas, Siqi Shen, Sarah Seraj, Larry An, Kenneth Resnicow, James Pennebaker, and Rada Mihalcea

In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Dec 2020

Abs Bib PDF

The ongoing COVID-19 pandemic has raised concerns for many regarding personal and public health implications, financial security and economic stability. Alongside many other unprecedented challenges, there are increasing concerns over social isolation and mental health. We introduce Expressive Interviewing – an interview-style conversational system that draws on ideas from motivational interviewing and expressive writing. Expressive Interviewing seeks to encourage users to express their thoughts and feelings through writing by asking them questions about how COVID-19 has impacted their lives. We present relevant aspects of the system’s design and implementation as well as quantitative and qualitative analyses of user interactions with the system. In addition, we conduct a comparative evaluation with a general purpose dialogue system for mental health that shows our system potential in helping users to cope with COVID-19 issues.
@inproceedings{welch-etal-2020-expressive, title = {Expressive Interviewing: A Conversational System for Coping with {COVID}-19}, author = {Welch, Charles and Lahnala, Allison and Perez-Rosas, Veronica and Shen, Siqi and Seraj, Sarah and An, Larry and Resnicow, Kenneth and Pennebaker, James and Mihalcea, Rada}, booktitle = {Proceedings of the 1st Workshop on {NLP} for {COVID}-19 (Part 2) at {EMNLP} 2020}, month = dec, year = {2020}, address = {Online}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2020.nlpcovid19-2.6}, doi = {10.18653/v1/2020.nlpcovid19-2.6} }