Sharp Waves: ILAE's epilepsy podcast
Sharp Waves: ILAE's epilepsy podcast
Language models reveal concerns of people with epilepsy in online discussions
Many of the most common concerns of people with epilepsy are topics they don't usually talk about with physicians. This is one of many findings of a study using AI to analyze more than 55,000 posts about epilepsy on Reddit. Sharp Waves spoke with the study's first author and two epilepsy advocates about the results and how they could help initiate important patient-provider conversations.
PLEASE NOTE: This episode includes discussion of suicidal ideation. If you are in crisis, call your local emergency number or crisis hotline.
The Reddit study is freely available online; no journal subscription is required:
Sharp Waves episodes are meant for informational purposes only, and not as clinical or medical advice.
Let us know how we're doing: podcast@ilae.org.
The International League Against Epilepsy is the world's preeminent association of health professionals and scientists, working toward a world where no person's life is limited by epilepsy. Visit us on Facebook, Instagram, and LinkedIn.
[00:00:00] Nancy Volkers: Hey everyone and welcome to another episode of Sharp Waves. Today we are walking through a study about the use of large language models in social media to try and get at patient concerns. So concerns that people with epilepsy may have that they may not be sharing with their physicians. We have a study author with us and we also have two other distinguished guests and I would like them to introduce themselves first and then we'll get started.
Dr. Fennig, would you like to start?
[00:00:34] Dr. Uriel Fennig: Yes. Thank you. Thank you for inviting me. It's an honor to be on this podcast. I'm a long-time listener. My name is Uriel Fennig, but everyone calls me Uri. And I'm an early-career neurologist. I'm in the transition between my residency and starting an epilepsy fellowship in my hospital, Sheba Medical Center in Israel.
[00:00:58] Dr. Laura M'Rabet: Hello. Thank you for inviting me as well. My name is Laura, Laura M’Rabet. I'm a caregiver of a son with severe epilepsy and I'm working for EpilepsyNL. And EpilepsyNL is a foundation and a patient advocacy organization for people with epilepsy and their relatives.
[00:01:17] Landis Wiedner: Hi, my name is Landis Wiedner.
I am the host of the What the EF podcast, which is a place for epilepsy “what the F” moments or adulting with epilepsy and covers a lot of the topics that Dr. Fennig reported in this research study. First of all, I'm just excited to be on this podcast because I'm also an avid listener. And then also, especially with this topic, or this research, just because I feel like it's so important moving forward.
[00:01:49] Nancy Volkers: Excellent. Thanks to everyone for being here today. And we're just going to jump into the questions. So, Dr. Fennig, if you could sort of set the stage for us—What is this study about? And why did you want to do a study like this one?
[00:02:03] Dr. Uriel Fennig: So, I think that as clinicians who treat people with epilepsy, we're usually only exposed to certain aspects of the patient's experience. And we're often focused on a certain set of issues that we may feel are the most important to address, like seizure frequency and medication side effects, et cetera. And this may be prudent in the clinical scenario and efficient, but many issues, which can sometimes be very central and important for the patients can be overlooked or not addressed in the clinical situation.
Other factors that contribute to this communication gap are, for instance, fear of judgment, let's say about using alcohol or drugs. Or even the patient assuming that the doctor won't be interested in hearing about this issue or the other, and of course not to mention the short and constrained appointments that don't always allow for elaboration.
And I do believe that to be better at treating epilepsy, we should strive to acknowledge and understand the many real-world and day-to-day issues that people with epilepsy deal with. And, of course, this isn't a new notion necessarily, but it's always good to look for new ways to bridge the communication gap with our patients.
And the way that we chose to do this was by studying a large online epilepsy community. I collaborated with a professor of computer sciences, Elad Yom-Tov, who has a special interest in studying online health-related behaviors. And using his technical expertise, we were able to use artificial intelligence models to analyze large masses of posts from the epilepsy community on Reddit, where people with epilepsy engage in peer-to-peer conversations and discourse, possibly showing us their various concerns in a more direct and unfiltered and uncensored manner, as opposed to the clinical setting.
I'll just introduce Reddit for a moment for those who don't know it. It's a network of many online communities called subreddits, and you can find a subreddit for practically any interest or topic. And one of these subreddits is about epilepsy, meaning it's an online community where people with epilepsy can post about anything and the community engages with the posts.
I kind of surfed the epilepsy subreddit and once I started to do so, I was actually fascinated just scrolling through the different posts and comments. It felt like I was kind of lifting a curtain or gaining some kind of a behind-the-scenes look at the lives and conversations of these people that usually I meet in a formal and clinical situation. And that really opened my mind to the potential that this sort of study can have to bring us closer to understanding the actual experiences of people with epilepsy.
And by applying AI tools, we would potentially be able to analyze and gain insights from this data on a massive scale.
[00:05:00] Nancy Volkers: I know there’s a major Methods section and there is large language models involved. So any listeners who are interested in that, I encourage you to read the paper, but I think we're going to jump to the results.
So you analyze something like 14 years of posts, which is a huge number of posts. What did you find? Can you sort of synthesize that for us? And was anything surprising to you or any trends or any patterns?
[00:05:28] Dr. Uriel Fennig: Well, what we set out to do was to basically answer the question, what do people with epilepsy talk about? So identifying the most common topics or themes was the main initial goal of our study.
One example of a prevalent topic was describing seizures. So describing seizures came in various forms. For instance, because seizures can be so diverse, a person may be asking, “My seizures don't seem typical. Is this even a seizure?”
And this may be one of the most powerful benefits of being in an online community, because chances are some people out of the thousands of users out there are going to confirm that, yes, they too have been experiencing something similar.
Another important theme that we came across was health care experiences. That's a name that we gave to indicate posts that were about a person's interaction with the health care system and with health care providers, for instance, complaints about a doctor's attitude or his decisions, or emergency room experiences, inpatient experiences, video EEG experiences, ambulances even.
I'll give a paraphrased example. “My doctor only seems interested in increasing my meds. I've been seeing this neurologist for years, and all they do is raise the dosage. No tests or anything. Recently, they even suggested I might just be sleepwalking. Is this normal for a neurologist to just up the meds without investigating further?”
And we did find, through sentiment analysis, which is another thing that we did, that posts concerning the health care experiences were mostly in the negative, but this does call for a separate and deeper analysis. I personally find this topic very interesting and crucial to investigate. So we may be looking into that in future studies.
Something else that we found and we describe in the paper is demographic differences and correlations with different kinds of themes. We found age-related and gender-related differences in topics. We found that women were more likely to bring up issues related to stigma, to emotional issues, and to health care experiences. And men were more likely to bring up drugs and alcohol and health care costs. Older users, which we defined as over 45, no offense to anyone, were predominantly concerned with topics related to health care costs, driving, and cognitive issues. The middle age group, between 26 and 45, were concerned more with fatigue and work-related issues, and the youngest group posted more about adjustment to a new diagnosis, and issues of stigma and drugs.
And these findings put together may help us in tailoring conversations when we encounter patients with these certain characteristics. For example, if I'm seeing a young female patient, I might proactively ask her about whether she's had an experience of stigma or social misconceptions about epilepsy. That may create space for talking about these issues and maybe to deal with them.
The last theme that I'll talk about is the theme of consultation, which was very prevalent. We labeled a consultation each time the poster was looking to consult with the community on various matters, and the fact that this was this was so prevalent reflects people's need for informal advice. People are looking for answers from their peers, perhaps about things that don't come up with the doctor, or perhaps about more day-to-day issues. Perhaps the person isn't satisfied with medical management, et cetera.
So the first thought that may come to mind is that people are consulting on medical matters with the lay community. And we're not saying that this is a negative thing, but at least it raises some concerns about the potential for misinformation and harm. But we did find something interesting about the engagement with these sorts of consultation posts. And we define engagement as the number of comments per post, or in other words, how much traffic a certain post created, how much did a post pique the community's interest.
We found that posts with the highest levels of engagement were about topics that are probably less likely to be addressed in the medical clinical setting. So posts about issues of stigma, drugs, alcohol use, relationship issues and emotional issues. And vice versa, things that would be more likely to come up in clinical visits, such as medication issues, showed lower engagement.
And this was a very interesting finding, because it suggested a dynamic that that peer advice is given where it's really needed. Users were more interested in replying and engaging with topics that they felt they have more to contribute to, where they could fill in the gaps left by formal medical advice.
And this suggests that informal sources of information and advice, such as Reddit, can complement rather than compete with professional health care advice.
[00:10:25] Nancy Volkers: Thank you so much, Dr. Fennig. That was a lot of information, and I'd like to ask Landis if you could respond. I know you've read the study and you've heard Dr. Fennig speak, so how do you see this study adding to the evidence of the need for some other ways to assess communication issues between providers and people with epilepsy?
[00:10:51] Landis Wiedner: Yeah, well, first of all, I just want to thank Dr. Fennig for doing this because I never thought of Reddit being such a great source of research and data.
As someone living with epilepsy, I recently joined the subreddit of epilepsy and in my personal experience, I'll go on to either find relief or comfort or something like that, and what I end up doing is what Dr. Fennig was talking about—engaging with a post. I'm just trying to look for someone who understands, but then I end up helping someone as well.
I think it does create a pretty solid community of consulting with your peers and also providing the lived experience that while, as amazing as my epileptologist is, it's something she can't provide. And I think for a neurologist to understand that there's that fear and that shame possibly of talking about it could maybe couch that conversation a little bit easier for the patient. Rather than the patient working up the courage to ask about it, it's actually the doctor bringing it up. I know that no matter how much—I am obsessed with my neurologist, I absolutely love her—I am still nervous every appointment. And I come in ready and armed with my list of questions, with my report, with everything, and she's wonderful and she listens. And she's an amazing human and I'm still nervous.
I am just so appreciative of this and I love that we're using a safe space, a safe anonymous space as a place to get more information about what's going on for people living with epilepsy because it is such a different and individualized condition.
And now by breaking it down to these subcategories, we can all become more aware of what everyone's dealing with.
[00:12:42] Nancy Volkers: Excellent. Thank you. Laura, we're going to move on to you and your response. And also, I believe you are actually already using this study in some of your work.
[00:12:55] Dr. Laura M'Rabet: I was really amazed by this study, Uri, because you used large language modeling to identify themes. We had the idea to go on socials, but if you go to Instagram or Facebook, it's very difficult to get the reports. So we have our own way of collecting stories.
I love stories. I collect them to see whether we can get out themes. And we thought about using large language modeling to get these themes and then you did it, so thank you for showing that it works.
I didn't find a lot of surprising themes, so it's really also a kind of validation that the themes you talk about every day, that they are really there and that a lot of people have these problems. And then on the other side, because epilepsy is very diverse, everybody has his own story, his own way of encountering the epilepsy, going through seizures is different. For me, I'm a caregiver, so I don't know how it is to have epilepsy, I just see it for my son. So it's a completely different angle that's also getting different themes.
One other thing is, and you touched upon it, Uri, is that OK, I'm a clinician, so how can I use this in my clinic and in my daily practice? And this is exactly what we are working on right at this moment. How can we get the right story from patients to tell that to the neurologist and to set up prompts like patient-reported outcome measurements, really individualized.
And I think that the new techniques of AI and large language modeling really give us the possibilities to use it also in your medical files, to get at the real targets of how can I improve your life, your life as a patient with epilepsy, because epilepsy is so complex. It's not just the medical part. It's also social welfare. It's mental resilience. Somewhere it has to come together. For me, it's a validation that this way of working really gives us possibilities in the near future.
[00:15:06] Dr. Uriel Fennig: So, a lot was said. I did want to address some things that Landis said. I don't know why, but I was surprised and actually very interested when you said that while you were participating and answering other people that it made you feel much better or empowered. That's something that I didn't even think about and we might investigate that further. I think it's a very powerful thing, not just sharing, but also engaging with other posts as something that is beneficial. Another thing that you mentioned was that physicians, we don't have the tools to treat psychiatric issues or depression and whatnot.
But we can do something. And that something, I believe, is just to create a space to talk about it, to create the validation. Because I think that the field is shifting and we're more and more realizing that epilepsy is a network disease. It's not just about seizures, but it comes with a lot of different things. And not only is having active seizures defining the disease, but it's also about depression. It's also about cognitive issues, memory issues.
And what we can do is as clinicians, at least we can be proactive about bringing up certain topics like stigma or emotional distress. Just saying these things, just mentioning them, at least, can create the opportunity to, in the end, maybe to give a referral to professional treatment, but the first thing is that it needs to be said, it needs to be addressed.
[00:16:43] Nancy Volkers: Maybe we go back and talk a little bit about the study limitations. What are the limitations of looking at this group, and is there concern that while you're identifying all these topics that are important to people with epilepsy on Reddit, what about people with epilepsy who aren't on Reddit?
[00:17:05] Dr. Uriel Fennig: Reddit's user base does not necessarily represent the general population. Reddit users tend to be younger, more male, more tech savvy, and more from the USA. A side note is that in the epilepsy subreddit, we found a more female predominance. Also, I believe that over time, more and more sub-populations are getting online. But, of course, these differences are there, and they're important.
Another issue is the notion of who is active. Who is doing the posting? Who is doing the commenting? When I was performing this study, I learned something important about online behavior, which is the notion of super users, or there's even a rule called the 1-9-90 rule. One percent of the people contribute a lot. And they're called the super users. Nine percent contribute from time to time and 90 percent are called lurkers. There are people who read posts, but don't post themselves. And this is a general kind of rule about online behavior—it doesn't necessarily apply to this forum or medical forums in general, but it does probably apply at least to some extent, and it may influence the data.
And another important thing is the issue of the accuracy and the reliability of language models. In the end, LLMs or large language models are far from perfect, and they can introduce mistakes into the data. They can misinterpret our language nuances, misunderstand context.
I think that over time, internet fluency is increasing and online presence is being more and more representative, but there are still gaps in the representation, especially among the populations of older adults or children, lower socioeconomic groups and non-English speakers.
And this may need to be addressed through more traditional and offline methods, like surveys and interviews and focus groups, but still keeping with the online space, the methodology could be improved to try and be more inclusive.
[00:19:03] Dr. Laura M'Rabet: The limitations you just mentioned are completely, are there, but even if you go for the old-fashioned offline methods like interviews or focus groups, you have the same problems.
Because you have just a small group, and as a moderator in the old-fashioned way, you can steer a little bit of discussion and temper the enthusiastic ones and stimulate the people who are not talking that much. But in the end, you have the same problems.
As I stated in the beginning, your method shows that you can really analyze large groups. Which really validates your results and your themes and this is why I like the method so much, with its limitations.
What we have encountered is that if you ask about stories, you get a couple of stories, and then you identify themes, and then these themes you put into a kind of election. What are the most important themes? And then you get the non-active people more active in the whole discussion, so you have a two-step way of first identifying the themes and then validating the themes and that's the way to work around it.
[00:20:18] Nancy Volkers: You could maybe use that clinically as well. I mean if you handed someone a list of like the 10 most common themes that were mentioned on Reddit and said, you know, “Which of these are a concern for you today?” You're sort of introducing the conversation without having to have the conversation and ask the questions, and then you get a sense of, “Oh, this person is very concerned about stigma today, or work issues.” And then you can sort of direct your attention to that.
Clinicians could feel overwhelmed, like, “Oh, now I have to ask about these 25 things,” right? And we've talked about ways they can refer and distribute and, you know, not take on all of this. But then on the other hand, as Dr. Fennig said, epilepsy is more than seizures. So how do you address the issues that are most important?
[00:21:09] Dr. Laura M'Rabet: That's exactly why we are so interested in this kind of research.
[00:21:15] Landis Wiedner: This is like the most hopeful, uplifting conversation, just in the fact of, you know, what can be done immediately. Because so many times with epilepsy research, sometimes there's not an immediate sort of reaction, but this could be an immediate thing that people start implementing in their clinics. And I'm just so appreciative.
[00:21:35] Nancy Volkers: So, the last issue I wanted to address is some analysis that you did in the study where you actually looked at other subreddits. There were some associations between what people were posting about in the epilepsy subreddit and risk for suicidal ideation. Dr. Fennig, can you talk a little bit about those results that you found in the study and then maybe some words of wisdom on how to ingest those for clinicians and for people with epilepsy?
[00:22:12] Dr. Uriel Fennig: Yes, of course. So, one of the great advantages of using Reddit for this type of analysis is that it's actually a network of many communities and forums, and users often post in several different subreddits, and this can allow for studying behavior.
Or, in other words, when does a person post in one subreddit in relation to posting in another? And what sort of topic is that person talking about in the epilepsy space? There are a lot of potentials for this sort of cross posting analysis, but we did decide to focus on associations with depression, suicide, and substance abuse. And I'll talk about the depression and suicide part.
So, the association of epilepsy with depression is well known, and suicide is an increased risk in people with epilepsy, about two and a half fold higher than the general population. We did find specific themes that when people brought them up in the epilepsy subreddit, they were significantly more likely to post in depression and suicide subreddits.
And these posts were about emotional issues, stigma and misconceptions, the postictal state, and work issues. Work issues was actually associated with the highest risk. These findings may show that these issues are especially distressing and are possibly accompanied by a sense of hopelessness.
And on the other side, there were topics with an inverse association with posting in suicide and depression. And these were discussions about surgical treatment and driving. Possible interpretation is that these topics may reflect a sense of hope, maybe due to positive outcomes of surgery or potential positive outcomes. And about driving, perhaps not being allowed to drive is often a temporary thing. So maybe that also can instill some hope in someone who is feeling depressed about that.
My personal take from this is try to keep these findings in mind when patients talk about it or bring up these topics. Let's say someone brings up work troubles. Maybe give this subject more gravity than they may have given before.
I think that mental health issues should not be an afterthought for our clinicians. We should be particularly mindful of patients experiencing issues like work-related difficulties or a high burden of the postictal state.
[00:24:34] Landis Wiedner: I have a question. Did these findings surprise you? Were these also a part of the “behind the curtain” metaphor? Or was this something that was a little bit like, “Oh, no, I kind of, I knew this.”
[00:24:47] Dr. Uriel Fennig: I mean, I wasn't surprised that emotional issues were associated with depression. But I was surprised, or maybe there are a lot of issues that I could potentially think about that could be associated with depression and they weren't, but work-related issues, that was something that I was, perhaps not surprised, but it was very informative for me, because it was the highest risk of all the others.
So now, it's kind of in the back of my mind all the time, that if someone brings up that they're having difficulties at work, or having trouble finding work, or having some trouble at work with their boss or their colleagues, like I said, I give that a little more gravity today. It's kind of a red flag, and I try to create space to talk about this.
[00:25:37] Dr. Laura M'Rabet: Another question I have related to this is. The severity of postictal state, is it associated with work problems? Or can't you, I don't think that you can get that out of the analysis, I think.
[00:25:51] Dr. Uriel Fennig: Yeah, it's a great question. I imagine that people miss work days because of being in a postictal state that can last sometimes for days, and that probably can create some problems with their employer or with their colleagues. And I actually, I don't know the answer. I would love to really delve into some of these issues and increase the resolution.
I mean, there's so many posts being analyzed. Over 50, 000 original posts and over 2 million comments. So, I'm sure we can find this kind of granularity, but we just have to think about it, have these ideas, like you just gave us. Which is great.
[00:26:30] Landis Wiedner: Laura, I just want to thank you. I didn't even think of that, of how the postictal and work just weave in together and postictal and probably a lot of those issues just weave in and are like almost, not the same thing, but postictal states affect these things.
I think postictal states also are something that don't get necessarily always addressed as thoroughly in a clinician's office. For me, mine can last four days, you know, and I go into a very intense depression, 50 percent of the time experiencing suicide ideation. And it's something that I have learned to strategize for.
I literally have a folder that I take out after a seizure that says, “You're going to be depressed. This is how you're feeling. It's okay. Here are the people to call. Here are the movies you like. Here are the food places to get food from. And if you're feeling suicidal ideation, here's another plan.” That starts with calling someone, going over to someone's house, getting outside, and ends with going to the emergency room if needed.
And I feel like those are conversations, they're just so urgent that need to happen in the clinician's office. And I just really appreciate both of you, how you're just like, “Well, yeah, work and post ictal, there’s a reason.”
And I was sitting there like, yeah, I missed like three days of work last week because I had a seizure. Just the layers of it. Even though it's this, we've got this data and it's numbered and we've got all the numbers, but there's still like the tapestry of research that I feel like is just being brought to life in this conversation with you two.
[00:28:02] Dr. Laura M'Rabet: Your tip, or what you just gave us, you already have something that's really practical, and that's something I didn't think of. We can really help people with just, “Okay, if you feel this way, it's normal to feel this way, and here is a plan that works for her, perhaps it also works for you. So what are your nicest films, where do you get your food,” and so this is really a great idea as well.
And if you don't talk about it, you don't get these kinds of things and ideas.
I really loved the conversation. I really loved your ideas. I loved talking to Uri about your research. Your research gave me new ideas and I loved the reaction of the stories of Landis.
[00:28:47] Landis Wiedner: Yeah, I want to just thank everyone here, all three of you, for coming together and creating such a beautiful conversation. It's just proof of what happens when, like, all the different sides connect and have these conversations, and we all look behind each other's curtains.
And I just hope that this is a catalyst not only for more research like this, but for any clinician listening, any patient listening, any caregiver listening to think about—these are the things that you know are prevalent and not only is it validation like Laura pointed out, but it's also something that can be like a guidepost as to next steps for what's best for you and your treatment or the person that you care about and love on their treatment.
[00:29:31] Dr. Uriel Fennig: I'll start by also thanking the three of you and it was really fascinating, this discussion and the ideas that you brought up were very insightful for me. I'm an early-career neurologist and starting my career in epilepsy and these kinds of discussions really helped me, I think, grow as a clinician and just be better at what I do, not only on the medical side, but try and really understand, help my patients.
And I think a take-home message is to be proactive about topics. And I'm talking about for the clinicians. Be proactive about certain topics like stigma or work-related issues and substance abuse—they're discussed a lot online. I didn't realize how much they were discussed, but they're discussed a lot online and not a lot at all in clinic. So, proactively addressing them could create more space for discussion.
We can't bring up a list of 30 issues in each appointment, but we can try to identify, using the paper, but using also other insights about which characteristics can be associated with certain topics. Younger patients may tend to be more concerned with stigma. The middle age group that we mentioned may be more concerned with work related issues. And I think that bringing up selected topics, at the minimum, will probably leave the patient with a feeling that they’re being listened to.
[00:31:02] Nancy Volkers: Thanks to all three of you so much. It was wonderful having you here and I really appreciate your time.