AI in HPE Scholarship: Reflections from an Expert Panel
Hey there, MedEdMentor community!
Geoff here, and I'm really excited to share with you all some brilliant insights I gained when I moderated a recent panel on AI in Health Professions Education (HPE) Scholarship. I am so grateful to have had the opportunity to learn from these amazing folks, and just as excited to share what they had to say with you all. Of note, this panel discussion occurred on August 1st, 2024. Time stamping anything AI related is really important given how quickly everything is moving.
Meet the Panelists
Before we dive in, let's introduce the stellar panel:
- Dr. Celia Laird O'Brien, PhD: Assistant Dean of Program Evaluation and Accreditation at Northwestern University's Feinberg School of Medicine
- Dr. Carla Pugh, MD, PhD: Thomas Krummel Professor of Surgery at Stanford Medicine and Director of the Technology Enabled Clinical Improvement (T.E.C.I.) Center
- Dr. Martin Pusic, MD, PhD: Associate Professor of Pediatrics and Emergency Medicine at Harvard Medical School and Director of the Research Education Foundation at the American Board of Medical Specialties
Use of AI in this Piece
It's important to be transparent about the use of AI in our work for multiple reasons (a theme you will read about below). Here's a step-by-step breakdown of my approach for this particular post:
- Transcription: I uploaded the audio file of the panel discussion to Otter.ai to generate an initial transcript.
- Manual Editing: I carefully reviewed and edited the transcript for accuracy, ensuring quotes were correctly attributed to each speaker.
- LLM Identification of Themes: I turned to AI for my first pass at theme identification. I provided the transcript to Claude, a large language model from Anthropic. I also described the purpose and setting of the panel as well as the expertise of the people on the panel. I then asked Claude to extract the important themes that were discussed. The results were disappointing based on my recollection of the conversation.
- Human Identification of Themes: Given the insufficient thematic analysis from Claude, I started from scratch. After a thorough reading of the transcript, I identified key themes that I found most important and interesting.
- AI Writing Assistance Setup: I returned to Claude for help with the writing process. To orient Claude to my writing style, I provided five examples (over 15,000 words) of my previous work that captured the conversational tone I was aiming for.
- Content Provision: I re-supplied Claude with the edited transcript, my personally-identified themes, and detailed descriptions of what stood out to me in the discussion related to these themes.
- Initial Draft Request: I asked Claude to write a first draft between 500-1500 words. This turned out to be insufficient for the depth of content we had.
- Comprehensive Draft: I requested a new draft. This version incorporated representative quotes from the transcript and stayed under 3000 words, while attempting to mimic my writing voice.
- Extensive Editing: I heavily edited this AI-generated draft to ensure it accurately represented the panel discussion and maintained what I thought was my authentic voice.
- “Final Polish”: I asked Claude to review and clean up my edited version for clarity and grammatical soundness. “Final polish” is in quotes as you’ll see based on the following steps.
- Panelist Reviews: I shared the piece with all three panelists to get their approval of the content and their quotes. Two of three were able to respond and gave the go-ahead.
- Back to The Drawing Board: As with all our written pieces, Greg and I share them with one another for critique before we share them with all of you. Greg provided some great feedback, with his main point being, “this reads like it was written by AI.” This was not the plan and not what you all deserve.
- Reorganization and Rewriting: I kept the ideas and the excellent points from the panelists, but created an overarching organization system, and rewrote the majority of the prose so that it was truly a piece worthy of our audience.
- FINAL Polish: Lastly, I sent it back through Claude to clean up any grammatical issues, then back to Greg for the thumbs up.
This process shows both the potential and the current state of AI and LLMs in particular. I used AI for each piece of this project. The only task with which I am sure it saved my time was transcribing the audio. Otherwise, I needed to either reperform or significantly alter the work of Claude. As you’ll see, the description of this AI-supported process ties in nicely to the overall topic!
Rogers' Diffusion of Innovation
Dr. Pusic brought up Rogers' Diffusion of Innovation Theory, which is pictured above and you will read more about later in this piece. This concept was developed to try to “explain how, why, and at what rate new ideas and technology spread.” Those that take to innovations quickly are represented on the left of the curve, with those that come later represented on the right. These various groups are all important in their own right, and this piece will highlight topics that encapsulate these various levels of acceptance, from the left to the right, after a high-level overview.
Note: Quotes have been edited for clarity and conciseness but maintain their original intent.
The Panelists Placed Themselves Across the Adoption Spectrum
One of the first things that struck me was the variety of approaches our panelists have towards AI, with Dr. Pusic furthest to the left (earliest adoption), followed by Dr. Pugh, and then Dr. O’Brien.
Dr. Pusic reports he is diving in headfirst. He enthusiastically explained, "I do use it. They say that AI will most help people with the things they struggle with. For example, I find titles really, really hard and laborious, and I hate doing it." He then described an iterative prompting process that helps him work towards more compelling titles for his manuscripts. He also mentioned creating an initial outline for a manuscript and creating an abstract from a full manuscript as examples of AI applications that he has employed.
Dr. Pugh employs AI for the sake of veracity. She shared, "I was doing a summary of engineers who were pioneers in haptics, and I wanted to see if it lists the people that I knew...like, do I have my list correct?" She also described an exercise in which she will divide her research group into two teams. One will use the tried-and-true method of statistical analysis without the help of AI, and the other will be free to employ AI as they see fit. The two teams need to "show their work," especially the team using AI so that everyone can analyze the queries that led to specific results. The full group will come together to compare their results, focusing on where each team may have gone wrong. FYI...Dr. Pugh says the team that errs has always been the team using AI (so far).
Dr. O'Brien takes a more cautious approach, using AI to help with writing for specific sentences, but not much beyond that. She expressed concern that over-reliance on AI might hinder her thought process, saying, "I think it was Lorelei Lingard who said that 'writing is thinking,' and really trying to craft what I'm saying helps me make my message more clear." This opening statement prefaced many moments where the panelists cautioned against outsourcing fundamental skills to AI as it may hinder development of those same skills.
Early Adoption
AI as a Writing Assistant
One of the most fascinating discussions centered around AI's role in writing. Dr. Pusic pointed out how LLMs really help him support mentees whose first language isn't English:
"I was involved in a master's program in which a number of the people are international, and writing for them does not come naturally, in English. It used to be that their mentors had to act as grade-12 English teachers at the same time they were worrying about the substance of the thing. And what I've seen is we get to focus more on the substance of the manuscript."
Dr. Pugh added to this theme and shared a brilliant workaround for a technically-minded mentee struggling with readability:
She even suggested using AI to conduct a mock "CNN interview" to help inject some personality into technical writing. I thought this speech-to-text combination with an LLM was a brilliant idea.
Dr. O'Brien brought some great perspective and raised a valid concern about AI potentially homogenizing writing styles, especially for novice writers still finding their "voice." She cautioned, "It's really inadvisable if you are not an expert, or if you are still learning in an area, to plug something in and ask for references or to help frame an argument." With these innovative technologies, cautious and thoughtful application is always advised.
Early Majority
The Future of Qualitative Research
The current abilities of AI tools to perform qualitative research might not knock your socks off, like when I asked Claude to extract themes from the transcript of this panel discussion. However, there's potential for some exciting developments as Dr. Pusic suggested a novel mixed-methods approach:
Exaggerating for effect, he said, "Every qualitative study that's ever been done has an N of 15. [This line got a good laugh from the audience]. Okay, N of 12 to 20, maybe 21, that sort of thing. But the notion is that qualitative research is super hard, right? The researchers have transcriptions from one hour of interviews of 15 people, and then they have to analyze that and identify the themes. That's several months of a person's effort. And therefore, we cap it. And theoretical sufficiency magically arrives at the 13th interview plus two more [clearly sarcasm]. I'm always a bit skeptical of that. And so, if we can use these tools, ATLAS.ti now has some sort of interface to a large language model that can make us more efficient then we can analyze a tsunami of text data."
Qualitative research can be incredibly rewarding and when engaging deeply with the data, one can gain an incredible amount of insight into a topic. But, anyone who has done a qualitative analysis knows the slog of this work. Like the writing process, doing the work is where the learning and thinking happen.
AI and Professional Identity Formation
A thread that appeared throughout our discussion was the idea that learning to incorporate AI into our work is a professional identity formation opportunity for all of us. Dr. O'Brien articulated this beautifully:
"I think that we, as academics and researchers, will also be going through a change in professional identity as these tools get more ubiquitous. And I think, right now, there's a lot of hesitation. But, as they become more advanced and we understand how much more productive one might be with this kind of assistance, that will be great!" She continued by asking rhetorically:
Our field is spending a lot of time on professional development aimed at teaching individuals how to use various AI tools. This technical training is important, but reimagining our roles as researchers, educators, and mentors in an AI-enhanced world is important work we need to be addressing now.
Late Majority
The Double-Edged Sword of AI and Diversity, Equity, and Inclusion
Our panel didn't shy away from the thorny issue of AI and diversity, equity, and inclusion (DEI). There are valid concerns about AI potentially perpetuating biases in existing data, but we also hope that it could address some DEI issues in research and writing. Dr. Pusic highlighted this duality:
"'Garbage in, garbage out.' So the first thing is to put less garbage in, right? One of my colleagues at the University of Miami ran a visual interpretation course where medical students go and look at art, and then ask them, 'what do you see?' And then, they examined a piece of art meant to bring out issues and tensions pertinent to DEI. And then they uploaded the image into ChatGPT and asked it to analyze the artwork. And the results came back, biased, biased, biased, biased." But, when the University of Miami educators specifically prompted ChatGPT to consider DEI issues in its interpretation of the image, it performed much better.
Similarly, Dr. Pusic noted the potential for AI to help him personally to identify and address biases:
The dual nature of AI in relation to DEI issues, its inherent biases based on its training data as well as its ability to identify bias in our work, is a dichotomy we will likely need to grapple with for a while.
Academia vs. Big Tech
Throughout our discussion, there was an undercurrent of concern about the role of big tech companies in shaping AI's future in academia. Dr. O'Brien captured this tension succinctly, “let's not forget that a lot of this, this whole new world, is going to be authored by for-profit companies. And the prevailing sentiment is, 'educational institutions are too bureaucratic, too slow, and too poor, and we're never going to keep up.'" She continued by articulating that we need to fight this perception and make sure we are staying involved and pushing the envelope.
Dr. Pugh added to this sentiment and tried to empower the audience, saying, “they want our data, so we've got a valuable card. They want our data. So be a businessperson and figure out your business strategy, and don't just give it away."
Dr. Pusic pointed out, as clinicians, compared to many other groups, we're uniquely positioned to navigate the potential dangers of AI:
He continued with a very astute analogy, "it's like using methotrexate for eczema. Whoever did that for the first time? Oh, my God! Right?!? Using a chemotherapeutic agent for a skin condition that can have a very high morbidity, but very low mortality? That’s a scary proposition. But they figured out that balance."
As the landscape of AI continues to evolve, we should not shy away from experimenting with these tools. Instead we should bring our cautious-yet-creative approach to ensuring academia remains among the leaders in the field.
The "Laggards"
The Role of Skepticism and Embracing the Naysayers
While much of our discussion focused on the potential of AI, our panelists also emphasized the importance of maintaining a healthy skepticism. Dr. Pusic drew an interesting parallel to the adoption of new technologies in general:
"Remember Rogers' Diffusion of Innovation? You know, the theory that has the badly named 'laggards' and leading-edge 'innovators' in a normal distribution, with most of us being somewhere in the middle? These two extremes each fulfill important functions. The 'innovators' bring the shiny new toy in front of us, but the 'laggards,' or 'late adopters' is probably a better term, are the ones who bring the proper skepticism."
He went on to stress the importance of protecting and valuing those who approach AI with caution:
As you have seen by this panel discussion, the conversation covered all the ground, from the early adopters all the way to the properly skeptical folks. I thought the topics covered represented the breadth of opinions, with many of the individuals on the panel bringing up ideas from multiple perspectives along Roger’s continuum. Just as we should have a balanced approach to AI as individuals, pushing the boundaries of what is possible while maintaining caution, balance in the field is also important to make sure we are growing with the technology in thoughtful ways.
Walking the Ethical Tightrope of AI
Lastly, much of what might make someone properly skeptical about AI falls into the ethics category. And all our panelists agreed: we're in uncharted territory, and the rulebook is still being written. But one thing's crystal clear – transparency is key.
Dr. Pusic emphasized the importance of ownership and responsibility: "There's kind of a 'Maginot Line,' if you will, in which you have to own every word of a manuscript. Even if ChatGPT started you off, you own all of this. You are responsible for any plagiarism or anything along those lines."
Dr. Pugh echoed this sentiment, stressing the need for clear guidelines: "I think that we need more guidelines on how to do it. I ask people to submit their queries to me so that I can look at the work that led to their output and learn alongside them. It's a technology that's not going to go away."
This ethical tightrope walking extends to the peer review process as well. While many journals are okay with AI-assisted manuscript drafting (with disclosure), they're putting the brakes on AI in peer review. But with reviewer shortages becoming a real headache, some of our panelists wondered if AI could lend a hand in the future.
Dr. Pusic mused, "I'm a deputy editor at Medical Education, and I cannot tell you how difficult it is in this day and age to get people to review. It used to be that you'd get four reviewers for every manuscript, and that was the standard. Now, I'm lucky if I can get two."
AI certainly has the potential for misuse and promoting academic dishonesty. Therefore, being clear about our applications and honestly sharing our successes and failures represents our best path forward as we navigate AI in HPE scholarship.
Wrapping Up: A Call to Action
So, where do we go from here? Well, if there's one thing our panelists agreed on, it's this: we need to keep experimenting, learning, and sharing our experiences. Dr. Pugh put it beautifully when she called for mentors to share their approaches to working with AI and mentees:
AI is here to stay, and it's evolving faster than we can imagine. But that doesn't mean we're passive observers. We need to engage, be creative, share our experiences, and work together to build the future of our field. And MedEdMentor is here to partner with you all in this journey.
What are your thoughts on AI in HPE scholarship? Have you experimented with AI tools in your own work? Please, share your thoughts via email. A few reflections from the MedEdMentor Community might make for a great follow-up piece!
Along these same lines, we welcome post ideas for The Shape of Scholarship (our blog) from all contributors, regardless of your native language or English proficiency. We're eager to amplify diverse voices and are happy to support you in developing your piece.
Until next time,
Geoff
Enjoyed this article?
Sign up for MedEdMentor to access our full library of resources.
Get started free