Introducing the MedEdMentor Paper Database

At the end of 2023, we launched our medical education specific literature search. Since then, you all have completed over 10,000 searches across our database of 46 journals containing 150,000 papers. Thank you for putting our tool to work!

But we have a lot of room for improvement. Our current approach is journal-based — if we include a journal, we have to include all of its papers. This creates a fundamental challenge: many important medical education papers are published in journals that mostly publish other content.

Take JAMA for example. While it published the original paper that introduced feedback in medical education, most of its papers are about clinical topics. With our current journal-based approach, we simply can't include journals like JAMA since their clinical content would overwhelm our search results.

To address this challenge, we developed a more precise approach. Together with Joseph Costello, Tony Artino, and Lauren Maggio, we developed a system to identify individual medical education papers. With this system, we no longer have to include all the papers from a journal.

We used our system to analyze 2.3 million papers from 151 journals. This process identified 119,000 medical education papers, which form the MEC (Medical Education Corpus).

Our tests show that 80% of the MEC is medical education content and that we captured 86% of medical education papers from those 151 journals. In comparison, our 2023 journal-based literature search was 47% medical education content.

Read on for more details, look at our preprint on medRxiv, or look through our data / technical info on Zenodo.

To build the MEC, first we had to decide what journals to pull from, then we needed to come up with a method to actually extract the medical education papers from our chosen journals.

To select which journals to pull medical education papers from, we created three groups of medical education journals, collectively named the MEJ (Medical Education Journals). The MEJ consists of:

• MEJ-Core — this group of journals consistently publishes medical education papers and forms a tight-knit citation network among themselves (e.g. Academic Medicine, Medical Education).
• MEJ-Adjacent — this group of journals occasionally publishes medical education papers and are moderately cited from MEJ-Core (e.g. JAMA, New England Journal of Medicine).
• MEJ-Peripheral — this group of journals rarely publishes medical education papers and are only sparsely cited from MEJ-Core. Since there are so many journals in this group, right now it exists in name only (we didn’t try to create a list of these journals).

The MEJ-Core currently consists of the MEJ-24, a set of 24 central medical education journals assembled using bibliometrics. With input from Lauren, Joe, and Tony (who all worked on the MEJ-24), we named this new set MEJ-Core so that it could expand beyond 24 journals as medical education evolves.

The MEJ-Adjacent consists of 127 additional journals. We included 102 journals that were cited 500 times or more in the MEJ-Core (Citation Impact in the diagram), and 25 additional journals from the AAMC MESRE Section's Annotated Bibliography.

The final list of journals is available on Zenodo.

Now comes the interesting part. We developed the MEC Classifier, a machine learning model that identifies medical education papers based on their titles, abstracts, and journal the paper is published in.

To train the model, we first gathered data for the model to learn from. We pulled all 2.2 million papers (later it became 2.3 million) in the MEJ-Core and MEJ-Adjacent from Semantic Scholar. Semantic Scholar is a search engine for scientific literature, but more importantly they expose their index of papers for free.

From these 2.2 million papers:

• we sampled 2,532 papers to train the model, and we labeled each of them as medical education or not
• we also separately sampled and labeled 1,500 papers to test the model (more on testing below).

Each paper was labeled by two different members of our team (medical education or not). If there were conflicting labels we used another team member as the tiebreaker. In general, we labeled papers as medical education if they met any of these criteria:

1. focused on undergraduate, graduate, or continuing medical education
2. involved premedical students, medical students, interns, residents, or fellows
3. otherwise appreciably pertained to medical education (e.g. surgical simulation)

All of our training data, testing data, and code is also available on Zenodo.

In our final step, we used our MEC Classifier to analyze all 2.3 million papers from our selected journals. It found 119,000 medical education papers, which form the MEC.

A snapshot of the MEC saved in December 2024 is freely available to download on Zenodo.

The MEC represents a major improvement over the only other way to identify medical education papers: PubMed’s Medical Subject Headings (MeSH).

MeSH terms are labels that PubMed assigns to papers to help organize them by topic. MeSH terms look like this at the bottom of paper pages on PubMed:

PubMed uses their own machine learning model to assign MeSH labels, called MTIX. About one-third of articles indexed via automation also receive human curation, but the majority of articles are automatically indexed.

For medical education, we found that only using "Education, Medical" isn't sufficient (a bit counterintuitive, right?). PubMed also labels medical education content under other MeSH terms too, including "Schools, Medical", "Students, Medical", and "Faculty, Medical."

We tested our system head-to-head against PubMed's MeSH labels using the 1,500 test papers that our machine learning model hadn't seen before.

For a fair comparison, we used only the subset of our test papers that were available on PubMed, which worked out to be 1,358 papers. (In general, Semantic Scholar contains about 10% more papers per journal than PubMed does, most of it appears to be things like news, announcements, and corrections)

Here are the results:

Our first metric is sensitivity, which focuses on the MEC Classifier's ability to retrieve medical education papers. Sensitivity was calculated as the proportion of actual medical education papers correctly identified by the method in question.

Within the 1,358 test papers available on PubMed, we had labeled 71 of them as medical education papers. To check sensitivity, we need to check how many of these papers are retrieved by each method:

• Combining all medical-education-related MeSH terms (including their child terms) retrieved only 47 of the 71 papers (66% sensitive).
• In contrast, the MEC Classifier retrieved 64 of the 71 papers (90% sensitive).

This increased capture by the MEC Classifier leads to huge downstream impacts on individual journals.

We looked at the top 10 medical education journals with the highest proportion of medical education content as determined by the MEC Classifier. We then compared against the proportion of the journal that was classified as medical education by MeSH.

We wanted to compare how much more we were capturing with the MEC Classifier on a journal-level basis.

The MEC is the first dedicated corpus of medical education papers. We think that its improved coverage compared to MeSH opens up new possibilities for medical education scholarship. We can now surface important papers from clinical journals that might otherwise be missed. Also, with a more complete picture of medical education literature, we can better analyze trends and identify emerging topics.

The MEC marks a significant step forward in making medical education literature more discoverable and useful. We're building upon this foundation with the MedEdMentor Paper Database. We think this database will be especially valuable if you have limited access to experienced librarians and mentors.

MedEdMentor's broader mission is to support scholarship across all of health professions education. While we've started our Paper Database with medical education papers, we'll eventually expand to create a comprehensive database for health professions education.