Searching Medical Education Journals on MedEdMENTOR

Information retrieval is concerned with taking a group of documents, applying a query, and ideally returning all relevant results, while returning no irrelevant results (in diagnostic terms, maximum sensitivity and specificity).

Using this lens, let's think through the existing solutions.

In medical education we often search PubMed, a fantastic resource. PubMed contains ~36 million papers, excellent for returning lots of relevant articles.

But at its core, PubMed is a database for biomedical literature. That means medical education researchers tend to find many irrelevant results.

For example, let's say you want to get a quick overview about feedback. Here is a naive search for "feedback," which returns 200,000 results, none of which are medical education related.

Now, those of you who are more library-science savvy may have pointed out that my search above could probably have been more effective with MeSH terms.

MeSH terms, or Medical Subject Headings is a controlled dictionary of terms used to label items on PubMed.

Here's the problem: for medical education, Maggio 2021 showed that MeSH terms are inaccurate.

For example, replicating Maggio's 2021 analysis, we found that Academic Medicine, often considered a core journal in the field, has 14,203 results in PubMed.

Google Scholar's index is estimated to be about 165 million documents. Their search algorithm is reported to use about 200 factors to rank documents "the way researchers do."

Google Scholar is certainly powerful and has its place, but aside from suffering from the similar problem of PubMed and returning irrelevant results because its index is so large, Google Scholar in particular may be prone to promoting bias.

Research reports using various forms of reverse engineering have shown that citation count is the heaviest factor in Google Scholar rankings (Beel 2009).

However, a recent report (Jordan 2022) pointed out the issues around using citation count as markers of relevance. Mainly that many fields have historically struggled with diversity, and so ranking by historical citation counts would only serve to amplify the existing voices, not bring up new voices.

Despite much of our training in biomedical science where replicability and objectivity are held as ideals, an academic search engine cannot truly be objective. (Why Your Medical Education Research Needs Theory)

Stating that a document is "relevant" always begs the question "relevant for who?"

Is it relevant for the average reader, the expert, or the novice? Is it relevant relative to our historical state or a future state?

Below is the perspective that MedEdMENTOR's search engine takes and provide transparency into how it works.

Given the continued explosion of medical education literature, and our goal to provide focus and clarity, we have adopted a Curator's mindset. To MedEdMENTOR, the best search contains the "core" of medical education and is not diluted with irrelevant results.

What that means in practice is that we prioritize minimizing irrelevant papers (maximizing specificity) with the tradeoff that we will not have every single possible relevant paper (diminished sensitivity).

We believe this approach is ideal for medical educators who are seeking guidance.

Now, what exactly are you searching when you search MedEdMENTOR?

First we began with previous work which utilized bibliometrics to delineate the core of medical education into 24 journals (Maggio 2021).

(Full journal list is at the end of the Lesson)

We then looked beyond the bibliometric core of the field, and noted that the AAMC MESRE's Annotated Bibliography of 60+ additional journals contained MEJ-24 along with many journals with high relevancy and low irrelevancy.

Combine these approaches, we have created the first-ever (to our knowledge) index of medical education articles, with the MEJ-24 at the core and selected journals from the Annotated Bibliography also available for search.

Utilizing Semantic Scholar, we collected and indexed 120,000 documents from 22 journals from MEJ-24 (Focus on Health Professional Education and BMJ Simulation and Technology Enhanced Learning is not indexed on Semantic Scholar), along with an additional 24 journals from MESRE’s Annotated Bibliography which when reviewed had high relevancy and low irrelevancy.

We will continue to modify this list and update our index over time.

Based on the above discussion related to Google Scholar, our initial search index does NOT utilize any citation counts for relevance.

For search ranking, we are currently utilizing a keyword based search, where the more of the keywords are present, the higher the search relevance. Keywords in the title are worth more than in the abstract. And the closer the keywords are to each other the more relevant the document is scored.

(A complete technical description: we are utilizing the open-source Postgres full text search engine, with title weighted A and abstract weighted D, with cover density = true.)