The NBME's 200+ page test writing guide contains 10 golden rules governing how they design USMLE questions. I've read it twice and will share the highlights here.
Introduction to the "Gold Book"
The NBME's Gold Book, Constructing Written Test Questions for the Basic and Clinical Sciences, is an inside look into how the NBME thinks about medical knowledge assessment. It touches upon which questions are the ones worth asking, as well as exactly how to (and how not to) write questions. Though the intended audience of the Gold Book is medical educators, I think certain implicit principles referenced over and over in it are helpful to students studying for the USMLEs, including Step 1, Step 2 CK, Step 3, and the NBME Shelf exams.
Definitions used by NBME
Items: A full multiple choice question, including the stem of the question, the lead in, the correct answer, and the distractors.
Stem: The first part of the item which leads to the question being asked, i.e., the lead in. Often, this takes the form of a clinical vignette or a description of an experiment.
Lead-in question: Also known as the question itself.
Options: The set of all answer choices.
Single best response: The single best option.
Distractors: The set of options excluding the single best answer.
The goals of the USMLEs
Step 1: to assess your understanding and ability to apply important concepts of the sciences underlying the practice of medicine, with emphasis on principles and mechanisms of underlying health, disease, and modes of therapy.
Step 2 CK: to assess your ability to apply medical knowledge, skills, and understanding of clinical science essential for the provision of patient care under supervision, with an emphasis on health promotion and disease prevention.
Step 3: to assess your ability to apply medical knowledge and understanding of biomedical and clinical science essential for the unsupervised practice of medicine, with emphasis on patient management in ambulatory settings.
The "Start with an Important Concept" rule
Before writing an item, an important testable concept should be decided. The entire item should be then be written with this concept in mind.
According to the NBME, items should 1) align with educational goals of the test, 2) focus on important topics more heavily than unimportant topics, 3) focus on important knowledge as opposed to trivial facts, 4) test application of knowledge as opposed to isolated fact recall, 5) not be tricky or overly complex.
Vignettes are often used by the NBME because they allow to test concepts rather than trivial facts.
The NBME designs their vignettes to focus on the following
• Application of knowledge using clinical vignettes to pose medical decisions or clinical decision making tasks in clinical situations
• Common or potentially catastrophic problems; avoid “zebras” and esoteric entities.
• Specific tasks the test-taker must be able to undertake at the next stage of training
• Areas in which clinical reasoning mistakes are often made
The set of important knowledge for USMLE exams is outlined in the official USMLE content outline. Items are written with this content outline in mind, with respect to the NBME's item writing guidelines, and in accordance with the goal of each exam.
The "Use A Vignette Template" rule
The NBME literally recommends using a template to write clinical vignettes. Here is the actual template.
First sentence:
• Age, gender (e.g., a 60-year-old woman)
• Site of care (e.g., the emergency department)
• Presenting complaint (e.g., shortness of breath)
• Duration of complaint (e.g., 3 days)
Subsequent sentences:
• History of present illness, past medical history, family history, social history, and review of systems if important and plausible for the scenario
• Physical findings
• Results of diagnostic studies (labs, imaging, tests)
• Initial treatment, subsequent findings
Lead-in question:
Should be focused and, together with the stem, allow one to list a homogeneous set of options and select the single best option without looking at the written options.
Options:
Should be homogeneous and plausible.
Each should be wholly true or false along one dimension.
There should be a single best answer which can be guessed without looking at the options.
The "Single Best Answer" rule
Options should be homogeneous, able to be judged entirely true or false based on a single dimension, contain a single best answer. Distractors may be wholly or partially wrong. The single best answer is the one that is true along the most number of dimensions. For example, if the answer is a diagnosis, the single best answer should be a unifying diagnosis consistent with as many components of the clinical vignette as possible.
Example: "A 60 year old female develops abrupt onset dyspnea two days after orthopedic surgery. Her blood pressure is 90/60, pulse 100, and temperature 37.0. A chest radiograph is performed and is normal. Which of the following is the most likely diagnosis?"
Homogeneity means that the options might include several reasonable causes of dyspnea e.g., Primary pulmonary hypertension, Emphysema, Pneumothorax, and Pulmonary embolism.
Each of the distractors is incorrect along one dimension. For example, emphysema and primary pulmonary hypertension are chronic and progressive diseases, so these options are wholly incorrect based on the dimension of chronicity. Pneumothorax and pulmonary embolism are both abrupt onset. However, pneumothorax is wholly incorrect on the dimension of imaging, as a significant pneumothorax would be detected on chest radiography. The single best answer is pulmonary embolism.
However, the simplest way to answer this question is to simply recognize that abrupt onset dyspnea and hypotension after a surgery, especially an orthopedic surgery, is concerning for pulmonary embolism. In other words, the question could have been answered without looking at the options at all.
Applying the Single Best Answer rule requires a strong fund of knowledge, so be sure to check out our posts on what resources you should use when preparing for the USMLE Step 1 and clinical shelf exams and USMLE Step 2 CK exams.
The "Cover The Options" rule
All USMLE questions should be able to be answered "from the stem and lead-in alone"..."without looking at the options".
The NBME guide states that if a lead-in is properly focused, you should be able to read the stem and the lead in, cover the options, logically generate a homogeneous set of options, and guess the correct answer.
The "All Relevant Facts" rule
Items stems should contain all the relevant facts necessary, and no data should be provided in the options.
If you are confused by a stem, do not look to the options for guidance; all relevant facts are contained in the stem, and the lead-in should allow you to use these facts, based on the important knowledge you have learned, to guess the correct answer.
The "Patients Do Not Lie" rule
Unless you are provided with a statement that the physician suspects something that contradicts what the patient has reported, you can assume that what the patient reported or what has been stated about the patient is correct.
The "Classic Cases" rule
The NBME discourages the use of real patient cases on the USMLEs, simply because real patients are too complex, and have non-classic presentations. The NBME will not try to mislead you by providing "red-herrings" in vignettes that are intended to lead you to an incorrect diagnosis. The USMLE items are written with a preconceived testable concept in mind. However, the NBME will sometimes provide "window-dressing" or extraneous information not needed to answer the item. In general, cases on the USMLE are classic cases. USMLE Step 2 CK contains more "window dressing" than does Step 1, and Step 3 contains more "window dressing" than Step 2 CK. You and your MedSchoolGuru will work together to make sure you first understand which findings are important, and then integrate those findings into a diagnosis, differential diagnosis, or clinical action.
The "No Trivia" rule
USMLE items are designed to test important concepts rather than trivial facts. If you got an item wrong on an NBME exam and think the tested concept is trivial, you are either misunderstanding the question, or you don't fully understand why the fact is important. Your MedSchoolGuru will review all of your NBMEs with you after you take them to help you why you got each question wrong and help you understand the concepts you need to know, as well as why they are important.
The "No Savvy Test Takers" rule
USMLE items are designed to prevent savvy-test takers from guessing the correct option based on technical flaws on how options are designed or how the lead-in question is phrased. Don't spend too much time trying to "outsmart" the test. Instead, try to understand the stem, and figure out, based on what you have been taught, if you can identify the important testable concept alluded to in the stem of the question. Think about this concept, read the lead-in, and cover up the answers. Come up with an answer in your own head, then look at the options to choose the answer that best matches the answer you came up with in your head.
Final thoughts
While this article helps you understand the blueprint underlying each USMLE question, and hints at a few good tricks to approaching USMLE questions (e.g., the "cover the answer" method), it does not provide a systematic approach to any USMLE question. MedSchoolGurus provides its tutees with a brief but comprehensive guide on how to approach any USMLE question. As you review questions with your Guru, your Guru will help you identify the cognitive pitfalls which lower your score the most, coach you on how to avoid these pitfalls, and use the MedSchoolGurus approach to any USMLE question to help develop and polish your own test-taking skills.
For more information about our shelf exam and USMLE tutoring services, check out medschoolgurus.com. To schedule a free 15 minute consultation session, visit medschoolgurus.com/contact.
