How-to articleMethodology & analyticsMedicine & healthcare

The Delphi method in medical education research: panels, rounds, consensus thresholds and reporting

The Delphi method gathers expert judgement through anonymous, repeated questionnaires with feedback between rounds. It is widely used in medical and health professions education to agree competencies, curricula and assessment criteria. This article explains how to design, run, analyse and report a Delphi study for a doctoral project.

What you will learn

  • Explain the defining features of the Delphi method and when it suits a research question
  • Design an expert panel and justify its composition and size
  • Plan rounds, rating scales, feedback and a consensus definition before data collection
  • Analyse quantitative ratings and qualitative comments across rounds
  • Report a Delphi study using current guidance such as CREDES, ACCORD and DELPHISTAR

What you will unlearn

  • Consensus in a Delphi study means 75% agreement.Thresholds vary between studies; what matters is that the definition is justified and fixed before the rounds begin.
  • A Delphi study discovers the truth about a question.It measures the degree of agreement among a chosen panel, so its findings depend on who is on the panel.
  • Participants are completely anonymous.They are anonymous to one another but known to the researcher, which is better described as quasi-anonymity.
  • You keep running rounds until everyone agrees.A stopping rule, such as a maximum number of rounds or stable responses, should be set in advance.

Skills you will practise

  • Choosing a consensus method
  • Panel selection
  • Questionnaire and rating-scale design
  • Defining consensus and stopping rules
  • Analysing ratings across rounds
  • Reporting with CREDES and DELPHISTAR

What the Delphi method is

The Delphi method was developed at the RAND Corporation in the 1950s to obtain the most reliable agreement possible from a group of experts on questions for which no definitive evidence existed (Dalkey & Helmer, 1963). Instead of meeting face to face, panel members answer a series of questionnaires. After each round the researcher summarises the group's responses and returns them, and members may revise their views in the light of that summary.

Four features define the method: anonymity among participants, iteration over several rounds, controlled feedback of the group's responses, and statistical aggregation of judgements (Hsu & Sandford, 2007; Hasson et al., 2000). Anonymity reduces the influence of dominant or senior individuals that affects committees, while feedback allows views to converge for reasons that can be documented. Participants remain known to the researcher, so the term quasi-anonymity is more accurate.

The method has spread from forecasting into the health and social sciences, and its use has grown across disciplines (Khodyakov et al., 2023; Niederberger & Spranger, 2020). In health professions education it is used to agree competencies, curriculum content, assessment criteria and the items of new instruments; one review discussed its role in radiography education research (John-Matthews et al., 2017).

Delphi method in medical education for PhD research scholars
Figure 1. The Delphi method in medical education research.

When a Delphi study fits your question

A Delphi study suits questions about what should be, rather than what is: which competencies a graduate needs, which topics belong in a curriculum, or which criteria define good performance. It is appropriate when evidence is limited or contested, when the relevant experts are geographically dispersed, and when you want each expert's view to count equally. It is less suitable when the question can be answered by existing data, or when you need rich discussion of reasons, for which focus groups or a nominal group technique may be better.

Common Delphi variants
VariantMain featureTypical use
Classical DelphiRound 1 asks open questions; items are generated from the answersExploring a new area
Modified DelphiRound 1 presents items drawn from a literature review or prior workRefining an existing list, as in many curriculum studies
Policy DelphiSeeks to map the range of views and arguments rather than reach agreementContested policy questions
Online or real-time DelphiRounds run on a web platform, sometimes with continuous feedbackLarge or international panels

Designing the expert panel

The panel is the most important design decision, because the result is the agreement of these people and no others. Define what counts as expertise for your question, for example years of teaching, curriculum leadership, clinical practice or recent graduation, and state how you identified and invited members. A heterogeneous panel, mixing educators, clinicians, students and patients, broadens perspectives but may make agreement harder; a homogeneous panel converges more easily but represents fewer views (Hasson et al., 2000).

There is no agreed formula for panel size, and published studies vary widely (Hsu & Sandford, 2007). Justify yours from the diversity of expertise you need, the precision you want and the attrition you expect. Attrition is a real risk: in the DELPHISTAR study, 91 experts completed the first round, 69 the second and 56 the third (Niederberger et al., 2024). Plan reminders and keep each round short.

Ethics review is still needed even though participants are professionals answering questionnaires. Your application should explain how panel members are recruited, how consent is obtained, how their identities are kept from one another, how comments are anonymised before feedback, and where data are stored. In medical education, panel members may include your own colleagues or students, so address any power relationships and make clear that declining or withdrawing has no consequences. If you use an online survey platform, check where it stores data and whether it allows you to return each member's previous ratings privately.

Running the rounds

  1. 1Define objectives and consensus rule
  2. 2Select and invite the panel
  3. 3Pilot the questionnaire
  4. 4Round 1: generate or rate items
  5. 5Analyse and feed back
  6. 6Round 2: re-rate with feedback
  7. 7Round 3 or stop
  8. 8Report results
Figure 2. Steps in running a Delphi study.
  1. Set the objectives and the rules. Before any data are collected, state the question, the rating scale, the definition of consensus and the stopping rule, and register them in your protocol (Diamond et al., 2014).
  2. Pilot the questionnaire with a few people similar to the panel to check clarity and timing.
  3. Round 1. In a classical Delphi, ask open questions and analyse the answers into items; in a modified Delphi, present items from your literature review and invite additions.
  4. Feedback. Return each item with the group's distribution or median and, where useful, anonymised comments, together with the member's own previous rating.
  5. Round 2 and later. Ask members to re-rate items that have not reached consensus, and to give reasons if they hold a view far from the group.
  6. Stop when the pre-specified rule is met, for example after a maximum of three rounds or when ratings change little between rounds.

Three rounds are common, partly because attrition rises with each additional round, but the number should follow from your stopping rule rather than from habit (Hsu & Sandford, 2007; Hasson & Keeney, 2011).

Defining and analysing consensus

A systematic review of Delphi studies found that consensus was often not defined, or defined only after the results were known, and recommended that the definition, the stopping criteria and the handling of items that do not reach consensus be stated in advance (Diamond et al., 2014). Percentage agreement is the most common definition. DELPHISTAR, for example, counted an item as agreed when at least 75% of respondents rated it 6 or 7 on a seven-point scale (Niederberger et al., 2024). Other studies use medians with interquartile ranges, or measures of stability between rounds (von der Gracht, 2012).

Quantitative analysis usually reports, for each item and round, the number of respondents, the distribution of ratings, the median and interquartile range, and whether the consensus rule was met. Qualitative comments are analysed thematically and used to reword items, explain disagreement and interpret the final list. Report items that did not reach consensus as findings in their own right, because persistent disagreement among experts is informative.

Worked example (illustrative)

A hypothetical doctoral researcher wants to agree the core content of a simulation-based curriculum for final-year medical students. She runs a modified Delphi: 40 items drawn from a scoping review are rated by a panel of 30 simulation educators, clinicians and recent graduates on a seven-point scale. Her protocol defines consensus as at least 75% of ratings of 6 or 7, sets a maximum of three rounds, and states that items rated 1 or 2 by at least 75% will be excluded. After round 1, 22 items reach consensus and three are excluded; the rest are fed back with medians and anonymised comments. She reports the response rate in each round and the items still unresolved after round 3. The numbers are invented to illustrate the steps.

Reporting a Delphi study

Reporting of Delphi studies has often been inconsistent; a review of studies selecting healthcare quality indicators found wide variation in how the method was applied and described (Boulkedid et al., 2011). Three guidance documents now help. CREDES gives recommendations for conducting and reporting Delphi studies, developed from a methodological review in palliative care (Jünger et al., 2017). ACCORD is a reporting guideline for consensus methods in biomedicine, itself developed through a modified Delphi (Gattrell et al., 2024). DELPHISTAR offers 38 reporting items for Delphi studies in the health and social sciences (Niederberger et al., 2024).

Elements to report in a Delphi study, consistent with CREDES, ACCORD and DELPHISTAR
ElementWhat to report
RationaleWhy a Delphi design, and which variant
PanelDefinition of expertise, recruitment, numbers invited and responding in each round
ItemsHow items were generated, piloted and worded
RatingScale, anchors and instructions
Consensus and stoppingPre-specified definition, stopping rule and handling of non-consensus items
FeedbackWhat information was returned to panel members between rounds
AnalysisStatistics used and how comments were analysed
ResultsItem-level results per round, including items not agreed
EthicsApproval, consent and how quasi-anonymity was protected

Strengths, limitations and common mistakes

The method's strengths are that it gives each expert an equal voice, reaches dispersed participants at low cost, documents how views changed and allows reflection between rounds. Its limitations are that results depend on panel selection, that attrition can bias later rounds, that it is demanding for the researcher and participants, and that feedback can push members towards conformity rather than genuine agreement (Hasson & Keeney, 2011).

  • Defining consensus after seeing the results.
  • Describing the panel only as "experts" without stating criteria.
  • Reporting only the items that reached consensus.
  • Ignoring attrition when interpreting later rounds.
  • Presenting panel agreement as evidence of effectiveness.

For design choices more generally, see designing experimental research and mixed methods; for problem framing in medicine, research problem identification in medical science.

Key takeaways

Your next step

Want to apply this to your own study?

  1. 1Tell us your topic, data and deadline
  2. 2A subject mentor reviews your work with you
  3. 3You get a plan and a quote
Research design

References

  1. Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011). Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS ONE, 6(6), e20476. https://doi.org/10.1371/journal.pone.0020476
  2. Dalkey, N., & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9(3), 458–467. https://doi.org/10.1287/mnsc.9.3.458
  3. Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology, 67(4), 401–409. https://doi.org/10.1016/j.jclinepi.2013.12.002
  4. Gattrell, W. T., Logullo, P., van Zuuren, E. J., Price, A., Hughes, E. L., Blazey, P., et al. (2024). ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Medicine, 21(1), e1004326. https://doi.org/10.1371/journal.pmed.1004326
  5. Hasson, F., & Keeney, S. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change, 78(9), 1695–1704. https://doi.org/10.1016/j.techfore.2011.04.005
  6. Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing, 32(4), 1008–1015. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x
  7. Hsu, C.-C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation, 12, Article 10. https://doi.org/10.7275/pdz9-th90
  8. John-Matthews, J. S., Wallace, M. J., & Robinson, L. (2017). The Delphi technique in radiography education research. Radiography, 23(Suppl. 1), S53–S57. https://doi.org/10.1016/j.radi.2017.03.007
  9. Jünger, S., Payne, S. A., Brine, J., Radbruch, L., & Brearley, S. G. (2017). Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliative Medicine, 31(8), 684–706. https://doi.org/10.1177/0269216317690685
  10. Khodyakov, D., Grant, S., Kroger, J., Gadwah-Meaden, C., Motala, A., & Larkin, J. (2023). Disciplinary trends in the use of the Delphi method: A bibliometric analysis. PLoS ONE, 18(8), e0289009. https://doi.org/10.1371/journal.pone.0289009
  11. Niederberger, M., & Spranger, J. (2020). Delphi technique in health sciences: A map. Frontiers in Public Health, 8, 457. https://doi.org/10.3389/fpubh.2020.00457
  12. Niederberger, M., Schifano, J., Deckert, S., Hirt, J., Homberg, A., Köberich, S., Kuhn, R., Rommel, A., Sonnberger, M., & the DEWISS network. (2024). Delphi studies in social and health sciences: Recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study. PLoS ONE, 19(8), e0304651. https://doi.org/10.1371/journal.pone.0304651
  13. von der Gracht, H. A. (2012). Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change, 79(8), 1525–1536. https://doi.org/10.1016/j.techfore.2012.04.013

How to cite this article

Choose your referencing style, then copy. Check your university's or journal's guide for local variations.

PhD Research Lab Academy. (2021, November 29). The Delphi method in medical education research: panels, rounds, consensus thresholds and reporting. PhD Research Lab. https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars/

PhD Research Lab Academy. “The Delphi Method in Medical Education Research: Panels, Rounds, Consensus Thresholds and Reporting.” PhD Research Lab, 29 Nov. 2021, www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars. Accessed [access date].

PhD Research Lab Academy (2021) The Delphi method in medical education research: panels, rounds, consensus thresholds and reporting. PhD Research Lab. Available at: https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars/ (Accessed: [access date]).

PhD Research Lab Academy. “The Delphi Method in Medical Education Research: Panels, Rounds, Consensus Thresholds and Reporting.” PhD Research Lab. November 29, 2021. https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars.

PhD Research Lab Academy, ‘The Delphi Method in Medical Education Research: Panels, Rounds, Consensus Thresholds and Reporting’ (PhD Research Lab, 29 November 2021) <https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars> accessed [access date].

PhD Research Lab Academy. The Delphi method in medical education research: panels, rounds, consensus thresholds and reporting [Internet]. PhD Research Lab; 2021 Nov 29 [cited [access date]]. Available from: https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars/

PhD Research Lab Academy, “The Delphi method in medical education research: panels, rounds, consensus thresholds and reporting,” PhD Research Lab, Nov. 29, 2021. [Online]. Available: https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars. [Accessed: [access date]].

@online{delphi_method_in_medical_education_for_phd_research_scholars_2021,
  author = {{PhD Research Lab Academy}},
  title = {{The Delphi method in medical education research: panels, rounds, consensus thresholds and reporting}},
  organization = {PhD Research Lab},
  date = {2021-11-29},
  url = {https://www.phdassistance.com/blog/delphi-method-in-medical-education-for-phd-research-scholars},
  urldate = {[access date]}
}

This article is for learning. Use it to understand the method and cite the original sources above in your own work; do not copy or submit it as your own.

The Delphi method in medical education researchNeed help applying it to your study?
Research design