What the Delphi method is
The Delphi method was developed at the RAND Corporation in the 1950s to obtain the most reliable agreement possible from a group of experts on questions for which no definitive evidence existed (Dalkey & Helmer, 1963). Instead of meeting face to face, panel members answer a series of questionnaires. After each round the researcher summarises the group's responses and returns them, and members may revise their views in the light of that summary.
Four features define the method: anonymity among participants, iteration over several rounds, controlled feedback of the group's responses, and statistical aggregation of judgements (Hsu & Sandford, 2007; Hasson et al., 2000). Anonymity reduces the influence of dominant or senior individuals that affects committees, while feedback allows views to converge for reasons that can be documented. Participants remain known to the researcher, so the term quasi-anonymity is more accurate.
The method has spread from forecasting into the health and social sciences, and its use has grown across disciplines (Khodyakov et al., 2023; Niederberger & Spranger, 2020). In health professions education it is used to agree competencies, curriculum content, assessment criteria and the items of new instruments; one review discussed its role in radiography education research (John-Matthews et al., 2017).

When a Delphi study fits your question
A Delphi study suits questions about what should be, rather than what is: which competencies a graduate needs, which topics belong in a curriculum, or which criteria define good performance. It is appropriate when evidence is limited or contested, when the relevant experts are geographically dispersed, and when you want each expert's view to count equally. It is less suitable when the question can be answered by existing data, or when you need rich discussion of reasons, for which focus groups or a nominal group technique may be better.
| Variant | Main feature | Typical use |
|---|---|---|
| Classical Delphi | Round 1 asks open questions; items are generated from the answers | Exploring a new area |
| Modified Delphi | Round 1 presents items drawn from a literature review or prior work | Refining an existing list, as in many curriculum studies |
| Policy Delphi | Seeks to map the range of views and arguments rather than reach agreement | Contested policy questions |
| Online or real-time Delphi | Rounds run on a web platform, sometimes with continuous feedback | Large or international panels |
Designing the expert panel
The panel is the most important design decision, because the result is the agreement of these people and no others. Define what counts as expertise for your question, for example years of teaching, curriculum leadership, clinical practice or recent graduation, and state how you identified and invited members. A heterogeneous panel, mixing educators, clinicians, students and patients, broadens perspectives but may make agreement harder; a homogeneous panel converges more easily but represents fewer views (Hasson et al., 2000).
There is no agreed formula for panel size, and published studies vary widely (Hsu & Sandford, 2007). Justify yours from the diversity of expertise you need, the precision you want and the attrition you expect. Attrition is a real risk: in the DELPHISTAR study, 91 experts completed the first round, 69 the second and 56 the third (Niederberger et al., 2024). Plan reminders and keep each round short.
Ethics review is still needed even though participants are professionals answering questionnaires. Your application should explain how panel members are recruited, how consent is obtained, how their identities are kept from one another, how comments are anonymised before feedback, and where data are stored. In medical education, panel members may include your own colleagues or students, so address any power relationships and make clear that declining or withdrawing has no consequences. If you use an online survey platform, check where it stores data and whether it allows you to return each member's previous ratings privately.
Running the rounds
- 1Define objectives and consensus rule
- 2Select and invite the panel
- 3Pilot the questionnaire
- 4Round 1: generate or rate items
- 5Analyse and feed back
- 6Round 2: re-rate with feedback
- 7Round 3 or stop
- 8Report results
- Set the objectives and the rules. Before any data are collected, state the question, the rating scale, the definition of consensus and the stopping rule, and register them in your protocol (Diamond et al., 2014).
- Pilot the questionnaire with a few people similar to the panel to check clarity and timing.
- Round 1. In a classical Delphi, ask open questions and analyse the answers into items; in a modified Delphi, present items from your literature review and invite additions.
- Feedback. Return each item with the group's distribution or median and, where useful, anonymised comments, together with the member's own previous rating.
- Round 2 and later. Ask members to re-rate items that have not reached consensus, and to give reasons if they hold a view far from the group.
- Stop when the pre-specified rule is met, for example after a maximum of three rounds or when ratings change little between rounds.
Three rounds are common, partly because attrition rises with each additional round, but the number should follow from your stopping rule rather than from habit (Hsu & Sandford, 2007; Hasson & Keeney, 2011).
Defining and analysing consensus
A systematic review of Delphi studies found that consensus was often not defined, or defined only after the results were known, and recommended that the definition, the stopping criteria and the handling of items that do not reach consensus be stated in advance (Diamond et al., 2014). Percentage agreement is the most common definition. DELPHISTAR, for example, counted an item as agreed when at least 75% of respondents rated it 6 or 7 on a seven-point scale (Niederberger et al., 2024). Other studies use medians with interquartile ranges, or measures of stability between rounds (von der Gracht, 2012).
Quantitative analysis usually reports, for each item and round, the number of respondents, the distribution of ratings, the median and interquartile range, and whether the consensus rule was met. Qualitative comments are analysed thematically and used to reword items, explain disagreement and interpret the final list. Report items that did not reach consensus as findings in their own right, because persistent disagreement among experts is informative.
Worked example (illustrative)
A hypothetical doctoral researcher wants to agree the core content of a simulation-based curriculum for final-year medical students. She runs a modified Delphi: 40 items drawn from a scoping review are rated by a panel of 30 simulation educators, clinicians and recent graduates on a seven-point scale. Her protocol defines consensus as at least 75% of ratings of 6 or 7, sets a maximum of three rounds, and states that items rated 1 or 2 by at least 75% will be excluded. After round 1, 22 items reach consensus and three are excluded; the rest are fed back with medians and anonymised comments. She reports the response rate in each round and the items still unresolved after round 3. The numbers are invented to illustrate the steps.
Reporting a Delphi study
Reporting of Delphi studies has often been inconsistent; a review of studies selecting healthcare quality indicators found wide variation in how the method was applied and described (Boulkedid et al., 2011). Three guidance documents now help. CREDES gives recommendations for conducting and reporting Delphi studies, developed from a methodological review in palliative care (Jünger et al., 2017). ACCORD is a reporting guideline for consensus methods in biomedicine, itself developed through a modified Delphi (Gattrell et al., 2024). DELPHISTAR offers 38 reporting items for Delphi studies in the health and social sciences (Niederberger et al., 2024).
| Element | What to report |
|---|---|
| Rationale | Why a Delphi design, and which variant |
| Panel | Definition of expertise, recruitment, numbers invited and responding in each round |
| Items | How items were generated, piloted and worded |
| Rating | Scale, anchors and instructions |
| Consensus and stopping | Pre-specified definition, stopping rule and handling of non-consensus items |
| Feedback | What information was returned to panel members between rounds |
| Analysis | Statistics used and how comments were analysed |
| Results | Item-level results per round, including items not agreed |
| Ethics | Approval, consent and how quasi-anonymity was protected |
Strengths, limitations and common mistakes
The method's strengths are that it gives each expert an equal voice, reaches dispersed participants at low cost, documents how views changed and allows reflection between rounds. Its limitations are that results depend on panel selection, that attrition can bias later rounds, that it is demanding for the researcher and participants, and that feedback can push members towards conformity rather than genuine agreement (Hasson & Keeney, 2011).
- Defining consensus after seeing the results.
- Describing the panel only as "experts" without stating criteria.
- Reporting only the items that reached consensus.
- Ignoring attrition when interpreting later rounds.
- Presenting panel agreement as evidence of effectiveness.
For design choices more generally, see designing experimental research and mixed methods; for problem framing in medicine, research problem identification in medical science.
Key takeaways
- The Delphi method combines anonymity, iteration, controlled feedback and aggregation of expert judgement (Dalkey & Helmer, 1963; Hsu & Sandford, 2007).
- Panel composition determines the result, so define expertise and report recruitment and attrition (Hasson et al., 2000; Niederberger et al., 2024).
- Fix the consensus definition and stopping rule before the first round (Diamond et al., 2014; von der Gracht, 2012).
- Report with current guidance: CREDES, ACCORD and DELPHISTAR (Jünger et al., 2017; Gattrell et al., 2024; Niederberger et al., 2024).
- Treat persistent disagreement as a finding, and do not present consensus as proof of effectiveness (Hasson & Keeney, 2011; Boulkedid et al., 2011).
