How-to article

Reliability and Validity in a PhD Thesis: Which Tests to Report and When

Learn which reliability and validity tests to use in a PhD thesis, how to interpret questionnaire results, and what measurement evidence to report.

Introduction

A questionnaire can produce consistent scores without measuring the intended construct. Likewise, a scale may have strong theoretical foundations but perform poorly in a particular sample. Reliability and validity therefore address related but different questions: Are the scores sufficiently consistent, and is there evidence supporting their intended interpretation?

Start With the Measurement Question

Before selecting a reliability or validity test, identify what needs to be established. A multi-item questionnaire, a clinical measurement, an observational rating and a single-item question do not require the same assessment.

Use this sequence to guide the analysis:

  1. Internal consistency: Do items intended to measure the same construct produce sufficiently consistent scores?
  2. Temporal consistency: Is the measure consistent, independent of the change in the actual attribute?
  3. Rater consistency: Is there any consistency in the ratings made by different raters?
  4. Factorial composition: Does the measure consist of the factorial composition?
  5. Construct validity: Is the interpretation supported by theoretical and empirical evidence?

These questions are not interchangeable. For example, test–retest reliability cannot establish whether questionnaire items represent the correct construct, and internal consistency alone cannot confirm a scale’s factor structure.

Cronbach’s Alpha vs McDonald’s Omega

Cronbach’s alpha and McDonald’s omega are commonly used to assess the internal consistency of multi-item scales [1, 2]. However, they rely on different assumptions and should not be treated as interchangeable measures of scale quality.

Measure

When it is useful

Main consideration

Cronbach’s alpha

Assessing internal consistency when its assumptions are reasonably appropriate

Interpretation is affected by the scale’s dimensionality and assumptions about item relationships

McDonald’s omega

Assessing internal consistency when items have different factor loadings

Requires an appropriate factor model; the selected omega coefficient must match the measurement structure

Alpha is most straightforward to interpret when items measure a common construct and the assumptions underlying its use are reasonable. When item loadings differ, omega may provide a more suitable reliability estimate under an appropriate model.

Neither coefficient proves one-dimensionality. A high alpha can occur when a scale contains correlated dimensions or many similar items. Researchers should examine the intended construct and factor structure rather than interpreting a coefficient in isolation.

Omega also has different forms. Omega total estimates the proportion of score variance attributable to all common factors in the specified model, whereas omega hierarchical estimates the contribution of a general factor when a suitable hierarchical or bifactor model is justified.

Report the coefficient selected, its estimate and confidence interval where available, the number of items, and the evidence supporting the scale’s structure. Avoid treating 0.70 as a universal pass mark; acceptable reliability depends on the purpose, consequences and context of measurement.

When Should You Use EFA or CFA?

EFA and CFA are statistical techniques which analyse the relationships among the variables and factors, but there is a difference in their purposes.

Method

Appropriate purpose

What to report

EFA

Exploring the possible factor structure when it is uncertain

Extraction method, rotation, factor-retention rationale, loadings and cross-loadings

CFA

Testing a theoretically specified factor structure

Estimator, model specification, factor loadings, fit indices and any justified model changes

EFA is useful when adapting an instrument or examining a structure that has not been established in the target context. Factor retention should consider theoretical meaning alongside evidence such as parallel analysis, rather than relying on the eigenvalue-greater-than-one rule alone.

CFA evaluates whether a specified model is reasonably consistent with the observed data. Depending on the model and estimator, researchers commonly report the chi-square statistic, CFI, TLI, RMSEA and SRMR [3]. Interpret these together, considering sample size, model complexity, estimator suitability and the nature of the data. No single fit index proves that a model is correct.

If a model is modified after examining the results, explain the theoretical rationale. Changes made solely to improve fit may capitalise on chance and require confirmation in independent data.

When Is Test–Retest Reliability Appropriate?

Test–retest reliability assesses score stability across two or more measurement occasions. It is appropriate when the underlying characteristic is expected to remain reasonably stable during the interval.

The interval matters: a very short gap may inflate agreement through recall, while a long gap may allow genuine changes in the construct. Report the interval, sample, measurement instrument and statistic used.

For continuous scores, an intraclass correlation coefficient (ICC) may be appropriate. Specify the ICC model, whether raters or occasions are treated as fixed or random, whether the target is consistency or absolute agreement [4], and whether the estimate applies to a single measurement or an average. These choices change the meaning of the result.

For categorical or ordinal data, the choice between weighted or unweighted kappa may be made based on the nature of the categories and whether there are varying levels of disagreement among them. The use of kappa is also influenced by category prevalence and marginal totals.

How Should You Assess Construct Validity?

Construct validity refers to whether evidence exists that supports the interpretation of scores as measuring what was intended. Construct validity rests on a cumulative body of evidence, not on a single statistic.

  • Content validity: Do the items represent the construct appropriately? Theory, literature review, expert evaluation, and target population input can be used as evidence.
  • Structural validity: Does the structure of the items support the dimensional structure? Evidence might come from EFA or CFA.
  • Concurrent validity: Do measures which should have a relationship to related constructs exhibit an appropriate relationship?
  • Discriminant validity: Are constructs which should be distinguishable theoretically also empirically distinguishable?
  • Criterion-related validity: Is there an appropriate relationship between scores and an external criterion?

In confirmatory factor or structural equation models, average variance extracted (AVE) and the heterotrait–monotrait ratio (HTMT) may contribute to convergent and discriminant validity assessments [5]. They should not be used as automatic pass/fail rules: interpretation depends on the model, constructs, uncertainty and theoretical expectations. A high HTMT value may indicate that two constructs are not sufficiently distinct, but it should prompt investigation rather than an unsupported conclusion.

Which Tests Should You Select and Report?

Use the following matrix to match the research question to the evidence required.

Research question

Suitable assessment

Essential reporting detail

Are multi-item scores internally consistent?

Alpha or omega

Estimate, confidence interval where available, item count and scale structure

Does the proposed factor structure fit?

EFA or CFA, as appropriate

Estimator, factor-retention or model rationale, loadings and relevant fit evidence

Are scores stable over time?

Test–retest ICC or another suitable agreement statistic

Measurement interval and statistic specification

Do raters agree?

ICC for appropriate continuous ratings; kappa for categorical ratings

Rater design, agreement definition and measurement scale

Is the construct interpretation supported?

Multiple sources of validity evidence

Theoretical rationale and relevant empirical findings

Choose the analysis that answers the measurement question, rather than running every test by default.

Worked Example: A Customer-Satisfaction Questionnaire

Suppose a PhD researcher measures customer satisfaction using four questionnaire items. Before calculating a reliability coefficient, the researcher should establish whether the four items are intended to measure one construct and whether that structure is defensible.

Assume an appropriate analysis produces an omega total estimate of 0.84, with a 95% confidence interval of 0.78–0.89. This suggests reasonably consistent scores under the specified model but does not establish that the scale is unidimensional or valid.

Factor structure analysis can be done either using EFA or CFA based on whether it is exploratory or confirmatory in nature. If there is support for the suggested factor structure based on the CFA test, provide evidence in terms of loadings and model fit. Assess whether satisfaction is related to constructs that it is expected to relate to without blurring its boundaries with perceived service quality construct.

If the HTMT values are 0.93 between satisfaction and service quality, then assess whether they have overlapping nature with respect to their definitions, items, factor relationships and the uncertainty involved.

The defensible conclusion combines reliability estimates, structural evidence and theoretically relevant validity findings. Each result supports a specific claim; none substitutes for the others.

What Should You Include in the Thesis?

A clear measurement section should document:

  • The instrument’s source, version, items, response options and scoring procedure.
  • The sample and any missing-data handling relevant to the analysis.
  • The reason for selecting each reliability or validity method.
  • The statistic, estimate, confidence interval where available and relevant assumptions.
  • Factor loadings, model-fit evidence or agreement specifications where applicable.
  • Any items removed, model changes made and the theoretical justification for those decisions.
  • The limitations of the evidence and the implications for interpreting study findings.

Report the results even when they do not meet expected thresholds. Explain what the evidence supports, what remains uncertain and how that affects the conclusions.

Need Support with Reliability and Validity Analysis?

PhD Assistance offers Data Analysis Services to help researchers select appropriate reliability and validity tests, interpret Cronbach’s alpha, McDonald’s omega and factor-analysis results, and assess measurement evidence.

Get tailored statistical support for your research objectives, questionnaire data and measurement model, with clear interpretations and thesis-ready reporting for your PhD or dissertation.

Conclusion

Reliability and validity testing should follow the measurement question. Alpha or omega may support internal-consistency claims; EFA and CFA address factor structure; ICC and kappa assess different forms of stability or agreement; and construct validity requires multiple complementary sources of evidence. A strong PhD thesis explains why each method was selected, reports its results transparently and avoids claiming more than the evidence establishes.

Frequently Asked Question

What is the difference between reliability and validity?

Reliability concerns the consistency of scores under specified conditions. Validity concerns the evidence supporting the interpretation and use of those scores for the intended purpose.

Is Cronbach’s alpha above 0.70 sufficient?

Not by itself. Interpretation depends on the research context, scale structure, assumptions and intended use. A coefficient above 0.70 does not prove one-dimensionality or validity.

Should I use Cronbach’s alpha or McDonald’s omega?

The choice depends on the scale’s measurement model and assumptions. Omega may be preferable when item loadings differ, provided the fitted model is appropriate.

Do I need both EFA and CFA?

Not always. EFA explores an uncertain structure, while CFA tests a specified structure. Using both may be justified during instrument development, ideally with independent data for confirmation.

Can one statistical test establish construct validity?

No. Construct validity requires a coherent argument supported by relevant evidence, such as content representation, factor structure, relationships with other constructs and external criteria.

Reference

  1. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334. https://doi.org/10.1007/BF02310555
  2. McDonald RP. Test Theory: A Unified Treatment. Lawrence Erlbaum Associates; 1999.
  3. Brown TA. Confirmatory Factor Analysis for Applied Research. 2nd ed. Guilford Press; 2015.
  4. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016;15(2):155–163. https://doi.org/10.1016/j.jcm.2016.02.012
  5. Henseler J, Ringle CM, Sarstedt M. A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science. 2015;43:115–135. https://doi.org/10.1007/s11747-014-0403-8
Your next step

Want help applying this to your research?

  1. 1Tell us your topic and stage
  2. 2We match a subject mentor
  3. 3You get a plan and a quote
All insights
Reliability and Validity in a PhD Thesis: Which Tests to Report and WhenPhD Research Lab Insights