Critical reviewMethodology & analyticsEducation

Methodological rigour in experimental second-language research: a critical review of Jahanbakhsh et al. (2025)

A synthesis of 791 intervention studies shows that reporting in second-language research has improved, but power analysis, data sharing and informative graphics still lag behind. This review explains the findings, places them in a decade of reform, and turns them into a design checklist for doctoral researchers.

What you will learn

  • Summarise the design and main findings of Jahanbakhsh et al. (2025)
  • Explain how methodological syntheses assess study quality and what their limits are
  • Place the findings in the history of reform in quantitative second-language research
  • Calculate and justify a sample size for an intervention study using field-specific effect sizes
  • Choose what to report, share and plot so that an experimental study meets current expectations

What you will unlearn

  • Twenty learners per group is enough for a classroom experiment.For a typical between-group effect of d = 0.40, twenty per group gives power of only about 0.23.
  • Cohen's small, medium and large benchmarks apply to language learning studies.Field-specific benchmarks for second-language research are larger than Cohen's general values.
  • A significant p value shows that the intervention worked well.Effect sizes with confidence intervals, and their practical meaning, are needed to judge how well.
  • A methodological synthesis is just a descriptive report.It is a systematic, empirical study of published research, with its own sampling, coding and reliability checks.

Skills you will practise

  • Power analysis
  • Effect size interpretation
  • Checking statistical assumptions
  • Reporting standards
  • Open data and materials
  • Data visualisation
  • Appraising methodological syntheses

The paper under review

Jahanbakhsh et al. (2025) report a methodological synthesis in Research Methods in Applied Linguistics, published online in October 2025. Their starting point is a familiar concern: research increasingly stresses robust methods, yet doubts remain about the quality of quantitative studies in second-language (L2) research. They set out to assess how far published intervention studies follow established methodological good practice.

The corpus comprises 791 interventionist quantitative articles published over 12 years in eight journals chosen for high impact and relevance to the main areas of applied linguistics. Interventionist studies are those that manipulate instruction or another treatment and measure its effect, so they carry the field's causal claims about teaching and learning.

Design and main findings

The authors developed a detailed coding scheme covering five areas: sampling and design, the types of statistical analysis used, the statistical assumptions that should be checked, reporting practices, and the visual presentation of data. Each article was coded against it, and the results were examined across journals and over time.

Summary of findings from the abstract and highlights of Jahanbakhsh et al. (2025)
AreaDirection reported by the authors
Design-related issuesImproved
Reporting practicesImproved, with marked gains between 2015 and 2018
Sampling, especially power analysisStill neglected, no significant improvement over time
Data sharingStill rare, no significant improvement over time
Data-rich, accountable visualsStill under-used, no significant improvement over time

The authors conclude that progress in methodological rigour is gradual. They present the results as a basis for guidelines, better training and a more critical reading of quantitative findings in applied linguistics.

Two features of the findings deserve emphasis. First, the areas that improved are largely those that journals can enforce at submission, such as reporting descriptive statistics and effect sizes. Second, the areas that stalled require effort before data are collected (power analysis), after publication (sharing data) or new skills (data-rich graphics). That pattern suggests that guidelines alone change what is reported more readily than how studies are planned, which is why doctoral training in design matters. This is an interpretation of the pattern, not a finding the authors tested.

A decade of reform in context

The paper continues a well-established line of research. Plonsky assessed designs, analyses and reporting in quantitative L2 research (Plonsky, 2013); his synthesis of 606 studies from two leading journals between 1990 and 2010 found increases in sample sizes, delayed post-testing and the reporting of effect sizes, reliability estimates and standard deviations, alongside continued heavy reliance on null hypothesis significance testing (Plonsky, 2014). Al-Hoorie and Vitta (2019) reviewed 150 articles across 30 L2 journals and found incomplete reporting of reliability, validity, non-significant results, effect sizes and assumption checks.

In 2015 Language Learning published guidelines for reporting quantitative methods and results (Norris et al., 2015) together with an analysis of what gets reported and recommendations for the field (Larson-Hall & Plonsky, 2015). Psychology's reporting standards were revised soon after (Appelbaum et al., 2018). The improvement in reporting that Jahanbakhsh et al. (2025) locate between 2015 and 2018 coincides with this period, although a synthesis cannot show that the guidelines caused it. Gass et al. (2021) review the same trajectory and describe a field that has matured statistically but still has work to do.

Replication remains thin. Marsden et al. (2018) found 67 self-labelled replication studies across 26 journals, a mean of about one replication for every 400 articles, and no direct replications. That matters for the findings on data sharing: without shared data and materials, close replication and re-analysis are difficult.

Strengths and limitations

Strengths

  • Scale and scope. Nearly 800 studies across eight journals and twelve years give a broad and recent picture.
  • Focus on interventions. Concentrating on experimental and quasi-experimental studies targets the work whose causal claims most influence teaching practice.
  • Coverage of neglected areas. Coding assumptions, data sharing and visual presentation, not only effect sizes and p values, reflects current open-science concerns.
  • Change over time. Tracking trends allows the authors to show where reform has and has not taken hold.

Limitations

  • Reporting is not conduct. A synthesis codes what articles report. A study that checked assumptions but did not say so is coded as not checking them, so the results describe transparency as much as practice.
  • Journal selection. Eight high-impact journals may not represent the wider field, including regional and practitioner journals and doctoral theses, where practice may differ.
  • Description rather than explanation. The study shows where practice lags but cannot say why: training, reviewer expectations, journal policy and access to participants are all plausible reasons that need separate investigation.
  • Quality as a composite. Treating many separate features as markers of rigour risks implying that every item matters equally, whereas an unpowered design is a more serious threat than a missing graph.

The three persistent weak spots

Power analysis

Power is the probability that a study detects an effect of a given size if it exists. Underpowered studies do not only miss real effects; when they do reach significance, the estimated effect tends to be inflated, because only the larger chance estimates cross the threshold. A literature built from many small studies can therefore overstate how well interventions work. A priori power analysis, based on a realistic effect size from previous research rather than a hoped-for one, is the remedy (Brysbaert, 2019). Where the sample is fixed by circumstances, a sensitivity analysis, stating the smallest effect the design could reliably detect, is the honest alternative.

Data sharing

Shared data allow errors to be found, analyses to be checked and studies to be combined in meta-analyses. Shared instruments and materials make replication possible (Marsden et al., 2018). In L2 research, the IRIS repository hosts instruments and materials, and general repositories such as the Open Science Framework can hold anonymised data and analysis code. Plan sharing at the ethics stage: participants must consent to it, and learner data such as recordings may need to be anonymised or shared only on request.

Informative visuals

Bar charts of group means hide the distribution of scores, the presence of outliers and the overlap between groups. Plots that show individual data points, distributions or paired change, with means and confidence intervals added, let readers judge the size and consistency of an effect for themselves. The synthesis found that such data-rich graphics remain uncommon (Jahanbakhsh et al., 2025), so using them is an easy way for a doctoral study to stand out.

Reading a methodological synthesis

When you cite a synthesis like this one in a literature review, check how the journals and years were chosen, how the coding scheme was developed, whether inter-coder agreement was reported, and whether findings are presented as proportions of studies or as trends tested over time. These details determine how far the conclusions generalise beyond the sampled journals. Report the synthesis's scope accurately when you cite it, rather than treating it as a verdict on the whole field.

What it means for your study

The findings translate directly into design decisions. The biggest single risk for a doctoral intervention study is low statistical power. Plonsky and Oswald (2014) analysed effect sizes from primary studies and meta-analyses in L2 research and found that Cohen's general benchmarks underestimate the effects typically observed; they proposed field-specific values of roughly d = 0.40, 0.70 and 1.00 for small, medium and large between-group differences. Brysbaert (2019) shows how quickly required sample sizes grow for effects of this size.

A worked power analysis (illustrative)

A doctoral researcher plans to compare a new vocabulary-teaching method with normal teaching in two groups, measuring learning on a post-test. The figures below are illustrative calculations, not results.

  1. Choose a plausible effect. Prior studies and the field benchmarks suggest a small between-group effect, d = 0.40, is realistic (Plonsky & Oswald, 2014).
  2. Calculate the sample. With a two-sided alpha of .05 and power of .80, an independent-samples comparison needs about 100 learners per group.
  3. Check the alternative. With 20 learners per group, the power to detect d = 0.40 is only about 0.23, so a non-significant result would be uninformative.
  4. Use the pre-test. Analysing the post-test with the pre-test as a covariate reduces the required sample. If pre-test and post-test correlate at about .50, the requirement falls to roughly 75 per group.
  5. Account for clustering. If whole classes rather than individual learners are assigned to conditions, the design effect of clustering must be added, which increases the sample again; multilevel or mixed-effects models are then needed for the analysis.
  6. Pre-register and report. State the effect size, its source, alpha, power and the resulting sample in the methods, and report the effect size with a confidence interval in the results (Norris et al., 2015).

What to report and common mistakes

Reporting checklist for an L2 intervention study
AreaWhat to reportSource
Sampling and powerA priori power analysis: effect size and its basis, alpha, power, target and achieved sampleNorris et al. (2015); JARS-Quant
DesignAssignment method, comparison group, pre-test and delayed post-test, attritionJARS-Quant
InstrumentsReliability for this sample and validity evidenceLarson-Hall and Plonsky (2015)
AssumptionsWhich were checked, how, and what was done when they failedAl-Hoorie and Vitta (2019)
ResultsDescriptive statistics, effect sizes with confidence intervals, exact p valuesNorris et al. (2015)
VisualsPlots showing individual data points or distributions, not only bar charts of meansJahanbakhsh et al. (2025)
Open materialsWhere data, instruments and analysis code can be obtainedMarsden et al. (2018)

Common mistakes

  • Choosing the sample by convenience and reporting no power analysis.
  • Interpreting a non-significant result from a small sample as evidence of no effect.
  • Using Cohen's general benchmarks to call an L2 effect "large".
  • Running many separate tests without addressing multiple comparisons (Al-Hoorie & Vitta, 2019).
  • Ignoring classroom clustering when whole classes receive the treatment.
  • Showing only bar charts of means, which hide spread and outliers.
  • Keeping data and instruments private when ethics approval would allow sharing.

Key takeaways

Your next step

Want to apply this to your own study?

  1. 1Tell us your topic, data and deadline
  2. 2A statistician reviews your work with you
  3. 3You get a plan and a quote
Quantitative analysis

References

  1. Al-Hoorie, A. H., & Vitta, J. P. (2019). The seven sins of L2 research: A review of 30 journals' statistical quality and their CiteScore, SJR, SNIP, JCR impact factors. Language Teaching Research, 23(6), 727–744. https://doi.org/10.1177/1362168818767191
  2. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3–25. https://doi.org/10.1037/amp0000191
  3. Brysbaert, M. (2019). How many participants do we have to include in properly powered experiments? A tutorial of power analysis with reference tables. Journal of Cognition, 2(1), 16. https://doi.org/10.5334/joc.72
  4. Gass, S., Loewen, S., & Plonsky, L. (2021). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching, 54(2), 245–258. https://doi.org/10.1017/S0261444819000430
  5. Jahanbakhsh, A. A., Banitalebi, Z., Larson-Hall, J., & Shiiba, A. (2025). Methodological rigor in quantitative L2 research: A focus on interventionist experimental studies. Research Methods in Applied Linguistics, 4(3), 100273. https://doi.org/10.1016/j.rmal.2025.100273
  6. Larson-Hall, J., & Plonsky, L. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(S1), 127–159. https://doi.org/10.1111/lang.12115
  7. Marsden, E., Morgan-Short, K., Thompson, S., & Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning, 68(2), 321–391. https://doi.org/10.1111/lang.12286
  8. Norris, J. M., Plonsky, L., Ross, S. J., & Schoonen, R. (2015). Guidelines for reporting quantitative methods and results in primary research. Language Learning, 65(2), 470–476. https://doi.org/10.1111/lang.12104
  9. Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4), 655–687. https://doi.org/10.1017/S0272263113000399
  10. Plonsky, L. (2014). Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. The Modern Language Journal, 98(1), 450–470. https://doi.org/10.1111/j.1540-4781.2014.12058.x
  11. Plonsky, L., & Oswald, F. L. (2014). How big is "big"? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. https://doi.org/10.1111/lang.12079

How to cite this article

Choose your referencing style, then copy. Check your university's or journal's guide for local variations.

PhD Research Lab Academy. (2026, February 13). Methodological rigour in experimental second-language research: a critical review of Jahanbakhsh et al. (2025). PhD Research Lab. https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review/

PhD Research Lab Academy. “Methodological Rigour in Experimental Second-language Research: A Critical Review of Jahanbakhsh Et Al. (2025).” PhD Research Lab, 13 Feb. 2026, www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review. Accessed [access date].

PhD Research Lab Academy (2026) Methodological rigour in experimental second-language research: a critical review of Jahanbakhsh et al. (2025). PhD Research Lab. Available at: https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review/ (Accessed: [access date]).

PhD Research Lab Academy. “Methodological Rigour in Experimental Second-language Research: A Critical Review of Jahanbakhsh Et Al. (2025).” PhD Research Lab. February 13, 2026. https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review.

PhD Research Lab Academy, ‘Methodological Rigour in Experimental Second-language Research: A Critical Review of Jahanbakhsh Et Al. (2025)’ (PhD Research Lab, 13 February 2026) <https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review> accessed [access date].

PhD Research Lab Academy. Methodological rigour in experimental second-language research: a critical review of Jahanbakhsh et al. (2025) [Internet]. PhD Research Lab; 2026 Feb 13 [cited [access date]]. Available from: https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review/

PhD Research Lab Academy, “Methodological rigour in experimental second-language research: a critical review of Jahanbakhsh et al. (2025),” PhD Research Lab, Feb. 13, 2026. [Online]. Available: https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review. [Accessed: [access date]].

@online{methodological_rigour_quantitative_l2_critical_review_2026,
  author = {{PhD Research Lab Academy}},
  title = {{Methodological rigour in experimental second-language research: a critical review of Jahanbakhsh et al. (2025)}},
  organization = {PhD Research Lab},
  date = {2026-02-13},
  url = {https://www.phdassistance.com/academy/critical-review/methodological-rigour-quantitative-l2-critical-review},
  urldate = {[access date]}
}

This article is for learning. Use it to understand the method and cite the original sources above in your own work; do not copy or submit it as your own.

Methodological rigour in experimental second-language researchNeed help applying it to your study?
Quantitative analysis