The paper under review
Jahanbakhsh et al. (2025) report a methodological synthesis in Research Methods in Applied Linguistics, published online in October 2025. Their starting point is a familiar concern: research increasingly stresses robust methods, yet doubts remain about the quality of quantitative studies in second-language (L2) research. They set out to assess how far published intervention studies follow established methodological good practice.
The corpus comprises 791 interventionist quantitative articles published over 12 years in eight journals chosen for high impact and relevance to the main areas of applied linguistics. Interventionist studies are those that manipulate instruction or another treatment and measure its effect, so they carry the field's causal claims about teaching and learning.
Design and main findings
The authors developed a detailed coding scheme covering five areas: sampling and design, the types of statistical analysis used, the statistical assumptions that should be checked, reporting practices, and the visual presentation of data. Each article was coded against it, and the results were examined across journals and over time.
| Area | Direction reported by the authors |
|---|---|
| Design-related issues | Improved |
| Reporting practices | Improved, with marked gains between 2015 and 2018 |
| Sampling, especially power analysis | Still neglected, no significant improvement over time |
| Data sharing | Still rare, no significant improvement over time |
| Data-rich, accountable visuals | Still under-used, no significant improvement over time |
The authors conclude that progress in methodological rigour is gradual. They present the results as a basis for guidelines, better training and a more critical reading of quantitative findings in applied linguistics.
Two features of the findings deserve emphasis. First, the areas that improved are largely those that journals can enforce at submission, such as reporting descriptive statistics and effect sizes. Second, the areas that stalled require effort before data are collected (power analysis), after publication (sharing data) or new skills (data-rich graphics). That pattern suggests that guidelines alone change what is reported more readily than how studies are planned, which is why doctoral training in design matters. This is an interpretation of the pattern, not a finding the authors tested.
A decade of reform in context
The paper continues a well-established line of research. Plonsky assessed designs, analyses and reporting in quantitative L2 research (Plonsky, 2013); his synthesis of 606 studies from two leading journals between 1990 and 2010 found increases in sample sizes, delayed post-testing and the reporting of effect sizes, reliability estimates and standard deviations, alongside continued heavy reliance on null hypothesis significance testing (Plonsky, 2014). Al-Hoorie and Vitta (2019) reviewed 150 articles across 30 L2 journals and found incomplete reporting of reliability, validity, non-significant results, effect sizes and assumption checks.
In 2015 Language Learning published guidelines for reporting quantitative methods and results (Norris et al., 2015) together with an analysis of what gets reported and recommendations for the field (Larson-Hall & Plonsky, 2015). Psychology's reporting standards were revised soon after (Appelbaum et al., 2018). The improvement in reporting that Jahanbakhsh et al. (2025) locate between 2015 and 2018 coincides with this period, although a synthesis cannot show that the guidelines caused it. Gass et al. (2021) review the same trajectory and describe a field that has matured statistically but still has work to do.
Replication remains thin. Marsden et al. (2018) found 67 self-labelled replication studies across 26 journals, a mean of about one replication for every 400 articles, and no direct replications. That matters for the findings on data sharing: without shared data and materials, close replication and re-analysis are difficult.
Strengths and limitations
Strengths
- Scale and scope. Nearly 800 studies across eight journals and twelve years give a broad and recent picture.
- Focus on interventions. Concentrating on experimental and quasi-experimental studies targets the work whose causal claims most influence teaching practice.
- Coverage of neglected areas. Coding assumptions, data sharing and visual presentation, not only effect sizes and p values, reflects current open-science concerns.
- Change over time. Tracking trends allows the authors to show where reform has and has not taken hold.
Limitations
- Reporting is not conduct. A synthesis codes what articles report. A study that checked assumptions but did not say so is coded as not checking them, so the results describe transparency as much as practice.
- Journal selection. Eight high-impact journals may not represent the wider field, including regional and practitioner journals and doctoral theses, where practice may differ.
- Description rather than explanation. The study shows where practice lags but cannot say why: training, reviewer expectations, journal policy and access to participants are all plausible reasons that need separate investigation.
- Quality as a composite. Treating many separate features as markers of rigour risks implying that every item matters equally, whereas an unpowered design is a more serious threat than a missing graph.
The three persistent weak spots
Power analysis
Power is the probability that a study detects an effect of a given size if it exists. Underpowered studies do not only miss real effects; when they do reach significance, the estimated effect tends to be inflated, because only the larger chance estimates cross the threshold. A literature built from many small studies can therefore overstate how well interventions work. A priori power analysis, based on a realistic effect size from previous research rather than a hoped-for one, is the remedy (Brysbaert, 2019). Where the sample is fixed by circumstances, a sensitivity analysis, stating the smallest effect the design could reliably detect, is the honest alternative.
Data sharing
Shared data allow errors to be found, analyses to be checked and studies to be combined in meta-analyses. Shared instruments and materials make replication possible (Marsden et al., 2018). In L2 research, the IRIS repository hosts instruments and materials, and general repositories such as the Open Science Framework can hold anonymised data and analysis code. Plan sharing at the ethics stage: participants must consent to it, and learner data such as recordings may need to be anonymised or shared only on request.
Informative visuals
Bar charts of group means hide the distribution of scores, the presence of outliers and the overlap between groups. Plots that show individual data points, distributions or paired change, with means and confidence intervals added, let readers judge the size and consistency of an effect for themselves. The synthesis found that such data-rich graphics remain uncommon (Jahanbakhsh et al., 2025), so using them is an easy way for a doctoral study to stand out.
Reading a methodological synthesis
When you cite a synthesis like this one in a literature review, check how the journals and years were chosen, how the coding scheme was developed, whether inter-coder agreement was reported, and whether findings are presented as proportions of studies or as trends tested over time. These details determine how far the conclusions generalise beyond the sampled journals. Report the synthesis's scope accurately when you cite it, rather than treating it as a verdict on the whole field.
What it means for your study
The findings translate directly into design decisions. The biggest single risk for a doctoral intervention study is low statistical power. Plonsky and Oswald (2014) analysed effect sizes from primary studies and meta-analyses in L2 research and found that Cohen's general benchmarks underestimate the effects typically observed; they proposed field-specific values of roughly d = 0.40, 0.70 and 1.00 for small, medium and large between-group differences. Brysbaert (2019) shows how quickly required sample sizes grow for effects of this size.
A worked power analysis (illustrative)
A doctoral researcher plans to compare a new vocabulary-teaching method with normal teaching in two groups, measuring learning on a post-test. The figures below are illustrative calculations, not results.
- Choose a plausible effect. Prior studies and the field benchmarks suggest a small between-group effect, d = 0.40, is realistic (Plonsky & Oswald, 2014).
- Calculate the sample. With a two-sided alpha of .05 and power of .80, an independent-samples comparison needs about 100 learners per group.
- Check the alternative. With 20 learners per group, the power to detect d = 0.40 is only about 0.23, so a non-significant result would be uninformative.
- Use the pre-test. Analysing the post-test with the pre-test as a covariate reduces the required sample. If pre-test and post-test correlate at about .50, the requirement falls to roughly 75 per group.
- Account for clustering. If whole classes rather than individual learners are assigned to conditions, the design effect of clustering must be added, which increases the sample again; multilevel or mixed-effects models are then needed for the analysis.
- Pre-register and report. State the effect size, its source, alpha, power and the resulting sample in the methods, and report the effect size with a confidence interval in the results (Norris et al., 2015).
What to report and common mistakes
| Area | What to report | Source |
|---|---|---|
| Sampling and power | A priori power analysis: effect size and its basis, alpha, power, target and achieved sample | Norris et al. (2015); JARS-Quant |
| Design | Assignment method, comparison group, pre-test and delayed post-test, attrition | JARS-Quant |
| Instruments | Reliability for this sample and validity evidence | Larson-Hall and Plonsky (2015) |
| Assumptions | Which were checked, how, and what was done when they failed | Al-Hoorie and Vitta (2019) |
| Results | Descriptive statistics, effect sizes with confidence intervals, exact p values | Norris et al. (2015) |
| Visuals | Plots showing individual data points or distributions, not only bar charts of means | Jahanbakhsh et al. (2025) |
| Open materials | Where data, instruments and analysis code can be obtained | Marsden et al. (2018) |
Common mistakes
- Choosing the sample by convenience and reporting no power analysis.
- Interpreting a non-significant result from a small sample as evidence of no effect.
- Using Cohen's general benchmarks to call an L2 effect "large".
- Running many separate tests without addressing multiple comparisons (Al-Hoorie & Vitta, 2019).
- Ignoring classroom clustering when whole classes receive the treatment.
- Showing only bar charts of means, which hide spread and outliers.
- Keeping data and instruments private when ethics approval would allow sharing.
Key takeaways
- Jahanbakhsh et al. (2025) find better design and reporting in 791 L2 intervention studies but little progress on power analysis, data sharing and informative visuals.
- Their results continue a line of syntheses showing gradual reform since 1990 (Plonsky, 2014; Al-Hoorie & Vitta, 2019).
- Field-specific effect sizes imply far larger samples than many classroom studies use (Plonsky & Oswald, 2014; Brysbaert, 2019).
- Replication is rare in the field, which makes open data and materials more important (Marsden et al., 2018).
- Follow published reporting guidelines for every intervention study (Norris et al., 2015; Appelbaum et al., 2018).
