Critical reviewResearch writingComputer science

Teaching responsible AI to doctoral researchers: a critical review of Spina et al. (2024)

A four-day PhD course at the University of Udine taught responsible AI through the everyday work of information access research. This review examines the published report of that course, separates what it shows from what it cannot, and draws out practices doctoral researchers can adopt in their own projects.

What you will learn

  • Identify exactly what kind of source Spina et al. (2024) is and what claims an event report can support
  • Summarise the sessions, activities and evaluation of the responsible AI course
  • Evaluate the course evaluation using a recognised framework for training outcomes
  • Relate the course design to research on AI ethics principles, responsible innovation and ethics education
  • Apply positionality and benefit-risk questions to your own AI-related doctoral project

What you will unlearn

  • A published course report is evidence that the course works.An event report describes what was done and how participants reacted; it does not measure learning or later behaviour.
  • Responsible AI is mainly a matter of following published ethics principles.Principles are widely shared but abstract, so researchers need practice in applying them to concrete design and evaluation choices.
  • Positionality statements belong only in qualitative social research.Reflecting on how background and discipline shape a project is useful in computing research too, as this course shows.
  • Ethics is a separate box to tick after the technical work is designed.Questions of who benefits and what can go wrong shape data, evaluation and deployment decisions from the start.

Skills you will practise

  • Critical appraisal of event reports
  • Responsible AI
  • Positionality and reflexivity
  • Fairness-aware evaluation
  • Benefit and risk analysis
  • Evaluating training outcomes

The source and what kind of document it is

The source is Spina et al. (2024), "Report on the hands-on PhD course on responsible AI from the lens of an information access researcher", published in SIGIR Forum (volume 58, issue 2, December 2024) in its Event Reports section. SIGIR Forum is the newsletter-style publication of the ACM Special Interest Group on Information Retrieval. The report has 24 authors: the lead instructor, Damiano Spina of RMIT University, five colleagues at the University of Udine, and a further group who appear to be course participants whose group work is reported.

Being precise about the genre matters for everything that follows. This is a descriptive report of a teaching event, with a short post-course questionnaire, not a research article that tests a teaching intervention. It should be judged on how well it documents and shares a course design, and cited for that, rather than as evidence of learning outcomes. The course materials were shared openly, and a condensed half-day version was later given as a tutorial at the ACM CHIIR 2025 conference (Spina, 2025).

Report on the hands-on PhD course on responsible AI
Figure 1. Spina et al. (2024).

What the report describes

The course ran at the University of Udine in May 2024 over four days, with in-person and online participants. Its stated aims were an enhanced understanding of the terminology and methods used by multidisciplinary teams working on responsible AI, and the application of these ideas to information access and retrieval research. Rather than teaching principles in the abstract, each session used existing information retrieval research as a case study (Spina et al., 2024).

Structure of the course reported by Spina et al. (2024), summarised
SessionFocusLinked group activity
1. Responsible AI: a multidisciplinary problemPositionality; how terms differ across disciplinesActivity 1: describe a research topic and write a positionality statement. Activity 2: cluster responsible AI keywords and identify terms that cause tension between disciplines
2. Mixed methods for presenting fact-checked contentMixed methods and participatory research, drawing on work with fact-checkersActivity 3: discuss how participatory methods could benefit your own research
3. Fairness and diversity: two sides of the same coin?Offline evaluation, axiomatic analysis of measures, diversity-aware evaluation and fairness-aware rankingActivity 4: examine fairness and diversity in the group case studies and how they would be evaluated
4. Information processing and physiological signalsEthical issues in lab studies using EEG, wristbands and eye trackingActivity 5: ask why the research is needed, who benefits, what can go wrong and how risk is managed

Of 23 participants, 20 took part in the activities and 14 completed the final questionnaire. Respondents were mostly early-stage PhD candidates, mainly in machine learning and mostly based in Italy. Half said they were not at all familiar with institutional ethics review, and most had not run a user study in the past five years. Sessions and activities were rated with magnitude estimation, a technique the Udine group has used for relevance judgements (Maddalena et al., 2017); on the normalised scale, sessions overall averaged about 0.76 and activities about 0.51, and eight respondents said the course exceeded their expectations. The authors acknowledge that time for activities was short and that not everyone completed the survey (Spina et al., 2024).

Strengths

  • Ethics embedded in the discipline. Responsible AI is taught through the measures, test collections and user studies that information retrieval researchers actually use, including fairness in ranking (Ekstrand et al., 2022). An analysis of 115 tech ethics syllabi recommended integrating ethics across the computing curriculum rather than isolating it (Fiesler et al., 2020); this course is a worked instance.
  • From principles to practice. AI ethics guidelines converge on a small set of principles such as transparency, fairness and responsibility (Jobin et al., 2019), but principles alone do not tell researchers what to do (Mittelstadt, 2019). The activities ask participants to make those principles concrete in their own projects.
  • Reflexivity made routine. Asking computing researchers to write positionality statements brings a practice from qualitative research, where it is used to make the researcher's influence visible (Holmes, 2020), into a field where it is still unusual.
  • Openness. The report publishes the activity prompts, participants' outputs and survey results, and the materials are openly available, so others can adapt and evaluate the design.

Limitations and methodological issues

A useful way to see the limits of the evidence is the four-level model of training evaluation: reaction, learning, behaviour and results (Kirkpatrick & Kirkpatrick, 2006). The report provides evidence at the first level only. Participants' ratings and satisfaction are reported; there is no measure of what they learned, no follow-up of whether they changed how they design or evaluate research, and no comparison group.

  • Small, partial sample. Fourteen respondents from 23 participants, with wide variation in ratings, cannot support general claims, and the views of the nine who did not respond are unknown.
  • Insider authorship. The report is written by the instructors and, it appears, by participants whose work is showcased. That is in keeping with a participatory spirit, but it means no independent voice assesses the course.
  • No baseline. Without a pre-course measure, the survey cannot show change; it shows how participants felt at the end.
  • Homogeneous cohort. One institution, mostly early-stage machine learning students from one country, limits what can be said about other disciplines, career stages or cultural settings.
  • Activities rated lower than sessions. The hands-on activities, the course's distinctive feature, received lower ratings than the lectures. The authors link this partly to limited time; it deserves further investigation before hands-on formats are promoted as superior.

A stronger evaluation of the same course would not need to be elaborate. Participants could write a short benefits and risks analysis of their own project before and after the course, scored by assessors who do not know which version they are reading. Their thesis proposals or ethics applications could be reviewed six months later for evidence of positionality, participatory elements or fairness-aware evaluation. And the reasons why activities were rated lower than sessions could be explored through brief interviews. Each step moves the evidence from reaction towards learning and behaviour, which is what educators deciding whether to adopt the design most need to know.

Ethically, the report attends to fairness, privacy, consent and misuse within the case studies, but it says little about the ethics of the course itself, for example how power differences between senior instructors and students shaped group discussion, or how participants consented to their outputs being published. These are points to address in future reports rather than flaws in the teaching.

How the report sits in the wider literature

Calls for AI ethics education have grown as AI systems have entered high-stakes settings (Borenstein & Howard, 2021). The course's design maps well onto the responsible innovation framework of Stilgoe et al. (2013), which asks researchers to anticipate impacts, reflect on their own assumptions, include wider voices and respond to what they learn. The report also cites the argument that responsible AI research should itself carry impact statements, because work intended to make AI safer can have unintended effects (Olteanu et al., 2023). Within information retrieval, earlier discussion of what the field should teach concentrated on core technical content (Markov & de Rijke, 2019); this course adds a responsible AI strand to that conversation.

Reading the course through the responsible innovation framework of Stilgoe et al. (2013)
Responsible innovation dimensionWhere it appears in the courseHow to apply it in a thesis
AnticipationActivity 5: what can go wrong and who benefitsAdd a benefits and risks subsection to your methods or discussion chapter
ReflexivityActivity 1: positionality statementsWrite a short positionality note and revisit it as the project changes
InclusionSession 2 and activity 3: participatory research with fact-checkersIdentify the users or communities affected and how they can inform the design
ResponsivenessActivity 4: rethinking evaluation to include fairness and diversityReport evaluation measures beyond effectiveness and explain changes made in response

What it means for doctoral researchers

Cite the report as a documented, openly shared course design for teaching responsible AI within information access research, with favourable participant reactions. Do not cite it as evidence that hands-on courses improve ethical practice, because it was not designed to show that.

The report is also useful well beyond information retrieval. Any doctoral researcher whose work trains, evaluates or deploys an AI system faces the same questions: whose data and judgements the system relies on, who is represented in the evaluation, and who carries the cost when it fails. The participant profile is a reminder of how common the gap is: half of the respondents were unfamiliar with ethics review, and most had not run a study with human participants. If that describes you, the course's prompts are a low-cost way to start, and your institution's research ethics office is the place to confirm what approval your project needs.

Worked example (illustrative)

A hypothetical second-year PhD student is building a model that ranks patient questions for a hospital helpline. Following the course's prompts, she writes a positionality note: she trained in computer science, has no clinical background and has used only English-language data, which may lead her to overlook how questions from older or non-native speakers are phrased. She then answers the four activity 5 questions. The research is needed because response times are long; patients and nurses benefit; ranking could systematically delay some groups; risk is managed by auditing ranking quality across age and language groups and by keeping a nurse in the loop. Each answer becomes a design decision, an evaluation measure or a limitation stated in the thesis.

Further Academy reviews discuss research on ChatGPT in higher education and supervision of PhD students with AI.

Common mistakes when writing about responsible AI in a thesis

  • Treating participant satisfaction with a course or tool as proof that it changed practice.
  • Listing ethics principles without showing how each affected a design, data or evaluation decision.
  • Reporting only effectiveness measures when the system affects different groups differently.
  • Writing a positionality statement once and never revisiting it as the project changes.
  • Leaving benefits and risks to a final paragraph instead of addressing them when choosing data and methods.
  • Citing an event report or tutorial as if it were a controlled evaluation.

Key takeaways

  • Spina et al. (2024) is an event report of a four-day PhD course, not an empirical test of learning outcomes.
  • The course embeds positionality, participatory methods, fairness and ethics in real information retrieval cases (Ekstrand et al., 2022).
  • Its evaluation reaches only the reaction level of training evaluation, from 14 of 23 participants (Kirkpatrick & Kirkpatrick, 2006).
  • The design is a practical response to the gap between AI ethics principles and research practice (Jobin et al., 2019; Mittelstadt, 2019).
  • Doctoral researchers can adopt its prompts on positionality, benefits and risks in their own theses (Stilgoe et al., 2013).
Your next step

Want to apply this to your own study?

  1. 1Tell us your topic, data and deadline
  2. 2A subject mentor reviews your work with you
  3. 3You get a plan and a quote
PhD consultation

References

  1. Borenstein, J., & Howard, A. (2021). Emerging challenges in AI and the need for AI ethics education. AI and Ethics, 1(1), 61–65. https://doi.org/10.1007/s43681-020-00002-7
  2. Ekstrand, M. D., Das, A., Burke, R., & Diaz, F. (2022). Fairness in information access systems. Foundations and Trends in Information Retrieval, 16(1–2), 1–177. https://doi.org/10.1561/1500000079
  3. Fiesler, C., Garrett, N., & Beard, N. (2020). What do we teach when we teach tech ethics? A syllabi analysis. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 289–295). Association for Computing Machinery. https://doi.org/10.1145/3328778.3366825
  4. Holmes, A. G. D. (2020). Researcher positionality: A consideration of its influence and place in qualitative research. A new researcher guide. Shanlax International Journal of Education, 8(4), 1–10. https://doi.org/10.34293/education.v8i4.3232
  5. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), 389–399. https://doi.org/10.1038/s42256-019-0088-2
  6. Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler. https://books.google.com/books/about/Evaluating_Training_Programs.html?id=BJ4QCmvP5rcC
  7. Maddalena, E., Mizzaro, S., Scholer, F., & Turpin, A. (2017). On crowdsourcing relevance magnitudes for information retrieval evaluation. ACM Transactions on Information Systems, 35(3), Article 19. https://doi.org/10.1145/3002172
  8. Markov, I., & de Rijke, M. (2019). What should we teach in information retrieval? SIGIR Forum, 52(2), 19–39. https://doi.org/10.1145/3308774.3308780
  9. Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501–507. https://doi.org/10.1038/s42256-019-0114-4
  10. Olteanu, A., Ekstrand, M., Castillo, C., & Suh, J. (2023). Responsible AI research needs impact statements too (arXiv:2311.11776). arXiv. https://arxiv.org/abs/2311.11776
  11. Spina, D. (2025). Responsible AI from the lens of an information retrieval researcher: A hands-on tutorial. In Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval. Association for Computing Machinery. https://doi.org/10.1145/3698204.3716479
  12. Spina, D., Roitero, K., Mizzaro, S., Della Mea, V., Da Ros, F., Soprano, M., Akebli, H., Falcon, A., Fasihi, M., Fiorin, A., La Barbera, D., Lizzio Bosco, D., Lunardi, R., Marturano, A., Muhammad, Z.-U.-D., Nascimben, F., Nottebaum, M., Pascoli, M., Popescu, M. H., ... Tremamunno, A. (2024). Report on the hands-on PhD course on responsible AI from the lens of an information access researcher. SIGIR Forum, 58(2), 1–29. https://doi.org/10.1145/3722449.3722456
  13. Stilgoe, J., Owen, R., & Macnaghten, P. (2013). Developing a framework for responsible innovation. Research Policy, 42(9), 1568–1580. https://doi.org/10.1016/j.respol.2013.05.008

How to cite this article

Choose your referencing style, then copy. Check your university's or journal's guide for local variations.

PhD Research Lab Academy. (2026, February 11). Teaching responsible AI to doctoral researchers: a critical review of Spina et al. (2024). PhD Research Lab. https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review/

PhD Research Lab Academy. “Teaching Responsible AI to Doctoral Researchers: A Critical Review of Spina Et Al. (2024).” PhD Research Lab, 11 Feb. 2026, www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review. Accessed [access date].

PhD Research Lab Academy (2026) Teaching responsible AI to doctoral researchers: a critical review of Spina et al. (2024). PhD Research Lab. Available at: https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review/ (Accessed: [access date]).

PhD Research Lab Academy. “Teaching Responsible AI to Doctoral Researchers: A Critical Review of Spina Et Al. (2024).” PhD Research Lab. February 11, 2026. https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review.

PhD Research Lab Academy, ‘Teaching Responsible AI to Doctoral Researchers: A Critical Review of Spina Et Al. (2024)’ (PhD Research Lab, 11 February 2026) <https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review> accessed [access date].

PhD Research Lab Academy. Teaching responsible AI to doctoral researchers: a critical review of Spina et al. (2024) [Internet]. PhD Research Lab; 2026 Feb 11 [cited [access date]]. Available from: https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review/

PhD Research Lab Academy, “Teaching responsible AI to doctoral researchers: a critical review of Spina et al. (2024),” PhD Research Lab, Feb. 11, 2026. [Online]. Available: https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review. [Accessed: [access date]].

@online{responsible_ai_phd_course_critical_review_2026,
  author = {{PhD Research Lab Academy}},
  title = {{Teaching responsible AI to doctoral researchers: a critical review of Spina et al. (2024)}},
  organization = {PhD Research Lab},
  date = {2026-02-11},
  url = {https://www.phdassistance.com/academy/critical-review/responsible-ai-phd-course-critical-review},
  urldate = {[access date]}
}

This article is for learning. Use it to understand the method and cite the original sources above in your own work; do not copy or submit it as your own.

Teaching responsible AI to doctoral researchersNeed help applying it to your study?
PhD consultation