The source and what kind of document it is
The source is Spina et al. (2024), "Report on the hands-on PhD course on responsible AI from the lens of an information access researcher", published in SIGIR Forum (volume 58, issue 2, December 2024) in its Event Reports section. SIGIR Forum is the newsletter-style publication of the ACM Special Interest Group on Information Retrieval. The report has 24 authors: the lead instructor, Damiano Spina of RMIT University, five colleagues at the University of Udine, and a further group who appear to be course participants whose group work is reported.
Being precise about the genre matters for everything that follows. This is a descriptive report of a teaching event, with a short post-course questionnaire, not a research article that tests a teaching intervention. It should be judged on how well it documents and shares a course design, and cited for that, rather than as evidence of learning outcomes. The course materials were shared openly, and a condensed half-day version was later given as a tutorial at the ACM CHIIR 2025 conference (Spina, 2025).

What the report describes
The course ran at the University of Udine in May 2024 over four days, with in-person and online participants. Its stated aims were an enhanced understanding of the terminology and methods used by multidisciplinary teams working on responsible AI, and the application of these ideas to information access and retrieval research. Rather than teaching principles in the abstract, each session used existing information retrieval research as a case study (Spina et al., 2024).
| Session | Focus | Linked group activity |
|---|---|---|
| 1. Responsible AI: a multidisciplinary problem | Positionality; how terms differ across disciplines | Activity 1: describe a research topic and write a positionality statement. Activity 2: cluster responsible AI keywords and identify terms that cause tension between disciplines |
| 2. Mixed methods for presenting fact-checked content | Mixed methods and participatory research, drawing on work with fact-checkers | Activity 3: discuss how participatory methods could benefit your own research |
| 3. Fairness and diversity: two sides of the same coin? | Offline evaluation, axiomatic analysis of measures, diversity-aware evaluation and fairness-aware ranking | Activity 4: examine fairness and diversity in the group case studies and how they would be evaluated |
| 4. Information processing and physiological signals | Ethical issues in lab studies using EEG, wristbands and eye tracking | Activity 5: ask why the research is needed, who benefits, what can go wrong and how risk is managed |
Of 23 participants, 20 took part in the activities and 14 completed the final questionnaire. Respondents were mostly early-stage PhD candidates, mainly in machine learning and mostly based in Italy. Half said they were not at all familiar with institutional ethics review, and most had not run a user study in the past five years. Sessions and activities were rated with magnitude estimation, a technique the Udine group has used for relevance judgements (Maddalena et al., 2017); on the normalised scale, sessions overall averaged about 0.76 and activities about 0.51, and eight respondents said the course exceeded their expectations. The authors acknowledge that time for activities was short and that not everyone completed the survey (Spina et al., 2024).
Strengths
- Ethics embedded in the discipline. Responsible AI is taught through the measures, test collections and user studies that information retrieval researchers actually use, including fairness in ranking (Ekstrand et al., 2022). An analysis of 115 tech ethics syllabi recommended integrating ethics across the computing curriculum rather than isolating it (Fiesler et al., 2020); this course is a worked instance.
- From principles to practice. AI ethics guidelines converge on a small set of principles such as transparency, fairness and responsibility (Jobin et al., 2019), but principles alone do not tell researchers what to do (Mittelstadt, 2019). The activities ask participants to make those principles concrete in their own projects.
- Reflexivity made routine. Asking computing researchers to write positionality statements brings a practice from qualitative research, where it is used to make the researcher's influence visible (Holmes, 2020), into a field where it is still unusual.
- Openness. The report publishes the activity prompts, participants' outputs and survey results, and the materials are openly available, so others can adapt and evaluate the design.
Limitations and methodological issues
A useful way to see the limits of the evidence is the four-level model of training evaluation: reaction, learning, behaviour and results (Kirkpatrick & Kirkpatrick, 2006). The report provides evidence at the first level only. Participants' ratings and satisfaction are reported; there is no measure of what they learned, no follow-up of whether they changed how they design or evaluate research, and no comparison group.
- Small, partial sample. Fourteen respondents from 23 participants, with wide variation in ratings, cannot support general claims, and the views of the nine who did not respond are unknown.
- Insider authorship. The report is written by the instructors and, it appears, by participants whose work is showcased. That is in keeping with a participatory spirit, but it means no independent voice assesses the course.
- No baseline. Without a pre-course measure, the survey cannot show change; it shows how participants felt at the end.
- Homogeneous cohort. One institution, mostly early-stage machine learning students from one country, limits what can be said about other disciplines, career stages or cultural settings.
- Activities rated lower than sessions. The hands-on activities, the course's distinctive feature, received lower ratings than the lectures. The authors link this partly to limited time; it deserves further investigation before hands-on formats are promoted as superior.
A stronger evaluation of the same course would not need to be elaborate. Participants could write a short benefits and risks analysis of their own project before and after the course, scored by assessors who do not know which version they are reading. Their thesis proposals or ethics applications could be reviewed six months later for evidence of positionality, participatory elements or fairness-aware evaluation. And the reasons why activities were rated lower than sessions could be explored through brief interviews. Each step moves the evidence from reaction towards learning and behaviour, which is what educators deciding whether to adopt the design most need to know.
Ethically, the report attends to fairness, privacy, consent and misuse within the case studies, but it says little about the ethics of the course itself, for example how power differences between senior instructors and students shaped group discussion, or how participants consented to their outputs being published. These are points to address in future reports rather than flaws in the teaching.
How the report sits in the wider literature
Calls for AI ethics education have grown as AI systems have entered high-stakes settings (Borenstein & Howard, 2021). The course's design maps well onto the responsible innovation framework of Stilgoe et al. (2013), which asks researchers to anticipate impacts, reflect on their own assumptions, include wider voices and respond to what they learn. The report also cites the argument that responsible AI research should itself carry impact statements, because work intended to make AI safer can have unintended effects (Olteanu et al., 2023). Within information retrieval, earlier discussion of what the field should teach concentrated on core technical content (Markov & de Rijke, 2019); this course adds a responsible AI strand to that conversation.
| Responsible innovation dimension | Where it appears in the course | How to apply it in a thesis |
|---|---|---|
| Anticipation | Activity 5: what can go wrong and who benefits | Add a benefits and risks subsection to your methods or discussion chapter |
| Reflexivity | Activity 1: positionality statements | Write a short positionality note and revisit it as the project changes |
| Inclusion | Session 2 and activity 3: participatory research with fact-checkers | Identify the users or communities affected and how they can inform the design |
| Responsiveness | Activity 4: rethinking evaluation to include fairness and diversity | Report evaluation measures beyond effectiveness and explain changes made in response |
What it means for doctoral researchers
Cite the report as a documented, openly shared course design for teaching responsible AI within information access research, with favourable participant reactions. Do not cite it as evidence that hands-on courses improve ethical practice, because it was not designed to show that.
The report is also useful well beyond information retrieval. Any doctoral researcher whose work trains, evaluates or deploys an AI system faces the same questions: whose data and judgements the system relies on, who is represented in the evaluation, and who carries the cost when it fails. The participant profile is a reminder of how common the gap is: half of the respondents were unfamiliar with ethics review, and most had not run a study with human participants. If that describes you, the course's prompts are a low-cost way to start, and your institution's research ethics office is the place to confirm what approval your project needs.
Worked example (illustrative)
A hypothetical second-year PhD student is building a model that ranks patient questions for a hospital helpline. Following the course's prompts, she writes a positionality note: she trained in computer science, has no clinical background and has used only English-language data, which may lead her to overlook how questions from older or non-native speakers are phrased. She then answers the four activity 5 questions. The research is needed because response times are long; patients and nurses benefit; ranking could systematically delay some groups; risk is managed by auditing ranking quality across age and language groups and by keeping a nurse in the loop. Each answer becomes a design decision, an evaluation measure or a limitation stated in the thesis.
Further Academy reviews discuss research on ChatGPT in higher education and supervision of PhD students with AI.
Common mistakes when writing about responsible AI in a thesis
- Treating participant satisfaction with a course or tool as proof that it changed practice.
- Listing ethics principles without showing how each affected a design, data or evaluation decision.
- Reporting only effectiveness measures when the system affects different groups differently.
- Writing a positionality statement once and never revisiting it as the project changes.
- Leaving benefits and risks to a final paragraph instead of addressing them when choosing data and methods.
- Citing an event report or tutorial as if it were a controlled evaluation.
Key takeaways
- Spina et al. (2024) is an event report of a four-day PhD course, not an empirical test of learning outcomes.
- The course embeds positionality, participatory methods, fairness and ethics in real information retrieval cases (Ekstrand et al., 2022).
- Its evaluation reaches only the reaction level of training evaluation, from 14 of 23 participants (Kirkpatrick & Kirkpatrick, 2006).
- The design is a practical response to the gap between AI ethics principles and research practice (Jobin et al., 2019; Mittelstadt, 2019).
- Doctoral researchers can adopt its prompts on positionality, benefits and risks in their own theses (Stilgoe et al., 2013).
