How-to articleResearch writingComputer science

How to identify a research problem in explainable artificial intelligence (XAI)

A step-by-step guide to finding a doctoral research problem in XAI: ten open problem areas, how to move from gap-spotting to a sharper problem, a worked example, and what a credible XAI study must report.

What you will learn

  • Describe the main families of XAI methods and the open problems surveys identify
  • Explain why evaluating explanations is the central unresolved problem in XAI
  • Move from a broad gap in the literature to a problem that challenges an assumption
  • Write an XAI research question that names the model, the explanation method, the users and the evaluation
  • List what an XAI study must report to be credible and reproducible

What you will unlearn

  • Interpretability always costs accuracy.Surveys find no general evidence for that trade-off, and interpretable models often perform comparably on structured data.
  • A SHAP or LIME plot shows how the model works.Post-hoc explanations approximate a model's behaviour; their faithfulness and stability must be tested, not assumed.
  • Adding an explanation method to a new data set is a research contribution.A contribution needs a problem: a failure, an untested assumption or an unmet user need that the study resolves.
  • Explanations are judged by how convincing they look.Explanations need functional tests and, where people use them, evaluation with real users on real tasks.

Skills you will practise

  • Problem formulation
  • Problematisation
  • Post-hoc explanation methods
  • Evaluation design for XAI
  • User study design
  • Reproducible ML reporting

What XAI is, and why finding a problem is hard

Explainable artificial intelligence (XAI) develops methods that help people understand why a machine learning model produces a given output. Some methods build models that are interpretable by design, such as sparse linear models or short decision rules. Others explain a trained model after the fact: LIME fits a simple local surrogate around one prediction (Ribeiro et al., 2016), and SHAP attributes a prediction to input features using Shapley values from cooperative game theory (Lundberg & Lee, 2017), with fast exact versions for tree ensembles (Lundberg et al., 2020).

Identifying research problems in explainable AI
Figure 1. Identifying research problems.

The field is now large. One survey screened 410 articles published between January 2016 and October 2022 and organised them along four axes: explaining the data, explaining the model, post-hoc explanation and the assessment of explanations (Ali et al., 2023). For a doctoral researcher, that volume is both an opportunity and a trap. Many papers apply an existing explanation method to a new data set and report attractive plots, and it is easy to propose one more. Examiners will ask what problem such a study solves. The steps below are designed to help you answer that question before you start.

Regulation has also raised the stakes. The EU Artificial Intelligence Act requires high-risk AI systems to be transparent enough for deployers to interpret their output and use it appropriately, and gives affected persons a right to an explanation of certain individual decisions (European Parliament & Council of the European Union, 2024). What counts as an adequate explanation in law and in practice is itself an open research question.

Step 1: Move from gap-spotting to problematisation

Most research questions are built by gap-spotting: finding a method, data set or population that the literature has not yet covered. Sandberg and Alvesson (2011) show that this habit tends to produce incremental studies that reinforce existing assumptions. Problematisation goes further: it identifies an assumption that a body of work takes for granted and asks what happens if it is wrong (Alvesson & Sandberg, 2011).

XAI has several assumptions worth challenging. One is that post-hoc explanations are faithful to the model they describe. Rudin (2019) argues that for high-stakes decisions this cannot be assumed, and that inherently interpretable models should be preferred where they perform comparably. Another is that interpretability must cost accuracy: the survey by Ali et al. (2023) found no scientific evidence for a general trade-off. A third is that a good explanation is one that looks plausible to its designer, rather than one that helps a defined user make a better decision. A problem built on testing one of these assumptions is stronger than one built on applying SHAP to a new data set.

Step 2: Survey the ten open problem areas

The ten problem areas below are a useful map, provided each is narrowed to a question one thesis can answer. The table restates each as an example question with a typical design; the questions are illustrations, not ready-made topics.

Ten open problem areas in XAI, with illustrative doctoral questions and designs
Problem areaWhat is unresolvedExample doctoral questionTypical design
Model-agnostic methodsWhether one method explains many model types equally wellDo LIME and SHAP attributions agree across gradient-boosted trees and neural networks trained on the same tabular task?Controlled comparison on benchmark data; agreement and faithfulness metrics
Evaluating explanationsNo agreed metrics or ground truth for explanation qualityCan synthetic data with known causal features serve as ground truth for ranking attribution methods?Synthetic benchmarks plus functional metrics such as deletion tests
Accuracy and interpretabilityWhether a trade-off exists in a given domainDoes a sparse interpretable model match a black-box model's accuracy on a clinical risk task?Benchmarking with cross-validation and calibration
Complex modelsExplaining very large models with many parametersHow stable are feature attributions for a deep image classifier under small input perturbations?Robustness experiments; stability metrics
High-dimensional and unstructured dataExplanations for images, text and sensor streamsWhich explanation formats help analysts interpret a time-series fault classifier?Method comparison plus small expert study
Privacy and securityWhether explanations leak information or can be manipulatedDoes releasing feature attributions increase the success of membership inference against a model?Attack experiments under a stated threat model
Human-centred explanationsMatching explanations to users' needs and decisionsDo counterfactual or attribution explanations lead loan officers to better override decisions?Randomised user study on a realistic task
Stability under driftExplanations that change as data shift over timeHow do SHAP attributions for a fraud model change under concept drift, and can that change signal drift early?Longitudinal analysis of production-like data streams
Cultural and societal factorsWhether explanation needs differ across groups and contextsDo patients in two health systems differ in what explanation of an AI triage result they find acceptable?Qualitative interviews or survey experiment
Reinforcement learningExplaining sequential decisions and policiesCan summaries of key states help users predict a trained agent's behaviour in new situations?Simulation environment plus user prediction task

Step 3: Put evaluation at the centre

Whatever area you choose, evaluation will be part of the problem. Ali et al. (2023) list XAI system design, generalisation of methods, user interaction, ground-truth evaluation and better tools among the open issues, and they group assessment of explanations into human-centred, application-grounded and functionally grounded approaches. Functionally grounded tests use computational proxies such as fidelity (does the explanation reflect what the model does?) and stability (does a small change in input produce a small change in explanation?). Human-centred and application-grounded tests involve people doing real tasks, and they answer a different question: does the explanation help?

A strong thesis usually combines both. A method that scores well on fidelity but confuses its users has not solved a practical problem, and a user study of an unfaithful explanation measures persuasion rather than understanding. Evaluation also depends on the field. Domain surveys catalogue the techniques used in areas such as healthcare (Chaddad et al., 2023), and Bank of England researchers have applied explanation methods to a default-risk model (Bracke et al., 2019); read the work in your target domain to learn who the users are and what decisions the explanations must support.

Step 4: Consider security, privacy and social context

Explanations reveal information about a model, and that can matter for security. Adversarial machine learning research catalogues evasion, poisoning and privacy attacks against models (Vassilev et al., 2025). Whether, and how far, releasing explanations changes exposure to such attacks is a researchable question, but it needs a clear threat model and responsible handling of any attack you demonstrate.

Social context matters too. Agreeing on principles such as transparency does not settle what an explanation should contain or who it should serve, and principles alone cannot guarantee ethical practice (Mittelstadt, 2019). Problems about who needs which explanation, in which setting, often call for qualitative or mixed methods, and they are as legitimate as algorithmic ones.

Worked example: from interest to problem statement (illustrative)

A doctoral researcher is interested in "XAI for credit scoring". That is an area, not a problem. The steps below show how it can be narrowed; the example is hypothetical.

  1. Area. Explanations for machine learning credit models.
  2. Gap-spotting version. "No study has applied SHAP to credit data from my country." This is a gap, but filling it would mainly reproduce known methods.
  3. Assumption found. Studies often present SHAP attributions as the reasons for a decision, assuming that they are stable. If small changes in training data or random seeds change the top-ranked features, the reasons given to an applicant could change for no substantive cause.
  4. Problem statement. "Feature attributions used to explain credit decisions may be unstable across equally accurate models, which undermines their use as reasons for individual decisions."
  5. Research questions. How stable are SHAP attributions across retrained, equally accurate models on a public credit data set? Does an interpretable scoring model give more stable reasons at similar accuracy? How do credit analysts judge the two kinds of explanation on realistic cases?
  6. Design. Repeated retraining experiments with stability metrics; an interpretable baseline; a small study with practitioners.

The final version tests an assumption (Alvesson & Sandberg, 2011), compares post-hoc and interpretable approaches (Rudin, 2019; Lundberg & Lee, 2017), combines functional and human-centred evaluation (Ali et al., 2023), and connects to a legal requirement to explain individual decisions (European Parliament & Council of the European Union, 2024).

Before committing, check feasibility. A user study needs access to the people who would really use the explanations, such as clinicians, analysts or loan officers, and ethics approval to involve them. Experiments with large models need computing time; repeated retraining multiplies it. Public data sets make results comparable but may not reflect your target setting, while private data may limit what you can publish and share. If any of these is out of reach, narrow the question until it is not, for example by replacing a field study with a carefully designed online experiment.

Step 5: Plan what you will report

XAI results are hard to compare because studies report different things. Reproducibility checklists developed for machine learning research ask for code, data, hyperparameters and the computing setup (Pineau et al., 2021). For an XAI study, add the items below.

What an XAI study should report
ItemWhat to report
ModelModel type, training data, performance on held-out data, random seeds
Explanation methodMethod, implementation and version, settings (for example, background data for SHAP)
Explained casesHow the explained instances were chosen; number of cases
Functional evaluationFidelity and stability metrics, with baselines such as random attributions
Human evaluationParticipants, task, conditions, outcome measures and ethics approval
Threat modelFor security or privacy work: attacker knowledge and capabilities
Code and dataRepository, licence and any access restrictions

Common mistakes

  • Treating "apply method X to data set Y" as a research problem.
  • Presenting post-hoc explanations as the model's true reasoning without testing faithfulness.
  • Assuming an accuracy and interpretability trade-off instead of measuring it.
  • Evaluating explanations only by visual appeal, or only with the research team as users.
  • Ignoring the intended users, their decisions and their context.
  • Reporting one random seed, so instability goes unnoticed.

Key takeaways

Your next step

Want to apply this to your own study?

  1. 1Tell us your topic, data and deadline
  2. 2A statistician reviews your work with you
  3. 3You get a plan and a quote
Problem identification

References

  1. Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023). Explainable artificial intelligence (XAI): What we know and what is left to attain trustworthy artificial intelligence. Information Fusion, 99, 101805. https://doi.org/10.1016/j.inffus.2023.101805
  2. Alvesson, M., & Sandberg, J. (2011). Generating research questions through problematization. Academy of Management Review, 36(2), 247–271. https://doi.org/10.5465/amr.2009.0188
  3. Bracke, P., Datta, A., Jung, C., & Sen, S. (2019). Machine learning explainability in finance: An application to default risk analysis (Staff Working Paper No. 816). Bank of England. https://www.bankofengland.co.uk/working-paper/2019/machine-learning-explainability-in-finance-an-application-to-default-risk-analysis
  4. Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023). Survey of explainable AI techniques in healthcare. Sensors, 23(2), 634. https://doi.org/10.3390/s23020634
  5. European Parliament & Council of the European Union. (2024). Regulation (EU) 2024/1689 of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689. https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
  6. Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30 (pp. 4765–4774). Curran Associates. https://neurips.cc/virtual/2017/poster/9253
  7. Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., & Lee, S.-I. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1), 56-67. https://doi.org/10.1038/s42256-019-0138-9
  8. Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501–507. https://doi.org/10.1038/s42256-019-0114-4
  9. Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2021). Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). Journal of Machine Learning Research, 22(164), 1–20. https://www.jmlr.org/papers/v22/20-303.html
  10. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1135-1144). ACM. https://doi.org/10.1145/2939672.2939778
  11. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5), 206–215. https://doi.org/10.1038/s42256-019-0048-x
  12. Sandberg, J., & Alvesson, M. (2011). Ways of constructing research questions: Gap-spotting or problematization? Organization, 18(1), 23–44. https://doi.org/10.1177/1350508410372151
  13. Vassilev, A., Oprea, A., Hamin, M., Fordyce, A., Anderson, H., & Davies, X. (2025). Adversarial machine learning: A taxonomy and terminology of attacks and mitigations (NIST AI 100-2e2025). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.100-2e2025

How to cite this article

Choose your referencing style, then copy. Check your university's or journal's guide for local variations.

PhD Research Lab Academy. (2024, January 7). How to identify a research problem in explainable artificial intelligence (XAI). PhD Research Lab. https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods/

PhD Research Lab Academy. “How to Identify a Research Problem in Explainable Artificial Intelligence (XAI).” PhD Research Lab, 7 Jan. 2024, www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods. Accessed [access date].

PhD Research Lab Academy (2024) How to identify a research problem in explainable artificial intelligence (XAI). PhD Research Lab. Available at: https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods/ (Accessed: [access date]).

PhD Research Lab Academy. “How to Identify a Research Problem in Explainable Artificial Intelligence (XAI).” PhD Research Lab. January 7, 2024. https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods.

PhD Research Lab Academy, ‘How to Identify a Research Problem in Explainable Artificial Intelligence (XAI)’ (PhD Research Lab, 7 January 2024) <https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods> accessed [access date].

PhD Research Lab Academy. How to identify a research problem in explainable artificial intelligence (XAI) [Internet]. PhD Research Lab; 2024 Jan 7 [cited [access date]]. Available from: https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods/

PhD Research Lab Academy, “How to identify a research problem in explainable artificial intelligence (XAI),” PhD Research Lab, Jan. 7, 2024. [Online]. Available: https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods. [Accessed: [access date]].

@online{research_problems_in_explainable_artificial_intelligence_methods_2024,
  author = {{PhD Research Lab Academy}},
  title = {{How to identify a research problem in explainable artificial intelligence (XAI)}},
  organization = {PhD Research Lab},
  date = {2024-01-07},
  url = {https://www.phdassistance.com/problem-identification/research-problems-in-explainable-artificial-intelligence-methods},
  urldate = {[access date]}
}

This article is for learning. Use it to understand the method and cite the original sources above in your own work; do not copy or submit it as your own.

How to identify a research problem in explainable artificial intelligence (XAI)Need help applying it to your study?
Problem identification