ExampleMethodology & analyticsComputer science

Example results chapter: deep learning models for detecting mental-health risk in tweets (annotated)

An annotated model results chapter comparing four deep learning text classifiers, showing how to define metrics correctly, report uncertainty, keep conclusions consistent with the tables and state the limits of social media labels. For PhD and master's researchers in computing and digital health.

What you will learn

  • Define confusion-matrix terms and metrics correctly for your own classes
  • Report confusion counts, derived metrics and confidence intervals together
  • Compare models without over-reading small differences
  • Recognise and report safeguards against data leakage
  • Limit conclusions to what the data and labels support

What you will unlearn

  • A false positive is a positive case with a correct label.A false positive is a negative case that the model wrongly predicts as positive.
  • The model with the highest accuracy is the best model.Small differences may fall within the uncertainty of the estimate and should not be ranked.
  • A ROC AUC of 0.5 means erratic performance.An AUC of 0.5 means the model discriminates no better than chance.
  • High accuracy on social media data shows the model detects mental illness.It shows the model separates the labelled groups in that dataset; labels and generalisation need separate evidence.

Skills you will practise

  • Classification metrics
  • Confusion matrices
  • Confidence intervals
  • Leakage control
  • Results reporting

About this example

This page is the results chapter of an example master's or early doctoral dissertation in computer science. The study trains four deep learning text classifiers to distinguish Twitter users who have publicly stated a mental health diagnosis from control users, and compares them. The chapter shows how to report machine learning results honestly: defining metrics correctly, giving uncertainty, keeping conclusions consistent with the tables and stating what was not done.

It is an illustrative teaching example, not a real student's work. All numbers in Tables 4.1 and 4.2 are illustrative data, created for teaching and checked for internal consistency; they describe no real dataset. Do not submit any part of it as your own work; your university's academic integrity rules apply.

The example: data, test design and metrics

4.1 Introduction

This chapter reports the performance of the four models described in Chapter 3: a long short-term memory network (LSTM) and an LSTM preceded by a one-dimensional convolutional layer (LSTM-CNN), each with an embedding layer trained from scratch or initialised with pre-trained 300-dimensional Word2Vec vectors. Section 4.2 recaps the test data; 4.3 defines the metrics; 4.4 presents results by model; 4.5 compares the models; 4.6 reports robustness checks and what they do not cover.

4.2 Test data

Positive users were identified by a self-reported diagnosis statement and controls were sampled from users with no such statement, a common but imperfect labelling strategy: self-disclosure is not a clinical diagnosis, and control groups may include undiagnosed users (Chancellor & De Choudhury, 2020; Ernala et al., 2019). Users, not tweets, were split into training (70%), validation (15%) and test (15%) sets, so that no user's posts appear in more than one set. The held-out test set contains 600 users, 300 in each class. The study used only public posts, removed usernames and links before analysis, reports no quotations that could identify a user and was approved by the university ethics committee, following published guidance for social media health research (Benton et al., 2017; Fiesler & Proferes, 2018).

4.3 Metrics

With the positive class defined as "self-reported diagnosis", a true positive (TP) is a positive user predicted positive; a false positive (FP) is a control user wrongly predicted positive; a false negative (FN) is a positive user wrongly predicted negative; and a true negative (TN) is a control user predicted negative. Accuracy is (TP + TN) / N. Precision is TP / (TP + FP), the share of positive predictions that are correct. Recall (sensitivity) is TP / (TP + FN). Specificity is TN / (TN + FP). F1 is the harmonic mean of precision and recall. Ninety-five per cent confidence intervals for accuracy use the Wilson method. Because the test set is balanced, accuracy is informative here; with imbalanced data, precision-recall curves are more informative than ROC curves (Saito & Rehmsmeier, 2015).

The example: results by model and comparison

4.4 Results by model

Table 4.1 gives the confusion counts for each model on the 600 test users, and Table 4.2 gives the derived metrics. All four models converged within the planned number of epochs, and training curves (Appendix D) showed no divergence between training and validation loss after early stopping.

Table 4.1. Confusion counts on the held-out test set (N = 600; 300 positive, 300 control). Illustrative data, for teaching only.
ModelTPFNFPTN
LSTM, trained embedding2703021279
LSTM-CNN, trained embedding2792115285
LSTM, Word2Vec embedding2762418282
LSTM-CNN, Word2Vec embedding2821815285
Table 4.2. Test-set performance (%), computed from Table 4.1. Illustrative data, for teaching only.
ModelAccuracy (95% CI)PrecisionRecallSpecificityF1
LSTM, trained embedding91.5 (89.0 to 93.5)92.890.093.091.4
LSTM-CNN, trained embedding94.0 (91.8 to 95.6)94.993.095.093.9
LSTM, Word2Vec embedding93.0 (90.7 to 94.8)93.992.094.092.9
LSTM-CNN, Word2Vec embedding94.5 (92.4 to 96.1)94.994.095.094.5

4.5 Comparison of models

The LSTM-CNN outperformed the plain LSTM with both embeddings (94.0% against 91.5% with trained embeddings; 94.5% against 93.0% with Word2Vec). Pre-trained Word2Vec embeddings gave slightly higher accuracy than trained embeddings for both architectures, a mean of 93.75% against 92.75%. The best model, LSTM-CNN with Word2Vec, achieved 94.5% accuracy, but its confidence interval overlaps those of the LSTM-CNN with trained embeddings and the Word2Vec LSTM, so these three models cannot be ranked with confidence on this test set. Only the plain LSTM with trained embeddings performed clearly below the best model. Errors were spread evenly between false positives and false negatives for all models.

The example: robustness checks and what they do not show

4.6 Robustness and limitations of the results

Two checks were carried out. First, the test set was screened for users or near-duplicate tweets that also appeared in the training data; none were found after the user-level split. Second, hyperparameters were tuned on the validation set only, so the test set was used once, for the results above; tuning on the test set, or reporting the best of many cross-validation runs, would bias the estimates upwards (Varma & Simon, 2006). Repeated cross-validation was not carried out because of computing limits, so the results reflect one split and their variability across splits is unknown.

The results describe performance on users from one platform, one period and one labelling method. Models of mental health trained on social media data often perform much worse on other datasets and platforms (Harrigian et al., 2020), and research in this area has repeatedly raised concerns about construct validity (Guntuku et al., 2017; Chancellor & De Choudhury, 2020). No claim is therefore made that the models detect mental disorders in general; they distinguish users who disclose a diagnosis from sampled controls in this dataset.

4.7 Summary of results

On a balanced, user-level test set of 600 users, all four models separated users who disclosed a diagnosis from controls with accuracy between 91.5% and 94.5%. Adding a convolutional layer improved the LSTM with both embeddings, and pre-trained Word2Vec embeddings gave a small average gain. The three strongest models could not be distinguished statistically on this test set. Chapter 5 discusses what these results mean for the research questions, how they compare with published work and what the labelling limits imply for any practical use.

How to adapt this to your own study

  1. Recap the test data, the unit of analysis and how the split prevents leakage.
  2. Define every metric once, in terms of your own positive and negative classes.
  3. Report confusion counts and derived metrics, with confidence intervals.
  4. Compare models only on differences your data can support, and say when intervals overlap.
  5. Check that every sentence in your comparison agrees with your tables.
  6. Report robustness checks done, and state clearly those not done.
  7. Limit conclusions to the population, platform and labels you studied.

Common mistakes in machine learning results chapters

  • Defining false positives or other metric terms incorrectly.
  • Calling accuracy "precision", or reporting one metric under another name.
  • Drawing a conclusion that the results table contradicts.
  • Splitting by post rather than by person, so the same person appears in training and test data.
  • Promising cross-validation in the results instead of reporting what was done.
  • Re-explaining architectures copied from tutorials instead of referring to the methods chapter.
  • Generalising from one platform and one labelling method to "mental disorders".
How this results chapter meets common examiner expectations.
What examiners look forWhere this example does itCheck in your own draft
Correct metric definitions4.3Is each term defined for your own classes?
Results that can be recomputedTable 4.1 confusion countsCould a reader recompute your metrics?
UncertaintyConfidence intervals in Table 4.2Do you show intervals or variability?
Consistency of text and tables4.5Does every sentence match the numbers?
Leakage and tuning controls4.2 and 4.6Is the test set truly unseen?
Honest limits and ethics4.2 and 4.6Have you stated label validity and ethics approval?

Related examples and guides

Key takeaways

Your next step

Want to apply this to your own study?

  1. 1Tell us your topic, data and deadline
  2. 2A statistician reviews your work with you
  3. 3You get a plan and a quote
Manuscript statistics

References

  1. Benton, A., Coppersmith, G., & Dredze, M. (2017). Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing (pp. 94-102). Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-1612
  2. Chancellor, S., & De Choudhury, M. (2020). Methods in predictive techniques for mental health status on social media: A critical review. npj Digital Medicine, 3, Article 43. https://doi.org/10.1038/s41746-020-0233-7
  3. Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., Boulesteix, A.-L., Camaradou, J. C., Celi, L. A., Denaxas, S., Denniston, A. K., Glocker, B., Golub, R. M., Harvey, H., Heinze, G., . . . Logullo, P. (2024). TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 385, e078378. https://doi.org/10.1136/bmj-2023-078378
  4. Ernala, S. K., Birnbaum, M. L., Candan, K. A., Rizvi, A. F., Sterling, W. A., Kane, J. M., & De Choudhury, M. (2019). Methodological gaps in predicting mental health states from social media: Triangulating diagnostic signals. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-16). ACM. https://doi.org/10.1145/3290605.3300364
  5. Fiesler, C., & Proferes, N. (2018). "Participant" perceptions of Twitter research ethics. Social Media + Society, 4(1). https://doi.org/10.1177/2056305118763366
  6. Guntuku, S. C., Yaden, D. B., Kern, M. L., Ungar, L. H., & Eichstaedt, J. C. (2017). Detecting depression and mental illness on social media: An integrative review. Current Opinion in Behavioral Sciences, 18, 43-49. https://doi.org/10.1016/j.cobeha.2017.07.005
  7. Harrigian, K., Aguirre, C., & Dredze, M. (2020). Do models of mental health based on social media data generalize? In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 3774-3788). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.337
  8. Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
  9. Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432. https://doi.org/10.1371/journal.pone.0118432
  10. Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91. https://doi.org/10.1186/1471-2105-7-91

How to cite this article

Choose your referencing style, then copy. Check your university's or journal's guide for local variations.

PhD Research Lab Academy. (2022, December 20). Example results chapter: deep learning models for detecting mental-health risk in tweets (annotated). PhD Research Lab. https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques/

PhD Research Lab Academy. “Example Results Chapter: Deep Learning Models for Detecting Mental-health Risk in Tweets (annotated).” PhD Research Lab, 20 Dec. 2022, www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques. Accessed [access date].

PhD Research Lab Academy (2022) Example results chapter: deep learning models for detecting mental-health risk in tweets (annotated). PhD Research Lab. Available at: https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques/ (Accessed: [access date]).

PhD Research Lab Academy. “Example Results Chapter: Deep Learning Models for Detecting Mental-health Risk in Tweets (annotated).” PhD Research Lab. December 20, 2022. https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques.

PhD Research Lab Academy, ‘Example Results Chapter: Deep Learning Models for Detecting Mental-health Risk in Tweets (annotated)’ (PhD Research Lab, 20 December 2022) <https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques> accessed [access date].

PhD Research Lab Academy. Example results chapter: deep learning models for detecting mental-health risk in tweets (annotated) [Internet]. PhD Research Lab; 2022 Dec 20 [cited [access date]]. Available from: https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques/

PhD Research Lab Academy, “Example results chapter: deep learning models for detecting mental-health risk in tweets (annotated),” PhD Research Lab, Dec. 20, 2022. [Online]. Available: https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques. [Accessed: [access date]].

@online{mental_disorder_prediction_using_deep_learning_techniques_2022,
  author = {{PhD Research Lab Academy}},
  title = {{Example results chapter: deep learning models for detecting mental-health risk in tweets (annotated)}},
  organization = {PhD Research Lab},
  date = {2022-12-20},
  url = {https://www.phdassistance.com/sample-work/mental-disorder-prediction-using-deep-learning-techniques},
  urldate = {[access date]}
}

This article is for learning. Use it to understand the method and cite the original sources above in your own work; do not copy or submit it as your own.

Example results chapterNeed help applying it to your study?
Manuscript statistics