About this example
This page is the results chapter of an example master's or early doctoral dissertation in computer science. The study trains four deep learning text classifiers to distinguish Twitter users who have publicly stated a mental health diagnosis from control users, and compares them. The chapter shows how to report machine learning results honestly: defining metrics correctly, giving uncertainty, keeping conclusions consistent with the tables and stating what was not done.
It is an illustrative teaching example, not a real student's work. All numbers in Tables 4.1 and 4.2 are illustrative data, created for teaching and checked for internal consistency; they describe no real dataset. Do not submit any part of it as your own work; your university's academic integrity rules apply.
The example: data, test design and metrics
4.1 Introduction
This chapter reports the performance of the four models described in Chapter 3: a long short-term memory network (LSTM) and an LSTM preceded by a one-dimensional convolutional layer (LSTM-CNN), each with an embedding layer trained from scratch or initialised with pre-trained 300-dimensional Word2Vec vectors. Section 4.2 recaps the test data; 4.3 defines the metrics; 4.4 presents results by model; 4.5 compares the models; 4.6 reports robustness checks and what they do not cover.
4.2 Test data
Positive users were identified by a self-reported diagnosis statement and controls were sampled from users with no such statement, a common but imperfect labelling strategy: self-disclosure is not a clinical diagnosis, and control groups may include undiagnosed users (Chancellor & De Choudhury, 2020; Ernala et al., 2019). Users, not tweets, were split into training (70%), validation (15%) and test (15%) sets, so that no user's posts appear in more than one set. The held-out test set contains 600 users, 300 in each class. The study used only public posts, removed usernames and links before analysis, reports no quotations that could identify a user and was approved by the university ethics committee, following published guidance for social media health research (Benton et al., 2017; Fiesler & Proferes, 2018).
4.3 Metrics
With the positive class defined as "self-reported diagnosis", a true positive (TP) is a positive user predicted positive; a false positive (FP) is a control user wrongly predicted positive; a false negative (FN) is a positive user wrongly predicted negative; and a true negative (TN) is a control user predicted negative. Accuracy is (TP + TN) / N. Precision is TP / (TP + FP), the share of positive predictions that are correct. Recall (sensitivity) is TP / (TP + FN). Specificity is TN / (TN + FP). F1 is the harmonic mean of precision and recall. Ninety-five per cent confidence intervals for accuracy use the Wilson method. Because the test set is balanced, accuracy is informative here; with imbalanced data, precision-recall curves are more informative than ROC curves (Saito & Rehmsmeier, 2015).
The example: results by model and comparison
4.4 Results by model
Table 4.1 gives the confusion counts for each model on the 600 test users, and Table 4.2 gives the derived metrics. All four models converged within the planned number of epochs, and training curves (Appendix D) showed no divergence between training and validation loss after early stopping.
| Model | TP | FN | FP | TN |
|---|---|---|---|---|
| LSTM, trained embedding | 270 | 30 | 21 | 279 |
| LSTM-CNN, trained embedding | 279 | 21 | 15 | 285 |
| LSTM, Word2Vec embedding | 276 | 24 | 18 | 282 |
| LSTM-CNN, Word2Vec embedding | 282 | 18 | 15 | 285 |
| Model | Accuracy (95% CI) | Precision | Recall | Specificity | F1 |
|---|---|---|---|---|---|
| LSTM, trained embedding | 91.5 (89.0 to 93.5) | 92.8 | 90.0 | 93.0 | 91.4 |
| LSTM-CNN, trained embedding | 94.0 (91.8 to 95.6) | 94.9 | 93.0 | 95.0 | 93.9 |
| LSTM, Word2Vec embedding | 93.0 (90.7 to 94.8) | 93.9 | 92.0 | 94.0 | 92.9 |
| LSTM-CNN, Word2Vec embedding | 94.5 (92.4 to 96.1) | 94.9 | 94.0 | 95.0 | 94.5 |
4.5 Comparison of models
The LSTM-CNN outperformed the plain LSTM with both embeddings (94.0% against 91.5% with trained embeddings; 94.5% against 93.0% with Word2Vec). Pre-trained Word2Vec embeddings gave slightly higher accuracy than trained embeddings for both architectures, a mean of 93.75% against 92.75%. The best model, LSTM-CNN with Word2Vec, achieved 94.5% accuracy, but its confidence interval overlaps those of the LSTM-CNN with trained embeddings and the Word2Vec LSTM, so these three models cannot be ranked with confidence on this test set. Only the plain LSTM with trained embeddings performed clearly below the best model. Errors were spread evenly between false positives and false negatives for all models.
The example: robustness checks and what they do not show
4.6 Robustness and limitations of the results
Two checks were carried out. First, the test set was screened for users or near-duplicate tweets that also appeared in the training data; none were found after the user-level split. Second, hyperparameters were tuned on the validation set only, so the test set was used once, for the results above; tuning on the test set, or reporting the best of many cross-validation runs, would bias the estimates upwards (Varma & Simon, 2006). Repeated cross-validation was not carried out because of computing limits, so the results reflect one split and their variability across splits is unknown.
The results describe performance on users from one platform, one period and one labelling method. Models of mental health trained on social media data often perform much worse on other datasets and platforms (Harrigian et al., 2020), and research in this area has repeatedly raised concerns about construct validity (Guntuku et al., 2017; Chancellor & De Choudhury, 2020). No claim is therefore made that the models detect mental disorders in general; they distinguish users who disclose a diagnosis from sampled controls in this dataset.
4.7 Summary of results
On a balanced, user-level test set of 600 users, all four models separated users who disclosed a diagnosis from controls with accuracy between 91.5% and 94.5%. Adding a convolutional layer improved the LSTM with both embeddings, and pre-trained Word2Vec embeddings gave a small average gain. The three strongest models could not be distinguished statistically on this test set. Chapter 5 discusses what these results mean for the research questions, how they compare with published work and what the labelling limits imply for any practical use.
How to adapt this to your own study
- Recap the test data, the unit of analysis and how the split prevents leakage.
- Define every metric once, in terms of your own positive and negative classes.
- Report confusion counts and derived metrics, with confidence intervals.
- Compare models only on differences your data can support, and say when intervals overlap.
- Check that every sentence in your comparison agrees with your tables.
- Report robustness checks done, and state clearly those not done.
- Limit conclusions to the population, platform and labels you studied.
Common mistakes in machine learning results chapters
- Defining false positives or other metric terms incorrectly.
- Calling accuracy "precision", or reporting one metric under another name.
- Drawing a conclusion that the results table contradicts.
- Splitting by post rather than by person, so the same person appears in training and test data.
- Promising cross-validation in the results instead of reporting what was done.
- Re-explaining architectures copied from tutorials instead of referring to the methods chapter.
- Generalising from one platform and one labelling method to "mental disorders".
| What examiners look for | Where this example does it | Check in your own draft |
|---|---|---|
| Correct metric definitions | 4.3 | Is each term defined for your own classes? |
| Results that can be recomputed | Table 4.1 confusion counts | Could a reader recompute your metrics? |
| Uncertainty | Confidence intervals in Table 4.2 | Do you show intervals or variability? |
| Consistency of text and tables | 4.5 | Does every sentence match the numbers? |
| Leakage and tuning controls | 4.2 and 4.6 | Is the test set truly unseen? |
| Honest limits and ethics | 4.2 and 4.6 | Have you stated label validity and ethics approval? |
Related examples and guides
- Academy guide: Reporting statistical results
- Academy review: Sentiment analysis and doctoral mental health
- Example results chapter: FDI and economic growth in Mauritius
- Service: manuscript statistics mentoring
- All sample work
Key takeaways
- Split data by person, not by post, and say so in the results (Kapoor & Narayanan, 2023).
- Define every metric correctly and report the counts behind it (Collins et al., 2024).
- Use the test set once; tuning on it biases performance upwards (Varma & Simon, 2006).
- Do not rank models whose confidence intervals overlap.
- State the limits of self-disclosure labels and of generalisation to other platforms (Harrigian et al., 2020; Ernala et al., 2019).
