ExampleResearch writingComputer science

Example proposal: detecting harmful speech in online videos from transcripts (annotated)

An annotated model proposal for a natural language processing project that ranks online videos by the likelihood of harmful spoken content. It helps master's and PhD scholars in computer science scope a machine-learning study, choose fitting data and design a sound evaluation.

What you will learn

  • Narrow a machine-learning project to what its method can actually detect
  • Turn processing steps into answerable research questions
  • Match training data to the data a system will meet
  • Design an evaluation that avoids leakage and suits rare classes
  • Plan ethical safeguards for public online data and annotators

What you will unlearn

  • Public videos can be used for research without ethical review.Public availability does not remove the need for ethics approval and protection of the people involved.
  • A model trained on toxic comments will work on speech.Written and spoken language differ, so performance must be tested on transcripts.
  • Accuracy is the best measure of a classifier.When harmful content is rare, precision and recall are far more informative.
  • Converting file formats is a research objective.It is preprocessing; objectives should be questions the study answers.

Skills you will practise

  • Scoping a study
  • Research questions
  • Evaluation design
  • Data leakage
  • Research ethics

About this example

This page is an example research proposal in computer science (natural language processing applied to online safety), written at the level of a master's dissertation proposal that could be extended into a doctoral project. It proposes a system that flags online videos whose spoken content may be harmful, so that human moderators can review them first. Each part of the proposal is followed by a "What to notice" note explaining what the passage does and why.

It is an illustrative teaching example, not a real student's work. Do not submit any part of it as your own work; your university's academic integrity rules apply.

The example: background, problem and aims

1. Background and problem

Video-sharing platforms host very large volumes of user-generated content, and a small proportion of it contains harassment, hate speech, threats or incitement. Platforms rely on a combination of user reports, automated filters and human moderators. Automated filters for text are well established, but much of the harmful material in videos is spoken rather than written, and spoken language differs from written comments in vocabulary, structure and noise. A practical question is therefore whether speech can be converted to text accurately enough, and classified reliably enough, to help moderators decide what to review first.

This proposal addresses only the spoken channel. Harmful visual content, such as violent or sexual imagery, cannot be detected from a transcript, and a complete system would need to combine audio, visual and textual signals (Baltrušaitis et al., 2019). Limiting the scope in this way makes the project feasible and allows its contribution to be measured precisely.

2. Aim and research questions

The aim is to develop and evaluate a transcript-based classifier that ranks videos by the likelihood that their speech contains harmful content, as a triage aid for human review. The research questions are:

  1. How accurately does automatic speech recognition transcribe the speech in a sample of user-generated videos, and how do transcription errors affect classification?
  2. How well do a lexical baseline and a fine-tuned transformer classifier identify harmful speech segments in transcripts, on channels not seen during training?
  3. Do error rates differ across dialects and speaker groups represented in the evaluation set?

The example: literature and rationale

3. Related work

Text classification for abusive and toxic language has moved from lexical features with linear models to pretrained transformer models, whose self-attention architecture captures context across a whole passage (Vaswani et al., 2017). Three findings from the wider literature shape this proposal. First, models trained on one source of text often perform worse on another, so a classifier trained on written comments cannot be assumed to work on transcribed speech. Second, language models can encode social biases, including associations between dialects or identity terms and offensiveness, which can lead to unequal error rates across groups (Gallegos et al., 2024). Third, machine-learning studies in security-related settings are prone to recurring pitfalls, including sampling bias, data snooping, inappropriate baselines and evaluation only in laboratory conditions (Arp et al., 2022). Detection of harmful content is also an adversarial, open-world problem in which the content being detected changes in response to detection, a long-standing difficulty for machine learning in security (Sommer & Paxson, 2010).

The example: method

4.1 Data

A sample of publicly available videos will be drawn from channels covering a range of topics, with the sampling frame and dates recorded. Audio will be extracted and segmented, and transcripts produced by an open-source speech-recognition system. A random subset of segments will be transcribed manually to measure word error rate (RQ1). Two trained annotators will label each transcript segment as harmful or not harmful under a written codebook with definitions and examples for each category; agreement will be measured and disagreements resolved by discussion. Existing written-comment corpora will be used only as a comparison condition, to show how much performance changes when a model trained on written text is applied to speech.

4.2 Models and evaluation

Two classifiers will be compared: a baseline using word and character n-gram features with logistic regression, and a pretrained transformer model fine-tuned on the annotated segments. Segment scores will be aggregated to a video-level ranking. To avoid leakage, all segments from the same channel will be placed in the same split, so the model is never tested on speakers it has already seen (Kapoor & Narayanan, 2023). Because harmful segments are expected to be a small minority, performance will be reported with precision, recall and the area under the precision-recall curve rather than accuracy alone (Saito & Rehmsmeier, 2015). Hyperparameter search budgets and variation across random seeds will be reported (Dodge et al., 2019), and error rates will be compared across dialect and speaker groups where the sample allows (RQ3).

  1. 1Sample videos and record sampling frame
  2. 2Extract and segment audio
  3. 3Automatic speech recognition; measure word error rate on a manual subset
  4. 4Annotate transcript segments with a codebook
  5. 5Train baseline and transformer classifiers on channel-level splits
  6. 6Rank videos for human review and analyse errors by group
Figure 1. Planned pipeline, from video to moderator review.
Table 1. Evaluation plan by research question.
QuestionMeasureWhy this measure
RQ1 transcription qualityWord error rate on a manually transcribed subsetShows how much noise the classifier receives
RQ2 classificationPrecision, recall, precision-recall AUC on unseen channelsHarmful content is rare; accuracy would be misleading
RQ2 robustnessChange in performance from written-text training to transcript trainingTests the domain-shift problem directly
RQ3 fairnessFalse positive and false negative rates by groupUnequal errors would harm some speakers more than others

The example: ethics, timeline and contribution

5. Ethical considerations

Although the videos are public, the people in them have not consented to research, and transcripts may contain personal information. The project will follow established internet research ethics guidance, store only the data needed, remove names and other identifiers from transcripts, and not quote material that could identify a speaker (franzke et al., 2020). Annotators will be exposed to distressing material, so the protocol will limit session length, allow withdrawal from categories of content, and provide access to support. Approval will be sought from the university research ethics committee before data collection, and the platform's terms of service will be checked.

6. Timeline

Table 2. Indicative ten-month plan.
MonthsWork
1 to 2Ethics approval; codebook drafting and pilot annotation
3 to 4Video sampling, audio extraction, speech recognition and word error rate measurement
5 to 6Full annotation and agreement analysis
7 to 8Model training and evaluation; subgroup error analysis
9 to 10Writing up, with a final month for revision

7. Expected contribution

The study will provide a measured account of how well transcript-based classification supports the triage of harmful spoken content, how much speech-recognition errors and domain shift reduce performance, and whether errors fall unequally on particular groups. A doctoral extension would add the visual channel and test multimodal models, evaluate the effect of the ranking on moderators' decisions, and study how performance changes over time as content adapts to detection.

How to adapt this to your own study

  1. State the decision or problem your system supports, and narrow the scope to what your method can detect.
  2. Write research questions that can be answered with evidence, not a list of processing steps.
  3. Use training data that matches the data your system will meet, or test the mismatch explicitly.
  4. Design splits that prevent leakage between training and test data.
  5. Choose evaluation measures that suit the class balance and the purpose.
  6. Plan for bias, ethics and annotator welfare from the start.
  7. If you build on a published design, cite it and say clearly what your study adds.

Common mistakes in this kind of proposal

  • Reproducing a published design without acknowledging it.
  • Training on one kind of text (written comments) and claiming results for another (speech).
  • Claiming to detect all harmful videos with a method that sees only one channel.
  • Comparing accuracy figures from studies of different tasks.
  • Reporting accuracy alone when the positive class is rare.
  • Treating ethics as a formality when the data involve real people and distressing material.
How this proposal meets common examiner expectations.
What examiners look forWhere this example does itCheck in your own draft
A clear, bounded problemSection 1Does your scope match what your method can do?
Answerable research questionsSection 2Could each question be answered with your data?
A review that informs the designSection 3Is every point in your review used later?
A sound evaluation designSection 4.2 and Table 1Have you prevented leakage and chosen fitting metrics?
Specific ethical safeguardsSection 5Are your safeguards specific to your data and people?
FeasibilityTable 2Is the plan realistic for your time and resources?
Originality and attributionSections 3 and 7Have you credited prior designs and stated what is new?

Key takeaways

  • Define the decision your system supports and scope the project to what the method can observe.
  • Test the gap between training data and real data rather than assuming it away.
  • Split data by source to prevent leakage, and report precision and recall for rare classes (Kapoor & Narayanan, 2023; Saito & Rehmsmeier, 2015).
  • Check for unequal error rates across groups (Gallegos et al., 2024).
  • Credit any published design you build on and say what your study adds.
Your next step

Want to apply this to your own study?

  1. 1Tell us your topic, data and deadline
  2. 2A subject mentor reviews your work with you
  3. 3You get a plan and a quote
Full dissertation mentoring

References

  1. Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2022). Dos and don'ts of machine learning in computer security. In Proceedings of the 31st USENIX Security Symposium (pp. 3971–3988). USENIX Association. https://www.usenix.org/conference/usenixsecurity22/presentation/arp
  2. Baltrušaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423-443. https://doi.org/10.1109/TPAMI.2018.2798607
  3. Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 2185–2194). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1224
  4. franzke, a. s., Bechmann, A., Zimmer, M., Ess, C., & the Association of Internet Researchers. (2020). Internet research: Ethical guidelines 3.0. Association of Internet Researchers. https://aoir.org/reports/ethics3.pdf
  5. Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024). Bias and fairness in large language models: A survey. Computational Linguistics, 50(3), 1097–1179. https://doi.org/10.1162/coli_a_00524
  6. Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
  7. Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10(3), e0118432. https://doi.org/10.1371/journal.pone.0118432
  8. Sommer, R., & Paxson, V. (2010). Outside the closed world: On using machine learning for network intrusion detection. In 2010 IEEE Symposium on Security and Privacy (pp. 305–316). IEEE. https://www.icsi.berkeley.edu/icsi/publication_details?n=2818
  9. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30 (pp. 5998–6008). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

How to cite this article

Choose your referencing style, then copy. Check your university's or journal's guide for local variations.

PhD Research Lab Academy. (2022, December 20). Example proposal: detecting harmful speech in online videos from transcripts (annotated). PhD Research Lab. https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos/

PhD Research Lab Academy. “Example Proposal: Detecting Harmful Speech in Online Videos from Transcripts (annotated).” PhD Research Lab, 20 Dec. 2022, www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos. Accessed [access date].

PhD Research Lab Academy (2022) Example proposal: detecting harmful speech in online videos from transcripts (annotated). PhD Research Lab. Available at: https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos/ (Accessed: [access date]).

PhD Research Lab Academy. “Example Proposal: Detecting Harmful Speech in Online Videos from Transcripts (annotated).” PhD Research Lab. December 20, 2022. https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos.

PhD Research Lab Academy, ‘Example Proposal: Detecting Harmful Speech in Online Videos from Transcripts (annotated)’ (PhD Research Lab, 20 December 2022) <https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos> accessed [access date].

PhD Research Lab Academy. Example proposal: detecting harmful speech in online videos from transcripts (annotated) [Internet]. PhD Research Lab; 2022 Dec 20 [cited [access date]]. Available from: https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos/

PhD Research Lab Academy, “Example proposal: detecting harmful speech in online videos from transcripts (annotated),” PhD Research Lab, Dec. 20, 2022. [Online]. Available: https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos. [Accessed: [access date]].

@online{nlp_based_advanced_method_of_detecting_harmful_videos_2022,
  author = {{PhD Research Lab Academy}},
  title = {{Example proposal: detecting harmful speech in online videos from transcripts (annotated)}},
  organization = {PhD Research Lab},
  date = {2022-12-20},
  url = {https://www.phdassistance.com/sample-work/nlp-based-advanced-method-of-detecting-harmful-videos},
  urldate = {[access date]}
}

This article is for learning. Use it to understand the method and cite the original sources above in your own work; do not copy or submit it as your own.

Example proposalNeed help applying it to your study?
Full dissertation mentoring