The creation of annotation corpora is a critical task in natural language processing which involves significant human effort. We summarized our experience of corpus annotation process in this repo Annotation Best Practice.

Moreover, we have published a work-in-progress book chapter Chapter 9: Best practices of annotating clinical texts for information extraction tasks to introduce more details.

The above figure comes from https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-020-1072-9/figures/1

More detailed description can be found in our paper:

• Assessment of the impact of EHR heterogeneity for clinical research through a case study of silent brain infarction
• Characterizing Chronic Pain Episodes in Clinical Text at Two Health Care Systems: Comprehensive Annotation and Corpus Analysis.