Evaluation of Syndromic Surveillance

Evaluation and strengthening of biosurveillance systems is acomplex process that involves sequential decision steps, numerous stakeholders, and requires accommodating multiple and conflicting objectives. Biosurveillance evaluation, the initiating step towards biosurveillance strengthening, is a multi-dimensional decision problem that can be properly addressed via multi-criteria-decision models.Existing evaluation frameworks tend to focus on “hard” technical attributes (e.g. sensitivity) while ignoring other “soft” criteria (e.g. transparency) of difficult measurement and aggregation. As a result, biosurveillance value, a multi-dimensional entity, is not properly defined or assessed. Not addressing the entire range of criteria leads to partial evaluations that may fail to convene sufficient support across the stakeholders’ base for biosurveillance improvements.We seek to develop a generic and flexible evaluation framework capable of integrating the multiple and conflicting criteria and values of different stakeholders, and which is sufficiently tractable to allow quantification of the value of specific biosurveillance projects towards the overall performance of biosurveillance systems.

Objective

To describe the development of an evaluation framework that allows quantification of surveillance functions and subsequent aggregation towards an overall score for biosurveillance system performance.

Malaria is a parasitic disease caused by Plasmodium falciparum. About 3.2 billion people worldwide are at risk of malaria. Children and pregnant women are particularly vulnerable to the disease. Sub- Saharan Africa carries a high share of the global malaria burden. Effective malaria surveillance system is essential in the control and elimination of malaria. Worldwide, there were an estimated 198 million cases of malaria in 2013 and 584,000 deaths. 

Objective

To describe the process of operation of the system and assess its key attributes, to determine the effectiveness and efficiency of the surveillance system and make appropriate recommendations to stakeholders for its improvement. 

USDA-APHIS-VS utilizes several continuous data streams to increase our knowledge of animal health and provide situational awareness of emerging animal health issues. In addition, USDA- APHIS-VS often conducts pilot projects to see if regular data access and analysis are feasible, and if so, if the information generated is useful. Syndromic surveillance was developed for three goals: a syndromic monitoring system to identify new diseases, as an emerging disease early warning system, and to provide situational awareness of animal health status. Current efforts focus on monitoring diverse data, such as laboratory accessions or poison center calls, grouped into syndromic or other health indicator categories, and are not intended to identify specific pre-determined diseases or pathogens. It is essential to regularly evaluate and re-evaluate the effectiveness of our surveillance program. However, there are difficulties when using traditional surveillance evaluation methods, since the objectives and outcomes of monitoring novel data streams from pilot projects are not easily measurable. An additional challenge in the evaluation of these data streams is the identification of a method that can adapt to various context and inputs to make objective decisions. Until recently, assessment efforts have looked at the feasibility of regular analysis and reporting, but not at the utility of the information generated, nor the plausibility and sustainability of longer term or expanded efforts. 

Objective

To implement a systematic and uniform approach to evaluating data sources for syndromic surveillance within the United States Department of Agriculture (USDA) Animal and Plant Health Inspection Services (APHIS) Veterinary Services (VS) group. 

The Centers for Disease Control and Prevention (CDC) uses the National Poison Data System (NPDS) to conduct surveillance of calls to United States poison centers (PCs) to identify clusters of reports of hazardous exposures and illnesses. NPDS stores basic information from PC calls including call type (information request only or call reporting a possible chemical exposure), exposure agent, demographics, clinical, and other variables.

CDC looks for anomalies in PC data by using automated algorithms to analyze call and clinical effect volume, and by identifying calls reporting exposures to pre-specified high priority agents. Algorithms analyzing call and clinical effect volume identify anomalies when the number of calls exceeds a threshold using the historical limits method (HLM). Clinical toxicologists and epidemiologists at the American Association of Poison Control Centers and CDC apply standardized criteria to determine if the anomaly is a potential incident of public health significance (IPHS) and then notify the respective health departments and PCs as needed. Discussions with surveillance system users and analysis of past IPHS determined that call volume-based surveillance results in a high proportion of false positive anomalies. A study assessing the positive predictive value (PPV) of this approach determined that fewer than four percent of anomalies over a five-year period were IPHS.1 A low PPV can cause an unnecessary waste of staff time and resources. We hypothesized that first stratifying call volume by exposure category would reduce the number of false positives. With the help of medical toxicologists, we created 20 toxicologically-relevant exposure categories to test this hypothesis. 

Objective

Our objective was to determine if the detection performance of current surveillance algorithms to detect call clusters is improved by stratifying by exposure category. 

The Department of Defense conducts syndromic surveillance of health encounter visits of Military Health System (MHS) beneficiaries. Providers within the MHS assign up to 10 diagnosis codes to each health encounter visit. The diagnosis codes are grouped into syndrome and sub-syndrome categories. On October 1, 2015, the Health and Human Services-mandated transition from ICD- 9-CM to ICD-10-CM required evaluation of the syndrome mappings to establish a baseline of syndrome rates within the DoD. The DoD data within the BioSense system currently utilizes DoD ESSENCE syndrome mappings. The Master Mapping Reference Table (MMRT) was developed by the CDC to translate diagnostic codes across the ICD-9-CM and ICD-10-CM encoding systems to prepare for the transition. The DoD ESSENCE and MMRT syndrome definitions are presented in this analysis for comparison. 

Objective

The transition from ICD-9-CM to ICD-10-CM requires evaluation of syndrome mappings to obtain a baseline for syndromic surveillance purposes. Two syndrome mappings are evaluated in this report. 

To obtain feedback and seek future directions for an ISDS initiative to establish and update research questions in Informatics, Analytics, Communications, and Systems Research with the greatest perceived impact for improving surveillance practice.