Evaluation of Syndromic Surveillance

In November 2002 a NATO summit meeting issued an initiative calling for member states to begin development of an interoperable disease surveillance system that had the ability to give early warning in the event of an attack on armed forces using weapons of mass destruction. In response, the French military have developed the “Projet de Surveillance Spatiale des épidémies au Sein des Forces Armées en Guyane” (2SE FAG), a prototype real-time syndromic surveillance system based on fever case reporting which has been in operation among armed forces personnel in French Guiana since October 2004. Between January and June 2006, French Guiana experienced the largest epidemic of dengue fever in its history. During that year, 2255 confirmed cases and many thousands more suspected cases were recorded among the civilian population. 2SE FAG issued an alert based on a rise in fever cases among armed forces personnel in week 2 of 2006, 5 weeks before a rise was noticed among the civilian population. Limited evaluations of the system have taken place in the past; this study represents a final evaluation of the system before its possible expansion.

Objective

The objective of this study was the evaluation of the syndromic surveillance system 2SE FAG which operates among armed forces personnel in French Guiana using the “Framework for Evaluating Public Health Surveillance Systems for Early Detection of Outbreaks,” published by CDC.

Rhode Island implemented the Real-time Outbreak and Disease Surveillance (RODS) system, developed in 1999 by the University of Pittsburgh’s Center for Biomedical Informatics. This system is based on real-time information from hospital emergency departments that is transmitted and analyzed electronically for the purpose of early detection of and situational awareness for public health emergencies. Through this system, chief complaint is reported in real-time. Diagnoses, coded in the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), are reported to the RI RODS system as they become available. Three hospitals are currently participating in a pilot implementation of the RI RODS system.

Preliminary work by a CDC Working Group (CDCWG) developed recommendations for syndrome definitions for use in syndromic surveillance programs. Ten syndromes, based on ICD-9-CM diagnosis codes, identified diseases associated with critical bioterrorism-associated agents or indicative of naturally occurring infectious disease outbreaks. As a component of the evaluation of the RI RODS system, we evaluated the RI RODS chief complaint classifier (CoCo) using ICD-9-CM codes and the CDCWG work as the gold standard.

Objective

This paper presents findings related to the evaluation of the CoCo used in the pilot implementation of a syndromic surveillance system in Rhode Island.

Efforts have been made to standardize and prioritize the description and evaluation of syndromic surveillance systems. Systematic information on the performance of existing systems can be used to assess and compare the value of these systems, and inform decisions regarding their use. 

The Michigan’s Emergency Department Syndromic Surveillance System (MSSS) is an implementation of an early version of the Real-time Outbreak and Disease Surveillance system developed by the University of Pittsburgh, which collects patient chief complaint data from emergent care facilities in real time. At the Michigan Department of Community Health the system has been in use since 2003. Alterations to the system and recruitment of data contributors have been ongoing. The primary stated purpose of the MSSS is earlier detection of outbreaks of severe illness, enabling a more rapid public health response and intervention to reduce the impact of public health threats.

Objective

This work describes key characteristics of MSSS and reports on its evaluation.

Immediately following September 11, 2001, the District of Columbia Department of Health began a syndromic surveillance program based on emergency room (ER) visits. ER logs are faxed on a daily basis to the health department, where health department staff code them on the basis of chief complaint, recording the number of patients in each of the following syndromic categories: death, sepsis, rash, respiratory complaints, gastrointestinal complaints, unspecified infection, neurological, or other complaints. These data are analyzed daily and when a syndromic category shows an unusually high occurrence, a patient chart review is initiated to determine if the irregularity is a real threat. 

A time series analysis of the data from this system has shown that with the application of a variety of detection algorithms, the syndromic surveillance data does well in identifying the onset of the flu season. In addition, simulation studies using the same data have shown that over a range of simulated outbreak types, the univariate and multivariate CUSUM algorithms performed more effectively than other algorithms. The multivariate CUSUM was preferred to the univariate CUSUM for some but not all outbreak types.

Objective

This paper evaluates an ER syndromic surveillance system based on simulation studies and comparisons with other surveillance systems.

CDC’s BioSense system provides near-real time situational awareness for public health monitoring through analysis of electronic health data. Determination of anomalous spatial and temporal disease clusters is a crucial part of the daily disease monitoring task. Spatial approaches depend strongly on having reliable estimated values for counts among the geographic sub-regions. If estimates are poor, algorithms will find irrelevant clusters, and clusters of importance may be missed. While many studies have focused on improved computation time and more general cluster shapes, our effort focused on finding anomalies that are correct according to available BioSense data history.

Objective

We applied spatial scan statistics to data from CDC’s BioSense system and examined the effect of the spatial prediction method on determination of anomalous disease clusters. The objectives were to decide on a reliable spatial estimation method for one BioSense data source and to establish criteria for making this decision using other sources.

The Real-time Outbreak and Disease Surveillance system collects chief complaints as free text and uses a naïve Bayesian classifier called CoCo to classify the complaints into syndromic categories. CoCo 3.0 has been trained on 28,990 manually clas-sified chief complaints. The free text chief com-plaints are challenging to work with, due to problems caused by linguistic variations such as synonyms, abbreviations, acronyms, truncations, concatenations, misspellings and typographic errors. Failure to correct these word variations may result in missed cases, thereby decreasing sensitivity of detection.

Objective

To determine whether preprocessing chief complaints before automatically classifying them into syndromic categories improves classification performance.

In 2004, the Indiana State Department of Health (ISDH) contracted with the Regenstrief Institute to build an information exchange infrastructure to support the collection of surveillance data. This pilot program involves implementation of electronic reporting in 46 of the state’s 114 emergency departments. Chief complaint data are collected and analyzed to identify clusters of disease earlier than a diagnosis can be confirmed or the disease reported to the ISDH. The system utilized the chief complaint coder CoCo to map the chief complaints into one of eight syndromes. This evaluation was completed after one-third of the pilot facilities were operational.

Objective

This evaluation was conducted to determine if any pilot hospitals have operational practices that may affect the ability of the Public Health Emergency Surveillance System to accurately and efficiently identify clusters of infectious disease in Indiana.

While traditional means of surveillance by governments, multi-national agencies, and institutional networks assist in reporting and confirming infectious disease outbreaks, these formal sources of information are limited by their geographic coverage and timeliness of information flow. In contrast, rapid global reach of electronic communication has resulted in the advent of informal sources of information on outbreaks. Informal resources include discussion sites, online news media, individual and organization reports and even individual search records. The earliest descriptions of the severe acute respiratory syndrome outbreak in Guangdon Province, south China came from informal reports. However, system development to date has been geared toward knowledge management and strategies for interpreting these data are underdeveloped. There is a need to move from simple knowledge reorganization to an analytic approach for disseminating timely yet specific signals.

Objective

Internet-based resources such as discussion sites and online news sources have become invaluable sources for a new wave of surveillance systems. The WHO relies on these informal sources for about 65% of their outbreak investigations. Despite widespread use of unstructured information there has been little, if any, data evaluation.

Syndromic surveillance aims to decrease the time to detection of an outbreak compared to traditional surveillance methods. Emergency department (ED) syndromic surveillance systems vary in their methodology and complexity and are usually based on presenting chief complaints. Prior work in ED-based syndromic surveillance has shown conflicting results on agreement between chief complaint and discharge diagnosis, which may be syndrome-dependent. The use of ED discharge diagnosis may improve surveillance validity if it can be done in a timely fashion.

Objective 

The purpose of this study is to characterize the relationship of emergency department chief complaint and final primary ICD-9 diagnosis assigned at the time of emergency department disposition for patients with symptoms and/or ICD-9 codes associated with influenza like illness (ILI) using an electronic medical record.

This study aims to evaluate the sensitivity, specificity and Positive Predictive Value (PPV) of body temperature measurements > 100.5 žF in relationship to laboratory confirmation of influenza and other ILI pathogens.