Evaluation

T-Cube is especially useful for rapidly retrieving responses to ad-hoc queries against large datasets of additive time series labeled using a set of categorical attributes. It can be used as a general tool to support any task requiring access to such data. From the application’s perspective it is transparent: it acts just like the database itself, but an incredibly quickly responding one. The authors had a chance to put T-Cubes into practical use as an enabling technology in applications requiring massive screening of multidimensional temporal data. These applications include two systems to support monitoring of food and agriculture safety and predictive analytics developed at the US Department of Agriculture and the Food and Drug Administration, as well as a system to monitor and forecast health of a fleet of aircraft operated by the US Air Force.

Objective

T-Cube, a data structure designed to efficiently represent large collections of temporal data has been shown to benefit surveillance applications involving monitoring sales of over-the-counter medications and emergency department visits. In this paper we present efficiencies which can be realized in practical applications of T-Cube beyond its original areas of deployment, and we advocate a widespread use of it as a technology which makes manual ad-hoc lookups as well as many kinds of complex automated analyses feasible.

One of the challenges facing developers and users of automated disease surveillance systems is being able to accurately evaluate the performance of their systems for the wide variety of public health threats that are possible. A variety of methods have been used in the past to create data sets for use in testing algorithm performance. Synthetic data has been created using agent-based simulations where data is created based on the hypothesized activity of individuals with contagious diseases. This data is only as accurate as the social models and variety of assumptions which must be made permit. Real data containing elevated levels of respiratory and gastrointestinal activity have been used to evaluate the ability of algorithms to detect the elevated levels. Routine unvalidated outbreaks are typically not public health emergencies and may not represent signals of interest. Another approach is to use real background data and inject a variety of different types of synthetic cases representing various types of outbreaks on top of that background.

With the introduction of the American Health Information Community (AHIC) Minimum Data Set (MDS), the public health surveillance community should have the potential to obtain greater specificity for alerts generated in automated systems. The introduction of these additional data elements increases the complexity of algorithms using linked data elements. Creating synthetic data sets that accurately estimate relationships among chief complaint, pharmacy, laboratory and radiology is an added complexity in creating synthetic outbreaks for performance evaluation.

Objective

The objectives of this presentation are to describe the need for synthetic data containing the elements of the AHIC MDS. Approaches for creating synthetic data with MDS data elements will be presented and methods for insuring maintenance of confidentiality will be discussed.

In the fall of 2001, the Bioterrorism Preparedness and Response (BT P&R) Unit initiated a syndromic surveillance system utilizing chief complaint data collected from Emergency Departments throughout Los Angeles County (LAC). Chief complaint data were organized into four syndromes (gastrointestinal, neurological, rash and respiratory) based on key words/phrases that appear in the patient’s record. Syndrome data are analyzed daily; counts for each syndrome are calculated and compared to a threshold to determine if a “signal” or aberration has occurred (EARS algorithm). A signal is defined as a case count elevated above threshold for a particular syndrome at an individual hospital.

Objective 

To describe the methods used by LAC, Department of Health Services, BT P&R Unit in determining the response to unusual disease/syndromic activity in LAC hospitals.

In 2007-2008, the authors surveyed public health officials in 59 state, territorial, and selected large local jurisdictions in the United States regarding their conduct and use of syndromic surveillance. Fifty-two (88%) responded, representing areas comprising 94% of the United States population. Forty-three (83%) of the respondents reported conducting syndromic surveillance for a median of 3 years (range = 2 months to 13 years). Emergency department visits were the most common data source, used by 84%, followed by outpatient clinic visits (49%), over-the-counter medication sales (44%), calls to poison control centers (37%), and school absenteeism (35%). Among those who provided data on staffing and contract costs, the median number of staff dedicated to alert assessment was 1.0 (range 0.05 to 4), to technical system maintenance 0.6 (range zero to 3); and, among the two-thirds who reported using external contracts to support system maintenance, median annual contract costs were $95,000 (range = %5,500 to $1 million). Respondents rated syndromic surveillance as most useful for seasonal influenza monitoring, of intermediate usefulness for jurisdiction-wide trend monitoring and ad hoc analyses, and least useful for detecting typical community outbreaks. Nearly all plan to include syndromic surveillance as part of their surveillance strategy in the event of an influenza pandemic. Two thirds are either "highly" or "somewhat" likely to expand their use of syndromic surveillance within the next 2 years. Respondents from three state health departments who reported they did not conduct syndromic surveillance noted that local health departments in their states independently conducted syndromic surveillance. Syndromic surveillance is used widely throughout the United States. Although detection of outbreaks initially motivated investments in syndromic surveillance, other applications, notably influenza surveillance, are emerging as the main utility.

In Connecticut, several syndromic surveillance systems have been established to detect and monitor potential public health threats: 1) the hospital admissions syndromic surveillance (HASS) system in 2001; and 2) the emergency department syndromic surveillance (EDSS) system in 2004. For the HASS, hospitals manually categorize unscheduled admissions into 11 syndrome categories and report these aggregate counts through an internet-based system daily to DPH; all 32 hospitals participate. For the EDSS, hospitals electronically report deidentified emergency department chief complaint data to DPH, and using a computerized algorithm, DPH categorizes this data into 8 syndrome categories; currently 17 hospitals participate. As part of pandemic influenza planning, there has been an increased focus on situational awareness at the state and national level; Connecticut would likely rely on these two systems for this purpose.

Objective

To evaluate the performance of the HASS and EDSS systems in reflecting seasonal influenza activity in Connecticut and, thus, their possible utility during a pandemic.

In October 2006, the Centers for Disease Control and Prevention funded four institutions, including Emory University, to conduct evaluations of the BioSense surveillance system. These evaluations include investigations of situations that represent actual or potential threats to public health in order to describe: 1) the pathways that health departments follow to assess and respond to such threats, 2) the role of various forms of surveillance, including BioSense and other syndromic surveillance systems, in enabling health departments to achieve critical milestones along these pathways, and 3) whether and how surveillance information informs healthcare practice during these events. We anticipate that these case studies will 1) identify approaches to improving BioSense and other syndromic surveillance systems, 2) describe the characteristics of events where syndromic surveillance is most apt to be useful, and 3) provide a baseline for assessing future impacts of advances in the development of BioSense and other forms of public health surveillance. This paper describes preliminary observations from initial case studies conducted by the Emory University team.

Objective

This paper describes preliminary observations from case study investigations of the uses of BioSense and other surveillance resources in public health practice.

On December 14th, 2006, a severe windstorm in western Washington caused hundreds of thousands of residents to lose power. On December 15, 2006, there was a surge in emergency department (ED) visits for patients presenting with signs of acute carbon monoxide (CO) poisoning. A Public Health investigation was initiated following the storm to determine the extent of CO poisoning due to the windstorm. A retrospective analysis was later undertaken to evaluate how well our syndromic surveillance system was able to identify patients who presented to area EDs with carbon monoxide poisoning.

Objective

We evaluated the performance of our ED syndromic data for detecting visits associated with CO poisoning.

Irregularly shaped cluster finders frequently end up with a solution consisting of a large zone z spreading through the map, which is merely a collection of the highest valued regions, but not a geographically sound cluster. One way to amenize this problem is to introduce penalty functions to avoid the excessive freedom of shape of z. The compactness penalty K(z) is a function used to reduce the scan value of irregularly shaped clusters, based on its geometric shape. Another penalty is the cohesion function C(z), a measure of the absence of weak links, or underpopulated regions within the cluster which disconnect it when removed. It was mentioned in that such weak links could be responsible for a diminished power of detection in cluster finder algorithms. Methods using those penalty functions present better performance. The geometric  compactness is not entirely satisfactory, although, because it has a tendency to avoid potentially interesting irregularly shaped clusters, acting as a low-pass filter. The cohesion function penalty method, although, has slightly less specificity.

Objective

We introduce a novel spatial scan algorithm for finding irregularly shaped disease clusters maximizing simultaneously two objectives: the regularity of shape and the internal cohesion of the cluster.

The Ontario Telehealth Telephone Helpline (henceforth referred to as “Telehealth”) was implemented in Ontario in 2001. It is administered by Clinidata, a private contractor hired by the Ontario Ministry of Health and Long-Term Care, 24 hours a day, 7 days a week, including holidays, at no cost to the caller. The calls are answered by registered nurses in both official languages from four calling centres that use identical decision rules (algorithms) and store all call information into one centralized data repository. The calls are usually approximately 10-minutes, patient based, and are directed by a nurse-operated electronic clinical support system.

Objective

Following the lead established by the UK’s NHS Direct Syndromic Surveillance system as well as the SARS Report’s desire to “broaden the information collection capacity of Telehealth as a syndromic surveillance tool,” we are retrospectively evaluating the value of Ontario’s Telehealth’s health helpline as a syndromic surveillance system. To date, there have been no published descriptions of Telehealth. This article endeavours to address this lacuna.

Developing and evaluating outbreak detection is challenging for many reasons.  A central difficulty is that the data the detection algorithms are “trained” on are often relatively short historical samples and thus do not represent the full range of possible background scenarios.  Once developed, the same dearth of historical data complicates evaluation.  In systems where only a count of cases is provided, plausible synthetic data are relatively easy to generate.  When precise location data is available, simple approaches to generating hypothetical cases is more difficult.

Advances in epidemiological modeling have allowed for increasingly realistic simulations of infectious disease spread in highly detailed synthetic populations. These agent-based simulations are capable of better representing real-world stochastic disease transmission process and thus show highly variable results even under identical initial conditions. Due to their ability to mimic a wide range outcomes and more fully represent the unknowns in a system, models of this class have become increasingly used to help inform decisions about public policies about hypothetical situations (eg pandemic influenza [1]).  This characteristic also makes them a powerful tool to represent the processes that create surveillance information.

Objective

Developing and evaluating detection algorithms in noisy surveillance data is complicated by a lack of realistic noise, meaning the surveillance data stream when nothing of public health interest is happening. These jobs are even more complex when data on the precise location of cases is available. This paper describes a methodology for plausible generation of such noise using agent-based models of infectious disease transmission based on highly resolved dynamic social networks.