The Environmental Sample Processor (ESP) Software Design: Software for Detection and Quantification of Microorganisms Danelle E. Cline, Thomas C. O’Reilly, Timothy Meese, Brent Roman, and Duane R. Edgington Monterey Bay Aquarium Research Institute 7700 Sandholdt Road Moss Landing, CA 95039 USA Abstract- The Environmental Sample Processor (ESP) instrument has been designed by the Monterey Bay Aquarium Research Institute (MBARI) for ocean sampling and monitoring. The ESP is an in situ sampling and processing device that enables near real-time detection of specific microorganisms through the application of molecular probes. The intended use of ESP is a 1 to 3 month deployment in 50 meters maximum depth for detection of harmful algal blooms. In this paper, we present an overview of the software architecture deployed on the ESP instrument. The ESP software design is applied on two prototype instruments with similar mechanical design, but different control electronics. Presented in this work is the software architectural framework used that allows for controlled start up, shutdown, task and event handling in a concurrent software environment. We discuss how object-oriented design patterns such as the Adapter pattern are used to solve design problems and how testing improved reliability. A description and examples are given of the flexible ESP macro language that allows scientists to automate chemical processing steps. And finally, an algorithm for DNA probe array image registration and data extraction involving low-pass filtering, connected components, rotational translation, and component recognition and interpretation is presented. INTRODUCTION The application of molecular probe assays to detect harmful algal bloom (HAB) species typically involve highly repetitive operations, require trained personnel, and are often difficult to perform outside the laboratory. It is highly desirable to have an instrument designed to collect samples and automate the application of probe assays in situ. Such an instrument used in conjunction with existing in situ chemical, physical, or optical sensors, would dramatically improve our ability to detect HABs within their biological context [1]. The impetus for building the Environmental Sample Processor (ESP) was to build a sampling device to enable in situ analysis, because an instrument with such analysis capabilities simply did not exist. Our target design for the ESP was to automate HAB detection. However, by adapting the ESP operation through its macro language, the ESP can be used for a wide range of water quality monitoring (e.g. pesticides, E. coli, etc.). The ESP was designed and built by a team of engineers at the Monterey Bay Aquarium Research Institute (MBARI) in conjunction with the molecular biology lab led by Dr. Chris Scholin. The ESP device and software are protected under a U.S. patent [2]. This paper presents an overview of the software architecture deployed on the ESP instrument. We first give a technical overview of the ESP instrument and operation. We then discuss the software architecture, ESP macro language, and design challenges. Lastly, we give an overview of the algorithm for processing filter images. II. TECHNICAL OVERVIEW A. Shipboard and Embedded ESP MBARI has fabricated two complete ESP instruments; one unit is for shipboard use and the other is an embedded unit for in situ use. The first ESP is a rugged, rack-mounted shipboard unit intended for short duration cruises to aid traditional techniques of sampling and processing for HAB detection. The shipboard unit was also intended to be a hands-on prototype the MBARI science and engineering staff could use to determine what worked and what didn’t. The second ESP built is an embedded instrument built for in situ use on a moored platform. All the design refinements from the shipboard unit are rolled into the embedded ESP, making the embedded ESP nearly identical mechanically to the shipboard ESP. B. ESP Operation The fundamental operation of the ESP is to autonomously filter discrete volumes of seawater through a filter membrane and to automate the application of DNA (or other molecular) probes to identify and quantify the captured microorganisms. Reference [1] gives further details on the ESP concept and chemistry that drove its design. (Please note that ESP is referred to as “Genosensor” in [1].) For real-time species detection using ESP, cells are broken open and their contents washed over species- Figure 2. Examples of the detection of the toxic, pennate diatom Pseudo-nitzschia australis with the shipboard ESP. The presence of the diatom results in the emission of light. P.australis was not detected in sample 1, which was collected in an area of upwelling, but was detected in sample 4, which was collected where the water was warmer. Both samples were taken in the Monterey Bay. Samples were collected for later whole cell analyses. Panel 1a was treated to show all species in the sample and sample 4a was treated with a probe that labels P.australis specifically. [1,2] Figure 1. Left, Environmental Sample Processor, Right, ESP enclosed in housing for deployment specific ribosomal RNA (rRNA)-targeted DNA probes. These probes are arrayed in a matrix of dots on the filter membrane. Subsequent processing steps causes these dots to fluoresce or luminesce, where each dot's intensity is proportional to a specific organisms abundance in the sample. Filter membranes are imaged with a charge coupled device (CCD) camera to produce an image similar to that in Fig. 2. An image-processing algorithm is then applied to the image to produce a report of presence and abundance of targeted organisms. C. Hardware Details The ESP is an electromechanical device featuring six motor-mechanical assemblies, a heating system, 28 valves, a cooled-CCD camera, and a 25 ml motor driven syringe for metering reagents and samples. Motion System There are six motion axes altogether: two filter elevators, a carousel, shuttle, filter clamp, and syringe. Each motor assembly consists of a lead screw assembly attached to small DC motors with integral shaft encoders. Each axis has a forward and reverse limit switch used to halt motion at the end of travel. Motors are controlled with a pulse width modulated control loop using incremental encoder feedback. Five out of the six motion assemblies are dedicated to positioning custom designed filter holders within a clamp assembly. The remaining motion assembly is a syringe plunger mounted on a lead screw, used for metering reagents and as a sample pump. Imaging System The ESP camera is a Santa Barbara Instruments Model ST-8E CCD designed for astronomy use. A Peltier junction cools the 1530 x 1020 pixel CCD detector to typically 10 to -17 degrees Celsius. Filter images are acquired with up to 5 minute exposure times. The output of the imaging software is a standard 16-bit Monochrome Tag Image File Format (TIFF) stream. The TIFF files are stored on the instrument's flash disk and may be uploaded to a client computer through a communication link. Housekeeping Sensors To detect leaks or electrical ground to seawater, the ESP includes a humidity sensor and ground fault circuitry. If humidity becomes abnormally high or a ground fault occurs, the system immediately shuts down. Heater System The filter holder containing the filter membrane is clamped between two film heaters. The temperature of the filter holder is sensed by an infrared detector operating at a distance of approximately 0.75 inches from the filter. A PID loop executing at 1 Hz in software uses the infrared sensor as feedback to maintain temperature within 1 degree of desired set point. User Interface Application Logic Protocol Layer (TCP-IP) Client Computer Power Management The ESP instrument transitions to a low-power state between samples. In the low-power state, the entire PC104 stack is powered off, except a timer circuit on the power supply board. During normal operation, the ESP powers itself on at a user programmed GMT time. Upon wakeup, pre-programmed macro files are loaded and executed in sequence. A typical sample takes 2 to 6 hours to process. To minimize heat and power dissipation within the enclosure, the entire CCD camera is powered on only 7 minutes before an exposure is taken to allow just enough time for the Peltier junction to cool the CCD. To further minimize power, motors are powered on only when in motion. Communications Two communications paths are built into the ESP design: a wireless and a direct path. For mooring communications, a wireless SLIP connection is made over a pair of wireless data transceivers (Freewave model DGR115W radio modems). This data link provides up to 115 KBaud throughput across a 20 mile range. For testing or pre-deployment configuration, connection is made with the PC-104 Ethernet adapter. Ethernet RF Modem ESP Instrument PC-104 Stack To speed up system development and meet space constraints, the ESP design includes nine PC-104 compliant electronics cards. The CPU module features a 80486DX4 CPU operating at 66MHz, 52 MB RAM, a 128MB compact flash disk, serial and parallel ports, floppy interface, IDE interface, keyboard, and clock. A standard Ethernet card is used as the link for software development and for instrument test and configuration. The remaining off-the-shelf PC-104 modules include a vehicle grade power supply, three servo control boards for motion control, two solid state relay cards to optically isolate and switch valves, and a multifunction A/D card for system sensors. One custom PC-104 form factor card was built for power switching, sensor conditioning and ground fault detection. Miscellaneous wiring harnesses and breakout boards provide the glue to the PC-104 stack. The operating system used is VxWorks 5.4 and software is developed using the Tornado 2.0 cross development environment. Transport Layer (SLIP) Transport Layer (SLIP) Protocol Layer (TCP-IP) Application Framework Macros and Macro Interpreter Alarm Handler Instrument Control Logic Data Logger Hardware Adapation Layer Operating System Adaptation Layer Operating System (NT/VxWorks) Figure 3. ESP Software Layers III. SOFTWARE ARCHITECTURE The ESP software is an object-oriented design, largely written in C++ and operates on the real-time operating system VxWorks version 5.4 (embedded version). In the classical sense of a real-time system, the ESP system is considered a ‘soft’ real-time system. At the highest layers of the software architecture (Fig. 3) are the user interface and communication protocols. Here, a simple menu based user interface communicates to the instrument through a wireless SLIP connection or a direct Ethernet connection. At the core of the ESP design is a software framework that allows for controlled startup, shutdown, task and event handling in a concurrent software environment. The application framework is a collection of classes built for instrument design. This framework was an in-house tested design we used to speed up development. Controlled by the framework is a set of classes that define the ESP macro language. The ESP language is a simple ASCII based language used to execute common algorithms that control the ESP. It gives flexibility to the science user by allowing lab-based chemical processing steps to be described with a simple text editor. Also controlled by the framework is the collection of classes that compose the instrument control logic. Control logic is invoked through callback functions called by a main event loop in the application framework. Figure 4. Adapter design pattern class structure Outside the framework are classes for alarm handling and data logging. These classes manage collecting and organizing error and status messages for display and storage. Beneath the application layer is the hardware adaptation layer. This adaptation layer is where we isolate the hardware-dependent parts of the program from the hardware-independent parts. This allows the ESP software to function on two different sets of control electronics, one for the shipboard and the other for the embedded ESP. Building the adaptation layer did require extra effort, yet allowed us to minimize the future overhead of maintaining two programs. Beneath the hardware adaptation layer is the operating system adaptation layer. Adding the operating system adapter layer allowed us to support both shipboard and embedded ESP operating systems (Windows NT and VxWorks respectively). A. Design Strategy Design for Changing Requirements It is critical that the software design adapt to incremental changes without requiring major software redesign. Rather than approach the design in terms of function, an objectoriented design strategy is used in the ESP. Given that we had a small design team, we needed to respond to changes without requiring significant rework. A good objectoriented design allowed us to change individual object representations without changing the entire design. Incorporate Software Reuse and Object-Oriented Design Patterns Software design patterns and reusable object-oriented designs are used where possible throughout the entire ESP design. By reusing previously tested designs, we shortened our development time and reduced maintenance time. The Application Framework, Data Logger and Alarm Handler are all examples of a software reuse in the ESP software design. Figure 5. Motor Adapter class structure By using design patterns we created a more flexible and elegant design. A design pattern can be described as the core solution to a general problem that occurs over and over again. In the software context, a software design pattern is a “description of communicating objects and classes that are customized to solve a general design problem in a particular context [3]. ” The intention of a software design pattern is to identify classes and class collaborations, not to detail the concrete solution. After close inspection, we found many applications for object-oriented design patterns including the Adapter, Factory, Command, and Singleton. A detailed description of the software patterns used in the ESP design can be found in [3]. Describing all the design pattern use in the ESP is beyond the scope of this paper. However, we will present one pattern – the Adapter pattern - to describe its use in the context of a real-world application. The Adapter pattern is used in seven different components in the ESP design. Basically, this pattern is applied when an interface doesn’t match the one needed. The Adapter provides one common interface for a client class to use. This was precisely the problem we had with incompatible hardware and operating system interfaces. We needed one common interface to access two sets of operating systems and two different hardware platforms (shipboard and embedded). The structure of the Adapter pattern in Fig. 4 is described in the Unified Modeling Language (UML) [4]. In the Adapter pattern, A Client class uses the adapter through a common Target interface. The Adapter adapts the interface of the Adaptee to its Target interface [3]. Fig. 5 shows the pattern applied to different motor driver interfaces. Notice in Fig. 5 that there are two participants to the adaptee - the AndiServo and CDMCWin classes. Also note that the implementation is slightly different from the textbook Adapter pattern. The Motor and MotorAdapter have a composition relationship (the Motor class is composed of a MotorAdapter object), rather than a Figure 6. Motor-Axis Class Heirarchy subclassed relationship (the Adapter subclasses the Target). B. Object Identification In the evolution of the ESP design we discovered that the real art in object-oriented design is identifying objects. Fortunately, there was a clear mapping between the ESP system and controlling objects, making the process of object identification relatively straightforward. However, this did not preclude the object (class) designs from evolving from their original design concept as is normal in any object-oriented design. As the ESP instrument evolved inheritance hierarchies and object structure were reviewed and revised accordingly. The process of identifying objects in the ESP design started with the ESP project statement of work document [7]. By identifying objects within the document described in the form of nouns (syringe, carousel, macro, valve manifold, etc.) we determined the system objects. By identifying verbs within document, we defined the operations and interactions between objects. For example, the software interprets a macro file, or the carousel rotates to align tubes. Objects were refined further by using UML drawings and listing out operations and attributes of each object. During this process, common factors between objects became evident, resulting in a more refined inheritance structure. The resultant design was a decomposition of the software into parts that map either into concrete hardware systems or more abstract system functions. A good example of the resultant object decomposition is the hierarchy of the motor-axis subsystem (Fig. 6). All the common functions and variables for each axis are factored out into an Axis class. At the root of the tree is the Motor class that provides a common interface for all motor commands common to each axis. The concrete Figure 7. ESP Application Framework class structure Active/task classes are shown in bold boxes implementations (Carousel, Syringe, etc.), can override the behavior of the Motor or Axis class to provide unique implementations specific to their design. At this level, unique ‘find home’ algorithms or custom jogging behaviors are implemented. C. Application Framework Design The framework used in the ESP application is an instrument application framework. In the object-oriented context, a framework can be considered a semi-complete collection of classes, built for a particular type of application like a user interface, a database, or, in this case, an instrument design. The application writer simply subclasses and overrides a few virtual functions from the framework to create an application. The framework handles all the details like control and synchronization though a main event loop. The framework used in the ESP design is sometimes referred to as the “three-task” model because of the three tasks in this framework: MicroApp, MicroEventHandler, and MicroPeriodicHandler. The framework uses a concurrent design pattern called the half-synchronous, half-asynchronous design pattern. This pattern is “An architectural pattern that de-couples synchronous I/O from asynchronous I/O in a system to simplify programming effort with degrading execution efficiency [5].” This pattern allows us to separate the asynchronous event like an operator sending a command, from the synchronous activity like monitoring a sensor at a specified rate. The design can respond to these different I/O actions between the concurrent MicroEventHandler and MicroPeriodicHandler tasks within a reasonable reaction time1. Fig. 7 shows the class structure of the Application 1 A reasonable response is sufficient; this framework meets a soft realtime system requirement. Framework with subclassed ESP application classes. MicroApp is the main class, responsible initializing the framework and spawning both MicroEventHandler and MicroPeriodicHandler tasks. System shutdown and reset are also handled in MicroApp. MicroEventHandler is the task responsible for handling events, such as those associated with coordinated motion control. MicroPeriodicHandler is responsible for periodic sampling of sensors. TaskSynchronizer handles inter-task communication between all three tasks in the framework. The subclassed ESP application classes are responsible for creating all the application specific callbacks and instrument control logic. IV. ESP MACRO LANGUAGE The ESP macro language is a simple, ASCII based language for controlling the ESP. A wide range of sample processing protocols can be described with the macro language, enabling the ESP to execute a wide range of water quality monitoring applications. Each macro executes a common algorithm used to run the instrument. There are 15 total macro commands. TABLE 1 EXAMPLE MACRO USAGE Macro Description LOADFILTER Load filter from carousel tube 1 SYRINGEPULL 5.0 Sample Pull 5.0 ml through “Sample” port HEAT 40 5 Heat to 40°C for 5 minutes ACQUIREIMAGE DARK, 3X3, 3000, "P.Australis test filter" Acquire a dark image, 3x3 binning, 3 second exposure time, label the image, “P. Australis test filter” Macros are combined in an ASCII text file to form a macro program. An example portion of a macro program is shown below. # This macro represents the steps taken # to create lysate as part one of the # two step Sandwich Hybridization # process PROGRAM: SANDHYBE1Test35 SYRINGE PULL 5.0 Sample DELAYSEC 10 SYRINGE PUSH ALL Waste SYRINGE PULL 5.0 Setrinse DELAYSEC 10 . . . END Macro programs are executed through a simple user interface, where they can be started, stopped, or resumed. Macro programs may also invoke other macro programs. However, operators and primitives like for loops and if then blocks are not supported in the ESP macro language. V. DESIGN CHALLENGES Building software to control a complex instrument poses significant design challenges. From the beginning, ESP engineers knew the device must be reliable under repeated unattended operation and run autonomously. The foremost problem to address was building a reliable instrument for operation amidst the rigors of an ocean deployment. Fundamentally, the ESP requires flawless operation because it has no redundant systems to use in the presence of failures. Recovery from any failure (software failure, mechanical jam, etc.) is unlikely at sea. To address the question of reliability, significant effort went into testing for repeatable mechanical operation and reliable, fault-tolerant software design. A. Lab Testing For the ESP to be reliable, it had to repeatedly execute a sequence of macros that involved coordinated motion control of 5 axes, heat control, image acquisition, control and sequencing of 24 valves, precision metering of reagents/sample, and autonomous data download. During early development with the shipboard ESP, carrying out repeated macros without mechanical jams or software failures proved to be problematic. To work through these problems, laborious lab tests anywhere from 2 hours to 20 hours were performed, primarily to exercise ESP function using simulated science macros. These lab tests resulted in many redesigns of mechanical, electrical, and software systems. B. Unit and Module Testing Software testing included both unit and module testing for fault tolerance. Software testing on the ESP can be broken in to three elements: design testing, fault/error detection testing, and error recovery testing. Our first tests were to test for normal operation against the detailed design. For example, a test might be designed to test all transitions in the motor state machine design against its paper design. This phase generally revealed flaws in the design, making it extremely valuable. When a test passed against its detailed design, we moved forward to generating fault test cases. The challenge in generating fault test cases was to induce faults to see if they manifest themselves as an error. For example, in the case of the motion system, to generate test cases, we asked questions like, when a motor over- currents, does it generate the correct exception, and is the exception propagated correctly? Error recovery testing was the last element of testing, and proved to be the most difficult. Error recovery was only possible in a few cases. At best, we could log an error and perform retries. Most cases resulted in faults that required some form of manual intervention. to propagate errors and to gracefully shutdown the system. We also unexpectedly found exceptions produce easy to read programs. C. Exception Handling Extracting the data from the ESP filters required an image-processing scheme. Unlike traditional sensor based instrument based on optical, voltage or current measurements, the data generated from the ESP measurement must be extracted from an image. As described in section II-B, the resultant reaction causes a spot in a matrix array to fluoresce or luminesce. Therefore, the data of interest is the location and average intensity of each spot. The X Y location of the “dot” is used to indicate the type of organism and the “dot’s” intensity indicates microorganism abundance. To date, all image-processing work has been done with Matlab and the Matlab Image Processing Toolbox on commercial filter grids manufactured by Beckman. Our intention is to implement image processing on-board the in-situ ESP, on MBARI custom-designed filter probe arrays. Adding exceptions improved robustness of the overall design and allowed for a graceful degradation of the system. Two classes of exceptions were used in the ESP design: application exceptions and run-time exceptions. Examples of application exceptions include ‘motor over current’, ‘motion time out’, macro file exceptions (‘bad argument count’, ‘argument out of range’, etc.). Run-time exceptions include exceptions resulting from either programmer error or system error. For example, runtime exceptions include ‘memory allocation failure’ and ‘task spawn failure’. Careful considerations were weighed to decide where exceptions were placed and how exceptions were handled. Rather than taking a hit in code bloat by adding excessive exception checks, exceptions were added only in the highest levels of the design, as close to the caller as possible. Exceptions were not placed in driver level code. Here, we used return codes. This avoided the perils of propagating exceptions through many levels to the error handling code. Exceptions were only handled in well-defined blocks because it is generally considered good practice to handle specific exceptions and not group them in a ‘catch all’ block. A ‘catch all’ block could leave some exceptions not handled, resulting in an unstable system. The following is an example of the use of a defined exception block in the Syringe class runFindHome() function: try{ do { … run find home algorithm }while(); } // end try catch(Axis::TimeOut,timeout) { //handle error and return } catch(Motor::PositionError,poserr) { //handle error and return } catch(Motor::OverCurrent,overcurrent) { //handle error and return } In summary, we found adding exception handling improved system reliability and created a simple structure VI. IMAGE PROCESSING The ESP image processing can be broken into the three classical processing phases: Image Segmentation, Description, and Recognition and Interpretation [6]. These phases are described below. A. Image Segmentation Basically, image segmentation means divide (segment) the image into its parts. For the filter processing this requires separating the "dots" and alignment markers from the background. Segmentation is the most critical step in automating any processing scheme. What we found particularly difficult was segmenting “dots” with light intensities. Only by using a combination of a priori knowledge of the expected intensity and expected location of the features, can data be separated from the background. In addition, it may be necessary to use an adaptive approach that successively loops through the processing algorithm, incrementally moving threshold levels, until valid data is found. B. Image Description The second phase is to describe the segmented features by their characteristics. Like image segmentation, tuning the algorithms in this phase requires a priori information about the expected location and size of “dots” and alignment marker features. To describe features, a 4connected component algorithm is applied and feature pixel coordinates are grouped in respective arrays. The arrays are processed to calculate feature area, average intensity, width, height, eccentricity, center coordinates, and radial distance. To find potential “dot” features, a filter is applied against each feature area and height to width ratio. Potential “dot” features are then subtracted from the entire feature set. The remaining features are filtered based on area, eccentricity, and radial error, to find potential alignment markers. C. Image Recognition and Interpretation The third phase is to interpret the subset of potential data and alignment marks to find the true alignment marks and “dots”. This is accomplished by differentiating features based on their size, shape and location. Alignment marks are long and thin and within a defined ring of the filter. “Dots” are generally round and within a circular area centered on the filter. By applying a filter using these distinguishing features, X and Y markers and “dots” are found. Alignment marks are used to re-orient the image by rotating to a normalized XY coordinate system, and by normalizing to a XY coordinate system, reported data can be post processed more intelligently. A raw data report might resemble the following: 2 data found Location normalized relative to calculated center (468,335) Area Average Standard Intensity Deviation Alexandrium 90 2424 40 P.australis 107 2317 35 Acknowledgments The authors would like to thank MBARI employees, Dr. Chris Scholin, Roman Marin III, Gene Massion, Scott Jensen, Mark Brown, Kevin Sullivan, and Karen Salamy for their valuable ideas and participation in testing of the software design and ESP system. References [1] Scholin, C., Massion, G., Mellinger, E., Brown, M., Wright, D., Cline, D.,“The Development and Application of Molecular Probes and Novel Instrumentation for Detection of Harmful Algae”,Ocean Community Conference ’98 Proceedings, Marine Technology Society, vol. 1 pp. 367-370. [2] Scholin, C.A., Massion, E.I., Wright, D.K., Cline, D.E., Mellinger, E.,Brown, M. 2001. Aquatic Autosampler Device. US Pat. No. 6187530. [3] Gamma, E. Helm, R., Johnson, R., Vlissides, J., Design Patterns: Elements of Reusable ObjectOriented Software, Addison-Wesley, 1995. [4] Booch, G., Rumbaugh, J., Jacabson, I., The Unified Modeling Language User Guide, Addison-Wesley, 1999. [5] Schmidt, D., “Concurrent Object-Oriented Network Programming: Distributed object Computing with Corba, Java, and C++,” course notes, U.C. Berkeley University Extension, San Francisco, CA, pp 190, November 1998. [6] Gonzalez, R., Woods, R., Digital Image Processing, Addison-Wesley, 1993. [7] Massion, G., Scholin, C., Genosensor Statement of Work, MBARI internal document, January, 1998.
© Copyright 2024