Jason M. Eichenholz*a,b, Nick Barnetta, Yishung Juanga, Dave Fishb, Steve Spanoc, Erik Lindsleyd, Daniel L. Farkasd

aOcean Optics Inc., 4301 Metric Drive, Winter Park FL 32825
bOcean Thin Films Inc., 16080 Table Mountain Pkwy, Suite 100, Golden, CO 80403
cFinger Lakes Engineering, 15 West Main, Suite A, Dryden, NY 1305
dDepts. of Surgery and Biomedical Science, Cedars-Sinai Medical Center, Los Angeles, CA 90048

ABSTRACT
Spectral imaging involves capturing images at multiple wavelengths resulting in a data cube (x, y, ?) that allows materials to be identified by its spectral signature. While hyperspectral imagers can provide high spectral resolution, they also have major drawbacks such as cost, size, and the copious amounts of data in the image cube. Typically, the complete hyperspectral data cube provides little additional information compared to only 3-8 discrete (multiwavelength) imaging bands. We present two new approaches and related technologies where we are able to acquire spectral imaging data stacks quickly and cost-effectively. Our two spectral imaging systems represent different approaches integrated with standard CCD and CMOS imagers: sequential rotating filter wheels (RFWs) and lithographically patterned dichroic filter arrays (DFAs). The RFW approach offers the ability for rapid configuration of a spectral system, and a whole new level of self-contained image acquisition, processing and on-board display. The DFA approach offers the potential for ultra compact imagers with acquisition of images of multiple wavelengths simultaneously, while still allowing for processing and display steps to be built into the camera. Both approaches lend themselves production of multi-wavelength/spectral imaging systems with differing features and advantages.

Keywords: Spectral, Hyperspectral, Imaging, Multiwavelength, filter wheel, hemoglobin, patterned dichroic, filter array

1. INTRODUCTION
Over recent years there have been many advances in spectral imaging and as technology continues to evolve the broad range of imaging applications is increasingly becoming apparent. Spectral/hyperspectral and multiwavelength imagers have been used in studies ranging from relatively simple color measurements to more demanding chemical and biological imaging.

The interaction of light with tissue varies considerably across the electromagnetic spectrum1-2 and spectral imaging utilizes this as a powerful tool for biomedical discrimination and measurements. Careful selection of spectral bands enables the assessment of a wide range of tissue and blood chromophores such as oxy-hemoglobin, deoxy-hemoglobin and bilirubin, and the choice of wavelengths in the near infrared spectrum compared to the visible spectrum provide some added depth discrimination to measurements3. In addition spectral imaging has the advantage that it can be applied noninvasively which is useful for applications such as assessment of burns or inflammatory skin conditions4-6.

Single point spectroscopy is widely used in biomedical applications for monitoring changes in scattering, absorption and fluorescence at the cellular and tissue level that occur as tissue progresses from a normal to a diseased condition7-8. Spectroscopic measurements can be used to compare measurements from one location to another or to track chromophore concentrations at one location against time. It is a natural progression to then apply spectral imaging to similar medical conditions where it can provide both spectral and spatial information about the distribution of features, as well as allowing the observation of how these distributions alter with time.

Spectral/hyperspectral imaging (more comprehensive than multiwavelength imaging) is a particularly powerful research tool generating images over many discrete wavelengths and producing rich data cubes that can then allow specific chromophores to be identified by the spectral signature at each pixel.

This ability is especially useful in a research environment where the spectral regions of interest are not well defined or known a priori. However generating meaningful data from such large data cubes can be computationally intensive, and in many cases the complete hyperspectral data cube provides little additional information compared to just using a few (typically 3-8) carefully selected spectral bands. Once the spectral regions of interest have been identified it is much easier to limit the measurement to a few spectral bands and by taking this approach it is possible to produce spectral imagers that are smaller and more portable, less complex and computationally intensive, and more cost-effective.

Unfortunately there a very few commercially available spectral/hyperspectral imagers, and these systems tend to be oneoff or research-based, put together using off-the-shelf parts and cameras and can cost anywhere from tens of thousands to millions of dollars per system, depending on the application, wavelength bands, and spatial and spectral resolution requirements. A typical multiwavelength imager essentially consists of either discrete filter wheels, mechanically diced thin-film dichroic filters mounted in front of an image sensor, or multiple cameras with bulk dichroic filters. Even for those touted as commercial systems, there is no real volume production pathway with significant price or reduced complexity enhancements at even as few as tens or hundreds of units.

We have developed two new multispectral imaging systems, the sequential rotating filter wheel (RFW) and the Dichroic Filter Array (DFA), that can be mass produced in a cost-effective manner while still being easily configured to various applications. Both approaches have (different) trade-offs, but both can leverage existing hyperspectral data sets to determine and fabricate a few carefully selected wavebands to produce cost effective solutions for both R&D and commercial spectral imaging applications.



Figure 1. The two different multispectral imaging implementations are shown above. (a) shows a monolithic RFW with four 9 nm FWHM visible wavelengths and two clear channels for image stabilization and (b) shows the DFA concept which captures all wavelength simultaneously for increase throughput.

2. SEQUENTIAL ROTATING FILTER WHEEL (RFW) CAMERA
2.1 Filter Wheels
The concept of using individual thin film filters in front of a CCD or CMOS camera is not new. Researchers have been doing this for years using discrete round or square filters and an off the shelf or customized opto-mechanical assembly. This type of implementation provides a flexible low cost multispectral/multiwavelength imager for R&D studies, but it has little chance of being successfully deployed in a clinical environment.

The fundamental difference between those systems and our RFW camera is the rotational speed capability of the wheel and the minimal gap between filters, combined with on board image stacking and processing. Ocean Thin Films has extensive experience designing and manufacturing sequential rotating filter wheel optical systems used in consumer HDTV projection video display products. These systems consist of the same core components used in this multiwavelength camera: custom dichroic filters in segmented or monolithic wheels combined with quiet long life motors. We have been supplying these assemblies in prototype to production quantities to customers for over ten years. Our current projection display filter assemblies are designed, built, and balanced to operate at more than 14,000 RPM far exceeding the requirements necessary for the lower rotational speeds used in this camera.

2.2 The camera system
The RFW camera system consists of a monochromatic high definition CMOS image sensor, optical filter wheel, motor with encoder, electronics for image capture, filter wheel synchronization, data analysis, and image output and a display screen. The data capture parameters of each image can be customized to compensate for differences in spectral response and intensity reaching the sensor due to changes such as quantum efficiency or filter bandwidth. Image stabilization algorithms are used to compensate for the spatial changes (motion) of objects within the captured images.

The image sensor is the Aptina MT9M032, 1/4.5-Inch 1.6Mp CMOS HDTV monochrome digital image sensor. This sensor was chosen due to the native HDTV resolution and the superior dynamic range (>70 db), low-light performance (2.1 V/lux-sec), the low dark current, and the programmable controls: gain, frame rate, frame size. The sensor has 1472
H x 1096 V active pixels. The pixel size is 2.2 ?m x 2.2 ?m. The imager is capable of 1080p output at 30 fps or 720p output at 60 fps. The sensor has good quantum efficiency from 375-875 nm.

The images are acquired from the CMOS sensors by a customized Finger Lakes Engineering high performance Falcon digital imaging processing system. The Falcon image acceleration board utilizes a Xilinx Spartan 3 FPGA and the Dualcore Texas Instruments DaVinci DM6446 Image Processor. The TMS320DM6446 CPU includes both a 297MHz ARM9 microprocessor and a 594MHz DSP, as well as a variety of connectivity options including Gigabit Ethernet,
USB, DVI (or Pixel Link) video output, and SD/MMC Media Card I/O Ports. The board facilitates and synchronizes the image capture, storage, stabilization and processing functions without the need for an external computer by performing all mathematical calculations, image synthesis (or false-coloring), and a final composite image display.

The filter wheel shown in Figure 1a consists of two broad visible segments from ~510 to 630 nm and four 9 nm FWHM bandpass filters with center wavelengths of 540, 561, 579, and 597 nm respectively. Figure 2 shows the filter pass bands overlaid with the key blood oxygenation curves of the initial application. The oxy and de-oxy hemoglobin molar extinction coefficient9 e in [cm-1/(moles/liter)] are plotted on the same graph as the filter transmission to help visualize the location of our filters with respect to the oxy and deoxy absorption features.



Figure 2. RFW pass bands overlaid with the key blood oxygenation curves of the initial application. Note the >75% pass band for the filters (Filter1-Filter4), as well as the visible blocking filter (VIS). Also shown are molar extinction curves in cm-1/(moles/liter)

The filter wheel rotational position information is synchronized with the electronics to determine which filter segment is within the light path. The CMOS imager has a global shutter which allows the light from each filter to be captured and then the data is stacked into the on-board memory and available for use by the on-board image synthesis and image stabilization algorithms as shown below in Figure 3.



Figure 3. Rotation of the filter wheel and stacking of the data into the image stack for image stabilization and processing

Each snapshot has a unique “sequence code” tagged within the frame during the CMOS imager readout, and the process is then repeated for the next filter wheel segment as the wheel rotates counter clockwise. When an entire image stack of Filters 1-4 has been collected, all four “colors” are then stabilized with respect to the center clear (in this case C1) which acts as the image fiducial for Stack 1. With the stack of 5 images aligned to within 20 pixels of each other using the image stabilization routines in the DaVinci processor, the camera then applies the mathematical processing to the stack.

This arrangement also has the advantage of being able to re-use two of the images from the previous stack in a new stack, “Stack 2”. Stack 2 used the recycled Filters 3 and 4, a new image fiducial (C2), and a new Filter 1 and 2. All four colors are stabilized with respect to C2. This process repeats with a new image stack being created after every half revolution of the wheel.

After mathematical processing any one of the “colors” or a composite image of more than one wavelength band can be generated by the algorithms in the DSP and the DSP drives the 1280 x 720 DVI video output on an external HDTV monitor such as an LCD or Plasma Display or a sub-sampled 640x480 output for viewing on the internal LCD drive. Alternately the data from the composite image, the entire stack, and/or each individually filtered image is available for internal storage within the device that can be ported to an external device via the Ethernet connection.

2.3 Biological images from the camera
We have processed and viewed images from multiple targets including currencies with different spectral signatures. The system has good low light sensitivity and is able to get good images of human skin with relatively low amounts of white LED lighting. In fact the yellow phosphor emission from high power white LEDs match very well with the ~500-600
nm spectral region of this camera. We performed some very preliminary biological imaging utilizing the five different filters in the filter wheel camera. Below in Figure 4 is an example of an image of one of the author’s fingers both with a commercially available consumer grade color digital camera (Figure 4a) and the same digit images thru the RFW camera with some image processing turned on in the camera (Figure 4b).



Figure 4: Standard color image of one of the authors fingers and a false color multispectral image showing a small subsurface cyst under the skin of the same finger

The processing algorithms used in the camera system are implemented in the following sequences:
• Noise Averaging
o In this first step, each image capture for each wavelength is added together with the previous sample of the image for the same wavelength. A stack of one current and five previous wavelengths are maintained for each original wavelength. The stack of individual wavelength stack is essentially averaged using a SINC function

• Region Averaging
o Each averaged image is divided into 32x32 pixel cells and an average intensity value is computed for the cell

• Region Expansion
o The regions of each cell are then compared between wavelength combinations to determine the dynamic range between each region. For example, a 32x32 averaged region of 540 nm and a 32x32 averaged region of 560 nm could be compared. The comparison determines the dynamic range as an integer between these two 32x32 cells.

• Thresholding
o The dynamic range value computed in the previous step is used as a threshold for colorization. Each individual pixel within the 32x32 cell is compared against the threshold value. Pixels below the threshold are mapped into an 8-bit color space for “negative” values. Pixels above the threshold are mapped into an 8-bit color space for “positive” values

• Pseudocolor visualization
The positive and negative values resulting in the previous step are then directly mapped to RGB channels to produce a false-color image.

As seen in Figure 4b, the false color multispectral image displays a change in oxygen levels near the location of a small sub-suface cyst under the skin of that finger. The cyst is scheduled to be removed in the near future and is only noticeable tactilely by the author. None of the author’s other fingers display this phenomena.

3. DICHROIC FILTER ARRAY (DFA) CAMERA
The second class of multispectral/multiwavelength imaging systems we developed utilizes lithographically patterned dichroic filter arrays (DFA). This type of imager offers the unique advantage of simultaneous measurement of each spectral channel, scalability to tens of Megapixel resolutions, compact size, and no moving parts. This implementation is perfect for either higher volume production such as in an airborne platform or if the user needs simultaneous image acquisition of all of the wavelengths in a snapshot mode. Once the individual multispectral filter performance is verified using the sequential filter wheel implementation described above in Section 2, the transition to the DFA is relatively straightforward.

3.1 The Dichroic Filter Array
The technology behind the fabrication of our DFA substrates has been describe elsewhere10-12 and we have fabricated a four wavelength DFA with ~20 nm FWHM filters in a Bayer like pattern at 750, 772, 802, and 834 nm as shown below
in Figure 5a.



Figure 5. (a) Bayer like pattern and (b) microscope image of the dichroic filter array.

The physical size of the DFA is 35 mm x 23 mm. There are 8.75 Million (3500x2500) individual filters on each DFA. Each individual pixel is 10 ?m x 10 ?m on a 10 ?m center to center spacing with a 1 micron border around the edge of each pixel resulting in an active area of 8 ?m x 8 ?m. A microscope transmission image of a small section of the DFA is shown above in Figure 5b.

Figure 6 below plots the transmission spectra of all four filters as well as the oxy and
de-oxy hemoglobin molar extinction coefficient [in cm-1/(moles/liter)], in a manner similar to Figure 2. Given the small sizes of each individual pixel, it is extremely difficult to make an absolute transmission measurement. The transmission spectra were made on a microscope with a collimated light source illuminating the filter array. The roll off of the intensity of the filters as the wavelengths gets longer is partially attributed to the decreased efficiency of the grating and detector in this spectrometer.



Figure 6. Overlay of DFA filter bands and blood oxygenation curves

3.2 The DFA-based camera
The DFA based multispectral imager setup shown below in Figure 7 incorporates two optical subsystems, the image acquisition lens that forms an intermediate image of the object on the DFA and a relay lens that reimages the DFA onto the CCD camera chip.



Figure 7. Picture of the DFA setup consisting of (from right to left) the imaging lens, the DFA, the relay lens, and the CCD camera. The DFA is re-imaged onto the full frame CCD with a 4x magnification.

This setup was initially developed to assist in the characterization of the DFA and subsequently utilized to perform some initial imaging tests. The imaging lens is a Nikon Nikkor 60 mm focal length lens. We mounted the DFA on a 5-axis gimbal mount to ensure good spatial orientation of the DFA with respect to the CCD.

The imaging lens is mainly for multispectral imaging and is not necessary for DFA characterization. It projects the image of an object on DFA and from there the 4x microscope objective relays the image to the CCD. Due to the potential for spectral shift of the thin film filters, it is important that the light hitting the filter is parallel to the optical axis. Therefore to work properly with the DFA, the imaging lens should possess telecentric properties on the intermediate image plane. The macro lens, AF Micro Nikkor 60mm, used in the setup appears to exhibit good telecentric performance.

The CCD is an e2v/Atmel Camelia M1 CL 8M. It has a full frame CCD featuring 2300 (H) x 3500 (V) pixels in 10 ?m square pixels. The Camelia camera is capable of 10 fps via the Camera Link output. The relay lens projects a DFA cell to a 4 x 4 pixels array on the CCD with less than 5 % crosstalk between adjacent spectral channels in the DFA. The raw CCD image is an interlaced image of 4 channels with each channel covering a 4 x 4 pixel bin. Splitting the images requires registration info of each channel on the CCD and is easily accomplished in our Matlab image acquisition program. The registration is a form of look-up-table that is collected as a part of system assembly and characterization process. To
ensure that the DFA is calibrated and aligned, the DFA is illuminated with a narrow band light source that matches one of the DFA channel and its CCD image is analyzed in the Matlab GUI. In the recorded CCD image, only those pixels that are mapped to the specific DFA channel are lit, the rest will be dark. In an ideal setup, the image relay lens would possess similar telecentric properties in the object plane. As the objective we utilized is an off the shelf 4x microscope objective without any known telecentric properties, we saw some crosstalk towards the outer field of view. For this reason, this specific setup has a limited field of view ~ 3 x 3 mm.

Illumination is provided by three LED arrays (i.e. 780, 810, 870 nm) and/or one off the shelf projector lamp bulb. In addition, 4 dichroic filters were prepared that when working in conjunction with LED can narrow the light source to match individual DFA channel.

3.3 DFA Image acquisition
We are able to acquire images directly into Matlab or save the files as uncompressed TIFF images. We have acquired numerous objects with this setup and are able to quickly de mosaic and split the single image into 4 different images at each of the four wavelengths.

We performed some preliminary biological imaging utilizing the four near IR wavelength bands. Given that the DFA filter bands were not specifically designed for hemoglobin measurement, we decided to begin with a rough oxygenation imager realizing full well that they were not ideal for this application. Images of the forearm were obtained from 3
healthy volunteers, and then processed using the following algorithm in Matlab.

Step 1 (image isolation): Each of the correspondingly filtered-pixels were extracted from the master image to form a virtual 4-channel image stack.

Step 2 (Systemic adjustments): Using both dark-field and white-field images obtained earlier during system calibration, the images were corrected for the system transfer functions as well as and illumination homogeneities. Additionally, both “dead” and “hot” pixels were suppressed.

Step 3 (estimated oxygenation levels): For each individual pixel, a rough qualitative estimate of oxygenation was based on combining 3 of the DFA channels: 750nm + 775nm - 835nm. The 805nm band was used as a control (black-and-white) image as it is very close to the isosbestic point (similar absorptions for both oxygenated and de-oxygenated blood).

The results of this process are shown in Figure 8 below. The top set images are the control bands, and the bottom images are the rough map shown with a “jet” type lookup table (blue=low, red=high intensity). Each column shows a different volunteer’s forearm. While the algorithm and spectral band selection can be improved, this process illustrates the fundamental ability of the DFA to capture 4 separate bands at once and then perform an analysis based on the mixed image.