Food safety and quality are becoming progressively important, and a failure to implement monitoring processes and identify anomalies in composition, production, and distribution can lead to severe financial and customer health damages. If consumers were uncertain about food safety and quality, the impact could be profound; hence, we need better ways of minimizing such risks. On the data management side, the rise of artificial intelligence, data analytics, the Internet of Things, and blockchain all provide enormous opportunities for supply chain management and liability management, but the impact of any approach starts with the quality of the relevant data. Here, we present state-of-the-art spectroscopic technologies including hyperspectral reflectance, fluorescence imaging as well as Raman spectroscopy, and speckle imaging that are all validated for food safety and quality applications. We believe a multimode approach comprising of a number of these synergetic optical detection modes is needed for the highest performance. We present a plan where our implementations reflect this concept through a multimode tabletop system in the sense that a large, real-time production-level device would be based on more modes than this mid-level one, while a handheld, portable unit may only address fewer challenges, but with a lower cost and size.
- multimode optical imaging
- food contamination
- hyperspectral imaging
- food quality
- multimode data management
- machine learning
There is a great need to assess the composition of food, quantitatively and reproducibly, in order to avoid any unintended scenarios, ranging from a product not being quite what it is stated to be (e.g., lesser quality fish or olive oil) to intentional adulteration (including by terrorist intent) to random contamination (such as by bacteria that can be lethal). These constitute the application domain of, respectively, food quality, food defense, and food safety. Given the place food occupies in society, and the possible extreme implications of any negative events, there is great interest in bringing the best testing to the task of ensuring the quality and safety of our food supply. Unfortunately, some of the currently prevalent methods (molecular/biochemical/biophysical, such as polymerase chain reaction (PCR), chromatography, mass spectrometry, etc.) are intrinsically too slow to yield results in real time, and also rely on random and very sparse sampling. We believe that the power of light as an investigational tool can be brought to the resulting challenge and focus on this possibility here.
Optical imaging is an approach rapidly growing in popularity and applications due to technological advances that have enabled the production of smaller, less expensive, more efficient, and faster light sources and detectors. These new technologies have facilitated the acquisition of more accurate optical image sets, yielding molecular, structural, and physiological information from targeted samples. There are many different optical measurement techniques used by industry and academic researchers alike, with each technology usually focusing on a specific property of light (intensity, polarization, wavelength, coherence, temporal change, etc.). We believe, however, that no single method can provide the comprehensive analysis of food that is required.
When applied to food samples, the accuracy of optical detection techniques can be limited due to factors such as low penetration depth and lack of contrast, especially for low biomarker concentrations. However, using a strategic combination of multiple optical detection technologies in an optical system that thus becomes multimode, the chemical and/or biological detection accuracy can be substantially improved. Each individual detection method can provide a specific and complementary (sometimes even synergetic) piece of information regarding the sample being examined. Thus, by combining a number of these methods, the impact of the individual limitations can be minimized, and their combined strengths may be harnessed to deliver highly specific results.
The advantages of multimode optical imaging include greatly reducing the time required for the initial detection and enumeration of contaminants, with minimal sample preparation, nondestructive evaluation, fast acquisition times, and visualization of the spatial distribution of numerous components simultaneously. These advantages are highly useful in detecting contaminants in food for assessing safety and quality, and the use of multiple modes of detection, properly combined, is essential for effectiveness and performance.
We summarize here optical technologies which are useful in food safety and quality applications, highlighting both successes and limitations, thus underscoring the usefulness of the new, multimode approach we propose.
2. Hyperspectral imaging
Hyperspectral imaging (HSI) is a growing platform technology that functions by integrating conventional imaging and spectroscopy to gain spatial and spectral information from an object . It is capable of capturing reflectance, transmittance, and fluorescence images in the visible and infrared regions with submillimeter spatial resolution  and high spectral resolution (10 nm). While HSI was originally developed for remote sensing , it has gained popularity in the field of food safety and analysis with new applications reported in fruits and vegetables [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 34, 37, 42], poultry [21, 22, 23, 24, 25], and meat [26, 27, 28]. Some advantages HSI has in comparison with other techniques such as RGB imaging, NIR spectroscopy, and multicolor imaging include being able to produce spatial and spectral information, multiconstituent information, and sensitivity to minor components .
HSI in the near infrared (NIR) can provide chemical composition of red meat such as prediction of fat, protein, and water content of lamb meat . Moreover, this method enables the detection of certain bacteria in food, such as E. coli . Fungal growth on food products is of particular concern due to the potential for detrimental effects on population health ranging from allergic reactions and respiratory problems to the production of mycotoxins. HSI has been deployed to identify fungal species such as Aspergillus flavus, Aspergillus parasiticus, Aspergillus niger, and Fusarium spp. which can produce mycotoxins, which are secondary metabolites that are toxic for humans and animals [36, 37].
A common source of contamination for fresh products and other raw materials used to produce food is fecal contamination; hence it would be highly desirable to develop an automatic inspection system for use in the field and on processing lines. Multispectral detection of fecal contamination on apples using HSI imaging was demonstrated by Kim et al. . A HSI system with a range of 450–851 nm was used to examine reflectance images of experimentally contaminated apples. Fecal contamination sites were evaluated using principal component analysis (PCA) with the goal of identifying two to four wavelengths that could be used in an online multispectral imaging system. As shown in Figure 1, their results showed that contamination could be identified using either of three wavelengths in the green, red, and NIR regions.
With the use of HSI in the spectral range of 400–1000 nm, E. coli loads in grass carp fish have been measured to evaluate microbial spoilage. In 2015, the researchers demonstrated that reflectance HSI in combination with multivariate analysis had the ability to rapidly and noninvasively quantify and visualize the E. coli loads in grass carp fish flesh during the spoilage process . Distribution maps, shown in Figure 2, were created to allow for visualization of E. coli contamination. These distribution maps were vital in that they provided more detailed information of postmortem spoilage development in grass carp flesh. One of the main advantages that HSI has over conventional spectroscopy methods is its ability to visualize distribution maps of the contamination in a pixel-wise manner. By multiplying the regression coefficients of the multiple linear regression model by the spectrum of each pixel in the image, a prediction map was generated for showing the distribution of E. coli within the fish flesh. The different E. coli loads were represented by different colors from blue to red. As E. coli load increased, the color of the images shifted from blue to red, reflecting the growth of bacteria.
In 2013, Feng et al.  presented HSI as a nondestructive tool for direct, quantitative determination of Enterobacteriaceae loads on chicken fillets. The authors developed partial least squares regression (PLSR) models and root mean squared errors. After a simplified model was developed, the PLSR model, it was used for predicting Enterobacteriaceae loads in every pixel of the image acquired from HSI, resulting in a new image called a “prediction map.” In this prediction map, a color scale was used to describe the different microbial loads in each spot of the sample. As shown in Figure 3, when the microbial loads increase, the images shift from a blue color to a more reddish one, this reflects the growth of bacteria on the chicken fillets.
Changes in temperature during cold storage of meat products can lead to undesirable microbial growths, which may affect food safety. A study of the spoilage of beef was reported by Peng et al. ; in this work, HSI was exploited to measure biochemical changes within the fresh beef. The research demonstrated that HSI showed potential for real-time and nondestructive detection of bacterial spoilage in beef.
Work performed by Barbin et al.  used HSI in the near-infrared range (900–1700 nm) to determine the total viable count and psychotropic plate count in chilled pork during storage. NIR hyperspectral images in the reflectance mode were captured every 48 h from each sample. Assuming that meat spoilage is evident at a microbial load of 107 CFU per gram or cm2, the author’s defined a cutoff point of 106 CFU/g as an acceptable threshold of freshness. By examining the spectral information that was obtained from the samples, a difference was observed in the wavelength range between 1300 and 1600 nm, where fresh samples had lower absorbance than spoiled samples (see Figure 4). This spectral region is commonly assigned to N-H stretch of proteins (amines and amides) and their interactions with water, and it could suggest the occurrence of proteolytic changes, which are recognized as the main indicator for the onset of spoilage in meat products.
In 2016, Everard et al.  presented fluorescence HSI coupled with multivariate image analysis techniques utilized for the detection of fecal contaminates on spinach leaves. Violet fluorescence excitation was provided at 405 nm, and light emission was recorded from 464 to 800 nm. Partial least square discriminant analysis (PLSDA) and wavelength ratio methods were compared for detection accuracy for fecal contamination. The PLSDA model had 19% false positives for nonfresh post storage leaves. A wavelength ratio technique using four wavebands (680, 688, 703, and 723 nm) was successful in identifying 100% of fecal contaminates on both fresh and nonfresh leaves.
Detection of fecal contamination on cantaloupes using HS fluorescence imagery was demonstrated by Vargas et al. . HS images of cantaloupes artificially contaminated with a range of diluted bovine feces were acquired from 425 to 774 nm in response to ultraviolet-A (320–400 nm) excitation. Evaluation of images at emission peak wavelengths indicated that 675 nm exhibited the greatest contrast between contaminated and untreated surface areas. Two-band ratios compared with the single-band images enhanced the contrast between the fecal contaminated spots and untreated cantaloupe surfaces.
Yang et al.  examined methods to classify fecal contamination on leafy greens. They utilized HS fluorescence imaging system with ultraviolet-A excitation (320–400 nm) for detection of bovine fecal contaminants on the abaxial and adaxial surfaces of romaine lettuce and baby spinach leaves. They applied six spots of fecal contamination to each of the 40 lettuce and spinach leaves. Their results showed that for both lettuce and spinach, the detection of fecal matter was best obtained using the ratio of the signal from 666 nm divided by that from 680 nm, (R values of 0.98 for romaine lettuce and 0.96 for baby spinach).
3. Raman spectroscopy and spectral imaging
Raman spectroscopy is a nondestructive spectroscopic technique, based on the vibrational properties of the constituent molecules, that provides molecular information about the sample under examination. The Raman signal results from molecules being excited by a small amount of incident light at a specific wavelength. The remitted light has some of its photons shifted to different wavelengths by the addition or subtraction of vibrational energy from some of the tissue intramolecular bonds . Contrast is achieved when the tissue molecular constituents differ enough that the Raman signals from two tissues have different wavelength distributions. Raman spectral imaging (RSI) intertwines Raman spectroscopy and digital imaging to visualize the composition and structure of a target, thereby having great potential for food safety and analysis . Historically Raman imaging systems have only been able to perform Raman measurement at a microscopic level and were unable to evaluate whole surfaces of individual foods. Recent studies have shown a benchtop point-scanning Raman chemical imaging system designed and developed for food safety research . Raman imaging is a highly specific and sensitive technique as it allows for the detection of particular chemicals at low concentrations, such as detecting melamine particles in dry milk. This technique has wide applications, and due to its specificity, it may help detect contaminants in food products of different sizes.
A study aimed at the detection and differentiation of important food and waterborne bacteria (E. coli, Staphylococcus epidermidis, Listeria monocytogenes, and Enterococcus faecalis) was performed by Fan et al.  using surface-enhanced Raman spectroscopy (SERS) coupled with intracellular nanosilver as SERS substrates. Variations observed in the spectral patterns of bacterial pathogens are due to the different quantity and distribution of cellular components like proteins, phospholipids, nucleic acids, and carbohydrates. SERS coupled with statistical analysis has become very useful in discriminating and detecting bacterial cells, spores, and viruses.
In another study, a portable Raman sensor system was presented with an integrated 671 nm microsystem diode laser as excitation light source for the rapid in situ detection of meat spoilage and bacteria . The system used in this chapter is an example of the reduction in form factor of enabled by recent advances and is comprised of three main components: a handheld measurement head with a laser driver electronics board, the Raman optical bench, and finally, a battery pack. This method was used to rapidly detect meat spoilage in specific pork cuts, musculus longissimus dorsi (LD) and musculus semimembranosus (SM). The authors were able to determine the total number of mesophilic aerobic microorganisms on the surface of the meat to show possible correlations of the bacterial growth with the measured Raman spectra. In 2007, the food industry faced substantial economic losses following the discovery of melamine, a nitrogen rich chemical, in human and pet foods . In one SERS study which employed SERS-active substrates, the concentration of melamine was measured in wheat gluten, chicken feed, and processed foods such as cake and noodles [49, 50].
4. Speckle imaging
Spoilage and poisoning of food products by microorganisms is a major issue in food safety and human health. As these microorganisms grow and become more active, they cause deterioration of food quality and cause food intoxication. Some of the microorganisms capable of such damage are bacteria, yeast, and mold. As detailed earlier, there have been many different technologies developed to detect harmful microorganisms in food products such as hyperspectral imaging, Raman spectroscopy, and high-performance liquid chromatography. All these methods have certain intrinsic short comings. Factors such as the need for a well-equipped laboratory, high-cost equipment, complicated procedures for sample preparation and long analysis times, and trained professional operators limit their widespread application in the food processing, transportation, marketing, and preservation in various food industries.
A technology that is finding increasing favor by circumventing many of these limitations is laser speckle imaging. Laser speckle imaging has been introduced in this field of application to monitor moving particles in optically inhomogeneous media by analyzing time-varying laser speckle patterns for applications such as measuring meat quality and detecting contaminants. Unlike multiple light scattering in meat which exhibits static and deterministic speckle intensity patterns, light paths associated with the movements of living microorganisms result in time-varying changes in the speckle intensity patterns. Therefore, by detecting the decorrelation in the laser speckle intensity patterns from tissues, the living activities of microorganisms can be detected.
Another advantage of this method is the ability to examine meats sealed with transparent packaging because this method detects time-varying signals in reflected laser beams and transparent plastic does not affect these. Furthermore, the technique can provide rapid assessment as bacterial colonies can be detected within a few seconds . Thus, this method provides an efficient and effective way to detect live bacteria in food products to avoid food toxicity. Speckle imaging systems have been demonstrated to indicate the presence of bacterial colonies and other contaminants in both food and water . Technology such as this may be very effective in the marketplace as food producers or consumers themselves may be able to use them to assess food safety. As mentioned, there are currently several approaches available for detecting low levels of microorganisms in food; however, they require complex equipment, high costs, invasive procedures, and skilled technicians to operate which all act to restrict its widespread adoption and use in the food industry .
Work performed by Yoong et al.  aimed to detect and quantify various levels of contamination using chicken breast meat samples. The meats contaminated with bacteria had significant decreases in the autocorrelation values over the time lag, whereas the control group (meat treated with a PBS solution) did not show any major changes. The meat treated with a high concentration of bacteria had more significant changes over the time lag compared with the meat treated with a low concentration of bacteria. Moreover, the decrease in the autocorrelation value was proportional to the concentration of the treated bacteria. The measured autocorrelation values were all statistically different from one another (p < 0.001), and the decreases in the autocorrelation were proportional to the concentration of bacteria. Thus, the authors were able to show that through various experimental validations, spontaneous bacterial activity caused strong decorrelation in laser speckle dynamics (Figure 5).
In 2014, Kim et al.  presented a label-free bacterial colony phenotyping technology called bacterial rapid detection using optical scattering technology (BARDOT), which can provide classification for several different types of bacteria. Recent experiments with colonies of Bacillus species using speckle imaging show a certain speckle formation that allows for the detection and identification of these bacterial species. As the center diameter of the Bacillus spp. colony grew from 500 to 900 microns, the average speckle area decreased twofold and the number of small speckles increased sevenfold. As Bacillus colonies grow, the average speckle size in the scatter pattern decreases and the number of smaller speckle increases due to the swarming growth characteristics of bacteria within the colony . Singh et al. showed the real-time detection and identification of Salmonella colonies grown from inoculated peanut butter, chicken breast, and spinach or from naturally contaminated meat using BARDOT technology (90–100% accuracy) in the presence of background microbiota from naturally contaminated meat .
5. Multimode hyperspectral imaging system
Due to the multicomponent nature of foods, their reflectance or fluorescence spectra are complex and chemometric methods using multivariate analysis are needed to extract contaminant-specific information. By varying both the excitation and detection wavelengths and measuring both reflectance and fluorescence emission properties of a food sample, we can fine-tune algorithms for specific foods and contaminants. It has been shown that for biological tissues, dual or multiple excitation fluorescence can increase the specificity and accuracy of classification and quantification of specific sources of fluorescence . Rasch et al.  showed the combination of different spectroscopic methods (such as fluorescence and NIR spectroscopy) becomes a promising approach to circumvent such single method inherent limitations and to use optical sensing for in situ mycotoxin detection. Additional chemometric tools are essential to eliminate disturbing factors and to extract the desired biochemical information with respect to contamination with fungi and/or mycotoxins.
An example of a multimode hyperspectral imaging system operates in fluorescence and reflectance modes as well as speckle imaging is shown in Figure 6 developed by SafetySpect Inc. The system uses spectral band sequential imaging on the detection side. To ensure high signal-to-noise level, camera and spectral selection filter integration time is optimized for each spectral band from visible to the near infrared. The illumination module uses two independent light sources to provide illumination for fluorescence excitation and reflectance measurements using three computer-controlled LED illumination rings. The UVA (375 nm) and blue/violet (420 nm) LED rings provide fluorescence excitation. White LEDs will be used for reflectance illumination. The HSi-440CO hyperspectral imaging system (Gooch & Housego, UK, originally developed by ChromoDynamics, Inc.) incorporated in the proposed system can image and analyze multiple signals in fixed and living cells at video rates. Its tunable filter can switch wavelengths within microseconds. The system acquires multiwavelength, high-spatial and spectral resolution image datasets, and can compute and display quantitative signal-specific images in near real time. The spectrally controllable image capture system can record spectral images of food samples in wavelengths ranging from 450 to 800 nm. The system is configured as a tabletop platform where illumination and detection will operate above the food sample.
In this system, time-varying speckle signals can be quantitatively addressed with the speckle correlation time. A sample containing living microorganisms will have a correlation time way shorter than a static one, and thus contaminated food will be less time-correlated as compared to fresh food due to the spontaneous motility of microorganisms. Correlation time of scattered light from samples, the presence and activity of microorganisms can be quantitatively analyzed.
Let us consider I (x,y,t) the image of the sample at time t. The correlation coefficient between two images of the sample at different times is given by the normalized autocorrelation function:
where T is the total acquisition time, δt the time difference, and τ the time lag. In the case of food contamination assessment, the sample is expected to be static and the correlation to be close to the unity. Every decorrelation effect is due, then, to the presence of live microorganisms moving across the sample.
There is inherent risk in food (preparing, selling, and consuming it), and we need better ways of minimizing such risk. The number of people who are sickened by problematic food is staggering (it is estimated that 1/6 of the US population is thus affected yearly), and the number of people who die (~3000/year) is unacceptable. If one examines the rather extensive risk management/mitigation literature, it is evident that certain fields of human endeavor (such as air travel) are doing a better job than others in minimizing the undesirable scenarios. A particularly pragmatic take on this field was provided by Dr. J. Reason , who developed an approach he termed the Swiss Cheese theory (Figure 7). Basically, he posits that we all want to insert countermeasures between us and hazards, to prevent harm, but because we are human and thus imperfect, these countermeasures are like a slice of Swiss cheese. The most logical and direct improvement is to “stack” the slices of cheese, as the holes do not align, and prevention is achieved. Translated to imaging for food safety, this calls for a multimode approach, which is what we propose (see Figure 8). The number of modes needed for good performance scales, naturally, with the difficulty of the problem, and we plan to have our implementations reflect this, in the sense that a large, real-time production-level device will be based on more modes than a mid-level (e.g., restaurant) one, while a handheld, portable unit may only address 80% of the challenges, but with ~20% of the cost and size.