Multi-Frame Superresolution Optical Coherence Tomography for High Lateral Resolution 3D Imaging

Abstract

We report that high lateral resolution and high image quality optical coherence tomography (OCT) imaging can be achieved by the multi-frame superresolution technique. With serial sets of slightly lateral shifted low resolution C-scans, our multi-frame superresolution processing of these special sets at each depth layer can reconstruct a higher resolution and quality lateral image. Layer by layer repeat processing yields an overall high lateral resolution and quality 3D image. In theory, the superresolution with a subsequent deconvolution processing could break the diffraction limit as well as suppress the background noise. In experiment, about three times lateral resolution improvement has been verified from 24.8 to 7.81 μm and from 7.81 to 2.19 μm with the sample arm optics of 0.015 and 0.05 numerical apertures, respectively, as well as the image quality doubling in dB unit. The improved lateral resolution for 3D imaging of microstructures has been observed. We also demonstrated that the improved lateral resolution and image quality could further help various machine vision algorithms sensitive to resolution and noise. In combination with our previous work, an ultra-wide field-of-view and high resolution OCT has been implemented for static non-medical applications. For in vivo 3D OCT imaging, high quality 3D subsurface live fingerprint images have been obtained within a short scan time, showing beautiful and clear distribution of eccrine sweat glands and internal fingerprint layer, overcoming traditional 2D fingerprint reader and benefiting important biometric security applications.

1. Introduction

Optical coherence tomography (OCT) [1, 2] is an advanced non-contact 3D imaging technique, providing subsurface cross-sectional tomographic images. It offers deeper penetration depth [3] and larger scan area [4] than confocal microscope imaging [5] as well as higher resolution [3] than ultrasonic imaging [6]. It is thus widely utilized in 3D imaging of eyes [7, 8], skins [9, 10, 11, 12], blood vessels [13], cartilages [14], and numerous biomedical applications.

With non-contact and non-invasive advantages, OCT has significant medical applications. There is also a huge potential of OCT for many non-biomedical applications that demands non-destructive testing and evaluations in micron scale resolutions [15]. For example, there is no preparatory steps for OCT sample imaging, instead of gold coating for SEM imaging; no coupling media as required for ultrasound imaging; no special safety precautions like X-ray. Also, the near infrared light source in OCT usually has no photo reactions with most materials, very safe for quality testing of damage in silica [16], glass-fiber reinforced polymer samples [17], strained polymer samples [18], microstructures [19, 20, 21, 22, 23], papers [24], oil paintings [25], film coatings [26], fastener flushness [27], and so on. Besides, the successful detection of embedded and hidden structures is another potential of OCT for security applications, such as 3D fingerprint identification defending against spoofing attack with fake fingerprints [28, 29, 30, 31]. However, compared with other imaging techniques such as microscopy and confocal microscopy, the low lateral resolution and high speckle noise restrict the OCT becoming a competitive imaging tool in some non-biomedical areas highly relying on en-face lateral image quality.

OCT imaging has two distinct resolutions namely axial resolution in the depth direction and lateral resolution in the en-face plane like microscopy. The axial resolution regards to the coherence length of the light source and thus can be improved by supercontinuum light [32] or extended broadband superluminescent diode (SLD) [33]. The lateral resolution is mainly restricted by diffraction limit [34], lateral sampling rate [35] and background noise [36]. The diffraction limit is the minimum focused spot size, determined by the numerical aperture (NA) of the OCT sample arm optics. Although a high NA optics could achieve a smaller focused beam spot size on the sample, the quick divergence of the beam size out of the focal plane reduces the depth of focus (DOF) of the OCT system, losing its main advantage over confocal microscope. Higher NA also limits the lateral field-of-view (FOV) due to the rapid off-axis degradation of the focusing performance, explained in our previous work [37, 38]. Therefore, it is crucial to overcome the complex trade-off among lateral resolution, axial DOF, and lateral FOV in the OCT imaging.

Adaptive optics (AO), an astronomical telescope technique, has been adopted in OCT systems to correct aberration wave front and thus improves the lateral resolution [39]. Except the high cost and a very limited FOV (maximum 1 × 1 mm²) [40, 41], AO technique in principle is to recover the original lateral resolution of OCT, which however is blurred by human eyes. Thus, it is not suitable for non-ophthalmic imaging like skins due to scattering blurring. A virtually structured detection (VSD) method [42] was reported to improve the lateral resolution by adding an electro-optic phase modulator (EOPM) in the reference arm. The EOPM shifts the light phase with multiple of π/2, and then the VSD algorithm fuses four phase shifted A-scans to one, achieving resolution doubling. It is a time consuming (taking ∼40 s for each image frame) technique which is infeasible for in vivo imaging and 3D imaging. Robinson et al. [43] register four sparse scanned summed voxel projection (SVP) of retina images to reconstruct a higher density en-face image in $y$-axis to improve the resolution and reduce motion errors, while the quality improvement does not overcome the traditional high density scan images. Digital image deconvolution processing is a potential technique to break diffraction limit and improve the resolution [44, 45, 46, 47]. The estimation of the ground true lateral point spread function (PSF) of the system is however very difficult and the actual PSF may be different in different samples and at different depth layers.

Background noise is another factor degrading the resolution and image quality of OCT systems. Different from white noise, the structure related speckle noise in OCT imaging is difficult to be suppressed by the multi-frame averaging [48, 49]. Szkulmowski et al. [36] introduced an interesting averaging algorithm with multiple shifted B-scans to remove the speckle noise. However, this approach introduces new ghost patterns in in vivo imaging, such as multiple ghost fingertip patterns in the output image, due to averaging multiple B-scans in different positions. Besides, simple averaging shifted images may penalize the high frequency signals and degrade the resolution. And the longer B-scan time is impractical for 3D imaging.

Lateral sampling rate [35] in scan-based OCT imaging is termed the scan matrix density. According to Nyquist-Shannon sampling theorem, the sampling frequency should double the sample frequency at least. Although increasing the scan matrix density could improve the lateral resolution, this method is at the expense of longer scan time and not suitable for time sensitive applications, such as in vivo imaging of fingerprint. Besides, high scan density cannot overcome the diffraction limit and reduce background noises.

In this book chapter, we report an effective multi-frame superresolution technique to significantly improve the lateral resolution and image quality of OCT without adoption of extra hardware and higher NA optics. Through adjustment of galvanometer scanners to introduce slightly shifts among different sparse sampled C-scans, the superresolution processing is then applied to generate a three times higher lateral resolution image with suppressed background noise, demonstrated by imaging a standard resolution target. The remarkable improvement of 3D in vitro imaging has been observed in a microstructure sample with 2–3 μm scale features. The image stitching technique helps us to reconstruct an ultrawide FOV and high lateral resolution 3D image. For in vivo imaging, the image registration method is used to estimate the unknown random shifts among different C-scans. The subsequent superresolution processing demonstrates high quality 3D and subsurface in vivo images of fingerprint, benefiting various security applications.

2. OCT system and superresolution principle

2.1 SD-OCT and lateral resolution limit

Our spectral domain optical coherence tomography (SD-OCT) (BIOptoscan OS-186, New Span Opto-Technology) is one kind of the most popular OCT systems in ophthalmic clinic applications, as schematically shown in Figure 1. It sends a broadband light from the SLD to a 2 × 2 optical fiber coupler. The SLD has a center wavelength ${\mathrm{\lambda }}_o$ of 860 nm and spectral bandwidth $\mathrm{\delta \lambda }$ of 100 nm (IPSDW0822–0314, InPhenix). One split beam is sent to the reference arm that is focused to a mirror and then reflected back to the fiber coupler. The other split beam in the sample arm is focused to the measurement sample and laterally scanned by a pair of galvanometer scanner mirror. The scattering signals from different depth layers of the sample collected by lens are sent back to the fiber coupler to interfere with the return beam from the reference arm, generating spectral interference patterns that are imaged by the optical spectrometer for computer signal processing. Each scattering depth would result in a near sinusoidal interference pattern in the frequency domain. The final spectral image looks complex due to mixing of all interference patterns with different periods from all sample depth layers. A fast Fourier transform processing of the mixed interference pattern in the frequency domain can beautifully retrieve a series interface layers inside the sample within the depth range of the SD-OCT, set by the combination of sample arm optics DOF and the spectral resolution of the optical spectrometer. This above processing yields the A-scan, the depth intensity profile $I\left(z\right)$ of one point in the lateral plane. Through galvanometer scanning in the transverse $x$ axis, we obtain the B-scan $x$-$z$ intensity image $I\left(x,z\right)$. By galvanometer scanning in both the transverse $x$ and $y$ axes line by line, we have the C-scan $x$-$y$-$z$ intensity 3D image $I\left(x,y,z\right)$.

Figure 1.

Schematic configuration of our SD-OCT system.

As discussed earlier, the focused beam spot size at full width half maximum (FWHM) of the SD-OCT imaging system is mainly limited by the NA of the sample arm optics [34] as

$$\mathrm{\delta x}=0.37\frac{{\lambda }_0}{\mathit{NA}}.$$E1

Here, NA is the ratio of the input collimated beam radius to the focal length of the sample arm lens in the air. The axial DOF is determined by [34].

$${\mathit{DOF}}_{\mathit{\text{axial}}}=\frac{0.565·{\lambda }_0}{{sin}^2\left[\frac{{sin}^{-1}\left(\mathit{NA}\right)}{2}\right]}.$$E2

Given our collimated beam diameter of ∼3 mm, the corresponding focusing spot size and axial depth range are listed in Table 1 for a few common lenses’ focal lengths. Here, before reaching spectrometer limitation, the axial depth range is set by the axial DOF.

Focal length (mm)	Beam spot size (μm)	Axial depth range (mm)	Lateral FOV (μm)
10	2.12	0.086
19	4.03	0.31	500 × 500
30	6.36	0.78	1400 × 1400
100	21.21	8.64*(2.86)	12,000 × 12,000

Table 1.

Four different lenses with their calculated beam sizes, as well as axial depth range and lateral FOV.

*

Our present optical spectrometer for the SD-OCT supports a maximum depth range of 2.86 mm in the air.

Also, the NA of optical system will influence the effective lateral FOV. In theory, the lateral FOV of the OCT is simply given [50] as:

$${\mathit{FOV}}_{\mathit{\text{lateral}}}=2·f·tan{\theta }_{\mathit{max}}.$$E3

Here, $f$ is the focal length of the sample arm lens and ${\theta }_{\mathit{max}}$ depends on both the radius of the lens and the acceptable maximum off-axis scanning angle of the galvanometer scanners.

Typically, the acceptable maximum galvanometer scanning angle ${\theta }_{\mathit{max}}$ could be quite large. However, due to off-axis focused beam aberration and Petzval field curvature [38], the FOV is quite limited. The displacement $∆x$ of an image point at height $h_i$ on the Petzval surface from the paraxial image plane is given by [51]:

$$∆x=\frac{h_i^2}{2}\sum _{j=1}^m\frac{1}{n_j{f}_j}$$E4

Here, $n_j$ and $f_j$ are the indices and focal lengths of the $m$ thin lenses forming the system. This equation implies that the Petzval surface is an unaltered value by changes in position or shapes of lenses and stops, but inversely proportional to the focal lengths. Detailed illustration and explanation could refer to our previous work [38]. Usually the field curvature from high NA lenses would rapidly blur the off-axis image and degrade the image quality at the edge. We thus need to find a suitable FOV with an acceptable off-axis image quality reduction for the selected sample lenses.

In order to quantify the influence of the field curvature to the focusing performance, we simulate the off-axis focusing degradation of three common lenses (Thorlabs, AC254-030-B, AC127-019-B, and AC254-100-B) by ZEMAX software, with two of them shown in Figure 2. We see that the focal spot size of 30 mm focal length lens remains almost unchanged when off-axis distance is less than 300 μm. With off-axis distance larger than 800 μm, the focusing degradation becomes obvious. At off-axis distance of 1000 μm, the diameter of focal spot size (measured at FWHM of the peak) is ∼16% larger than that at the center position. The increased off-axis focused beam spot size would significantly degrade the OCT lateral resolution. With 1400 × 1400 μm² areal scan imaging (700 μm off-axis distance), the lateral resolution at the image edge is considered acceptable.

Figure 2.

The curve shows the focus spot size change with the off-axis distances (radius). The scan coordinate is shown at right. Thorlabs AC254-030-B 30 mm (A) and AC127-019-B 19 mm (B) focal length lenses are used in these simulations.

For 19 mm focal length lens (Thorlabs, AC127-019-B) simulated in Figure 2(B), a 500 μm off-axis distance would lead to ∼39% larger in focal spot size. Thus, the optimized single C-scan FOV has to be limited to 500 × 500 μm² area (250 μm off-axis distance), only one eighth of the 30 mm focal length lens. For 100 mm focal length lens (AC254–100-B), the image quality is usually acceptable in a 1.2 × 1.2 cm² [38] (not shown here), but losing lateral resolution due to large spot size as we discussed above. In practice, unavoidable spherical and coma aberrations would further degrade the image quality.

Clearly, smaller focused beam spot size would improve the lateral resolution but at the expense of the DOF and lateral FOV. For example, although a lens with 10 mm focal length would provide the smallest focused spot size of 2.12 μm, its poorest axial DOF of 0.086 mm makes the SD-OCT incompetent to the confocal microscopy which maximally provides a depth range of about 0.2 mm [3]. Also, the ultrashort focal length will restrict the effective lateral FOV to 200 × 200 μm² due to rapid off-axis degradation. With a 100 mm focal length lens, the focused beam spot size is about 21 μm, verified by using a laser beam profiler (BP-5.0, New Span Opto-Technology). Although this focused beam can offer the long depth range of 2.86 mm set by the spectrometer as well as 1.2 × 1.2 cm² lateral FOV, its large spot size does not provide desirable lateral resolution, not suitable for imaging of fine structures. Therefore, without special and high cost hardware design and improvement to overcome diffraction limitation, we should consider image processing method to improve the lateral resolution of SD-OCT. If the lateral resolution can be improved to several μm with great image quality and maintain the predominant depth range of 2.86 mm, the processing based lateral resolution improvement technique could benefit security imaging applications such as sub-surface fingerprint reader. Using a 30 mm focal length lens, our goal is to improve the lateral resolution to ∼2 μm to approach that of confocal microscope which could benefit micron scale structural imaging.

2.2 Improving lateral resolution by high density scanning

For a large FOV image with short scan time, the focused beam spot scan matrix is usually set to one by one without spot positional overlapping, as illustrated in Figure 3(b), like spatially separated pixel array in an image sensor. The presence of spot spacing results in under sampling and loss of spatial image features. Without demanding smaller focused beam spot size for preserving a long DOF and a large FOV, a double or higher density scan matrix with partial scan beam spot overlapping could improve the SD-OCT lateral resolution to some extends explained in Nyquist-Shannon sampling theorem, but at the expense of reducing the lateral FOV, as illustrated in Figure 3(c). Besides the FOV reduction, this high-density scanning method has its resolution limitation and cannot suppress the background noise, discussed in Section 3. In this chapter, the low density scanning means one by one scan array without nearby spot overlapping and the high density scanning means adjacent scan spots would be partially overlapped. For example, four times high density scanning means each scan spot have half spot overlapping with four neighboring ones (top, bottom, left and right).

Figure 3.

An illustration image of a high resolution target (a) and its output result under a low density scan imaging (b). (c) 4-time higher density scan with the same scan matrix size as (b) but half spot overlapping of adjacent ones yields higher resolution image while with a reduced FOV. (d) 4-time larger scan matrix leads the same scan density as (c) and the same FOV as (b), but 4-time longer scan time. Here, we apply four colors to show overlapping of adjacent scan spots.

In order to avoid FOV reduction, a larger C-scan matrix may be applied to the sample sacrificing the scan time as shown in Figure 3(d), which however is unacceptable for time sensitive in vivo imaging of live tissues due to random tissue motion and vibration during the long scan time. To illustrate the temporal motion of live tissue, we performed 100 repeated sparse B-scans of 128-spot to image a human skin. The 100 sets of such B-scan without scan spot overlapping completed within 0.55 s and the fast Fourier transform was calculated later. The comparison of the 1st and the 100th B-scan images shows no observable image shifts as shown in Figure 4(a), demonstrating that a 100 × 128 or 128 × 128 (we also verified) C-scan is fast enough for typical live tissue in vivo SD-OCT imaging without concerning motion errors in one C-scan. Similarly, we performed 100-frame repeated 512-spot B-scans with the same FOV as above and compared the 1st and the 100th images as shown in Figure 4(b). We observed obvious image positional shifts during the 100 × 512 scan period indicating some tissue motion during this period. These two experiments were repeated several times with similar results. The scanner optics was held steady during the image acquisition. This indicates that 100 × 512 C-scan, not so high density, is already inadequate for reliable in vivo imaging of live skin tissues owing to live body motion and vibration during the long scan time. Needless to say, in vivo tissue image misalignments are expected in 512 × 512 or 1024 × 1024 C-scans due to much longer scan time. Take the retina C-scan imaging as an extreme example, unintended eye quick motions can result in clear image artifacts and misalignments, indicated at the green arrow line in Figure 4(c). Thus, the most reliable way for in vivo imaging is to scan the sample as fast as possible, avoiding any motion artifacts and errors in one C-scan set. In our experiment, a 128 × 128 C-scan within 0.7 s acquisition time could effectively prevent most motion errors, guaranteeing the data reliability in one C-scan. For unavoidable motion shifts among multiple in vivo C-scan sets, an image registration method will be used to align them.

Figure 4.

SD-OCT in vivo B-scan on a live human skin with 100 repeated frames of (a) 128-spot density and (b) 512-spot density. Same FOV is applied to the two scans. We then overlapped the 1st and the 100th frames to compare the differences by green and magenta colors. (a) Due to the short scan time, the horizontal positions of the hair (in the blue and yellow squares) between the 1st and the 100th images remain almost unchanged. (b) There are obvious shifts between the first and the last images due to slower 512-spot B-scan, in which tissue motion occurs during the scan. (c) The green arrow indicates the discontinued line of the two micro-vessels caused by eye motion during the C-scan. 100 mm focal length lens was used in these experiments. The scale bars in (a), (b), and (c) are 500 μm.

2.3 Improving lateral resolution by the multi-frame superresolution for in vitro imaging

The multi-frame superresolution is an image processing technique, studying image degradation models (such as optical blur, motion effect, down-sampling, and additive noise) and then recovering the high resolution image from multiple low quality images based on the superresolution algorithm and the sub-pixel information differences among these images, overcoming the resolution limit of the hardware. Figure 5 illustrates how these effects result in a low quality image during the conventional camera image acquisition. To recover the high resolution image, a series reversed methods such as up-sampling, motion/pixel shift estimation and compensation, deconvolution, and denoising are applied.

Figure 5.

Conceptual illustration of image degrading effects during conventional camera image acquisition.

In a SD-OCT system, the main degradations of an ideal lateral image $S\left(x,y\right)$ is the optical blurring $H\left(x,y\right)$ caused by the lateral PSF of OCT optics and the down-sampling $\downarrow$ due to the sparse scan matrix and the large focused beam spot size. Each isolated focused beam spot is treated as a pixel in conventional camera imaging. The motion effect can be generally ignored when imaging non-biomedical samples such as microstructures [19, 37] since both the sample arm and the samples are stable. The motion effect should be considered for imaging of in vivo tissue such as fingerprint [31, 38] due to potential live body vibrations. According to the superresolution principle [52, 53, 54], the resolution improvement comes from the effective sub-pixel information differences among multi-frame low resolution images, as illustrated in Figure 6. Without introducing sub-pixel shifts among images, the stationary multi-frame imaging and processing would mainly contribute to minimize temporal noises [48]. For SD-OCT superresolution, the conventional sub-pixel shift now called sub-spot-spacing shift is due to different imaging principles.

Figure 6.

The illustration shows our superresolution reconstructed image from multiple low resolution frames with sub-pixel shifts. (a) The first $x$-$y$ plane image I0 with 5 × 5 pixels is set as position 0, as reference. (b) The second image I1 shifts half-pixel to the right of I0. (c) The third image I2 shifts half-pixel to the bottom direction. (d) The last image I3 shifts half-pixel to the right-diagonal direction. (e) Four times pixel resolution (10 × 10) image is obtained after superresolution processing.

Figure 6 illustrates how to apply multi-frame superresolution technique to SD-OCT imaging. A reference 5 × 5 pixel image in one depth layer of a C-scan with no scan spot overlapping is shown in Figure 6(a). As we introduced above, each pixel represents a focused scan beam spot on the sample. To satisfy multi-frame superresolution requirement, we intentionally introduce slight differences in a series of control voltage matrix of scanners, creating a sequence of C-scans with sub-spot-spacing shifts (equivalent to sub-pixel shifts) in the $x$-$y$ lateral plane, as illustrated in Figure 6(b)–(d). The superresolution processing of the four lateral images but with sub-spot-spacing shifts reconstructs a higher pixel resolution image of that depth layer, exhibited in Figure 6(e).

The image resolution covers two concepts: one is pixel resolution which is equivalent to dots per inch or sampling rate in conventional terminology and the other is spatial resolution which is defined as the smallest discernible detail in an image [55], one example as Rayleigh criterion. Figure 6 is obviously a result of lateral pixel resolution improvement leading to lateral spatial resolution improvement by the increment of sampling rate. In theory, simply reducing the step size of the sub-spot-spacing shift and increasing non-identical image sets for multi-frame superresolution processing can continuously improve the lateral pixel resolution. However, there is a spatial resolution limit owing to optical diffraction limit, system noises, stability of interference pattern and so on. In other words, when the sampling rate is high enough, further increment would not be helpful to lateral resolution improvement. Thus, finding an effective relationship among the lateral resolution improvement, the sub-spot-spacing shift, and the number of image frames would be critical to identify a desired resolution improvement without unnecessary image frames and associated excess acquisition time. Without particularly indicated, the lateral resolution improvement discussed in this book chapter represents lateral spatial resolution improvement.

Figure 6 illustrates four C-scans having multi-directional 1/2-sub-spot-spacing shifts. For easier explanation, the four shifts are simplified as four blocks in Figure 7(b), showing the shifts directions and space relative to the first non-shift C-scan as reference 0. Mathematically speaking, the three shifts should be represented as (0.5, 0), (0, 0.5), and (0.5, 0.5) in $x$ and $y$ coordinates. Compared with the traditional four-frame shift strategy in Figure 7(b), we experimentally found that more shifts (gray ones) in Figure 7(c) and (d) in addition to red shifts lead to better image quality. The gray shifts in Figure 7 provide more information for superresolution processing, suppressing background noises in OCT imaging. By using 1/4-spot-spacing shift as in Figure 7(d) red points, the superresolution technique can improve the lateral pixel resolution by 16 times in principle. Similarly, a series 1/8-spot-spacing shift C-scans (not shown) can improve the lateral pixel resolution by 64 times. Simplifying the shift strategy introduced later in the chapter, we name the Figure 7(c) as 1/2-spot-spacing shift step and maximum 1/2-spot-shift, and Figure 7(d) as 1/4-spot-spacing shift step and maximum 3/4-spot-shift, and so on.

Figure 7.

(a) Take the first C-scan as position 0, for reference. (b) This is a simple illustration of Figure 6, showing the traditional shift strategy for four times pixel resolution improvement. (c) Our shift strategy for four times pixel resolution improvement, including additional gray shifts as indicated by yellow arrows, which can provide better output image quality. (d) Similar as (c), we use a smaller 1/4-spot-spacing shift to increase the pixel resolution by 16 times.

Considering an ideal high quality lateral image $S\left(x,y\right)$ degraded by a pure translational motion with space invariant blur and additional noise as $V\left[x,y\right]$, one of the acquired low resolution lateral image $I\left[x,y\right]$ at a selected depth layer in a C-scan is modeled as

$$I\left[x,y\right]=\left[F\left(H\left(x,y\right)⨂S\left(x,y\right)\right)\right]\downarrow +V\left[x,y\right].$$E5

Here, $F$ is the motion operator due to sub-spot-spacing shifts among multiple C-scans discussed above. $H\left(x,y\right)$ is the PSF of the sample arm optics, blurring the image. $⨂$ is the convolution operator. $\downarrow$ is the discretizing down-sampling operator due to the sparse scan matrix and finite spot size.

According to the image degradation model in Eq. (5), we can recover the high resolution image $\widehat{S}\left(x,y\right)$ with a series slight shifted $I\left[x,y\right]$’s by mathematical processing. Generally speaking, the recovering processing is minimizing the errors between the model and all the measurement values. We estimate the approximate high resolution image $\widehat{S}$ in a minimum $L_p$ norm problem [53] as

$$\widehat{S}=\underset{S}{\text{ArgMin}}\left[\sum _{k=1}^N{‖D{F}_k\mathit{HS}-I_k‖}_p^p\right].$$E6

Here, $I_k$ is the $k$th input low resolution image. $F_k$ is the motion operator for the $k$th low resolution image. $D$ is the down-sample operator which can be simply determined as $\frac{1}{8}$, $\frac{1}{4}$ or $\frac{1}{2}$ by how many times the sampling rate improvement (such as 8, 4 or 2 times) and the total input frame number captured. $H$ is the optical blur operator or PSF. The noise $V\left[x,y\right]$ is an additive term and can be suppressed by multi-frame superresolution processing, which thus is not included in Eq. (6). Besides, we define $G=\mathit{HS}$ as the image $S$ convolved with a PSF, due to the complexity of the deconvolution problem in OCT imaging system. We would solve the deconvolution problem later [44]. Rewriting Eq. (6) [53] we have

$$\widehat{G}=\underset{G}{\text{ArgMin}}\left[\sum _{k=1}^N{‖D{F}_{k}G-I_k‖}_p^p\right].$$E7

Eq. (7) is a minimization of $L_p$ norm problem that can be separated into two steps: reconstruct a non-deconvolved high resolution image $\widehat{G}$ from a series of low resolution image frames ${\sum }_{k=1}^N{I}_k$ and then find a proper PSF to eliminate optical blur $H$ and recover the expected image $\widehat{S}$ from $\widehat{G}$.

If $p=1$, it is a $L_1$ norm problem, or a least-absolute problem. If $p=2$, it is a $L_2$ norm problem, or a least-square problem. $L_1$ norm is robust to outliers but may penalize the high frequency signals. In most OCT applications presented in this chapter, we notice that the background noises are usually temporal noise along with structure related speckle noise without significant outliers. Both the temporal noise and the speckle noise can be suppressed by processing with adjacent pixels [36] and the average of multiple lateral images. Therefore, we applied a kind of $L_2$ norm called normalized convolution (NC) algorithms introduced by Knutsson et al. [56] and Pham et al. [57] to process the designated shifted images in Figure 7 to improve the lateral resolution of our SD-OCT system.

We select the NC algorithm [56, 57] instead of other steepest descent algorithms because it considers the relation of a center pixel with neighborhood encompassing N pixels (for example, the radius of 4 pixels). And the final value of each output pixel is optimally solved [58] by adjacent ones, effectively reduce the structure related speckle noise. In experiment section, through shifted C-scans and the NC algorithm, we demonstrated that our superresolution technique can significantly reduce the background noises in final lateral and 3D images. Besides, due to the shift compensation for all low resolution frames, our method avoids ghost patterns observed in output images. Additionally, this kind interpolated method has good tolerance to the incomplete input frames lack of some shifts. For example, even lack of I3 in Figure 6, we still can estimate the output image according the neighborhood pixels in incomplete input images.

After the interpolation algorithm, the next step is to find a proper PSF to recover the expected image $\widehat{S}$ from $\widehat{G}$. There are numerous reports on various deconvolution methods to improve OCT image resolution [45, 46, 47]. Lucy-Richardson deconvolution [47, 59, 60] with a proper Gaussian PSF appears to be a widely accepted solution for recovering blurred images,

$${\widehat{S}}_{m+1}\left(x,y\right)={\widehat{S}}_m\left(x,y\right)\left[H\left(-x,-y\right)⨂\frac{\widehat{G}\left(x,y\right)}{H\left(x,y\right)⨂S_m\left(x,y\right)}\right],$$E8

where ${\widehat{S}}_m\left(x,y\right)$ is the estimate of the undistorted image in $m$th iteration. The deconvolution process starts with ${\widehat{S}}_0\left(x,y\right)=\widehat{G}\left(x,y\right)$. The original input image $\widehat{G}\left(x,y\right)$ is obtained from Eq. (7). $H\left(x,y\right)$ is the lateral PSF of the system. The Gaussian PSF is a common selection [45, 46, 47] owing to the focused beam spot lateral profile following a certain Gaussian distribution. However, the spot profile may not keep the circular symmetry for off-axis scanning. Considering the scattering inside a sample, the focused beam may not retain near Gaussian distribution. Thus the blind deconvolution [61, 62] might be a better solution, which uses maximum a posteriori probability (MAP) algorithm to automatically estimate the irregular PSF in the input image and then deblur it, avoiding the limitation of the regular PSF and exhibiting better performance in the final image. In this book chapter, we applied the blind deconvolution method introduced by Krishnan et al. [62]. In theory, the resolution limit of an optical system is determined by diffraction limit [63], which is related to the PSF. Thus, it is possible to break the diffraction limit and further improve the spatial resolution of optical systems through deconvolution with a correct PSF. Although the $\widehat{S}$ deconvolved from a Gaussian or estimated PSF would show obvious resolution improvement to $\widehat{G}$, these methods may lead to some ringing artifacts and reduce the output image quality. Also, the deconvolution methods are usually sensitive to the noise floor which further restricts their applications, explained later in the experiment Section 3. In this chapter, we thus focus on the first step to reconstruct a high quality image $\widehat{G}$, but also provide deconvolved images for readers to compare.

2.4 Estimating the unknown shifts to improve lateral resolution by multi-frame superresolution for in vivo imaging

The above superresolution processing is suitable for SD-OCT imaging of static samples such as microstructures where the sub-spot-spacing shifts $F_k$ are intendedly set. For in vivo SD-OCT imaging of live tissues such as fingerprint identification, the shifts $F_k$ are unknown due to live body motion and vibration, making the superresolution processing difficult. An effective estimator is critical to accurately estimate the shifts before superresolution processing. We decompose the unknown spatial shifts into two directions: one is in the depth $z$-axis and the second is in the lateral $x$-$y$ plane. Herein, the rotational angle motions could be ignored for fingerprint reader.

In the $z$-axis, the height shifts among multiple C-scans can be corrected by some obvious features, like comparing the top positions of multiple 3D images. While in the lateral $x$-$y$ plane, without any simple indicators, an advanced shift estimator is desired. To improve the estimation accuracy, we firstly average multiple lateral images along the $z$-axis to enhance the contrast of key features in the $x$-$y$ plane. Then a popular image registration algorithm—multi-modal volume registration [64] is applied to estimate the shifts among these averaged lateral images. According to the registration algorithm, we seek to maximize the mutual information between the reference image $u$ and test image v:

$$\widehat{T}=\underset{T}{\text{ArgMax}} I\left(u\left(x\right),v\left(T\left(x\right)\right)\right).$$E9

Here, $T$ is a transformation from the reference image to the test image. $v\left(T\left(x\right)\right)$ is the test image associated with the reference image $u\left(x\right)$ after transformed with $T$. We treat $x$ as a random variable over coordinate locations in $u$ and $v$. The best transformation $\widehat{T}$ can be estimated by algorithms [64, 65, 66] to maximize the mutual information $I$ between $u$ and $v$.

This $\widehat{T}$ is considered as the motion operator $F_k$ in Eq. (4) for the $k$th low resolution in vivo lateral image to the reference one. After the approximation of all shifts $F_k$, the following superresolution processing as described in Section 2.3 would be applied for the lateral resolution improvement. Here, the spatial shifts among multiple C-scans are caused by random body motions and vibrations, and we do not introduce any intended sub-spot-spacing shifts.

2.5 SD-OCT image acquisition and superresolution processing

The SD-OCT image acquisition is a lateral spot scanning image acquisition process where the depth tomographic information in $z$-axis is obtained intrinsically for each scan point. Our superresolution processing is to analyze and improve the lateral resolution in $x$-$y$ plane. Thus, we need to transfer $z$-axis information of numerous points to multiple $x$-$y$ plane layers. First, we perform the SD-OCT C-scan, acquiring multiple B-scan images $I\left(x,z\right)$ in the $x$ direction at different $y$ (see Figure 8(a)—left). These B-scans can be arrayed in sequence to generate a 3D matrix $I\left(x,y,z\right)$ as shown in Figure 8(a)—middle. We then retrieve a sequence of 2D $x$-$y$ images $I\left(x,y\right)$ at different depth $z$ as shown in Figure 8(a)—right, for later processing. The lateral resolution improvement is to use several $x$-$y$ images at an identical $z$ position (see Figure 8(b)—middle) but from slightly lateral shifted C-scans (A, B, C, etc. from Figure 8(b)—left) to perform multi-frame superresolution processing, yielding a higher lateral resolution image $\widehat{S}\left(x,y\right)$ as in Figure 8(b)—right. Repeat the process in Figure 8(b) layer by layer for all depth $z$ layers can yield a higher lateral resolution 3D image in the whole space, not shown.

Figure 8.

Extract lateral images from OCT tomography images to reconstruct higher lateral resolution images. (a) Left: The original B-scan tomography images $I\left(x,z\right)$ at different $y$ positions. Middle: Array these B-scans to form a 3D image $I\left(x,y,z\right)$. Right: Extract $x$-$y$ lateral images I(x, y) at different depth $z$ layers to generate a new lateral image stack. (b) Left: A series new lateral shifted image stacks (labeled blue A, B, C) obtained from the processing of (a). Middle: Exact multiple x-y images I(x, y) from the left stacks at the identical depth $z$. Right: Superresolution processing is applied to these $I\left(x,y\right)$ images at the same depth $z$ to generate a high lateral resolution image. Repeat process (b) for all $z$ depth layers can yield a high lateral resolution 3D image stack, not shown.

3. Experiments and results

3.1 Lateral resolution, image quality, and efficiency improvement

We compare the performance of our superresolution technique with designated shifts to other traditional methods, such as high density scan and multiple frames averaging, in three aspects: lateral spatial resolution, image quality, and scan time.

1. Lateral spatial resolution: as we mentioned in Section 2, spatial resolution represents the ability to distinguish the smallest discernible detail in the object, such as closed line pairs, which is an important indicator to all imaging systems. A standard negative resolution targets (R3L3S1N—Negative 1951 USAF Test Target, Thorlabs), as partly shown in Figure 9, is used to evaluate the resolution improvement. This resolution target provides 10 groups (−2 to +7) with 6 elements per group, offering a highest resolution of 2.19 μm. Considering our beam spot size in Table 1, group 4–5 and 6–7 are suitable for resolution testing of our OCT system with 100 and 30 mm focal length lenses, respectively. The resolution (the gap between two lines, the same as the width of 1 line) of group 4–7 is listed in Table 2.

   The resolution target is with a negative clear tone glass pattern. The chrome area appears dark because of blocking the backlight illumination while the transparent patterns are bright. Usually the SD-OCT system is more sensitive to reflectivity enhancement than reduction, and thus a resolution target with sudden reflection reduction is better for judging the resolution limit of the system. Successfully imaging and distinguishing these fine patterns is an effective way to demonstrate both the high lateral resolution and high sensitivity of our technique.

2. Image quality: we take the peak signal-to-noise ratio (PSNR) and the dynamic range (DR) as two indicators to evaluate the image quality improvement. The PSNR definition is given as [36, 67]:

   $$\mathit{\text{PSNR}}=20{log}_{10}\frac{{\mathit{Max}}_{\mathit{\text{signal}}}}{{\mathit{STD}}_{\mathit{\text{noise}}}}.$$E10

   Here, ${\mathit{STD}}_{\mathit{\text{noise}}}$ is the standard deviation (STD) of the background noise. Higher PSNR means higher image quality and lower noise. Usually, an acceptable image quality should be with PSNR >20 dB. The DR is defined as [68]:

   $$\mathit{DR}=20{log}_{10}\frac{{\mathit{Max}}_{\mathit{\text{signal}}}}{{\mathit{RMS}}_{\mathit{\text{noise}}}}.$$E11

   Here, ${\mathit{RMS}}_{\mathit{\text{noise}}}$ is root mean square (RMS) of dark noise. Higher DR means we can distinguish more details in both dark and bright areas of an image. For an OCT system, we expect to extract more information of deep layers, imaging weak structure signal from the noise.

3. Scan time: in order to compare the scan time of different methods in a simple way, we take the scan time of 64 × 64 matrix as unit 1 (∼0.18 s) for reference. Higher density 128 × 128 scan takes 4 units. Superresolution with 9 shifted low density C-scans of 64 × 64 takes scan time of 9 units. In experiment, we buffer the scan data and perform the fast Fourier transform subsequently to ensure the shortest scan time. Shorter scan time is very important for in vivo 3D imaging avoiding motion errors and artifacts [69]. Even for 3D imaging of static non-biomedical samples, a short scan time would still be needed to reduce the waiting time and improve the work efficiency, especially in the mass production.

Figure 9.

(a) The image of group 4–7 of the negative 1951 USAF test target is taken by ZEISS SteREO Discovery.V20 microscope. Due to the back illumination, the transparent patterns appear bright white and the chrome portion is dark. (b) The enlarged middle portion of (a) showing details of group 6–7.

Element	Group number
	4		5		6		7
	Density of line pairs (lp/mm)	Width of 1 line (μm)	Density of line pairs (lp/mm)	Width of 1 line (μm)	Density of line pairs (lp/mm)	Width of 1 line (μm)	Density of line pairs (lp/mm)	Width of 1 line (μm)
1	16.00	31.25	32.00	15.63	64.00	7.81	128.00	3.91
2	17.96	27.84	35.90	13.92	71.80	6.96	143.70	3.48
3	20.16	24.80	40.30	12.40	80.60	6.20	161.30	3.10
4	22.63	22.10	45.30	11.05	90.50	5.52	181.00	2.76
5	25.40	19.69	50.80	9.84	101.60	4.92	203.20	2.46
6	28.51	17.54	57.00	8.77	114.00	4.38	228.10	2.19

Table 2.

Lookup table of negative 1951 USAF test target.

Using a 100 mm focal length lens with a $\mathit{NA}$ of 0.015 we performed SD-OCT imaging of the resolution target shown in Figure 9(a). In Figures 10 and 11, a set of OCT lateral images are compared, which were acquired by different scan matrixes and processing methods but with the same FOV of ∼1 × 1 mm². All the images were taken in the same experiment with the same focusing condition and light source power. The output images were uniformly set as 8-bit gray TIFF format for comparison.

Figure 10.

OCT lateral images of the negative resolution target (group 5–7 region) with the same fixed FOV (∼1 × 1 mm²) but different acquisition methods. The scan matrix is shown in column 1 and the corresponding scan time is in column 2. The whole OCT lateral image is exhibited in column 3. The blue square in the image shows distinguishable and indistinguishable elements, enlarged in column 4 for further resolution limit comparison. The background noise in the selected ROI (the red square region in column 3) with STD, PSNR, RMS and DR values is enlarged in column 5. The original low density scan with 64 × 64 non-overlapped spot array is shown in row (A) and set its scan time as reference unit 1. OCT images acquired with different scan matrixes are given in rows (B)–(F). The averaged result of five 1024 × 1024 frames is given in (G). The reconstructed image by superresolution processing from 961 shifted 64 × 64 low resolution input images (1/16-spot-spacing step and maximum 15/16-spot-shift, forming a 31 × 31 shift matrix similar to Figure 7(d)) without deconvolution processing (H) and with Lucy-Richardson deconvolution processing an optimized Gaussian PSF (I) or with the blind deconvolution (J). We average the same 961 low resolution input images with up sampling and shift compensation, shown in (K). All the four images (H)–(K) have the same translation shift parameters and the same output image size. The optimized Gaussian PSF and the estimated PSF by the blind deconvolution are shown at the right bottom of resolution image in (I) and (J). (L) the inverted $x$-axis horizontal profiles of the yellow lines in the center of patterns G5E6 of (G) and (H), demonstrating the effective resolution improvement in (H). By the visual comparison, the image resolutions in (I) and (J) are obviously better than that (H). *The statistical values are for reference only since much fewer pixels in ROIs of (A)–(C), compared to the pixel numbers in (D)–(K). SR in (H) is short for superresolution. SR w LR de in (I) is short for superresolution with Lucy-Richardson deconvolution. SR w blind de in (J) is short for superresolution with blind deconvolution.

Figure 11.

A list of superresolution processed images with much fewer input C-scans. Superresolution processed images (A) and (B) with scan strategies as in Figure 7(b) and (c), respectively. Superresolution processed images (D) and (E) with scan strategies as in Figure 7(d) without and with gray shifts, respectively. Image (C) is the Lucy-Richardson deconvolution of image (B) using an optimized Gaussian PSF and the Gaussian PSF is shown at the right bottom of (C). Superresolution with blind deconvolution reconstructed image (F) from the same input images as (E) and the estimated PSF is shown at the right bottom of (F). *The values are for reference only due to low pixel numbers in ROIs. SR in (A)–(E) is short for superresolution. SR w LR de in (C) is short for superresolution with Lucy-Richardson deconvolution. SR w blind de in (F) is short for superresolution with blind deconvolution.

In Figure 10, the scan matrix is given in the first column (such as 64 × 64) and the corresponding scan time (taking 64 × 64 scan time as unit 1) in the second column. Column 3 is the OCT lateral image of the resolution target. Column 4 shows the enlarged image of the blue area of the column 3, comparing the barely distinguishable element (the first row) and the indistinguishable element (the last row). Here, we simplify Group i Element j on the resolution target as GiEj. The background noise image of the red region in column 3 is enlarged in column 5 with detailed noise statistics (STD, PSNR, RMS and DR values). The comparisons on lateral resolution, image quality, and scan time in Figures 10 and 11 are summarized in Table 3.

A. Lateral spatial resolution. Figure 10(A) is the reference low resolution image with 64 × 64 scan matrix. Thus, there is no beam spot overlapping like Figure 3(b). We barely see the resolution element in G4E3 which spatial resolution is about 25 μm. Such low resolution is due to the low scan density or undersampling. When increasing the scan matrix to 128 × 128 (B), 256 × 256 (C), 512 × 512 (D), 1024 × 1024 (E), and 2048 × 2048 (F) within the same fixed FOV, lateral resolution is obviously improved, indicating the higher scan matrix density in general can contribute to the lateral resolution improvement. However, increasing the scan matrix density from 1024 × 1024 to 2048 × 2048, we only observe slight improvement. Further increasing the scan matrix density will not contribute to the lateral resolution but significantly prolong the scan time. From this trend, the maximum resolution is barely seen in 1024 × 1024 lateral image (E) as G5E4 line width of 11.05 μm which is close to our focused beam spot radius of 10.5 μm.

Scan time^a	High density scanning					Multi-frame superresolution
	Scan matrix^b	Spot spacing	Lateral resolution (μm)	PSNR^c (dB)	DR^c (dB)	Low resolution C-scans	Shift	Lateral resolution (μm)	PSNR^c (dB)	DR^c (dB)
1	64^2	1	24.80	22.54*	15.21*	1	0	24.80	22.54*	15.21*
4	128^2	1/2	15.63	19.92*	13.04*	4	1/2	13.92	23.24*	15.22*
9	—	—	—	—	—	9	1/2	12.40	25.66*	17.27*
						9de^4	1/2	11.05–12.40	19.67*	15.24*
16	256^2	1/4	12.40	19.23*	12.54*	16	1/4	11.05	27.39*	19.19*
49	—	—	—	—	—	49	1/4	9.84–11.05	31.93*	22.48*
						49de^d	1/4	9.84	24.10*	19.95*
64	512^2	1/8	11.05–12.4	18.58	12.06	—	—	—	—	—
225	—	—	—	—	—	225	1/8	9.84	32.58	22.87
						255de^d	1/8	7.81–8.77	27.40	21.77
256	1024^2	1/16	11.05	17.50	11.50	—	—	—	—	—
961	—	—	—	—	—	961	1/16	8.77	31.50	23.88
						961de^d	1/16	7.81	27.16	22.32
1024	2048^2	1/32	9.84–11.05	17.60	11.70	—	—	—	—	—
1280	1024^2 × 5	1/16	9.84	23.39	15.53	—	—	—	—	—

Table 3.

Comparison of the lateral resolution and image quality VS scan time by different methods.

^a

The scan time here take the original low density 64 × 64 C-scan without spot overlapping as unit 1 for reference.

^b

64² means 64 × 64, 128² means 128 × 128, and so on.

^c

The higher, the better.

^d

“de” means superresolution with deconvolution processing shown in Figures 10(I), (J) and 11(C), (F).

*

The values are only for reference.

The $\mathit{NA}$ of the measurement arm lens is 0.015 with focused beam spot size of about 21 μm. The focal length of the lens is 100 mm.

Except the scan density, further increasing lateral resolution should consider suppressing the background noise. We applied the traditional multi-frame averaging approach to average five of 1024 × 1024 scanned lateral images, resulting in an improved image in (G) showing visibility of G5E5 of 9.84 μm line width while G5E6 still indistinguishable as the profile in (L) left. Averaging more frames such as 10 would further reduce the noise but cannot improve the lateral solution to G5E6 (not shown here). Also, 10-frame lateral averaging takes too much scan time, unacceptable in a practical OCT 3D imaging.

Compared with high scan density and multi-frame averaging methods, our superresolution processing with designedly shifted low resolution C-scans can effectively improve the lateral resolution. Figure 10(H) shows our superresolution processed image with 961 input low resolution shifted C-scans (1/16-spot-spacing step and maximum 15/16-spot-shift). It is a 31 × 31 shifted scan matrix similar as the 7 × 7 matrix in Figure 7(d). From the enlarged resolution image in column 4, we can distinguish G5E6 of 8.77 μm line width which is also verified in (L) right. Besides, in order to verify the effectiveness of the superresolution algorithm, we up sampled the 961 input low resolution images to the same image size as (H) by bicubic interpolation, and then averaged them with shift compensation. Although the output image (K) has the same image size of (H), the spatial resolution is terrible, barely observing 12.40 μm line width pattern (G5E3), worse than both the high density scan and the multi-frame averaging. This comparison demonstrates that the lateral resolution improvement is from both sub-spot-spacing shifted information and the superresolution algorithm, not only more data collection.

After reconstructing the non-deconvolved high resolution image (H) from a series of low resolution images, further lateral resolution improvement should be achieved by Lucy-Richardson deconvolution processing of image (H) with an optimized Gaussian PSF in (I) or by blind deconvolution processing shown in (J) as we discussed in Section 2, both clearly exhibiting G6E1 of 7.81 μm line width without additional hardware configuration. Also, the superresolution with deconvolution processing obviously enhances the contrast of the resolution element. All three lines in G5E6 in (I) and (J) are much clearer than in (H), indicating the effectiveness of deconvolution methods. The optimized Gaussian PSF was selected by iteration changing of Gaussian parameters to achieve the best output image. Different from the Lucy-Richardson deconvolution with a manually selected Gaussian PSF, the blind deconvolution can automatically estimate an optimized irregular PSF and thus deblurred the image in (H) with less ringing artifacts, although it still introduces some additional noise to the background. Thus, for the following deconvolution processing, we mainly use the blind deconvolution algorithm. We also attached the Gaussian PSF or estimated irregular PSF at the right bottom of the deconvoluted image.

Compared with original C-scan (A), our superresolution technique improves the lateral resolution from 25 to 8.77 μm (H) (without deconvolution processing) and to 7.81 μm (with deconvolution processing, in (I) and (J)), a factor of ∼3 times improvement. According to the above discussion, we can summarize that for lateral resolution improvement, the superresolution technique with shifted low density C-scans is better than multi-frame averaging of several high density C-scans, which is better than one set simple high density C-scan. The superresolution with deconvolution processing will further improve the lateral resolution.

According to Rayleigh criterion [63], the resolution limit of an optical system is restricted to half of the focused spot size. Our present beam spot radius was measured as ∼10.5 μm, similar to the 9.84 μm line width of G5E5 in Figure 10(G). This agrees well with the theory of diffraction limit. We can actually observe the 8.77 μm line width pattern of G5E6 in (H), which is slightly better than the spot radius due to increase of pixel density, reduction of noise, and enhancement of image contrast by our superresolution technique.

The resolution of an optical system is physically restricted by the diffraction limited, or PSF in other words. Dense patterns cannot be distinguished are due to finite spot size blurring. The digital deconvolution processing with a proper PSF can break the diffraction limit for resolution and sharpness improvement. Superresolution processing with Lucy-Richardson deconvolution using an optimized Gaussian PSF in Figure 10(I) or with blind deconvolution in (J) can clearly exhibit the G5E6 line width of 8.77 μm with higher image contrast and further show the next group element G6E1 with 7.81 μm line width, both breaking the diffraction limit and significantly improving the lateral resolution.

B. Image quality. Simple high density scan did not change the image quality. Taking the six images in Figure 10(A)–(F) as examples, all their PSNRs were almost lower than 20 dB, demonstrating that the increase of scan density did not do any help to the image quality. Actually, with the exactly same focusing condition and light source power, the six images should have very similar quality. Although we see a little better PSNR and DR in (A)–(C), that is due to not enough pixel numbers in region of interest (ROI) which reduces the statistics reliability. Thus, for image quality comparison with other methods, we use scan matrix of 1024 × 1024 in (E) and 2048 × 2048 in (F) as reference.

Through five-frame averaging, the PSNR in (G) is improved to 23.39 dB, better than the value 17.50 dB in (E). A 10-frame averaging can further reach 27.75 dB (not shown) but it is still lower than 30 dB and doubling the scan time of five-frame averaging. The superresolution processed image in (H) can achieve 31.50 dB PSNR, almost doubling the dB values of the high density scan results in (E) and (F). Although all the images in Figure 10 have the same 8-bit gray range between 0 and 255, we recognize that the superresolution processed image shows better contrast and thus looks brighter. That is because the higher DR and lower background noise in (H), (I) and (J) makes brighter appearance to human eye observation. The DR value of (H) also doubles the values of (E) and (F) in dB unit. The superresolution with deconvolution processed images in (I) and (J) decrease a little in the image quality as compared to (H), because of the increased background noise by the deconvolution processing. Here, we simply summarize the image quality comparison that the superresolution processing is better than the multi-frame averaging which is better than high density scan. The superresolution with deconvolution improves the image resolution but slightly reduces the image quality.

Besides, we noticed that (K) has the best PSNR and DR value among all the images of Figure 10, which comes from averaging the 961 up-sampled low resolution images (the same input images as (H)) with shift compensation. The STD value is only 1.63, exhibiting very smooth background without obvious noises. If only focusing on the image quality values, we may be misled that the average of up-sampled images can provide better background noise suppression than the superresolution technique. However, this method penalizes the high frequency signal, resulting in a poor resolution of 12.4 μm, even worse than the high density scan in (E), which is not an acceptable method.

C. Scan time. From column 2 of Figure 10(F)–(J), it is easy to summarize that the scan time of the superresolution technique is faster than both the high density scan and the multi-frame averaging. Superresolution processing provides much better image resolution and quality with less scan time. Figure 10 compares the resolution limit of different methods and thus takes long scan time. For example, the present scan time of Figure 10(A) for FOV of ∼1 × 1 mm² takes 0.18 s while that of (F), (G), and (H) take 3.41, 4.27, and 3.2 min, respectively. If enlarging the FOV to ∼3 × 3 mm² area and keeping the same scan density of (F), (G), and (H), these methods would take 30.7, 38.4, and 28.8 min scan time (excluding fast Fourier transform calculation), too long for many applications. In practice, we need to consider acceptable scan time for in vivo imaging and the effectiveness of the experiments.

To reduce the scan time, we compare a list of superresolution processed images in Figure 11 with much fewer input C-scans than Figure 10(H). Also, there are two different shift strategies applied in this experiment similar as Figure 7(b) and (c) to demonstrate the additional gray shifts in Figure 7(c) are needed for higher lateral resolution and image quality. Although the red shifts are enough for sampling rate improvement by superresolution processing, those additional gray shifts could contribute to image noise reduction, the lateral spatial resolution and overall image quality improvement. In Figure 11(A), the pattern G5E3 is indistinguishable, processed with Figure 7(b) scan strategy. While with more shifts as Figure 7(c) strategy, the pattern of G5E3 in Figure 11(B) is clearly visible and we can further partly distinguish the G5E4 pattern. Similarly, the G5E5 in Figure 11(D) is not obvious with red shifts only in Figure 7(d). After including the additional gray shift patterns, the G5E5 pattern in Figure 11(E) becomes visible. Thus, these gray shifts can effectively improve the lateral resolution as well as reduce the background noise by about 20–70% in STD and RMS, overcoming the reconstructed image with red shifts only.

The superresolution processing with Lucy-Richardson or blind deconvolution has demonstrated its contribution to the resolution improvement again, shown in Figure 11(C) and (F) as compared to (B) and (E), respectively. The deconvolution also introduces some degradation to image quality, increasing the background noise similar as in Figure 10(I) and (J). It is important to note that the superresolution with deconvolution does not spend any extra scan time.

Comparing with Figure 10(B)–(E), the results in Figure 11 clearly show the advantage of our multi-frame superresolution processing with less scan time while offering much better lateral resolution and image quality. Reducing from 961 to 49 shifted C-scans, it only takes 9.8 s to see the 9.84 μm line width pattern in Figure 11(F), while the 1024 × 1024 high density scan in Figure 10(E) takes about 51 seconds to barely observe the 11.05 μm line width pattern with lower image quality. Similarly, Figure 11(A)–(E) provide higher lateral resolution and better image quality with shorter scan time than Figure 10(B)–(E). Clearly, our superresolution technique has demonstrated its superior performance in lateral resolution and image quality improvement with shorter scan time.

Based on the above experiments, the lateral resolution and image quality vs. scan time are summarized in Table 3. Obviously, the multi-frame superresolution technique can achieve much better lateral resolution and image quality with less scan time than high density scanning and multi-frame averaging.

Except the experiment with 100 mm focal length lens above, we also checked the performance of a 30 mm focal length lens, which focuses the collimated beam to the diameter of ∼6 μm, very suitable to image the patterns group 6–7 in the resolution target of Figure 9(b). Figure 12(A)–(D) exhibits the original, the high density scanned, the average of multiple high density scans, and our superresolution with deconvolution processed images, respectively. The original low density scan (64 × 64) cannot distinguish any pattern, except the G6E1 with line width of 7.81 μm in Figure 12(A). With extremely higher scan density of 2048 × 2048 (taking 1024 scan time units) or averaging of five 1024 × 1024 scanned images (1280 time units), the 3.10 μm (G7E3) and the 2.76 μm (G7E4) become barely visible as shown in Figure 12(B) and (C). After the multi-frame superresolution with blind deconvolution processing of 961 shifted low resolution images (similar as Figure 12(A), with 1/16-spot-spacing shift step and maximum 15/16-spot-shift), we can see the 2.19 μm patterns (G7E6) as in Figure 12(D). The lateral resolution has been significantly improved from 7.81 μm (the original sparse scan in Figure 12(A)) to 2.47 μm (superresolution processing without deconvolution, not shown) and to 2.19 μm (superresolution processing with blind deconvolution in Figure 12(D)), about 3–3.5 times improvement. Compared with other methods like the high density scan and the multi-frame averaging, our superresolution technique exhibits superior advantage in lateral resolution improvement again. Our technique also shows the apparently better image quality than other methods: PSNR and DR of 103.7 and 137.9% (without deconvolution) and 65.2 and 106.3% (with deconvolution, Figure 12(D)) higher than high density scan (Figure 12(B)) in dB unit; PSNR and DR of 50.9 and 60.5% (without deconvolution) and 22.4 and 39.2% (with deconvolution, Figure 12(D)) higher than the multi-frame averaging (Figure 12(C)) in dB unit. Similar as the experiment of using 100 mm focal length lens, the use of 30 mm focal length lens demonstrates again that our superresolution technique can offer higher lateral resolution and better image quality with less scan time than the high density C-scan and the multi-frame averaging method.

Figure 12.

The OCT lateral images of the resolution target in Figure 9(b) were taken by a 30 mm focal length achromatic lens using (A) 64 × 64 scan matrix, (B) 2048 × 2048 scan matrix, (C) 1024 × 1024 scan matrix with 5 frame averaging, and (D) the superresolution and blind deconvolution processed image from 961 shifted 64 × 64 low resolution images with 1/16-spot-spacing step and maximum 15/16-spot-shift. The estimated PSF is attached at the right bottom of (D). All the four images have the same FOV of about 250 × 250 μm².

The present Lucy-Richardson deconvolution with a Gaussian PSF or the blind deconvolution with an estimated PSF however have some problems: Although the deconvolution effectively improves the lateral resolution, it introduces some artifacts in Figures 10(I), (J), 11(C), (F) and 12(D), which may not be acceptable for some critical applications. The artifacts are from both imperfect PSF selection and the discrete Fourier transform. And they cannot be avoided in the advanced blind deconvolution.

To our observation, the deconvolution methods are sensitive to the noise level. If background noise is as low as Figure 10(K), the deconvolution processing will not introduce obvious artifacts (not shown here, referring to our previous work [11]). However, the method in Figure 10(K) is harmful to the spatial resolution. Practically, it is difficult to obtain a penetrated lateral image with so smooth background as well as maintaining high resolution due to various scattering mediums in the samples.

When a focused beam penetrating into a sample, the scattering would alter the cross-section profile of the beam. The optimized lateral PSF thus may be different in different samples and at different depth layers [45]. Even with advanced blind deconvolution, the ground true PSF [70] of the system at that depth layer is still difficult to find. We also could notice that the estimated PSFs in Figures 10(J), 11(F) and 12(D) are different.

Considering the above issues, we would not apply the deconvolution processing to the following OCT experiments of functional samples. However, it is important to point out that the superresolution technique with deconvolution processing can break the diffraction limit, improve the sampling rate and suppress the background noise together to significantly improve the lateral resolution and image quality.

3.2 Improved lateral resolution imaging of microstructure samples

Thus far, we have successfully demonstrated the effective lateral resolution and image quality improvement by the multi-frame superresolution processing with shifted low resolution C-scans. This processing can offer better image quality with less scan time than high density C-scan images and is especially suitable for imaging micron scale fine structures [19, 37, 71, 72, 73, 74].

We examined 3D imaging of a microstructure sample in Figure 13, in which the particle size is about 3 μm. The two left images of Figure 13(A) and (B) are the original sparse scan lateral SVP images of the same sample using 30 and 19 mm focal length lens with 1300 × 1300 and 500 × 500 μm² lateral FOV, respectively. Even with 19 mm focal length lens and ∼4 μm focused spot size, the microstructures are still invisible. After superresolution processing of 225 low resolution shifted frames with 1/8-spot-spacing step and maximum 7/8-spot-shift (using 15 × 15 shift matrix with arrangement similar to Figure 7(d)), we are able to observe clearly those microstructures and wavy surface caused by the imperfect fabrication in exposure and developing. This wavy surface is difficult to be seen in microscope imaging without topographic imaging capability. As our previous report, the multiple-frame superresolution processing can improve the lateral resolution of our SD-OCT with 19 mm focal length lens by ∼3 times, achieving 1–2 μm [37]. Although 19 mm focal length lens can provide better image resolution than 30 mm focal length lens due to smaller focal spot, it sacrifices the lateral FOV and axial DOF of the system. This trade-off should be considered when imaging different samples. In this experiment, the superresolution enhanced 30 mm focal lens system has provided good enough resolution ability to exhibit the details of the sample.

Figure 13.

Lateral SVP imaging of a microstructure sample without (left) and with (right) multi-frame superresolution processing. (A) 30 mm and (B) 19 mm focal length lens are used in the sample arm of our SD-OCT system. Except the ability of observation of ∼3 μm particles, the right images show wavy surface with various imperfect fabrications, which is difficult to be observed in microscopy imaging. The scale bars in (A) and (B) are 100 μm.

Except for better human vision, the lateral resolution and image quality improvements further benefit various machine vision algorithms, providing more details for feature detection. Our previous work has reported the superresolution assisted image stitching for achieving an ultra-wide lateral FOV. Taking Figure 14 as an example, we scanned a multi-layer microfluidic sample by the high density scan and our multi-frame superresolution with shifted C-scans introduced above. All the structures are visible in Figure 14(A) left, however with a lot of speckle noises. Applying the advanced SURF [75] feature detection algorithm to the left two adjacent SVP images, there are no correct feature pairs found between them. And the incorrect matching information fails the following image stitching, overlapping two left images as Figure 14(A) right. Actually, there is only 30% shared region for the left two images. This failure is because most machine vision algorithms are not robust to periodic structures and noisy background. After superresolution processing, the image quality is significantly improved as in Figure 14(B) left, although with the same pixel resolution. The improved images offer much more correct feature pairs, supporting the following image stitching algorithm to reconstruct a wide lateral FOV image successfully at right. This comparison demonstrates the superresolution technique would be an effective pre-processing for subsequent machine vision algorithms.

Figure 14.

(A) Failed image stitching from two noisy images. (B) The superresolution processed images provide more correct matched feature pairs and successfully help the subsequent stitching algorithm to reconstruct a wide FOV image.

As we discussed in Section 2.1, each lens has its lateral FOV limitation due to Petzval field curvature. For example, 1400 × 1400 μm² optimized lateral FOV for 30 mm focal length lens guarantees the overall high resolution for the whole C-scan region. However, this lateral FOV is obviously not enough to image a large sample with centimeter scale sizes. To overcome this drawback, we scan 6 nearby partial overlapped regions of a microstructure sample by a 30 mm focal length lens. Each local C-scan covers a FOV of 1300 × 1300 μm² and is enhanced by the superresolution processing. One of the SVP images is shown in Figure 13 (A) right. After repeating the image stitching layer by layer introduced in our previous work [37, 38], we generated a 3.2 × 2.3 mm² wide FOV seamless 3D image with high lateral resolution, as shown in Figure 15(A). Wide FOV images at three selected depth layers are shown in Figure 15(B). If enlarge the selected two B-scans (positions of the two arrows in the top view) in Figure 15(C), all adjacent parts are also stitched very well without any discontinuities. The details of the image stitching are given in our papers [37, 38].

Figure 15.

(A) Top view of a 3.2 × 2.3 mm² microstructure image stitched by six 1300 × 1300 μm² images. (B) Selected three depth layers in the stitched high lateral resolution 3D image. (C) Two B-scans in the stitched 3D image are selected by the arrows in (A).

Again, we stitched 10 close-by C-scans with 500 × 500 μm² FOV, imaged by a 19 mm focal length lens, to reproduce a 2.10 × 1.15 mm² wide FOV 3D image in Figure 16. Due to short focal length lens, this figure stitched by more C-scans only covers 1/3 area of Figure 15, although with higher lateral resolution of <2 μm [37]. The top view, selected layers, and selected B-scans are exhibited in Figure 16(A)–(C), respectively. The wide FOV images could be enlarged for stitching performance and image quality checking by readers. In principle, there is no limitation on lateral FOV enlargement by this image stitching technique while maintaining the needed high lateral resolution SD-OCT imaging by the multi-frame superresolution processing. While, for fully review the microstructure sample, the 19 mm lens need to image more than 30 adjacent regions due to small lateral FOV, spending at least 5 times more scan time than using a 30 mm focal length lens, thus it is only suitable for ultra-high lateral resolution imaging.

Figure 16.

(A) Top view of a 2.10 × 1.15 mm² microstructure image stitched by ten 500 × 500 μm² images. (B) Selected three depth layers in the stitched high lateral resolution 3D image. (C) Two B-scans in the stitched 3D image are selected by the arrows in (A).

3.3 Improved lateral resolution imaging of in vivo 3D fingerprint

The previous section has successfully demonstrated the superresolution processing enhanced 3D imaging for static samples. Actually, this quick and high quality 3D imaging technique is very suitable for time sensitive security applications such as in vivo 3D fingerprint identification. The traditional high density scan spends long time and easily leads to motion errors during the scanning. Using our sparse scan method, the SD-OCT only takes 2.8 s to acquire one 256 × 256 C-scan (excluding fast Fourier transform processing time), fast enough to avoid most motion errors within one C-scan cycle, assisted by a finger holder to reduce the potential body motions and vibrations. The in vivo unintended tissue movements lead to unknown spatial shifts among multiple C-scans. In order to apply the multi-frame superresolution technique to a series of in vivo sparse C-scans, the unknown shifts $F_k$ should be solved first. As we discussed in Section 2, we decompose these unknown spatial shifts into two directions: the depth direction $z$ and the en-face lateral plane $x$-$y$. The $z$-axis differences of two C-scans could be estimated by comparing their top positions. For more complex lateral intensity distribution, we utilize the effective multi-modal volume registration [64] to estimate the shift amounts in $x$- and $y$-axis for each two SVP images, which provides better lateral details. We also overlap the test image and the reference one with the shift compensation to double check the correctness of estimated lateral shifts. After collecting the $x$-, $y$-, $z$-position shifts information which produces best overlapping quality, the multi-frame superresolution processing is then performed layer-by-layer to improve the lateral resolution and reconstruct a high quality 3D image. The details of the estimation performance and overlap quality are given in our published work [11].

As discussed above, OCT has great potential in security applications, such as in vivo 3D fingerprint reader. Currently, fingerprint identification has been a dominant biometry technique, occupying about two-thirds of the biometry identification market [76]. Conventional optical or capacitive acquisitions of fingerprints can only capture a two-dimensional (2D) image of the surface, which have lots of limitations like pressure dependent skin distortions, skin damages, and wet or fuzzy fingerprints. More seriously, the traditional 2D fingerprint acquisition and analysis are not robust to detect fake fingerprint on spoofing attacks and identity thefts. Our superresolution enhanced SD-OCT could provide high quality 3D image to overcome 2D fingerprint reader. To demonstrate this idea, we examine in vivo 3D fingerprint imaging of a thumb (a 33-year-old male volunteer) to show the advantages. Successful imaging of subsurface eccrine sweat glands can serve as a good indictor to the SD-OCT image resolution and effectively defense fake fingerprint attacks which do not have these internal glands. Figure 17 shows two SVP images (covering about 5 × 5 mm²) of the eccrine sweat glands layer, which is the gap between the external and internal fingerprint layers, illustrated in Figure 18. The eccrine glands grow under the dermis and open out through the sweat pores on the surface. From the top view of the scanned fingerprint, these glands should appear as the dot style distribution through the whole region. However, due to the low resolution, the SVP image of the original sparse C-scan could not show the eccrine sweat glands distribution clearly. The enlarged yellow and blue local regions in Figure 17(A) only barely exhibit some brighter pixels, which cannot be distinguished from the background noise. After superresolution processing with 10 of such C-scans, the reconstructed eccrine sweat glands layer shows much higher lateral resolution and image quality. For example, the five gland spots in yellow and blue selected regions of Figure 17(B) can be clearly observed. We are also able to see the low contrast internal structures in the red and pink selected regions of Figure 17(B) which however cannot be imaged well in the original C-scans like image in Figure 17(A).

Figure 17.

(A) One original low resolution SVP image of the eccrine sweat glands layer. Below are the enlarged images of selected region with the yellow, blue, red and pink colors. (B) the superresolution processed image showing the improvement in the enlarged local images. The scale bars in (A) and (B) are 500 μm. 100 mm focal length lens is used here.

Figure 18.

(A) An original sparse B-scan covered about 5 mm scan width on a thumb. The yellow and blue rectangles are the enlarged areas pointed by the yellow and blue arrows, respectively. (B) One B-scan image extracted from the superresolution processed 3D image, at the same position of (A). The scale bars in (A) and (B) are 500 μm. 100 mm focal length lens was used.

The layer by layer superresolution processing also improves the B-scan image quality. Figure 18(A) shows an original low resolution fingerprint B-scan image in the same C-scan of Figure 17(A). We only observe the external fingerprint pattern but with very blurred images of the eccrine sweat glands and the internal fingerprint structures. The two right side images are enlarged areas pointed by the yellow and blue arrows. The yellow rectangle image shows a blurred eccrine sweat gland but we cannot distinguish the helical structure. The blue square image does not exhibit any eccrine sweat glands. After the same superresolution processing as the Figure 17(B), we extracted one B-scan image shown in Figure 18(B) from the final high quality 3D image (Figure 19(A)) at the same position of Figure 18(A). In Figure 18(B), the helical structure of the eccrine sweat gland marked by the yellow arrow is clearly visible and enlarged at the right side. The three eccrine sweat glands have different intensity because their centers are not in the same B-scan plane. The superresolution processed B-scan exhibits excellent image quality with 49.9% PSNR and 50.6% DR improvement in dB unit. The improvement from Figure 18(A) and (B) can be clearly visualized. After separating the multi-layer fingerprint 3D image (Figure 19(A)) into three layers: external fingerprint layer, eccrine sweat glands layer and internal fingerprint layer, the curved layer images are shown in Figure 19(B)–(D), respectively. The distribution of eccrine sweat glands in the whole scan area are beautifully displayed in Figure 19(C). The application of colormap brings the gland distribution clearer than the gray scale mapping in Figure 17. The 3D fingerprint structure is shown in both Figure 19(B) and (D). Our superresolution enhanced SD-OCT successfully reconstructs the high quality in vivo 3D subsurface fingerprint image. According to other reports, the surface external fingerprint is actually a replicate of the 300 μm lower internal fingerprint structure (the primary ridges) [77]. The high quality imaging of the internal fingerprint with the same features as the surface could be a significantly improved fingerprint identification technique, benefitting from existing large fingerprint database and avoiding the heavy database rebuilding work for other biometric techniques such as iris scanning [78] and face recognition [79], as well as effectively defending against fake fingerprints without such inner structures.

Figure 19.

Superresolution enhanced SD-OCT in vivo 3D imaging of a male thumb fingerprint. (A) A 3D side view shows the multi-layers of a fingerprint. (B) 3D imaging of the external fingerprint layer, processed by our superresolution technique. (C) A 3D top view of the eccrine sweat glands layer after our superresolution processing. Each spot is an eccrine sweat gland. (D) A 3D top view of the internal fingerprint layer improved by superresolution processing, showing the same structure as the external fingerprint. 100 mm focal length lens was used here.

4. Conclusion

In conclusion, a high lateral resolution and high image quality SD-OCT 3D imaging has been achieved by the multi-frame superresolution technique, with shorter scan time than traditional methods. Through adjusting the matrix of control voltages to the galvanometer scanners, we intendedly introduce designed sub-spot-spacing shifts to low resolution C-scans for static sample imaging. After the multi-frame superresolution processing of these shifted C-scan images, about 3 times lateral resolution improvement has been demonstrated by imaging a standard resolution target, from 25 to 7.81 μm and from 7.81 to 2.19 μm with sample arm lens NA of 0.015 and 0.05, respectively. Significant background noise reduction and image quality improvement without sacrificing the axial DOF and lateral FOV have also been attained. Moreover, the improved lateral resolution and image quality could further benefit various machine vision algorithms sensitive to the noise, providing more features. In combination with our previous work, an ultra-wide lateral FOV and high image resolution and quality OCT has been implemented for static non-medical applications, such as imaging a large microstructure sample.

We present that Lucy-Richardson deconvolution with an optimized Gaussian PSF and the advanced blind deconvolution may potentially break the diffraction limit to further improve the lateral resolution of OCT systems. Although the PSF is highly dependent on samples and depth layers as well as the deconvolutions are sensitive to noise levels, we show the conceptual significance of our superresolution with the following deconvolution in lateral resolution improvement.

For in vivo imaging of biometry identification, due to the concern of live body unintended vibration, the multi-volume registration algorithm is used to estimate translational shifts in $x$-$y$ plane without introducing sub-spot-spacing shifts. Then the same multi-frame superresolution processing with the estimated shifts successfully improve the lateral resolution for in vivo imaging. The in vivo layered 2D lateral images, B-scan tomography images and 3D images of a live fingerprint have shown remarkable lateral resolution and image quality improvement, compared to original C-scan images. The high quality imaging of internal fingerprint and the eccrine sweat glands could effectively defend fake fingerprint on spoofing attacks and identity thefts in important security applications.

Although the present study depends on a SD-OCT system, the superresolution technique is able to work with other scan based OCT imaging system including time domain OCT and swept source OCT, benefiting various medical and non-medical OCT imaging applications.

Acknowledgments

We thank New Span Opto-Technology for providing the SD-OCT system.

Conflict of interest

The authors declare no conflict of interest.