Confocal microscopy has gained great popularity in the observation of biological microstructures and dynamic processes. Its resolution enhancement comes from shrinking the pinhole size, which, however, degrades imaging signal‐to‐noise ratio (SNR) severely. Recently developed super‐resolution method based on the pixel reassignment technique is capable of achieving a factor of 2 resolution improvement and further reaching twofold improvement by deconvolution, compared with the optical diffraction limit. More importantly, the approach allows better imaging SNR when its lateral resolution is similar to the standard confocal microscopy. Pixel reassignment can be realized both computationally and optically, but the optical realization demonstrates much faster acquisition of super‐resolution imaging. In this chapter, the development and advancement of super‐resolution confocal microscopy through the pixel realignment method are summarized, and its capabilities of imaging biological structures and interactions are represented.
- super resolution
- confocal microscopy
- pixel reassignment
- computational realization
- optical realization
Better understanding of biological processes at the cellular and subcellular level is closely dependent on the direct visualization of the cellular microstructures. Among the various microscopic techniques, fluorescence microscopy takes advantage of the abilities to observe in real‐time the molecular specificities in living biological samples down to the cellular and/or subcellular scale, and thus has found broad applications in the investigations of cell biology and neuroscience. However, the spatial resolution of conventional microscopy is optically diffraction‐limited, restricting its lateral resolution to be ∼250 nm and axial resolution to be ∼600 nm (primarily determined by the numerical aperture of microscopic objective), respectively. As a result, it is very challenging to resolve the subcellular structures by the conventional microscopic technologies because their microstructures are comparable to (even finer than) the diffraction‐limited resolution.
Fortunately, a number of novel fluorescence microscopic techniques with super‐resolution capability have been established to break down the optical diffraction limitation in recent years, allowing the observation of many cellular and subcellular structures that are always not resolvable by the conventional fluorescence microscopy. For example, by sharpening the point‐spread function of the microscope with the suppression of the fluorescence emission on the rim of a focused laser spot, stimulated emission depletion (STED) microscopy breaks the optical diffraction limitation and achieves resolution as high as ∼30 nm . Localization‐based techniques, such as stochastic optical reconstruction microscopy (STORM) and photoactivated localization microscopy (PALM), enable imaging at a resolution of ∼20 nm [2, 3]. Structured illumination microscopy (SIM) applies spatially structured light illumination for shifting the high spatial frequency to the low‐frequency range, which thus can be collected by microscopy . These methods achieve an order of magnitude improvement in spatial resolution over the conventional fluorescence microscopy. Therefore, the super‐resolution microscopic technology opens up new windows for observing the previously unresolved cellular structures and provides great potentials for elucidating biological processes at the subcellular and molecular scale .
Among these high‐resolution fluorescence microscopic techniques, confocal microscopy, the first super‐resolution imaging technique, is one of the most widely used imaging approaches with moderately enhanced spatial resolution. Utilizing a focused laser as an excitation source in combination with a pinhole in front of the detector for blocking out out‐of‐focus signals, confocal microscopy is able to improve the spatial resolution by a factor of in principle. However, instead of its super‐resolution capability, the sectioning capability is more impressed because the spatial resolution with a factor of improvement is hardly accessible in the standard confocal microscopy. The resolution of confocal microscopy relies on the pinhole diameter, that is, higher resolution comes from the smaller sized pinhole filter. Such a small pinhole rejects the unwanted out‐of‐focus light, while parts of the desired in‐focus emission are filtered out simultaneously. As a result, the signal‐to‐noise ratio (SNR) is drastically decreased as the pinhole size shrinks, which, in turn, practically deteriorates the spatial resolution. Instead, the fluorescence efficiency within the biological samples is often weak, so a relatively large pinhole diameter is typically chosen concerning the imaging SNR. Therefore, the standard confocal microscopy is practically unable to provide super‐resolution imaging.
In order to achieve spatial resolution improvement and better imaging SNR simultaneously in confocal microscopy, light/fluorescence signals should be detected with a nearly closed pinhole array instead of a single pinhole . The images acquired by each pinhole within the array have the same resolution but different SNR levels . To overcome this limitation, a method applying the pixel reassignment technique is proposed by reasonably summing the signals from each nearly closed pinhole together, which enables simultaneous improvement of resolution and SNR. In this chapter, we present the state‐of‐the‐art super‐resolution techniques based on the pixel reassignment. Section 2 gives the principle of pixel reassignment firstly, and then two different operations realizing the pixel reassignment. Also, some representative super‐resolution images in biological specimens are summarized in this section. At last, some advances in super‐resolution confocal microscopy through the pixel reassignment will be discussed.
2. Super resolution by pixel reassignment
The concept of pixel reassignment is firstly proposed more than two decades ago to solve the drawbacks in standard confocal microscopy . As we know, the reduction of the pinhole diameter down to zero allows the finest lateral resolution in confocal microscopy in theory, which, however, generates fluorescent images with a very low SNR due to the dramatically degraded light collection efficiency. Although the pinhole size can be adjusted to one Airy unit for better imaging SNR, the lateral resolution is sacrificed. Instead of a single pinhole, a pinhole array is used for the light detection, followed by a reconstruction algorithm for the image formation. As a result, the standard confocal microscopy with the pixel reassignment operation is capable of enhancing its lateral resolution simultaneously with higher imaging SNR.
2.1. Principle of pixel reassignment
Pixel reassignment demonstrates great potentials for improving both lateral resolution and imaging SNR. Instead of summing the signals directly as the conventional imaging technologies, each signal is reassigned to a particular location where the signal most probably comes. Figure 1(a) gives the principle of the pixel reassignment in terms of excitation and detection point‐spread function (PSF) . The excitation PSF (PSFex, labeled by blue line) represents the distribution of the corresponding excitation focus. At a displaced pinhole, detection PSF (PSFdet, labeled by green line) is centered on the detection axis with a distributed probability of signal detection around that pinhole. The effective PSF (PSFeff, labeled by red line) is contributed from the overlap (multiplication) of PSFdet and PSFex. The well‐aligned pinhole is coaxial with the excitation focus, realizing the maximal signal detection probability. As the pinhole detector is far away from the axis of the excitation focus, the signal acquisition probability decreases because of their less overlying; consequently, these nearly closed pinhole detectors induce lower‐SNR image.
In the pixel reassignment implementation, a camera (similar with a pinhole array), rather than a point detector, is commonly employed because its individual pixels are considered as infinitely narrow pinhole. Neglecting Stocks shift in single‐photon fluorescence and assuming identical PSFdet and PSFex, a maximal probability of signal acquisition (i.e. PSFeff) is at the midway of the peaks of PSFdet and PSFex. Figure 1(b) gives two methods for the pixel reassignment operation, either twofold local contraction of the excitation focus without altering the distance between them (panel in lower left of Figure 1(b)), or twofold increasing the distance between the foci while maintaining their original size (panel in lower left of Figure 1(b)) . By reassigning the signals from all pixels within the detector array (i.e. all displaced pinholes as shown in Figure 1(a)) to the particular location, a sharper and higher‐SNR image is eventually achieved.
Pixel reassignment technique is able to improve the resolution to a factor of without sacrificing SNR, and the resolution can be further improved by deconvolution algorithm up to a factor of 2 [9, 10]. Although the spatial resolution of the pixel reassignment technique is still lower compared with other super‐resolution methods, such as STED and STORM [1–3], it overcomes some of their shortcomings. This technique inherits all advantages of the standard confocal microscopy, including high‐speed imaging rate, acceptable excitation intensity, optical sectioning capability, and a broad choice of fluorescent dyes and/or proteins, making it a readily accessible technology in a variety of biological investigations.
The pixel reassignment can be considered as an alternative method of SIM, theoretically achieving the same spatial resolution improvement compare with standard SIM through point‐like illumination feature. In contrast, the technique demonstrates better feasibility over the standard SIM, that is, the pixel reassignment operation can be easily implemented both computationally and experimentally (optical system adaptation). Unlike computational mode that is always time‐consuming in raw data processing, the pixel reassignment realized with optical means is capable of obtaining super‐resolution images with fast imaging acquisition. More details on these two different methods for realizing the pixel reassignment are represented as below.
2.2. Computational realization of pixel reassignment
2.2.1. Image scanning microscopy
Image scanning microscopy (ISM), proposed by C. Müller and J. Enderlein in 2009, is a super‐resolution microscopic technique based on the pixel reassignment . This system is modified from a standard confocal microscopy that replaces the point detector (normally a photomultiplier tube) with an Electron multiplying CCD (EMCCD) camera (labeled 9) as shown in Figure 2(a). The camera takes an image of each spatial position of the scanning focus, and then an algorithm of the pixel reassignment processing is utilized by summing the raw images to reconstruct an ISM image, which improves the resolution from 244 nm to 198 nm laterally.
Further, deconvolution function is used to improve its lateral resolution up to 150 nm, 1.63‐fold better than the image from raw data, as shown in Figure 2(b) and (c), respectively. Note that the pinhole in ISM (labeled 8) filters the out‐of‐focus light signals, maintaining the optical sectioning capability as the standard confocal microscopy. In this work, the realization of the lateral resolution improvement up to 198 nm does not entirely rely on the pinhole because of its relatively large diameter, which, however, gives a high imaging SNR. Therefore, with the computational pixel realignment ISM is able to provide images with optimization of both spatial resolution and imaging SNR.
2.2.2. Multifocal structured illumination microscopy
ISM demonstrates multiple advantages, including the optical sectioning capability as the standard confocal microscopy, the enhanced lateral resolution, and the high fluorescence collection efficiency . However, it is subjected to slow frame rate due to the EMCCD camera (imaging acquisition of 10 ms with each scanning position), and is time‐consuming for visualizing the three‐dimensional (3D) microstructures.
In order to speed up the imaging acquisition, Shroff et al. developed multifocal structured illumination microscopy (MSIM) by using a sparse lattice of excitation foci (similar to swept‐field or spinning disk confocal microscopy) in 2011 . As shown in Figure 3, MSIM applies a digital micromirror device (DMD) for generating the sparse lattice illumination patterns. After a series of reconstruction steps (open‐source software), MSIM enables 3D subdiffractive imaging with resolution doubling, indicating a lateral resolution at 145 nm and an axial resolution at 400 nm. Moreover, it provides the capability of significantly fast imaging acquisition at one 2D image per second.
For super‐resolution MSIM, the data acquisition and processing are implemented as below (please refer to Figure 4 for detailed procedures). First, the sample is excited with a sparse, multifocal excitation pattern. Second, the resulting fluorescence image is recorded with a camera, and then the digital pinholes around each fluorescent focus are applied for rejecting the out‐of‐focus emission. Afterwards, the pixel reassignment with 2× scaling is used to process the resulting image. Repeat the above procedures for the entire imaging region fully illuminated. Eventually, a super‐resolution image with ‐fold resolution improvement is obtained through the digital summation of all such pinholed and scaled images. Twofold resolution improvement is further achieved with deconvolution.
The resolution improvement of MSIM is demonstrated by imaging antibody‐labeled microtubules in human osteosarcoma (U2OS) cells embedded in Fluoromount as shown in Figure 5. Compared to the wide‐field images, the multifocal‐excited, pinholed, scaled, and summed (MPSS) images have both higher resolution and better contrast (Figure 5(b)). In Figure 5(d), the full‐width at half maximum (FWHM) of light intensity of microtubules is estimated at about 145 nm in MSIM images, giving a twofold resolution enhancement compared with the image from wide‐field microscopy (∼299 nm). Moreover, the frame rate of acquiring an image with field of view at 48 × 49 μm is up to 1 Hz in MSIM, indicating more than 6500‐fold faster acquisition over the ISM technology .
2.3. Optical realization of pixel reassignment
The pixel reassignment implemented by the computational means is capable of doubling the resolution than wide‐field imaging [9, 11]. The limitation, however, is that the methods are fundamentally time‐consuming compared to the standard conventional microscopy because a large number of raw images are essentially acquired and processed. Recently, optically realized pixel reassignment has been developed to overcome the limitations by adapting the optical imaging system instead of digital data‐processing operations, which produces images with comparable improvement in the spatial resolution [8, 10, 12].
2.3.1. Instant structured illumination microscopy
Instant structured illumination microscopy (ISIM) is developed by Shroff et al. in 2013 that is analogous to MSIM, while its pixel reassignment process operates optically instead of the digital computation procedures . As shown in Figure 6, the DMD used in MSIM is replaced with a converging microlens array. As a result, a multifocal excitation pattern is generated in ISIM. Correspondingly, a matched pinhole array is added to physically reject the out‐of‐focus emissions. With this modification, the optical pixel reassignment is realized based on the matched microlens array for twofold local contraction of each fluorescent focus. The fluorescence emission pattern is imaged onto a camera by galvanometer scanning. Eventually, the pinholed and scaled images are optically summed, enabling ‐fold resolution enhancement.
ISIM demonstrates 3D super‐resolution imaging with a lateral resolution of 145 nm and an axial resolution of 350 nm, nearly comparable with MSIM. Moreover, the 100 Hz frame rate comes from the optical operation of pixel realignment in ISIM, allowing super‐resolution real‐time imaging (almost 100‐fold faster than MSIM). Taking into account the data processing duration, the speed‐up factor exceeds 10000. In addition, the low illumination power in ISIM (∼5–50 W/cm2) mitigates photobleaching. As a result, ISIM can perform imaging over tens of time points without obvious photobleaching or photodamage. In Figure 7, the rapid growth (∼3.5 μm/s) of endoplasmic reticulum (ER) is monitored by ISIM even though less than 140 ms in the formation and growth of new ER tubules. The biological processes blur in previously developed technologies, such as MSIM and ISM [9, 11]. The capabilities make ISIM a powerful tool for time‐lapse super‐resolution imaging in living biological samples.
2.3.2. Re‐scan confocal microscopy
Rescan confocal microscopy (RCM) is another optical realization of the pixel reassignment technique, proposed by Luca et al. in 2013 .Compared with ISIM, it is more easily accessible to build an RCM because this system can be readily modified from a standard confocal microscopy as shown in Figure 8. The optical pixel reassignment in RCM is realized as below. The focal length of the lenses L2 and L3 is adapted for twofold local contraction of the fluorescent focus spot. Alternatively, the final fluorescence image is twofold magnified while maintaining the original fluorescence foci size.
This process is accomplished by reasonably changing the angular amplitude of the rescanner. The ratio of angular amplitude of the two scanners, expressed by the sweep factor M, changes the properties of the rescan microscope. For M = 1 the microscope has the same lateral resolution with a wide‐field microscope, defined by the well‐known optical diffraction limit; it achieves the super resolution for M = 2. The rescanner is used to deliver the fluorescence emission onto the camera pixels. The camera is in the exposure status for optical summation of the fluorescent focus during rescanning.
The lateral resolution improvement of RCM is quantified by imaging 100‐nm fluorescent beads. FWHM is found to reduce from 245 nm (15 nm) in wide‐field imaging to 170 nm (±10 nm) in RCM imaging, indicating an improvement by a factor of without deconvolution. Also, the resolution improvement is concluded by visualizing fluorescently labeled microtubules of HUVEC cell in Figure 9(a)–(f). To demonstrate the capability of RCM for monitoring dynamics, the time‐lapse imaging of living HeLa cells expressing EB3‐GFP with the growing end of microtubules is observed by RCM. As shown in Figure 9(g), RCM is able to track the fast dynamics (0.5 μm/s) with multiple advantages of improved resolution, high sensitivity, and sufficient imaging rate (1 fps).
2.3.3. Two‐photon instant structured illumination microscopy
RCM improves resolution by a factor of compared with wide‐field imaging while possessing optical sectioning capabilities as the traditional confocal microscope . Two‐photon excitation offers better optical sectioning capability based on the nonlinear effect. Infrared excitation light minimizes the optical scattering in the tissue, and the fluorescent signals come only from two‐photon absorption. These advantages effectively increase the penetration depth and simultaneously suppress the background signal, making the two‐photon excitation technique an ideal imaging tool for the thick samples.
Two‐photon instant structured illumination microscopy (2P ISIM) is a combination of RCM and two‐photon excitation technique, presented by Shroff et al. in 2014, as shown in Figure 10(a) . Similarly, an additional scanning component is introduced in 2P ISIM for the optical realization of pixel reassignment. In Figure 10(b)–(d), 2P ISIM provides better resolution than the diffraction‐limited two‐photon excitation mode by imaging the microtubules. Applying the deconvolution, the lateral resolution is further improved in Figure 10(c). 2P ISIM is quantified by ∼150 nm in the lateral resolution and by ∼400 nm in the axial resolution, respectively, with 100‐nm diameter fluorescent beads as imaging targets. A factor of 2 (with deconvolution) resolution enhancement is obtained compared with the conventional two‐photon wide‐field imaging (∼311 nm).
To demonstrate the enhanced penetration ability of 2P ISIM in living thick samples, embryos of transgenic Caenorhabditis elegans expressing GFP‐H2B are imaged in Figure 11. Both imaging resolution and contrast severely degrade at depths of more than ∼15 μm from the coverslip surface in 1P illumination due to strong scattering in deep tissue (Figure 11(a), (b)). The degradation is not compensated by increasing of the exposure time, which, however, mainly leads to high background noise. Two‐photon excitation of 2P ISIM effectively suppresses the out‐of‐focus emission. Thus, the subnuclear chromatin structures are clearly observed up to the depth of ∼30 μm in Figure 11(c), (d), where the fluorescence signals slightly reduce as the depth increases.
In this chapter, we represent the super‐resolution confocal microscopy (and two‐photon microscopy) realized through the pixel reassignment methods computationally and optically. These demonstrate multiple advantages of resolution improvement, high fluorescence collection efficiency, optical sectioning capability, and fast imaging acquisition, which thus is able to investigate biological structures and processes at the cellular and even macromolecular level with 3D spatial scale. Additionally, because the method is directly established based on the standard confocal microscopy and/or two‐photon microscopy, it mitigates the requirements in fluorescent probes and/or labeling methods that are always indispensable in some super‐resolution fluorescence microscopic technologies, such as STORM and PALM [2, 3].
More importantly, the development of these techniques is not limited in the laboratorial stage. In 2015, the first commercial setup, LSM 800, is established by Carl Zeiss , which, in principle, is based on ISM but replaces the EMCCD camera with a 32‐channel linear GaAsP‐PMT array (i.e. Airyscan detector as shown in Figure 12). The highest imaging speed of LSM 800 with 512×512 pixels is up to 8 Hz, tremendous faster than ISM. Therefore, we expect that the super‐resolution microscopy based on the pixel reassignment technique has great potentials for boosting imaging acquisition speed, and therefore further provides better understanding in intracellular molecular interactions and dynamic processes within living biological specimens.
In addition to the issue of imaging acquisition speed, multicolor fluorescence microscopy is desired for investigating the interactions between different structures or biomolecules via labeling them with distinct colors. The possible interactions can be revealed by the co‐localization of the different dyes and/or proteins. The standard fluorescence microscopy, however, might give inaccurate co‐localization due to the diffraction‐limited resolution. In combination with the pixel reassignment, the multicolor imaging technique is anticipated to provide a high‐resolution imaging of the biological interaction within live cells.
In MSIM and ISIM based on the pixel reassignment approach [9, 10], both super‐resolution imaging capability and color differentiation have been demonstrated, which have the advantages of easily configured optical system and weak cross‐talk effect between the different colors. Switching laser lines for the excitation of different fluorophores might induce spatial mismatch in the images. Therefore, it is more preferable for simultaneously exciting all fluorophores and synchronously collecting their fluorescence signals. Multiple detectors with appropriate dichroic mirrors and emission filters can be used to collect the different fluorescence signals with different detection channels. Alternatively, an imaging spectrometer can be applied to record the spectral feature of these fluorophores.
Synchronous imaging decreases the fluorescence photobleaching probability due to low light exposure, benefiting to long‐term monitoring of living samples. However, cross‐talk of the different fluorophores always occurs because of the broad and overlapping excitation and emission bands of fluorophores. Although the cross‐talk effects can be removed by selecting dyes with appropriately wide and non‐overlapping emission spectra, the dyes are often inaccessible, which thus restricts its application in multicolor imaging. Linear spectral unmixing analysis is a solution to eliminate the cross‐talk effect in spectral imaging . The spectrum of the mixed fluorescent signal is expressed as a linear integration of the component dye spectra , and therefore the concentration or intensity of the fluorescence from each dye can be precisely analyzed. Based on the data analysis, both spatial mismatch and cross‐talk effect are mitigated in multicolor imaging of live cells.
In Figure 13, we establish a multicolor RCM with simultaneous excitation of different fluorophores and synchronous collection of their fluorescence. Linear spectral unmixing analysis is implemented for the spectral differentiation of the live cells stained with different dyes. SYTO 82‐labeled nucleus and LysoTracker Red‐stained lysosomes within live bEnd.3 cells are imaged by RCM with a spectrometer as the spectral detector. The nucleus and lysosomes are captured simultaneously, followed by the linear spectral unmixing analysis based on the known spectral features of these two dyes (severely overlapping as shown in Figure 13(e)). Figure 13(b)–(d) gives a clear separation of the two kinds of subcellular organelles. This approach is very powerful in investigation of the dynamic interactions of the subcellular structures.
This work was supported in part by the National Natural Science Foundation of China grant Nos. 61505238 and 11504042, Daqing Normal University Youth Foundation No. 12ZR12, Daqing Normal University doctor Foundation No. 15ZR03 doctor Foundation, Natural Science Foundation Project of Heilongjiang Province Nos. A200506 and QC2015066, Science and Technology Research Project of Heilongjiang Province Education Department No. 12543002, and Guidance of Science and Technology Plan Projects of Daqing City No. szdfy‐2015‐59.