High-Quality Seamless Panoramic Images

Image mosaicing is the creation of a larger image by stitching together smaller images. Image mosaicing has many different applications, such as satellite imagery mosaics (Chon et al., 2010,) the creation of virtual reality environments (Szeliski, 1996; Chon et al., 2007; Brown and Lowe, 2007,) medical image mosaics (Chou et al., 1997,) and video compression (Irani et al., 1995; Irani and Anandan, 1998; Kumar et al., 1995; Lee et al., 1997; Teodosio and Bender, 1993).

Image mosaicing has four steps: 1) estimating relative pose among smaller images to project on a plane or specific defined surfaces, 2) projecting those images on the surface, 3) correcting photometric parameters among projecting images, and 4) blending overlapping images.
The first step has two categories: 2D plane images (For example, scanned maps in sections and orthoimages) based methods and perspective image based methods. Orthoimages are generated by correcting each image pixel from a perspective image with a help of a Digital Elevation Model (DEM,) produced by a stereo camera system, a laser/radar aerial system or obtained of photogrammetric works. To align the images with the DEM, we need to estimate absolute orientation with ground control points (GCP.) In the case of perspective images, we need to estimate relative pose using homography, affine transformation, colinear conditions, and coplanner condition with feature correspondences. (Hartley and Zisserman & Books, 2003;McGlone et al. & Books, 2004;Szeliski & Books, 2011) The second step projects all images onto a specific defined surface, such as 2D plane, cylinder (Chen, 1995;Szeliski, 1996), sphere (Coorg and Teller, 2000;Szeliski and Shum, 1997), and multiple projection planes (Chon et al., 2007.) In the case of cylinder and sphere, we assume that images are captured by fixing the position of a camera and rotating it at the position. And all images are projecting onto a cylinder or a sphere surface.
In the third step, we have to balance photometric parameters among projected images because exposure affects the photometric parameters of each image. At the last step, we have to create one image from multiple overlapping images.
Ideally each sample (pixel) along a ray would have the same intensity in every image that it intersects, but in reality this is not the case. Even after gain compensation some image edges are still visible due to a number of un-modeled effects, such as vignetting (the decrease of intensity towards the edge of the image,) parallax effects due to unwanted motion of the optical center, registration errors due to an incorrect or approximate camera model, radial distortion, and so on.
Because of this, a good blending strategy is important (Brown and Lowe, 2007;Goldman, 2011;Kim and Pollefeys, 2008;Zomet et al., 2006.) In particular, applying seam-line detection algorithms before applying blending algorithms is an effective strategy when using image data that has parallax effects, because a seam-line detected by optimal path finding algorithms passes equal-depth pixels or similar colors pixels overlapping images. a) Vehicle system b) Quad cycle system c) 4 stereo fish-eye images d) Image and 3D range panoramas e) Applications of ArcGIS and AutoCAD Map3D Fig. 1. EARTHMINE Inc.'s 3D mobile mapping systems, 3D panorama generated from eight fish-eye images captured using the system, and its applications.
In this chapter, we introduce luminance balancing using gamma correction, Dijkstra's algorithm for seam-line detection, our proposed fast color blending algorithm on seam-lines, and color balance and saturation implemented on EARTHMINE Inc.'s mobile mapping system (MMS) that generates 3D panoramic images as shown in Fig 1.

Color balance
A major challenge in merging multiple shots into a single panorama is that each individual image experiences its own exposure setting, even if each camera (when there are multiple cameras) is color calibrated. A common scenario that may cause such discrepancy is when the shots are taken, some cameras face towards the light source (e.g. the sun), and others face away from the light source. Due to the different exposure to the physical light source, some images will be brighter than others. Merging these images together inevitably causes a visible seam at the merging points.
To overcome this problem, we describe an algorithm that detects the differences in the luminance and chrominance in each image and attempts to equalize such differences. First, we assume that there exist overlapping regions between neighboring images. For the set of images, we assume there exist a single correction factor per channel (one luminance channel that governs image intensity and two chrominance channels that governs the color of the image) per image such that the luminance/chrominance of the overlapping areas match up.
The ideal approach would be to find corresponding pixels between the overlapping areas and find the optimal correction factors using these correspondences. Specifically, we can write the optimization process in the following form However, finding pixel correspondences between pairs of images is non-trivial and most currently available algorithms such as SIFT (Lowe, 2004) are very computationally expensive. For applications that have limited processing power (e.g. image stitching on mobile platforms) or real time processing requirements, such approach is inadequate.
Brown and Lowe (2007) and Xiong and Pulli (2010) pointed out that because luminance/chrominance is a global entity, instead of using pixel correspondences, we can approximate them using the mean luminance / chrominance value between the overlapping region. Eq ( Note that there is a trivial solution to Eq(2.2), where when the γs are set to 0, the objective function is at its minimum. To prevent this, we add a regularization term in the form of 1γ, such that this term is non-zero when γ is 0. In addition, this term prevents the luminance / chrominance from changing too much, which is often desired. where w d and w p are weights for the two terms.
The human visual system perceives brightness according to the Weber-Fechner law where the perceived brightness is not a linear function of the physical brightness but rather follows a log function (red line in figure 2.1; blue line indicates a linear response). To account for this effect, we linearize the luminance channel with a power function such that the brightness perception curve is linear (some cameras have this linearization in the post-processing step. If so this step can be omitted where P is the number of overlapping pixels. We used a correction factor of 2.2 according to common practice (Poynton 2002). Xiong and Pulli (2009) also used a very similar objective. The final objective over all the images is thus In the above equation, we assume that there are only two overlapping images (left and right) for a given image. While this is often the typical scenario, it is straightforward to generalize to any arbitrary amount of images. This objective function thus can be optimized with any of-the-shelf optimization package.

Seam line detection
To create a seamless panorama from a multiple stereo fish-eye camera system, with a system like the EARTHMINE Inc, is necessary an optimal seam-line detection algorithm (see Figs.

1(c) and 1(d).)
Dijkstra's algorithm to find an optimal path within a cost space (Bellman, 1957;Dijkstra, 1959.) A seam-line can be detect by using the Dijkstra's algorithm with a cost map built by a cross correlation between two overlapping images (Milgram, 1975(Milgram, , 1977Davis, 1998;Efros and Freeman, 2001.) Chon et al. (2010) proposed seam-line algorithm explained with the waterway on the terrain structure. a) High water level creates many possible paths between two points A and B. b) Lowered water level makes the maximum cost smaller. Chon et al. (2010) proposed a novel algorithm for selecting a seam-line that at first identifies a subarea within two overlapping images such that in the subarea, at least a seam line exists and the maximal mismatch score in this subarea is minimized. Let the degree of mismatching of pixels between two images on the overlapped region form a cost field, which can be expressed in a 3D plot. If a threshold is set with all cost values under the threshold filled with ``water'', then the 3D plot will look like the ones in Fig. 4, where Fig.  3.1(a) uses a larger threshold. The plots show some ``water ways'' between points A and B, and the threshold defines the water level. Fig. 3.1(b) shows that the allowed path is not near the shortest one. The technique then further applies Dijkstra's algorithm to find an optimal path within the restricted subspace. In this optimization phase, a cost conversion is applied to make a higher cost (the mismatch score) larger. This enables the search to find a possibly longer seam-line with less highly mismatched pairs.
Kerscher (2001) proposed a method called the "twin snake algorithm" to detect seam-lines. The algorithm starts with two initial vertical lines as control points of the snakes on the overlapping images. The snakes have two energy terms which are internal and external energies in general (Kass et al., 1987;Leymarie and Levine, 1993;Tiilikainen, 2007;Williams and Shah, 1990). The sum of the mismatching values on those lines and the relationship between neighbor control points are called internal energy and external energy, respectively. The twin snake algorithm builds an energy function * snake E from four terms: the internal term int E , the photometric term pho E , and the external force ext E . The energy is calculated for each vertex v(s) and integrated over the whole length of the snake: The internal energy int E tries to preserve a smooth shape for the curve. Photometric energy pho E usually evaluates edge strength or similar measures in the examined image and tries to pull the snake to salient image features. External energy ext E can be introduced by user interaction and are responsible for globally controlling and guiding the snake evolution (Kerscher, 2001.) The curve with minimum energy as shown in Fig. 3.2 is determined to be the optimal seam line. This algorithm cannot completely overcome the local minima problem, and it requires a high computation load. Even though Chon et al. (2010) avoid the local minima problem, it requires a high computation cost as well because of finding the best threshold. www.intechopen.com

Cost space
We adopts Dijkstra's algorithm to find an optimal seam-line on a cost space built by using the normalized cross correlation (NCC) between two overlapping images for pixel (u,v) (Lewis, 1995 The cost value approaches 0.0 for similar pixel points and 1.0 for dissimilar pixels.

Dijkstra's algorithm
Dijkstra's algorithm is a global optimization technique that determines the optimal path on the cost space by taking the local minimum operation at each node. To apply to optimal seam line searching, each pixel in an overlapping area is associated with a node, which has 8 neighboring nodes, with four in diagonal directions. Let the node at which we are starting be called the initial node. Let (u,v) specify a node and (k,l) be a neighboring node of node (u,v). NBR(u,v) indicates the set of neighboring nodes of node (u,v). Let the cost uv,kl d be a path cost between two nodes (u,v) and (k,l). Let the global minimum cost Di(u,v) be a accumulated cost from the starting node to (u,v). Dijkstra's algorithm will assign some initial costs to avoid some areas and will try to improve them step by step.
Assign to every node a tentative cost Di: set it to zero for the starting node and to infinity for all other nodes.
1. Mark all nodes as unvisited. Set the starting node as current. For current node (u,v), consider all its unvisited neighbors and calculate their tentative cost Di.
For example, if current node (u,v) has cost of 6, and a path cost uv,kl d is 2, the cost to node (k,l) through node (u,v) will be 6+2=8. If this cost is less than the previously recorded cost, overwrite the cost. All unvisited neighbors are added to an unvisited set. 2. When we are done considering all neighbors of the current node, mark it as visited. A visited node will not be checked ever again; its cost recorded now is final and minimal. 3. The next current node will be the node with the lowest cost in the unvisited set. 4. If all nodes have been visited, finish and then find the path with the minimum cost between the starting and goal nodes.   3.5 shows four seam-lines detected on four overlapping images using Dijkstra's algorithm on the cost spaces. Each overlapping image is generated from fish-eye images as shown in Fig. 1(c) using a sphere projection method (Coorg and Teller, 2000;Genner, 2006;Kim et al., 2004;Szeliski and Shum, 1997;Yakimovsky and Cunningham, 1978.) The size of each overlapping image is about 30 % of a panoramic image.

Fast color blending on seam-lines
Even though color transitions among all overlapping images are smoothed, the color matching only provides an approximate match. Because color differences among the images corrected by using the color matching method are not enough, Xiong and Pulli (2010) proposed an effective blending method with fast processing speed and high blending quality.
The method calculates the color differences between each p and all points D(p) on each of a seam lines c m and then interpolate D(p) at a pixel q of the blending image (see Fig. 4.1(a)).
Finally, the color value C(q) at pixel q are changed as C(q)= C(q) + D(p). Additionally, they proposed the creation of a color difference distribution process to enforce color consistency for 360-degree panorama. They attenuate the color of the pixel q in the blending area on the current scan line with

The proposed method
If overlapping images are well aligned, Xiong and Pulli (2010) proposed method works perfectly. However, poor image alignment due to parallax, registration errors, and radial distortions can lead to poor blending results in general.
To cope with this problem, we propose filtered color differences using multiple major color differences detected on a seam-line. When the seam points are well aligned, the difference between the neighboring points will be smoothly changed. Because seam points with misalignment cause sudden changes, we simply filter out these seam points and keep good aligned seam points. And we then replace the misaligned seam points with interpolated color differences using the ends of two neighbor seam points on a seam-line. We detect smoothly changed color differences using a median filter and the standard deviation of the changes. Fig. 4.2(a) shows an overlapping image that is located at the rightmost side of the panoramic image shown in Fig. 3.5. Fig. 4.2(b) shows a graph of color differences corresponding to the seam points of the seam line shown in Fig 4.2(a). If the seam points are well aligned, the differences will be smooth like the differences for seam points in the sky area.

Detection of good aligned seam points
Before applying Eq (4.2), we have to remove the color differences of poorly aligned seam points. To detect smooth changing parts, we use a threshold based on the standard deviation of the color differences. The graph in Fig 4.2(c) depicts the standard deviations of the color differences for each channel. The standard deviation of poorly aligned seam points is higher than that of the sky and road.
To evaluate whether seam points are poorly aligned or well aligned, we first build a histogram that accumulates the standard deviation in the horizontal direction, shown in Fig  4.2(c) as the thick green curve, and then find the peak point '*' in the thick green. A dotted red horizontal line in Fig. 4.2(c) chosen by the peak point is a reference standard deviation. We add a margin into the reference standard deviation to calculate a threshold, a red horizontal line in Fig. 4.2(c). We cannot remove the color differences in a dotted circle in Fig. 4.2(b) using only this single threshold, because the standard deviation of the color differences of seam points in the circle www.intechopen.com is low like the reference standard deviation. To remove the color differences of those seam points on the seam-line, we use the same process to build the second histogram using the color differences of the remaining seam points, which are determined by using the first histogram built by using the standard deviation.
After detecting a peak point on the second histogram as a reference of color difference, two thresholds are calculated by adding and subtracting a margin. In our experiments, the margin is set as 15. If a single reference is only applied, bad blending will occur as shown in Fig 4.3.

Fig. 4.3. Our extended algorithm with a single peak point
To suppress this problem, we choose multiple peak points in the histogram. If a peak point is including over 20% of the number of all seam points on the seam-line, the peak point becomes one of the references of color difference. Figs. 4.4(a) and 4.4(b) show color blending results by using one and two peak points chosen by our algorithm with each second histogram, respectively.
As the final step, we have to replace the color differences of seam points detected as poorly aligned seam points during the previous step. A line or curve is built by using both two ends of neighbor seam points from well-aligned seam points.
Figs. 4.5(a) and 4.5(b) show panoramic images created by using method of Xiong and Pulli (2010) and our extended method, respectively. When comparing the area in the dotted circle in Fig. 4.5(a), our extended method made a better result then that by Xiong and Pulli's method. However, our extended method did not perform as well within the dotted square in Fig. 4.5(b). This is originated from big differences among the values of the red, green, and blue channels of one or two end points of the remained seam points as reference color differences. If we change color differences into gray differences, this phenomenon will be disappear. However, subtle color differences among RGB channels could not recovered. We apply a median filter with a couple of seam points around the two end seam points to remove the incorrect differences. Fig. 4.6(a) shows an image including the phenomenon that is an overlapping image in the dotted square in Fig. 4.5(b) and a graph of color differences. Figs. 4.6(b) and 4.6(c) show a graph of filtered color differences and a final result using the filtered color differences, respectively. www.intechopen.com

Color balance and saturation
To make the final panorama more aesthetically pleasing, we apply additional processing to increase contrast, color balance, and color saturation. The additional processing also has the added benefit of making the collection process more tolerant to different lighting conditions.
We achieve color/white balance by using existing auto-white balance algorithm (employed in many image-editing applications, such as Adobe Photoshop). This algorithm computes histograms for each color channel (red, green, and blue) and then in each channel "discards" the 0.5% darkest and 0.5% lightest pixel colors from the histogram and then stretches the resulting histogram to [0, 255] effectively making all pixels that fell in the discarded ranges "black" and "white" respectively.
The above process has two effects. First, it increases the apparent contrast of the image. Second, it can cause a noticeable hue shift because the algorithm operates on the color channels individually. This is desirable for panoramas taken outdoors in natural light because it makes overly "cool" scenes appear warmer and vice versa--providing a good overall aesthetic improvement robustly under varying weather conditions. On the other hand, when the algorithm processes scenes with significantly dark scenes (such as panoramas in tunnels or those taken at night under artificial lighting) the hue shift will be much more pronounced and often produces very poor results. Thus, this color balance algorithm is only applied to images that is taken under decent lightly conditions. Figure

Conclusion
In this chapter we introduced a novel panorama-stitching algorithm. We described each stage of the algorithmic pipeline in detail, namely a channel-correction algorithm that normalizes luminance and chrominance of the projected images prior stitching; a seam-line finding algorithm that finds the optimal transition between two overlapping images; a color blending algorithm that blends the seam-lines; and finally, a color enhancement algorithm that adjusts the overall contrast, color, and saturation of the panoramic image.
In the algorithm, we particularly address the shortcomings of previous approaches, where poor results are often observed from the color-blending algorithm due to poorly aligned points from the individual images on the seam-lines. Our proposed algorithm removes outliers on the seam-lines due to misalignment by using histograms, color differences, and standard deviation of the color differences. The outliers are replaced by linearly interpolated points using the color differences of neighbor well-aligned seam points.
We have tested the proposed algorithm on millions of outdoor panoramas and it has been proven that the algorithm is robust under most lighting and weather conditions. The algorithm described in this paper is not only bounded to Earthmine's collection system but any system that generates panorama images.

References
Bellman, R., (1957). Dynamic Programming. Princeton Univ. Press. Brown, M. and Lowe, D., (2007 Photogrammetry is widely accepted as one of the best surveying methods to acquire tridimensional data without direct contact with the object, but its high operational costs in equipment and personnel somewhat limit its application in mapping. However, with the development of digital photogrammetry in the 1990's, it was possible to introduce automated processes and reduce the personnel costs. In the following years, the cost of computer hardware, digital cameras and positioning sensors has been lowering, making photogrammetry more accessible to other engineering fields, such as architecture, archeology and health fields. This book shows the results of the work of researchers from different professional backgrounds, which evaluate the uses of photogrammetry, including issues of the data, processing, as well as the solutions developed for some surveying types that can be extended to many applications.