Stereo Algorithm with Reaction-Diffusion Equations

Three-dimensional depth reconstruction from a pair of stereo images needs to find reliable stereo correspondence between left and right images. Stereo disparity refers to difference of positions between two corresponding points on the stereo images. Although there are many possibilities in finding the stereo correspondence, the human vision system successfully reconstructs three-dimensional depth distribution with binocular vision. Thus, we can expect that solving the stereo correspondence problem brings a key mechanism to understand the human vision system. The human vision system can solve the stereo correspondence problem from random-dot stereograms, which refer to a pair of stereo images consisting of only a randomly dotted pattern. Julesz demonstrated that the randomly dotted pattern is enough for the human vision system to solve the stereo correspondence problem (Julesz, 1960). When the human vision system is exposed to random-dot stereograms, it perceives three-dimensional structure emerging spontaneously. This implies that the human vision system has a module being able to detect stereo disparity from only a randomly dotted pattern and not requiring key information such as edges and feature points, which generally appear in natural scenes. Several researchers presented stereo algorithms to solve the stereo correspondence problem from random-dot stereograms in the early period of computer vision research. In particular, Marr and Poggio presented a representative stereo algorithm named the cooperative algorithm (Marr & Poggio, 1976). Their motivation to approach the stereo correspondence problem exists in the biological aspect of the human vision system; their algorithm consists of multi-layered cell networks. Their most important proposal to the stereo algorithm is the two famous constraints: continuity and uniqueness. The authors have been approaching several subjects in image processing and computer vision research, by utilizing reaction-diffusion equations as a biologically motivated tool (Nomura et al., 2007). We call the group of algorithms utilizing reaction-diffusion equations the reaction-diffusion algorithm. The previous research done by the authors presented several algorithms for edge detection, segmentation (grouping) and stereo disparity detection by utilizing the FitzHugh-Nagumo type reaction-diffusion equations. This chapter presents a stereo algorithm utilizing multi-sets of the FitzHugh-Nagumo type reaction-diffusion equations. We associate each set of the equations with each of possible disparity levels; a particular grid point of the set represents existence or non-existence of its associated disparity level. The filling-in process originally built in the reaction-diffusion O pe n A cc es s D at ab as e w w w .ite ch on lin e. co m

equations realizes the continuity constraint; the multi-sets mutually connected via a mutualinhibition mechanism realize the uniqueness constraint. Although the human vision system can detect stereo disparity from only random-dot stereograms, the authors believe that feature points such as edges detected from image brightness distribution help stereo algorithms to achieve more reliable stereo disparity detection. An integration mechanism of edge information into the stereo perception is interesting from the scientific point of view. According to this, the authors furthermore propose to integrate edge information into the stereo algorithm. As mentioned above, the authors have also presented an edge detection algorithm utilizing the reaction-diffusion equations (Nomura et al., 2008). Thus, in this chapter the stereo algorithm with edge information is fully realized with the reaction-diffusion equations biologically motivated. Finally, from the engineering point of view we need to evaluate quantitative performance of the stereo algorithms presented here. We apply the stereo algorithms to test stereo image pairs provided on the Middlebury website (http://vision.middlebury.edu/stereo/). Results of the performance evaluations show that the stereo algorithm utilizing the reactiondiffusion equations with edge information achieves better performance in areas having depth discontinuity, in comparison to the stereo algorithm without edge information. However, in other areas not having depth discontinuity, edge information obtained for image brightness distribution is useless. Thus, the performance of the stereo algorithm is not improved. In addition, other state-of-the-art stereo algorithms achieve much more performance. Thus, future work needed for the reaction-diffusion algorithm is to improve its performance also in other areas not having depth discontinuity.

Previous stereo algorithms
It is essentially difficult to detect stereo disparity from only a pair of stereo images, since image brightness distribution does not provide enough information to identify a particular point in image. There exists ambiguity in finding stereo correspondence between stereo images; the stereo correspondence problem is the typical ill-posed problem. Researchers have devoted their efforts to obtain reliable stereo correspondence with several different algorithms, such as, the template-matching algorithm and the cooperative algorithm. In the template-matching algorithm that finds stereo correspondence with a crosscorrelation function, we need to extend the size of its correlation window for reducing the ambiguity lying on the stereo correspondence problem. However, the larger size of the correlation window causes unreliable solution to the stereo correspondence problem, in particular, in areas having depth discontinuity. Thus, on the one hand, the larger correlation window size assuming the spatial uniformity of disparity distribution is necessary to reduce the ambiguity; on the other hand, such the spatial uniformity can not be assumed, in particular, in areas having depth discontinuity. The two requirements of reducing ambiguity and preserving depth discontinuity is mutually exclusive. Marr and Poggio imposed the two constraints: uniqueness and continuity on the cooperative algorithm (Marr & Poggio, 1976). The uniqueness constraint states that a particular point on a stereo disparity map has only one stereo disparity level except for transparent objects; the continuity constraint states that neighboring points on a stereo disparity map share same or similar disparity level(s) except for object boundaries. They tried to detect stereo disparity by taking account of the two constraints with a biologically motivated multi-layered network model, each grid point of which was considered as a cell activated by neighboring cells locating on the same layer and inhibited by other cells locating on other layers of disparity levels. The continuity constraint causes error of stereo disparity in areas having depth discontinuity. Thus, the problem arising from the depth discontinuity is one of the most significant issues not only in the template-matching algorithm, but also in other stereo algorithms including the cooperative algorithm. Since the original cooperative algorithm is designed for random-dot stereograms, it is not applicable to natural stereo images. More recently, Zitnick and Kanade presented a cooperative algorithm designed for natural stereo images (Zitnick & Kanade, 2000). In addition, the cooperative algorithm can detect occlusion areas in which an object occludes another object in either of two stereo images; it is generally difficult for stereo algorithms to detect occlusion areas. Other algorithms such as the belief-propagation algorithm and the graph-cuts algorithm are well known as state-of-the-art stereo algorithms. The Middlebury website is providing the ranking table showing which stereo algorithm achieves the best performance for test stereo image pairs with respect to the bad-match-percentage error measure. According to the ranking table, those state-of-the-art algorithms achieve much better performance, in comparison to the cooperative algorithm. The authors are interested in biologically motivated algorithms and thus contribute to the stereo correspondence problem by utilizing the reaction-diffusion algorithm. The algorithm presented by the authors is rather similar to the cooperative algorithm and is linked directly with mathematical models of information transmission and pattern formation observed in biological systems.

Modeling pattern formation processes with reaction-diffusion equations
Biological systems self-organize spatio-temporal patterns for information transmission (Murray, 1989). Typical examples of self-organized patterns are impulses propagating along a nerve axon and a spiral pattern and a target pattern observed in a two-dimensional system of slime mould. These self-organized patterns have been explained with a mathematical model of reaction-diffusion equations, which are generally described with time-evolving partial differential equations. Each of the equations consists of a diffusion term coupled with a reaction term describing its corresponding phenomenon such as, for example, a response to stimuli given by other neighboring cells or to external stimuli in biological systems. The next set of equations describes reaction-diffusion equations with the two variables u (x,y,t) and v(x,y,t) in two-dimensional space (x,y) and time t, as follows: where D u and D v are diffusion coefficients and f (u,v) and g (u,v) are reaction terms. The symbol ∂ t refers to the partial differential operator ∂/∂t and ∇ 2 refers to the Laplacian operator ∇ 2 =∂ 2 /∂x 2 +∂ 2 /∂y 2 . The FitzHugh-Nagumo type reaction-diffusion equations describe impulses propagating along a nerve axon in a biological system (FitzHugh, 1961;Nagumo et al., 1962); the equations have the reaction terms f (u,v) and g (u,v), as follows: www.intechopen.com where a and b are constants and ε is a positive small constant (0<ε<<1). A set of ordinary differential equations derived from Eqs. (1) and (2) under D u =D v =0 indicates two different types of system behavior such as the bi-stable system or the mono-stable system as shown in Fig. 1. When the set of the equations is bi-stable, it has the two stable steady states, one of which locates at the origin (0,0) and the other of which locates at another point (u s ,v s ) having u s close to one. Turing found that a condition causes a spatial static pattern such as a spot or stripe pattern with a set of reaction-diffusion equations (Turing, 1952), even if diffusion terms in the equations induce spatial homogeneity of the two variables u and v. When the diffusion coefficient D v is much larger than the diffusion coefficient D u , such the spatial static pattern appears. The Turing condition refers to the condition that causes the spatial static pattern in reaction-diffusion equations and the Turing pattern refers to the static pattern. Several selforganized patterns observed in biological systems have been explained as the Turing pattern and with the Turing condition (Kondo & Asai, 1995). We need to emphasize that the Turing pattern is due to the rapid diffusion on the inhibitor variable v, in comparison to the slow diffusion on the activator variable u (D u <<D v ).
The steady states A locating at (0,0) and C locating at (u s ,v s ) are stable; the steady state B is unstable. Depending on the parameter values a and b, the system of f(u,v)=0 and g(u,v)=0 becomes mono-stable or bistable. When the system is bi-stable, a solution converges either of the two stable states. The excited state denotes that the system is in the red area; the rested state denotes that the system is in the blue area. Kuhnert et al. found that a chemical reaction system can realize visual functions such as edge detection, segmentation and image memory (Kuhnert, 1986;Kuhnert et al., 1989). The chemical reaction system is described with a set of reaction-diffusion equations having activator and inhibitor variables. Thus, their experimental reports made a strong impact on us and highly motivated us to develop image processing and computer vision research with reaction-diffusion equations.

Modeling and realizing visual functions with reaction-diffusion equations
After the reports by Kuhnert et al., many other researchers have also done much efforts on modeling and realizing visual functions with reaction-diffusion equations. Asai et al. realized a reaction-diffusion system with the large-scale integrated circuits technology for image processing; Adamatzky et al. furthermore proposed a novel computer architecture and named it the reaction-diffusion computer (Adamatzky et al., 2005). Ueyama et al. proposed a reaction-diffusion system as a mathematical model of the human visual perception (Ueyama et al., 1998). They tried to explain a visual phenomenon observed in motion perception with a reaction-diffusion equation. In image processing and computer vision research, we usually utilize the Gaussian filter or a diffusion equation for reducing noise or selecting scales in patterns. For example, the edge detection algorithm proposed by Marr and Hildreth utilizes the difference of two Gaussians (Marr & Hildreth, 1980), each of which is alternatively expressed by a diffusion equation. Perona and Malik proposed an edge detection algorithm by utilizing an anisotropic diffusion (Perona & Malik, 1990). The authors have tried to build models of several visual functions such as edge detection, grouping and stereo disparity detection by utilizing reaction-diffusion equations. The authors believe that reaction-diffusion equations become a basic tool for realizing visual functions from the engineering point of view, and a basic model for understanding the human visual functions from the scientific point of view. This is the motivation for the present research work done by the authors by utilizing reaction-diffusion equations in image processing and computer vision research. This chapter presents a stereo algorithm by utilizing multi-sets of the FitzHugh-Nagumo type reaction-diffusion equations (1) and (2), on which the authors impose the Turing like condition (D u <<D v ). The grouping mechanism built in the reaction-diffusion equations works as the continuity constraint required in the stereo algorithm. In addition, the Turing like condition helps to prevent over-smoothing around corners in structure of stereo disparity distribution.

Stereo algorithm with reaction-diffusion equations
Upon acceptance of the two constraints: continuity and uniqueness proposed by Marr and Poggio (1976), the authors present a stereo algorithm that utilizes the FitzHugh-Nagumo type reaction-diffusion equations (1) and (2). In order to realize the stereo algorithm, we consider multi-sets of the equations, each set of which is associated with a possible disparity level, and each set of which governs areas having the associated disparity level. In addition, each set inhibits other sets from having other disparity levels via a mutualinhibition mechanism. A set of the reaction-diffusion equations organizes propagating waves, which are applicable to the realization of the continuity constraint (Fig. 2). A wave triggered by an external stimulus at a point in space begins to propagate. When the wave collides with another wave triggered at another point in the space, the two waves become one, if the system of the equations is bi-stable. This is a filling-in process that fills in undefined areas between the two triggered positions. Let us give an output of a matching cost function to the set of the reaction-diffusion equations as its external stimulus. Then, we can understand that the set realizes the continuity constraint.  The mutual-inhibition mechanism built in the multi-sets of the reaction-diffusion equations realizes the uniqueness constraint. Each set organizes propagating waves, for which the parameter a in Eq.
(2) controls their propagating speed. Thus, when we change the parameter value a, we can observe waves propagating at different speeds. If we choose a large value of a, we can inhibit the waves from propagating. Thus, each set of the reactiondiffusion equations utilized in the stereo algorithm is as follows: where n=0,1,⋅⋅⋅,N-1 denotes the index of possible disparity levels and μ is a constant. We provide an output of a matching cost function for s n (x,y) as an external stimulus to the set; the matching cost function returns one for a stereo image pair exactly matched with a disparity level d n and zero for an un-matched pair. The original reaction term f (u,v) in Eq.
(2) has a constant parameter a. In contrast to this, the above modified version of the reaction term in Eq. (4) depends on the state of the other set having the maximum value u max as follows: where a 0 and a 1 are constants and Ω i denotes an inhibition domain for the uniqueness constraint (Zitnick & Kanade, 2000). For preserving sharp corners existing in stereo disparity distribution, we impose the Turing like condition (D u <<D v ) on each of the reaction-diffusion equations in Eq.
(3). The condition induces rapid diffusion of the inhibitor variable v n , in comparison to diffusion of the activator variable u n . The rapid diffusion of the inhibitor prior to the diffusion of the activator prevents a wave from propagating into undefined areas. The phenomenon preventing the propagation of the wave becomes remarkable for a curved wave front having negative curvature. Thus, the wave propagation inhibited by itself in a disparity level helps a sharp corner governed by another disparity level to survive. The authors name this phenomenon due to the Turing like condition the self-inhibition mechanism. By computing Eqs. (3), (4) and (5)

Integration of edge information into the stereo algorithm
Areas having depth discontinuity cause much error in stereo disparity detection. If the areas are given in advance of the stereo disparity detection, information of the areas helps stereo algorithms to achieve better performance around the areas. However, the areas are generally unknown. Although we can not directly link edge areas existing in image brightness distribution with the areas having depth discontinuity, we can expect that information of the edge areas becomes a key to success in stereo disparity detection in the areas having depth discontinuity. The present chapter furthermore proposes the integration of edge information into the stereo algorithm presented above. We obtain edge information from either of stereo images with an edge detection algorithm and then feed the edge information to the stereo algorithm. The authors previously proposed the edge detection algorithm by utilizing the reaction-diffusion equations (Nomura et al., 2008). Let the two distributions u e (x,y) and v e (x,y) be the results of the edge detection algorithm; they represent spatial distributions of the activator variable u(x,y,t) and the inhibitor variable v(x,y,t) obtained at a point (x,y) and at time t after enough finite duration of time needed for convergence. In edge areas, a set of solutions u e (x,y) and v e (x,y) is in the excited state nearly equal to the stable steady state (u s ,v s ) (see also Fig. 1). Thus, we integrate the edge information (u e ,v e ) into the stereo algorithm as follows: where the terms describe anisotropic diffusion, according to the results of the edge detection. Since (1-u e ) becomes almost zero in edge areas, slow diffusion across an edge line prevents disparity information from propagating into other areas having other disparity levels. Thus, we can expect that the slow diffusion brings better performance around areas having depth discontinuity.

Experimental results
This experimental section reports performance of stereo algorithms. The stereo algorithms evaluated here are as follows: the reaction-diffusion algorithm without the integration of edge information (RD s ), the reaction-diffusion algorithm with the integration of edge information (RD s+e ) and the cooperative algorithm proposed by Zitnick and Kanade (ZK) (Zitnick & Kanade, 2000). The two algorithms RD s and RD s+e utilize a normalized crosscorrelation function as a matching cost function; the normalized cross-correlation function computes similarity between two areas, each of which consists of five points including the target point and its nearest four points. In order to show how much the two reactiondiffusion algorithms improve initial disparity maps, we also evaluated the initial disparity maps (CC 5 ). We utilized the four pairs of test stereo images: MAP, TSUKUBA, SAWTOOTH and VENUS for the performance evaluation of the stereo algorithms (Scharstein & Szeliski, 2002). The Middlebury website provides all of the four image pairs as well as their ground truth data of stereo disparity maps and depth discontinuity areas. Figure 3 shows the four pairs of left and right stereo images and their initial disparity maps obtained by only the crosscorrelation function (CC 5 ). The two error measures: the root-mean-square error measure and the bad-match-percentage error measure quantitatively evaluate stereo disparity maps obtained by the stereo algorithms. The root-mean-square error measure R (pixel) evaluates absolute difference between the true disparity map M t (x,y) and an obtained disparity map M(x,y,t) in a domain F as follows: where F denotes the number of pixel sites belonging to the domain F. The bad-matchpercentage error measure B (%) evaluates the ratio of the number of bad match pixel sites, which have absolute error larger than δd (pixel), to the number of total pixel sites belonging to the domain F as follows: www.intechopen.com where the function σ(S,δd) is the threshold function which returns one for S≥δd and zero for S<δd. The domain F in Eqs. (8) and (9) refers to the domain consisting of all pixel sites except for borders and occlusion areas (all), the domain consisting of untextured areas (untex.) or the domain consisting of depth-discontinuity areas (disc.). The Middlebury website also provides definitions of the three domains. Borders of 10 pixels (18 pixels for TSUKUBA) and occlusion areas were ignored in the present evaluation.
The reaction-diffusion algorithms RD s and RD s+e utilize time-evolving partial-differential equations, for which the finite difference method provides a set of linear equations with the finite differences δx, δy and δt for space (x,y) and time t. The parameter values δx=δy=1/5, δt=1/100, D u =1.0, D v =3.0, a 0 =0.13, a 1 =1.5, b=10, ε=10 -2 , μ=3.0 were chosen for the present experiments in RD s and RD s+e . The reaction-diffusion algorithm with edge information (RD s+e ) requires edge detection results. Figure 4 shows the edge detection results obtained by the reaction-diffusion algorithm designed for edge detection. See the literature (Nomura et al., 2008) for the edge detection algorithm and its parameter values utilized here. These results were fed to the algorithm RD s+e through the anisotropic diffusion terms, as described in Eq. (7). Figure 5 shows the ground truth disparity maps, disparity maps obtained by RD s+e and their absolute error maps. Table 1 shows error values evaluated for the disparity maps obtained by RD s , RD s+e and ZK, as well as for the initial disparity maps denoted by CC 5 . With respect to the bad-match-percentage error measure B (%), we can confirm that the algorithm RD s+e achieves better performance than RD s in areas having depth discontinuity for all of the four stereo image pairs. In particular, for the stereo image pair SAWTOOTH, RD s+e achieves better performance for all of the three evaluation domains: "all", "untex." and "disc.". However, RD s is better than RD s+e in the evaluation domains: "all" and "untex." for the stereo image pair TSUKUBA. The algorithm RD s+e assumes that edge areas obtained for brightness distribution coincide with areas having depth discontinuity. When a stereo image pair does not satisfy the assumption, RD s+e provides rather much error in stereo disparity detection. When comparing the reaction-diffusion algorithms RD s and RD s+e with the cooperative algorithm ZK, the reaction-diffusion algorithms achieve better performance for SAWTOOTH and VENUS and the cooperative algorithm achieves better performance for TSUKUBA. That is, we confirm that overall performance is almost the same. Future work required for the stereo algorithm RD s+e is to improve its performance not only for the areas having depth discontinuity, but also for other areas including untextured areas. In order to do this, we need to feed edge information obtained for an initial disparity map instead of image brightness distribution to the stereo algorithm RD s+e . Since initial disparity maps have much noise, we need to confirm how much the edge detection algorithm works correctly for the noisy maps. Furthermore, the authors are considering a new vision system that dynamically integrates the edge detection algorithm with the stereo algorithm. First, we obtain edge information from temporarily detected stereo disparity distribution; second, we compute one time step of the stereo algorithm with the integration of the edge information. By iterating these two steps alternately, we expect to achieve better performance of stereo disparity detection in the reaction-diffusion algorithm.
(a-1) (a-1) (a-2) (a-3)  Borders and occlusion areas of disparity maps were ignored in the performance evaluation. See Fig. 3 for the stereo image pairs and Fig. 4 for the edge detection results utilized here. (a-1), (a-2), (c-1), (c-2), (d-1) and (d-2) visualize the stereo disparity maps in the disparity range of min=0 (pixel) and max=31 (pixels); (b-1) and (b-2) visualize those in the disparity range of min=0 (pixel) and max=15 (pixels). (a-3), (c-3) and (d-3) visualize the error distribution maps in the range of min=0.0 (pixel) and max=2.0 (pixels); (b-3) visualize that in the range of min=0.0 (pixel) and max=5.0 (pixels).  Table 1. Results of quantitative performance evaluations obtained for the reaction-diffusion stereo algorithms RD s and RD s+e and the cooperative algorithm ZK as well as for the initial disparity maps denoted by CC 5 . We evaluated the algorithms in all areas (all), un-textured areas (untex.) and depth discontinuity areas (disc.); we ignored borders and occlusion areas in the performance evaluations. Underlined scores denote the best performance among the algorithms. See Eq. (8) for the root-mean-square error measure R (pixel) and Eq. (9) for the bad-match-percentage error measure B (%). The scores evaluated for ZK were resulting from the disparity maps provided on the Middlebury website.

Conclusion
This chapter presented a stereo algorithm utilizing multi-sets of the FitzHugh-Nagumo type reaction-diffusion equations. The stereo algorithm realizes the two constraints: continuity and uniqueness, as follows. Each set of the reaction-diffusion equations self-organizes propagating waves, the collision of which works as the continuity constraint. The mutual inhibition mechanism connecting the multi-sets works as the uniqueness constraint. In addition, the authors imposed the Turing like condition on each set of the reaction-diffusion equations; the condition helps the stereo algorithm to preserve small features such as sharp corners in stereo disparity distribution. In order to improve the performance of the stereo algorithm in areas having depth discontinuity, the authors additionally proposed the integration of edge information into the stereo algorithm. That is, the algorithm weakens the diffusion processes of the reactiondiffusion equations in areas having edges, which are detected by also the reaction-diffusion algorithm designed for edge detection from image brightness distribution. We evaluated quantitative performance of the two stereo algorithms with/without the integration of edge information, by applying them to the well known test stereo image pairs. Results of the performance evaluations show that the algorithm with the integration of edge information achieves better performance than the stereo algorithm without the integration of edge information in the areas having depth discontinuity. Overall performance of the stereo algorithms presented here is almost equivalent to that of the cooperative algorithm proposed by Zitnick and Kanade. According to the Middlebury website, other state-of-thewww.intechopen.com art stereo algorithms achieve much better performance, in comparison to the reactiondiffusion algorithm. The authors believe that it is possible to improve the performance of the reaction-diffusion algorithm by dynamically integrating the edge detection algorithm into the stereo algorithm. Thus, we need to improve the performance of the reaction-diffusion algorithm designed for stereo disparity detection.