19 Green's Functions of Matching Equations: A Unifying Approach for Low-level Vision Problems

Green's functions are a traditional technique for solving inhomogeneous differential equations which has found several applications in pure and applied science, as, for instance, in Electrodynamics or Quantum Mechanics (Hassani, 2002). Given a one-dimensional linear differential operator, x , and a set of boundary conditions, the solution to the inhomogeneous differential equation ) x ( g ) x ( f x = can be expressed as


Introduction
Green's functions are a traditional technique for solving inhomogeneous differential equations which has found several applications in pure and applied science, as, for instance, in Electrodynamics or Quantum Mechanics (Hassani, 2002).Given a one-dimensional linear differential operator, x , and a set of boundary conditions, the solution to the inhomogeneous differential equation can be expressed as where D is the domain of interest, and where the operator ) x , x ( G 0 , called the Green's function, is the solution to the equation under the same boundary conditions, with ) x -x ( 0 denoting Dirac's delta function.Operating on both sides of (1) with x , and making use of equation ( 2), we obtain where the sieving property of the delta function has been used, what proves that the function f(x), given by (1), is indeed the solution sought.It should also be noted that, if ) x , x ( H 0 is another integral operator, satisfying is also a solution to the original differential equation, i.e., it satisfies ) x ( g ) x ( f x = . Recently, the use of Green's functions of image matching equations has been proposed as a suitable means for approaching several visual computing problems, including shape from shading (Torreão, 2001;Torreão, 2003;Torreão & Fernandes, 2004), edge detection (Torreão &Amaral, 2002 and2006), motion simulation (Ferreira Jr. et al., 2004), and video interpolation (Ferreira Jr. et al., 2005).
Image matching equations have been used in computer vision and image processing for modeling such processes as stereoscopy (Barnard, 1986) and optical flow (Horn & Schunck, 1981).For instance, if 1 I and 2 I are two images of a dynamic scene, captured at consecutive times by a static camera, the optical flow constraint can be expressed as the intensity conservation condition where U(x,y) and V(x,y) are the optical flow components along directions x and y, respectively.The goal is then to use such image matching condition for estimating U and V, what is generally done by first expanding equation ( 6) in a Taylor-series up to first order in the flow, and using it along with other constraints (that would express, for instance, the smoothness of the flow components), in order to allow the solution of such ill-posed problem.
In the Green's function approach, on the other hand, a different use is made of the matching equation: assuming that the flow field is known (e.g., a uniform or an affine flow), equation ( 6) is solved for the matching image 2 I , given 1 I .For instance, assuming uniform flow along the direction arctan = (i.e., U(x,y) = u and V(x,y) = v, for both u and v constants, ), and taking a second-order Taylor-series expansion, the matching equation becomes . The solution to the above can be expressed as is the Green's function to equation (7), i.e., it is the solution to that equation when a delta function is substituted for 1 I on its right-hand side.If we want bounded solutions over an infinite domain , u G will take the form (Torreão, 2001) , otherwise.It will thus be a causal, shift-invariant operator.
Different kinds of Green's functions will result from different flow assumptions.If, instead of the uniform flow, we considered a one-dimensional affine model, with , for constant 0 u and 1 u and with , the matching equation would become whose Green's function, again if we require bounded solutions over an unbounded domain, will be (Ferreira Jr. et al., 2004) , otherwise.In equation ( 11), the parameter U x is defined as , and corresponds to the fixed point of the affine transformation, since we have . The parameters α and β are given as The Green's function, in this case, is a shift-variant operator which remains bounded over a domain , so long as we take 2 < u < 0 1 .Over finite domains, this solution is Still another form of Green's function results from considering the matching equation under the guise which is a variant of the affine matching condition, identical to equation (10) up to first order in 0 u .The bounded Green's function for the above, over a domain >> , as (Torreão & Fernandes, 2004) , otherwise.It can be easily verified that, similarly to our first affine form, All the Green's functions considered can be interpreted as point spread functions which generate motion through a linear model: when filtering an input image, they induce a displacement of the image features, accompanied by a loss of high frequencies which can be interpreted as motion blur.The potential of this for motion synthesis in computer graphics is evident, and has been extensively explored, based on the filter ) 1 ( U G (Ferreira Jr. et al., 2004 and2005).
Here, we will be mainly concerned with the computer vision applications of the approach, which have been based on the forms u G and ) 2 ( U G .Such applications also stem from the motion induction capabilities of these Green's functions.For instance, given a single input image, a second image can be generated which simulates the photometric stereo pair to the input, representing the same scene under displaced illumination.This has been used as the basis to the Green's function shape from shading (Torreão, 2001), which extends, to singleimage reconstruction, a two-image photometric-stereo approach (Torreão & Fernandes, 1998).Similarly, a depiction of the scene under a displaced point of view can also be simulated from a given image, what has led to the Green's function photometric motion (Torreão & Fernandes, 2004), also extending, to the single-input case, a multi-image process (Torreão et al., 2007).Signal differentiation is another computer-vision/image-process application where the Green's functions of matching equations have found use (Torreão &Amaral, 2002 and2006).This comes along naturally, when we remember that the first-order derivative of a signal can be approximated through the difference of displaced versions of it.Finally, we will here introduce a new application area for our Green's functions, by showing that, through their means, displaced versions of binocular image pairs can be generated whose local degree of matching yields a reliable measure of stereoscopic disparity.The remainder of this chapter is organized as follows: in Section 2 and Section 3, we review the Green's function approaches to signal differentiation and to shape from shading, both based on the uniform-matching Green's function, u G ; in Section 4, we review the Green's function photometric-motion process, based on the affine Green's filter, ) 2 ( U G (which, for simplicity, will henceforth be referred simply as U G ), and, in Section 5, we introduce the use of the same filter for stereoscopic disparity estimation.Our final remarks close the chapter in Section 6.

Signal Differentiation through Green's Functions
The Green's function approach to signal differentiation is based on the following rationale: If ) x ( I′ is the derivative of a signal ) x ( I , it can be expressed as u 2 According to the Green's function approach summarized above, an estimate of the signal , can be obtained as (see equation ( 8)) where u G (see Fig. 1) is the uniform-matching Green's function, as presented in (9).The identity in equation ( 16), valid up to second order in u, results from inverting the matching relation ) , which is a special case of equation ( 6).
Similarly, a signal ) x ( I + , which is an estimate of I(x + u), can be obtained as where it can be easily verified that the filter Using relations ( 16) and ( 17) in equation ( 15), and applying the commutative property of the convolution, there results the derivative estimate and we have thus arrived at a linear operator, , which, in the limit of u → 0, becomes the impulse response of a differentiator.) x ( D u turns out to be a special case of Deriche's well-known edge-detector (Deriche, 1987) A more general form for our differential operator can be found if we allow for scale factors in the matching equations.For instance, we could consider the relations to obtain the derivative estimate as a generalized version of our differentiator.Multiscale derivative estimates can then be obtained through linear combinations such as where the a are real-valued constants, satisfying 1 a = .For instance, we may consider just two terms in the above expansion, to get From equations ( 18) and ( 21), we thus see that, in such case, our derivative estimate will be obtained by convolving the input signal with the operator (see Fig. 2) where F(x), for x > 0, is given by In (Torreão & Amaral, 2006), a study was carried out which determined the values for a and η leading to maximum overall performance by the filter D(x), as measured through the (ΣΛ) SRC index introduced by Canny, where Σ, Λ and SRC denote, respectively, the detection, localization, and single-response measures (Canny, 1986).With a=1.1 and η=3.5, a (ΣΛ) SRC of 3.547 was achieved, beating the performance of alternative approaches, such as Sarkar and Boyer's filter, whose best mark is 3.388 (Sarkar & Boyer, 1991).Another advantage of the operator D(x), as also proven in (Torreão & Amaral, 2006), is that it allows simple recursive implementations, being realized as an infinite impulse response filter with only two poles and a single zero.Fig. 3 shows examples of the use of operator D(x) for edge detection.In such 2D applications, D(x) was employed in the direction perpendicular to the edges sought, while a projection function -chosen here as the integral of D(x) -was used in the direction parallel to the edges.Non-maxima supression and hysteresis thresholding have also been employed, in the usual fashion (Canny, 1986).

Green's Function Shape from Shading
Shape from shading (SFS) and photometric stereo (PS) are 3D shape estimation processes that take shading images as inputs -that is to say, they work with textureless images where a smooth gradient of intensities is observed, resulting solely from the orientation of the observed surfaces.SFS estimates surface orientation from a single shading image, while PS works with two or more monocular shading inputs, acquired under different illuminations.Both processes have been traditionally based on the so-called image irradiance equation, which relates the intensity at each image point to the surface gradient at the corresponding location in the scene, via the reflectance map function (Zhang et al., 1999) where y Z q and , x are the gradient components of the observed surface, Z(x,y), and where R is the reflectance map.
In (Torreão & Fernandes, 1998), an approach to photometric stereo was introduced, called the disparity-based photometric stereo (DBPS), whereby a pair of PS images are matched, similarly as a stereoscopic pair (Barnard, 1986), to yield a disparity map from which the shape of the observed surfaces can be recovered.Such disparity map results from the displacement of the irradiance pattern over the imaged surface, due to the change in illumination direction, a displacement that can be generally modeled as a non-uniform rotation, as proven in (Torreão, 2003).
DBPS is based on a pair of equations: a linear image irradiance equation, and a matching (optical flow) equation, that take the form is the difference of the input images, and where (u,v) denotes the optical flow, or disparity field.
Equating the two expressions for ΔI above, there results a differential equation on Z, whose approximate solution can be found as so long as the disparity component u is found by matching the input images along a direction such that k As proven in (Torreão, 2001), the DBPS approach can be extended to the single-input case, with the so-called Green's function shape from shading (GSFS).Here, the idea is to assume that the disparity field is uniform, and to solve the matching equation -considered up to second order in u, such as in ( 7) -for the matching image, 2 I , via Green's function.It has been shown that, in such case, the estimated depth map takes the form where a, b and c here are constants, the operation * denotes a convolution, the matching pair to the input image 1 I , and u H is the homogeneous integral operator x > , with 0 H u = , otherwise (see Fig. 4).As can be easily verified, u H satisfies the homogeneous form of equation ( 7), i.e., Besides the matching constant u, which must be chosen a priori, the single free parameter in equation ( 31) is 1 k , and this can be estimated from the input data, as described in (Torreão, 1999).For this purpose, we take into consideration the fact that the displacement of the irradiance pattern over the scene, due to the change in illumination (it should be kept in mind that we are simulating a photometric stereo situation, via the Green's function) can be modeled as a non-uniform rotation.This allows the introduction of a least-squares structurefrom-motion formulation that yields .k 1 Fig. 5 shows examples of shape estimation via GSFS.

Green's Function Photometric Motion
Photometric motion is a shape estimation process introduced by Pentland, based on his observation that, for surfaces in rotation relative to the camera, the photometric effects of the motion (i.e., the intensity change of a moving point) can prove more relevant than the geometric effects, due to projective distortion (Pentland, 1991).In his formulation, Pentland considered a quadratic expansion of the reflectance map, supposed symmetric and separable, and he also assumed that regions of approximately linear motion could be identified, allowing the registration of corresponding points in successive frames.Under such conditions, Pentland found that the intensity difference of registered points could be described by a linear reflectance map, and he thus used his linear shape from shading algorithm (Pentland, 1990) to obtain shape estimates of the imaged scene.
An alternative formulation of photometric motion has been recently introduced in (Torreão et al., 2007), along similar lines as followed for the disparity-based photometric stereo.A distinctive feature of this formulation is that of being based on the intensity change, due to the motion, at a fixed location in the image plane, and not, as in Pentland's approach, at a given point on the moving surface.This has the advantage of not requiring warping for the registration of corresponding points in the image sequence.Similarly as DBPS, our novel approach to photometric motion relies on two expressions for the intensity change, due to the motion, at a given point in the image plane, one of them a , similarly as in DBPS, we arrive at the expression , of successive frames in the image sequence.In the above equation, we have used where stands for the ratio v/u.Now, again as in DBPS, we must couple equation ( 36) with an image matching equation, in order to find a closed-form expression for the depth map Z(x,y).The appropriate matching equation is here found to describe an affine optical flow field, taking the form Equating ( 36) and ( 38), we find a diferential equation on Z, whose solution is given by where is an arbitrary constant, provided that the term in u 2 ∂ is neglected.
Through the Green's function approach, the above-described photometric motion formulation (whose results can be appreciated in (Torreão et al., 2007)) can be extended to the single-input case.In order to do this, we require a Green's function that will relate the matching image to the input image according to equation ( 38).Since that is a 1D expression, we may, without loss of generality, take the matching direction as x, to obtain where 1 I is the input image, 2 I is the image derived from it through the Green's function, and 0 u and 1 u are two constants representing, respectively, the disparity map and its derivative at x = 0 (in the general case, these will mean u u 0 = and u u 1 ∂ = ).By comparing equation ( 40) to equation ( 13), we find that, up to first order in 0 u , our photometric-motion Green's function will take the form of U G in equation ( 14) (see Fig. 6).

Green's Function Stereoscopy
In a binocular vision system, scene features project at different positions in the two cameras, giving rise to the so-called binocular disparities, which constitute the primary cue for stereo vision (Barnard, 1986).Assuming a horizontal imaging configuration, a pair of left and right images which are projections of the same 3D scene should be related as where d ≡ d(x,y) here denotes the disparity map.The above is simply a special case of the matching equation ( 6), and, based on this, we can propose a Green's function approach to stereoscopic disparity estimation: Given the pair of binocular images, we can filter each of them through the appropriate Green's function, to induce different rightwards and leftwards shifts, aiming at the elimination of their intrinsic binocular disparities.By evaluating the degree of matching between the shifted inputs, for instance by computing the squared magnitude of their difference, we can then obtain an estimate of the disparity information encoded by the original stereo pair.
We have implemented such approach using the affine Green's filter of equation ( 14), keeping its σ parameter fixed, and varying U x , in order to obtain different image shifts.Our preliminary results have proven encouraging, as shown by Figure 8 below.

Conclusion
We have reviewed the computer vision applications of Green's functions of image matching equations.Green's functions of both uniform-and affine-matching second-order differential equations have been considered, and we have illustrated their use for the computer vision problems of edge detection, monocular shape estimation, and stereoscopy.The Green's filters considered are essentially point-spread functions which have proven able to model the image-plane projection of a broad class of motions, along with their associated blur effects (Ferreira Jr. et al., 2004).As shown here, such motion modeling capability makes them suitable for a unifying approach to several low-level vision processes, whose full consequences still remain to be explored.This work has been supported by CNPq-Brasil.
Example of the use of operator D(x) for edge detection.(a) : input image.(b) and (c) : edges obtained for u = 0.05 and u = 1.0.
Fig. 4. Filter ) x ( H u , as a function of x/u. Examples of shape estimation via GSFS.(a) and (b) : input images.(c) and (d) : estimated depth maps.
matching (optical flow) equation, and the other involving photometric (reflectance map) considerations.Assuming a uniformly rotating surface, with angular velocity components A and B, along the x and y directions,

7 .
Fig. 6.Green's function ) x , x ( G 0 U , for 0 x 0 = , 1 x U = and . 1 u 0 =Using this to filter the input image, we obtain its matching pair 2 I , which, substituted for I in equation (39), yields the shape estimate Z(x,y).Figure7illustrates results of this approach.