At the present time, many studies are being conducted working toward the implementation of an Intelligent Traffic System (ITS). One field of this research is driving support systems, and many studies are being conducted to develop systems which identify and recognize road signs in front of the vehicle, and then use this information to notify the driver or to control the vehicle (Doermann et al., 2003, Gavrila, 1998, Hsu & Huang, 2001, Kimura et al., 2005, Mo & Aoki, 2004, Piccioli et al., 1996, Uchida et al., 2006, Uchimura et al., 2000, Yamaguchi et al., 2007).
Development of a system which can provide road information to the driver at any time is already underway. This system uses wireless communication with special narrow-band signal transmitters installed on the roadside, a technology which has already been commercialized with ETC. With the construction of this type of infrastructure, it is believed that there will be a change in the method of providing road sign information from the current method of providing visual information. However, much time will be required before this infrastructure covers all roads in local areas, and it is likely that as long as vehicles are driven by human drivers, road signs will never disappear as a means of providing traffic information.
On the other hand, the increase in traffic accidents accompanying the increasing amount of traffic has become a serious problem for society. The occurrence of traffic accidents is particularly high under special road conditions, such as at the entrance to a one-way street, sharp curves, and intersections without traffic signals. One possible countermeasure is to install "STOP," "NO LEFT TURN," and other traffic signs in order to notify the driver of the road conditions and other traffic information. However, there remains the possibility that the driver will, depending on his/her state of mind, fail to notice the sign while driving. A serious accident is possible if the driver fails to notice a sign such as "DO NOT ENTER," "STOP," or "ONLY DESIGNATED DIRECTION PERMITTED." It is said that a driver who is operating a vehicle is relying 80 - 90% on visual information to understand the environment outside the vehicle.
Predictions state that approximately 40% of traffic accidents could be prevented by reducing forward inattention among the drivers. It is possible that accidents can be prevented by utilizing an automatic road sign recognition system to provide traffic information to the driver, including information about the road in front of the vehicle. Image recognition technology which can correctly identify the external environment by means of images obtained from an on-board camera would reduce the burden on the driver and is expected to be an effective approach to driver support aimed at improving traffic safety. To date, studies have been conducted concerning development of a variety of driver support systems, primarily centered on the recognition of circular road traffic signs. However, there has been insufficient research concerning methods of recognizing road traffic signs of other shapes. In addition, previous studies have not considered changes in the signs resulting from changes in external illumination, and as a result there are problems with the effectiveness of identification and with the processing time (Inoue et al., 2003, Mastuura et al., 2002, Uchimura et al., 1998).
In order to implement a system which can recognize road signs and provide this information to the driver, both a method of identifying the areas in the camera images which contain signs and a method of recognizing the specific types of signs will be necessary. For the sign area identification method, a method using a genetic algorithm (Uchimura et al., 1998) and a method using active networks (Yabuki et al., 2000) have been proposed, and a consistent level of identification accuracy has been reported for these methods. However, the problem is the processing speed of the on-board system which is required to achieve real-time performance. Many solutions have been proposed, most of them involving identification of specific sign colors. One example is a method of identifying circular signs based on an RGB subtraction method (Mastuura et al., 2002) in order to rapidly identify the colors (specific colors) (Zin & Hama, 2005) used on the signs. For the road sign recognition method, a template matching method that is broadly used in the image processing fields, as well as improved versions of this method (Chen & Kao, 2008, Gavrila, 1998, Zhang & Ahao, 2007, , Zhang 2008), have been proposed. However the problem with these methods is that they involve a large amount of image processing, including enlargement, reduction, and rotation, which increases the processing time. With a method based on wavelet transformation or other processes for feature identification in frequency bands, accomplishing high-speed processing is highly difficult. In contrast, a method has been proposed in which the outlines of road signs contained in an image are directly identified as vectors without resizing the identified road sign image, with recognition then being performed by judgments which match the vector representations in the results (Yamauchi & Takahashi, 2003). This method has been reported to be robust in response to different image sizes and some degree of shape rotation, and also able to be carried out at high speed.
All of these proposed road sign identification and recognition methods are processes performed using an instantaneously-acquired static image, and it will be necessary for the processes to be performed for each frame if the method is to be commercialized. This means that correct recognition will not be possible in static images where temporary changes, such as a change in the light conditions or the sign being hidden by an object, have occurred. As a result, at such times an incorrect recognition result will be presented to the driver.
The method which this paper proposes assumes the use of an on-board camera, and involves processing of a moving image, allowing effective processing in cases when the sign is temporarily hidden by another object. In addition, this paper proposes a simple algorithm for sign detection and recognition that reduces the amount of required calculation. This will allow real-time processing on a software base with a general-purpose personal computer, a step which will be necessary in order to install a road sign recognition system onto a vehicle in the future. First, sign candidates are identified in the moving image based on the color-difference signals in the sign color information, and then the identified signs are recognized based on the sign external dimensions and the aspect ratio of the pattern dimensions on the sign. Nine kinds of signs, such as "STOP," "NO ENTRY" and "NO PASSAGE,” were prepared, each 1/3 the size of the actual signs. Experiments were then carried out for recognition of these signs under various kinds of measurement conditions. These road signs consisted of a maximum of four colors and relatively simple designs.
2. System configuration
Figure 1 shows the configuration of the road sign detection and recognition system. The scene which included the sign was captured by a camera and sent to a computer. Then the sign recognition program, which was written by the authors of this study, analyzed the moving image, detected the sign, and recognized the sign based on the criteria described in Sec. 3. The result of this recognition was then displayed in the top left corner, and the recognized sign was shown colored using light-blue and yellow-green colors.
3. Configuration of the detection and recognition program
Three types of signs were used: "white on a red background," "white on a blue background," and "black on a yellow background." The configuration of the detection and recognition program is explained below, using the example of "white on a red background" signs such as "STOP," "NO ENTRY," and "NO PASSAGE," as shown in Figure 2.
The program consisted of three main function blocks: "Capture of the image," "Detection," and "Recognition and display of results," as shown in Figure 3.
3.1 Capture of the dynamic image
In the "Capture of the image" block shown in Figure 3, the RGB signals were converted to YCbCr color-difference signals (Miyahara & Yoshida, 1988, Rehrman & Priese, 1998) as shown in Figure 4, where Y (intensity) = 0.299R + 0.587G + 0.114R, Cb (color difference signal for blue) = 0.5B - 0.169R - 0.331G, and Cr (color difference signal for red) = 0.5R - 0.419G - 0.08B. These signals were output to the "Detection" block.
The upper and lower sides of the 320 240 original image were trimmed away in order to resize the image to 320 180. This was done because there is a low probability of signs existing in the top and bottom areas of the image.
The "white on a red background" signs each had a simple white pattern in the red area. In the "Detection" block, white and red areas with values that exceed the threshold values were extracted from the image by using the Y and Cr signals. The program calculated the ratio of the height of the red area to that of the white area, and the ratio of the width of the red area to that of the white area. The procedure for calculation of these ratios is described in detail below.
Images using the Y and Cr signals are shown in Figure 5. By setting the threshold value for Y to 0.57, the Y-signal image was transformed into a binary image. This binary image also included many white areas other than the one white pattern in the sign. In other words, it was not an extraction of just the sign. By setting the threshold value for Cr to 0.56, the Cr-signal image was also transformed into a binary image.
The white area in the Cr-signal binary image represented the red area of the sign as shown in Figure 6(a). Dilation (enlargement) and erosion (contraction) operations were performed several times on this binary image, in which the white area of the image corresponds to the red area of the sign, until there were no black areas remaining inside this white area. Next, this image with no black area inside the sign was ANDed with the Y-signal binary image. In this way, only the white area in the sign was extracted, as shown in Figure 6(b).
By using the red and white areas, the ratio of the height of the red area to that of the white area (called the height ratio), and the ratio of the width of the red area to that of the white area (called the width ratio), can be calculated as shown in Figure 7. The height and width ratios for the "STOP," "NO ENTRY" and "NO PASSAGE" signs are shown in Table 1. These values are expressed as a range because the image is affected by the light conditions and/or the image processing operations.
3.3. Recognition and display of results
The "Recognition and display of results" block distinguished between these three kinds of signs by using the height and width ratios from the previous block. When only the height ratios were used, the ranges of the ratios for "STOP" and "NO ENTRY" partially overlapped. In contrast, because the width ratios of these three signs are different, the signs could be distinguished using only the width ratios. However, the width ratios among signs are not always different, and if recognition is restricted to just one ratio, some signs may be unrecognizable. We are certain that both of these ratios are useful when recognizing signs.
The following concept was adopted for the recognition algorithm. As shown in Table 2(a), three thresholds each were set for the height and width ratios, and numbers were assigned to each of the thresholds so that the product of code "a" times code "b" corresponded to a particular sign, as shown in Table 2(b). Here, the three ranges were determined based on the ranges of measured values shown in Table 1.
Nine kinds of road signs, each 1/3 the size of the actual signs, were prepared. The “white on a red background” signs were "STOP," "NO ENTRY," and "NO PASSAGE." The “white on a blue background” signs were "ONLY BICYCLE," "CROSSING," and "NO ENTRY EXCEPT DESIGNATED DIRECTIONS." The “black on a yellow background” signs were "INTERSECTION," "ROAD CROSSING," and "UNDER CONSTRUCTION." Experiments were then carried out for recognition of these signs under different measurement conditions indoors and outdoors. These road signs consisted of a maximum of four colors and relatively simple designs. The signs were detected based on their color information, and were recognized based on the ratios of the component colors of the signs.
The dynamic image used was a 24-bit color image of size 320 240 pixels. The processing time was 33 ms/frame (CPU: Pentium 4 1.6 GHz).
Using the proposed processing method, the "ROAD CROSSING" sign was processed for recognition. The result was as shown below. The original RGB image shown in Figure 8(a) was first transformed into a YCrCb image. Then, threshold values were set for the Y, Cr, and Cb signals, and various kinds of processing were used to extract the yellow and black areas as shown in Figure 8(b) and (c). For the yellow area of the sign and the black area inside the sign, the processing also detected the top left corner coordinates of rectangles with heights and widths that match the extracted yellow and black areas of the sign. Each of these areas was surrounded by a square, to which a color was added. Light blue was used for the yellow area and yellow-green for the black area. The name of the sign ("ROAD CROSSING") was indicated in the top left corner, as shown in Figure 9.
Displaying the recognized sign in this manner has the following advantage. If the result is not displayed, then when the recognition fails or is mistaken, the observer is unable to judge which signs were and were not recognized during the processing.
The purpose of this study was to simplify the identification and recognition algorithms and allow correct sign recognition. It showed that the system is able to satisfy this objective at the current processing time of 33 ms/frame. In comparison with methods previously proposed, this method was able to recognize a greater number of sign types while maintaining the same or better processing speed. The methods for comparison are the following.
Reference  (Test environment: Pentium 1.2 GHz, Recognized signs: round signs, Processing time: approx. 0.15 s/frame)
Reference : (Test environment: Pentium 166 MHz, Identified signs: round signs, Processing time: approx. 1.5 s/frame)
Reference : (Test environment: Pentium 120 MHz, Recognized signs: round signs, Processing time: approx. 35 s/frame)
Reference  (test environment: Pentium 4 2.2 GHz, Identified signs: round and triangular signs, Processing time: approx. 0.1 s/frame).
This study proposed a processing method for extracting the features of road signs and recognizing what each of these road sign means. Nine kinds of signs were used, consisting of a maximum of four colors and relatively simple designs. This method focuses on the ratio of the component colors. We verified that the method is able to correctly recognize the signs in real time. With the proposed method, good identification and recognition results were achieved with a moving image obtained under normal light conditions, and at faster processing times than the methods previously proposed.
In the future, consideration must be given to the following issues. First, recognition will be impossible if there are two signs with height and width ratios which are nearly the same. In this case, other information, such as the sign configuration, needs to be added. Second, the recognition results in this experiment deteriorated when there was an object with the same color pattern in the vicinity, and under special light conditions such as when the sign was backlit. Third, the angle of the camera with respect to the sign affects the shape of the image. Measures to address these issues need to be considered for the next step. We intend to consider new corrections and recognition methods in order to improve the recognition results for images obtained under poor photographic conditions.