Open access

Feature Extraction and Recognition of Road Sign Using Dynamic Image Processing

Written By

Shigeharu Miyata, Akira Yanou, Hitomi Nakamura and Shin Takehara

Published: 01 December 2009

DOI: 10.5772/7047

From the Edited Volume

Image Processing

Edited by Yung-Sheng Chen

Chapter metrics overview

4,060 Chapter Downloads

View Full Metrics

1. Introduction

At the present time, many studies are being conducted working toward the implementation of an Intelligent Traffic System (ITS). One field of this research is driving support systems, and many studies are being conducted to develop systems which identify and recognize road signs in front of the vehicle, and then use this information to notify the driver or to control the vehicle (Doermann et al., 2003, Gavrila, 1998, Hsu & Huang, 2001, Kimura et al., 2005, Mo & Aoki, 2004, Piccioli et al., 1996, Uchida et al., 2006, Uchimura et al., 2000, Yamaguchi et al., 2007).

Development of a system which can provide road information to the driver at any time is already underway. This system uses wireless communication with special narrow-band signal transmitters installed on the roadside, a technology which has already been commercialized with ETC. With the construction of this type of infrastructure, it is believed that there will be a change in the method of providing road sign information from the current method of providing visual information. However, much time will be required before this infrastructure covers all roads in local areas, and it is likely that as long as vehicles are driven by human drivers, road signs will never disappear as a means of providing traffic information.

On the other hand, the increase in traffic accidents accompanying the increasing amount of traffic has become a serious problem for society. The occurrence of traffic accidents is particularly high under special road conditions, such as at the entrance to a one-way street, sharp curves, and intersections without traffic signals. One possible countermeasure is to install "STOP," "NO LEFT TURN," and other traffic signs in order to notify the driver of the road conditions and other traffic information. However, there remains the possibility that the driver will, depending on his/her state of mind, fail to notice the sign while driving. A serious accident is possible if the driver fails to notice a sign such as "DO NOT ENTER," "STOP," or "ONLY DESIGNATED DIRECTION PERMITTED." It is said that a driver who is operating a vehicle is relying 80 - 90% on visual information to understand the environment outside the vehicle.

Predictions state that approximately 40% of traffic accidents could be prevented by reducing forward inattention among the drivers. It is possible that accidents can be prevented by utilizing an automatic road sign recognition system to provide traffic information to the driver, including information about the road in front of the vehicle. Image recognition technology which can correctly identify the external environment by means of images obtained from an on-board camera would reduce the burden on the driver and is expected to be an effective approach to driver support aimed at improving traffic safety. To date, studies have been conducted concerning development of a variety of driver support systems, primarily centered on the recognition of circular road traffic signs. However, there has been insufficient research concerning methods of recognizing road traffic signs of other shapes. In addition, previous studies have not considered changes in the signs resulting from changes in external illumination, and as a result there are problems with the effectiveness of identification and with the processing time (Inoue et al., 2003, Mastuura et al., 2002, Uchimura et al., 1998).

In order to implement a system which can recognize road signs and provide this information to the driver, both a method of identifying the areas in the camera images which contain signs and a method of recognizing the specific types of signs will be necessary. For the sign area identification method, a method using a genetic algorithm (Uchimura et al., 1998) and a method using active networks (Yabuki et al., 2000) have been proposed, and a consistent level of identification accuracy has been reported for these methods. However, the problem is the processing speed of the on-board system which is required to achieve real-time performance. Many solutions have been proposed, most of them involving identification of specific sign colors. One example is a method of identifying circular signs based on an RGB subtraction method (Mastuura et al., 2002) in order to rapidly identify the colors (specific colors) (Zin & Hama, 2005) used on the signs. For the road sign recognition method, a template matching method that is broadly used in the image processing fields, as well as improved versions of this method (Chen & Kao, 2008, Gavrila, 1998, Zhang & Ahao, 2007, , Zhang 2008), have been proposed. However the problem with these methods is that they involve a large amount of image processing, including enlargement, reduction, and rotation, which increases the processing time. With a method based on wavelet transformation or other processes for feature identification in frequency bands, accomplishing high-speed processing is highly difficult. In contrast, a method has been proposed in which the outlines of road signs contained in an image are directly identified as vectors without resizing the identified road sign image, with recognition then being performed by judgments which match the vector representations in the results (Yamauchi & Takahashi, 2003). This method has been reported to be robust in response to different image sizes and some degree of shape rotation, and also able to be carried out at high speed.

All of these proposed road sign identification and recognition methods are processes performed using an instantaneously-acquired static image, and it will be necessary for the processes to be performed for each frame if the method is to be commercialized. This means that correct recognition will not be possible in static images where temporary changes, such as a change in the light conditions or the sign being hidden by an object, have occurred. As a result, at such times an incorrect recognition result will be presented to the driver.

The method which this paper proposes assumes the use of an on-board camera, and involves processing of a moving image, allowing effective processing in cases when the sign is temporarily hidden by another object. In addition, this paper proposes a simple algorithm for sign detection and recognition that reduces the amount of required calculation. This will allow real-time processing on a software base with a general-purpose personal computer, a step which will be necessary in order to install a road sign recognition system onto a vehicle in the future. First, sign candidates are identified in the moving image based on the color-difference signals in the sign color information, and then the identified signs are recognized based on the sign external dimensions and the aspect ratio of the pattern dimensions on the sign. Nine kinds of signs, such as "STOP," "NO ENTRY" and "NO PASSAGE,” were prepared, each 1/3 the size of the actual signs. Experiments were then carried out for recognition of these signs under various kinds of measurement conditions. These road signs consisted of a maximum of four colors and relatively simple designs.

Advertisement

2. System configuration

Figure 1 shows the configuration of the road sign detection and recognition system. The scene which included the sign was captured by a camera and sent to a computer. Then the sign recognition program, which was written by the authors of this study, analyzed the moving image, detected the sign, and recognized the sign based on the criteria described in Sec. 3. The result of this recognition was then displayed in the top left corner, and the recognized sign was shown colored using light-blue and yellow-green colors.

Figure 1.

Configuration of the road sign detection and recognition system, consisting of a camera, a computer, and a display

Advertisement

3. Configuration of the detection and recognition program

Three types of signs were used: "white on a red background," "white on a blue background," and "black on a yellow background." The configuration of the detection and recognition program is explained below, using the example of "white on a red background" signs such as "STOP," "NO ENTRY," and "NO PASSAGE," as shown in Figure 2.

The program consisted of three main function blocks: "Capture of the image," "Detection," and "Recognition and display of results," as shown in Figure 3.

Figure 2.

Examples of "white on a red background" signs.

Advertisement

3.1 Capture of the dynamic image

In the "Capture of the image" block shown in Figure 3, the RGB signals were converted to YCbCr color-difference signals (Miyahara & Yoshida, 1988, Rehrman & Priese, 1998) as shown in Figure 4, where Y (intensity) = 0.299R + 0.587G + 0.114R, Cb (color difference signal for blue) = 0.5B - 0.169R - 0.331G, and Cr (color difference signal for red) = 0.5R - 0.419G - 0.08B. These signals were output to the "Detection" block.

Figure 3.

The program consisted of three function blocks: "Capture of the image," "Detection," and "Recognition and display of results."

The upper and lower sides of the 320 240 original image were trimmed away in order to resize the image to 320 180. This was done because there is a low probability of signs existing in the top and bottom areas of the image.

Figure 4.

Image YCrCb signals, converted from RGB signals.

3.2. Detection

The "white on a red background" signs each had a simple white pattern in the red area. In the "Detection" block, white and red areas with values that exceed the threshold values were extracted from the image by using the Y and Cr signals. The program calculated the ratio of the height of the red area to that of the white area, and the ratio of the width of the red area to that of the white area. The procedure for calculation of these ratios is described in detail below.

Images using the Y and Cr signals are shown in Figure 5. By setting the threshold value for Y to 0.57, the Y-signal image was transformed into a binary image. This binary image also included many white areas other than the one white pattern in the sign. In other words, it was not an extraction of just the sign. By setting the threshold value for Cr to 0.56, the Cr-signal image was also transformed into a binary image.

Figure 5.

Image YCrCb signals, converted from RGB signals.

The white area in the Cr-signal binary image represented the red area of the sign as shown in Figure 6(a). Dilation (enlargement) and erosion (contraction) operations were performed several times on this binary image, in which the white area of the image corresponds to the red area of the sign, until there were no black areas remaining inside this white area. Next, this image with no black area inside the sign was ANDed with the Y-signal binary image. In this way, only the white area in the sign was extracted, as shown in Figure 6(b).

Figure 6.

Color areas of the sign.

By using the red and white areas, the ratio of the height of the red area to that of the white area (called the height ratio), and the ratio of the width of the red area to that of the white area (called the width ratio), can be calculated as shown in Figure 7. The height and width ratios for the "STOP," "NO ENTRY" and "NO PASSAGE" signs are shown in Table 1. These values are expressed as a range because the image is affected by the light conditions and/or the image processing operations.

Figure 7.

Definition of height ratio and width ratio

"White on a red background" sign Height ratio Width ratio
STOP 0.190 - 0.233 0.532 - 0.574
NO ENTRY 0.217 - 0.241 0.821 - 0.885
NO PASSAGE 0.710 - 0.793 0.717 - 0.768

Table 1.

Height and width ratios for "white on a red background" sign

3.3. Recognition and display of results

The "Recognition and display of results" block distinguished between these three kinds of signs by using the height and width ratios from the previous block. When only the height ratios were used, the ranges of the ratios for "STOP" and "NO ENTRY" partially overlapped. In contrast, because the width ratios of these three signs are different, the signs could be distinguished using only the width ratios. However, the width ratios among signs are not always different, and if recognition is restricted to just one ratio, some signs may be unrecognizable. We are certain that both of these ratios are useful when recognizing signs.

The following concept was adopted for the recognition algorithm. As shown in Table 2(a), three thresholds each were set for the height and width ratios, and numbers were assigned to each of the thresholds so that the product of code "a" times code "b" corresponded to a particular sign, as shown in Table 2(b). Here, the three ranges were determined based on the ranges of measured values shown in Table 1.

Range of height ratio a Range of width ratio b
0.18 and 0.81 1 0.18 and 0.91 1
0.181 - 0.7 2 0.181 - 0.8 1
0.71 - 0.8 4 0.81 - 0.9 1.5
(a)
Road Sign A B Code (= a b)
UNKNOWN 1 1 1
STOP 2 1 2
NO ENTRY 2 1.5 3
NO PASSAGE 4 1 4
(b)

Table 2.

Height and width ratios for "white on a red background" sign

Advertisement

4. Experiment

Nine kinds of road signs, each 1/3 the size of the actual signs, were prepared. The “white on a red background” signs were "STOP," "NO ENTRY," and "NO PASSAGE." The “white on a blue background” signs were "ONLY BICYCLE," "CROSSING," and "NO ENTRY EXCEPT DESIGNATED DIRECTIONS." The “black on a yellow background” signs were "INTERSECTION," "ROAD CROSSING," and "UNDER CONSTRUCTION." Experiments were then carried out for recognition of these signs under different measurement conditions indoors and outdoors. These road signs consisted of a maximum of four colors and relatively simple designs. The signs were detected based on their color information, and were recognized based on the ratios of the component colors of the signs.

The dynamic image used was a 24-bit color image of size 320 240 pixels. The processing time was 33 ms/frame (CPU: Pentium 4 1.6 GHz).

Using the proposed processing method, the "ROAD CROSSING" sign was processed for recognition. The result was as shown below. The original RGB image shown in Figure 8(a) was first transformed into a YCrCb image. Then, threshold values were set for the Y, Cr, and Cb signals, and various kinds of processing were used to extract the yellow and black areas as shown in Figure 8(b) and (c). For the yellow area of the sign and the black area inside the sign, the processing also detected the top left corner coordinates of rectangles with heights and widths that match the extracted yellow and black areas of the sign. Each of these areas was surrounded by a square, to which a color was added. Light blue was used for the yellow area and yellow-green for the black area. The name of the sign ("ROAD CROSSING") was indicated in the top left corner, as shown in Figure 9.

Figure 8.

Processed railroad crossing sign.

Figure 9.

Result displayed with colors which easily indicate the location of the recognized sign.

Displaying the recognized sign in this manner has the following advantage. If the result is not displayed, then when the recognition fails or is mistaken, the observer is unable to judge which signs were and were not recognized during the processing.

The purpose of this study was to simplify the identification and recognition algorithms and allow correct sign recognition. It showed that the system is able to satisfy this objective at the current processing time of 33 ms/frame. In comparison with methods previously proposed, this method was able to recognize a greater number of sign types while maintaining the same or better processing speed. The methods for comparison are the following.

  • Reference [10] (Test environment: Pentium 1.2 GHz, Recognized signs: round signs, Processing time: approx. 0.15 s/frame)

  • Reference [11]: (Test environment: Pentium 166 MHz, Identified signs: round signs, Processing time: approx. 1.5 s/frame)

  • Reference [12]: (Test environment: Pentium 120 MHz, Recognized signs: round signs, Processing time: approx. 35 s/frame)

  • Reference [2] (test environment: Pentium 4 2.2 GHz, Identified signs: round and triangular signs, Processing time: approx. 0.1 s/frame).

Advertisement

5. Conclusion

This study proposed a processing method for extracting the features of road signs and recognizing what each of these road sign means. Nine kinds of signs were used, consisting of a maximum of four colors and relatively simple designs. This method focuses on the ratio of the component colors. We verified that the method is able to correctly recognize the signs in real time. With the proposed method, good identification and recognition results were achieved with a moving image obtained under normal light conditions, and at faster processing times than the methods previously proposed.

In the future, consideration must be given to the following issues. First, recognition will be impossible if there are two signs with height and width ratios which are nearly the same. In this case, other information, such as the sign configuration, needs to be added. Second, the recognition results in this experiment deteriorated when there was an object with the same color pattern in the vicinity, and under special light conditions such as when the sign was backlit. Third, the angle of the camera with respect to the sign affects the shape of the image. Measures to address these issues need to be considered for the next step. We intend to consider new corrections and recognition methods in order to improve the recognition results for images obtained under poor photographic conditions.

References

  1. 1. Chen C. C. Kao D. S. 2008 DCT-based zero replacement reversible image watermarking approach, International Journal of Innovative Computing, Information and Control, 4 11 3027 3036 .
  2. 2. Doermann D. Liang J. Li H. 2003 Progress in camera-based document image analysis, Proc.of IC-DAR’03, 606 616 .
  3. 3. Gavrila D. M. 1998 Multi-feature hierarchical template matching using distance transforms, Proc. of I EEE International Conference on Pattern Recognition, 439 444 .
  4. 4. Hsu S. Huang C. 2001 Road sign detection and recognition using matching pursuit method, Image and Vision Computing, 19 3 119 129 .
  5. 5. Inoue Y. Ishikawa N. Nakajima M. 2003 Automatic recgnition of road signs (in Japanese), Techical report of IEICE, ITS2003-98, 57 62 .
  6. 6. Kimura S. Tano H. Sakamoto T. Shiroma H. Ohashi Y. 2005 Traversal region and obstacle detection using tilted-plane-based stereo vision (in Japanese), Journal of the Robotics Society of Japan, 23 2 59 66 .
  7. 7. Matsuura D. Yamauchi H. Takahashi H. 2002 Extracting circular road sign using specific color distinction and region limitation (in Japanese), IEICE Trans. D-II, J85-D -II, 6 1075 1083 .
  8. 8. Miyahara M. Yoshida Y. 1988 Mathematical transform of (R, G, B) color space to Munsel (H, V, C) color data, SPIE visual Communication and Image Processing, visual Communication, 1001 650 657 .
  9. 9. Mo G. Aoki Y. 2004 A recognition method for traffic sign in color image (in Japanese), IEICE Trans. D-II, J87-D -II, 12 2124 2135 .
  10. 10. Piccioli G. Michieli E. De Parodi P. Campani M. 1996 Robust method for road sign detection and recognition, Image and Vision Computing, 14 3 209 223 .
  11. 11. Rehrmann V. Priese L. 1998 Fast and robust segmentation of natural color scenes, Proc. of the 3rd Asian Conference on Computer Vision, 1 598 606 .
  12. 12. Uchida S. Iwamura M. Omachi S. Kise K. 2006 Category data embedding for camera-based character recognition (in Japanese), IEICE Trans. D-II, J89-D -II, 2 344 352 .
  13. 13. Uchimura K. Kimura H. Wakiyama S. 1998 Extraction and recognition of circular road signs using road scene color image (in Japanese), IEICE Trans. A, J81-A , 4 546 553 .
  14. 14. Uchimura K. Wakiyama S. Fujino M. 2000 Extraction of circular traffic sign using limited color indication (in Japanese), IEICE Trans. D-II, J83-D -II, 2 855 858 .
  15. 15. Yabuki N. Matsuda Y. Kataoka D. Sumi Y. Fukui Y. Miki S. 2000 A study on an atumatic stop of computation in active net (in Japanese), Technical report of IEICE, 99 575 69 76 .
  16. 16. Yamauchi H. Takahashi H. 2003 A road sign recognition technique by tracing outline vectors, Journal of the Institute of Image Information and Television Engineers, 57 7 847 853 .
  17. 17. Yamaguchi H. Kojima A. Miyamoto T. Takahashi H. Fukunaga K. 2007 A Robust road sign recognition by superposing extracted regions from successive frame (in Japanese), IEICE Trans. D-II, J90-D -II, 2 494 502 .
  18. 18. Zhang Z. Ahao Y. 2007 Multiple description image coding based on fractal, International Journal of Innovative Computing, Information and Control, 3 6 B), 1615 1623 .
  19. 19. Zhang X. 2008 The improvement of a feature-based image mosaics algorithm, International Journal of Innovative Computing, Information and Control, 4 10 2759 2764 .
  20. 20. Zin T. T. Hama H. 2005 A robust road sign recognition using segmentation with morphology and relative color, Journal of the Institute of Image Information and Television Engineers, 59 9 1333 1342 .

Written By

Shigeharu Miyata, Akira Yanou, Hitomi Nakamura and Shin Takehara

Published: 01 December 2009