Open access peer-reviewed chapter

Machine Vision for Inspection: A Case Study

By Brandon Miles and Brian Surgenor

Submitted: November 15th 2010Reviewed: April 12th 2011Published: August 17th 2011

DOI: 10.5772/21546

Downloaded: 2484

1. Introduction

Automated inspection systems have the potential to significantly improve quality and increase production rates in the manufacturing industry. Machine vision (MV) is an example of one inspection technology that has been successfully applied to production lines. A wide variety of industrial inspection applications of MV systems can be found in the literature. For example, Lee et al. (2007) applied a MV system to bird handling for the food industry. Reynolds et al. (2004) looked at solder paste inspection for the electronics industry. Gayubo et al. (2006) developed a system to locate tearing defects in sheet metal.

There has also been numerous laboratory based work on MV systems with practical applications. Jackman et al. (2009) used a vision system to predict the quality of beef. Kumar (2003) worked on the detection of defects in twill weave fabric samples. Garcia et al. (2006) checked for missing and misaligned electronics components. Hunter et al. (1995) confirmed circularity in brake shoes. Kwak et al. (2000) identified surface defects in leather.

Although the range of applications is broad, they all tended to adopt the same image processing system with four main stages. The first is image acquisition. This is followed by preprocessing of the image, including applying various filters and selecting regions of interest. The third stage is feature extraction where individual features are extracted from the image. Finally a classifier is used to determine whether a given part is acceptable or not.

The automotive industry presents a particularly challenging environment for MV based inspection. With changing lighting conditions in a dirty environment there is a need for robust and accurate classifiers to perform accurate inspection. Feature selection routines can be used to improve the results of an ANFIS based classifier for automotive applications (Miles and Surgenor, 2009). Although there is potential for good results with this approach, it can take hours of processing time to compute an accurate solution (Killing et al., 2009).

This chapter presents the results of a project where six classification techniques were examined to see if development time could be reduced without sacrificing performance. As a case study, the problem of fastener insertion to an automotive part known as a cross car beam was investigated. Images taken from a production assembly line were used as the source of the data. The types of classifiers under investigation were:

  1. a Neural Network based processor,

  2. Principle Component Analysis to reclassify the input feature set and

  3. a direct Eigenimage approach to avoid the need to extract features from each image.

These methods were compared in terms of classification accuracy. An additional data set was also used to test the performance of these classifiers in detecting orientation defects in addition to presence and absence of clips. The results of these investigations are presented with a comparison of the performance on different datasets.

2. Techniques

Data for classifiers can be generated from input images in a variety of ways. The first is a feature based classifier. In this approach, features such as lines, holes and circles are extracted from an image. Numerical values are then obtained from these features, such as the x, y coordinates of a circle. These values can then be used as inputs to either a traditional Neural Network or a Neuro-Fuzzy System such as ANFIS. Principle Component Analysis (PCA) can be used to reclassify features to generate a data set with fewer inputs.

A second method to generate input values involves the application of Eigenimages. Eigenimages are generated from a set of training images. New images can be expressed as combinations of these Eigenimages. Coefficients are assigned to all the Eigenimages used to express a given set of input images. These coefficients can then be used to train either a Neural Network or an ANFIS system.

The six specific types of classifiers under investigation for this study are summarized in Table 1. They are grouped into Feature Based classifiers and Eigenimage Based classifiers along with ANFIS versus Neural Networks. This allows for a comparison between different classification techniques namely Neural Networks and ANFIS. However more importantly, it compares the Eigenimage based approach, which works directly on the pixels to produce classifier inputs, with the feature based approach, where features are first identified and then a classifier is trained.

ANFISNeural Network
Feature Based
Feature based with ANFISFeature based with a Neural Network
Feature based with PCA and ANFISFeature based with PCA and a Neural Network
Eigenimage Based
Eigenimage based with ANFISEigenimage based with a Neural Network

Table 1.

Summary of the six classifiers under investigation.

2.1. ANFIS

ANFIS possesses a full Fuzzy Inferencing Structure. A fuzzy structure is established as a preliminary model of the system. It can then be updated with additional inputs. Thus, it is trainable like a Neural Network. In this application, the ANFIS system was implemented using the MATLAB Fuzzy Toolbox. The reader is referred to (Roger Jang, 1993) for further background on this technique. The ANFIS technique was used as the benchmark in previous work reported in Killing et al. (2009).

2.2. Neural networks

The Neural Network used in this study was a Multi Layer Perceptron (MLP) network with one hidden layer. The hidden layer had a sigmoidal activation function and the output layer had a linear activation function. This approach was taken because the output value needed to be a scalar instead of the more common binary value. Twenty hidden nodes were used to minimize the possibility of over training the system while giving it sufficient size to be useful.

2.3. Principal component analysis

PCA can be used to reclassify the data in terms of maximized variance between input data and output data. By reorganizing the inputs into components ranked by variance the input data set can be reduced in size by keeping only the components with a high degree of variance. In this case, as chosen by experience, only the components comprising the top 95% of the variance were kept. This was typically five to seven inputs.

To calculate the principle components Singular Value Decomposition (SVD) was used. The sample covariance matrix S (calculated from the supplied data) was used to find the principle components:


where X¯is X calculated around the sample means or X¯=(Xc)and c is the mean of X. Once S is calculated the Eigenvalues and eigenvectors of S can be calculated and the principle components can subsequently be found based.

2.4. Eigenimages

The Eigenimage classifier offers a more direct approach without the need to extract features from images. Grey scale images can be represented as vectors of pixels. In this way the dataset X can be generated by vectors of images where X = [x 1 , x 2 , x 3 … x n ] for n sample images. SVD can then be performed identifying the principle components [e 1 , e 2 , e 3, …, e n ].

These Eigenimages or principle components are a series of images that when combined are able to represent the entire dataset of images. It is often desirable to only select the principle components that have the largest variances E = [e 1 , e 2 , … e k ]. In this case, it is possible to project a given image onto the Eigenspace constructed from these principle components. This projection generates an Eigenpoint from the image x with a set of project coefficients p = [p 1 , p 2, … p n ]. This projection is calculated by:


Once this projection is known the set of points p can be used to classify the images. Note that p presents a smaller set of input data than the entire image, which presents benefits computationally. In this way the PCA technique acts like a feature detector, producing numerical values from an image. The theory of EigenImages is discussed further in Sun et al. (2007) and Ohba and Ilkeuchi (1997). Both a Neural Network and ANFIS can be trained based on these principle components.

2.5. Features

Lines, Holes and Circles can be found in an image using a Hough transform. Additionally large colour blobs can be located based on locations of certain colours of pixels. Also given a specific region of interest in the images the average red, green, blue and grey scale intensity values can be found. A radial hole method as detailed in Miles and Surgenor (2009) has also been used. These features are found relative to the centre of the beam where possible.

Two new features have been introduced in order to help improve the accuracy of the results. These are a Generalized Hough Rectangle feature and PCA colour feature.

2.5.1. Hough rectangle feature

Lines, Holes and Circles can be found in an image using a Hough transform. Additionally One of the extensions of the generalized Hough transform (GHT) is using it to find rectangles. The symmetry that rectangles have can be employed to help locate them. First the gradient direction and magnitude of all the pixels are calculated. Say the rectangle has side lengths A and B, where A>B. The sides A are oriented in the direction of the major axis of the rectangle. The sides B are oriented in the direction of the minor axis of the rectangle. Assume that the angle of A is between 0 and 90º, then for a given edge pixel, if it’s gradient is between 0 and 90º cast votes for a rectangle with a centre on a line at ±B/2 pixels away from the edge pixel in the direction of the edge pixel’s gradient. Alternatively if the directional gradient of the edge pixel is between 90º and 180º then cast votes for a rectangle centred on a line at ±A/2 from the edge pixel.

Then in another plane accumulate votes assuming that the angle of the major axis is between 90º and 180º. If the directional gradient of an edge pixel is between 90º and 180º cast votes for rectangles centred on a line A/2 pixels away from the edge pixel in the direction of the edge pixel’s gradient. Alternatively if the directional gradient of the edge pixel is between 0 and 90º then cast votes for a rectangle centred on a line at ±B/2 from the edge pixel.

This approach will produce two prominent peaks in one of the two accumulators. The largest one is for the major axis and the smaller one is for the minor axis. Where these two peaks intersect should be the centre of the rectangle and the direction should be known because of the slope of the line. Figure 2 illustrates this technique. See Chapter 14 in Davies (2005) for further details.

Figure 1.

Illustration of the Hough rectangle method.

2.5.2. PCA colour feature

As an additional source of information, it is possible to apply PCA to the three red, green and blue elements of an image. The result is three new colour components that can be reclassified in terms of maximum order of intensity (Lee et al., 2007). The image is represented as a vector of pixels each with red green and blue values. By applying PCA to this vector new colour components can be generated. Applying SVD to the sample correlation matrix will generate eigenvectors and Eigenvalues. These eigenvectors can then be used to translate the image into its new colour components. The Eigenvalues of these new colour components can be used as numerical inputs.

3. Case study

The part in question is a cross car beam, which is the metal support located behind the dashboard of an automobile. See Figure 2. The beam is a stamped metal part and a stamped metal radio bracket is welded to this beam. As part of the assembly process four small rectangular clips must be inserted into the radio bracket. These four clips serve as locations for securing the radio unit to the cross car beam. Figure 3 illustrates the manufacturing cell used to inspect each part.

Figure 2.

Cross car beam containing the radio bracket.

This is a safety critical part. It is essential to ensure that these clips are properly installed, because of the consequences that could result from of an unsecure radio unit during a collision. In order to ensure the presence of these clips a machine vision system was developed to automatically inspect each part before it leaves the manufacturing cell that is dedicated to attached the clips. As illustrated in Figure 3, the PLC that controls the manufacturing cell communicates with a standalone PC that controls the vision system.

3.1. Setup

A two camera system was installed on the production assembly line in order to inspect the part. This is shown in the Figure 4(a). The part is pictured in Figure 4(b). The two digital firewire cameras capture 1024 x 768 images. For a lighting solution an LED ring light with a diffuser was chosen. The ring lights are visible on the front of the cameras in Figure 4(a). The decision to use ring lights was made based on the need for a lighting solution that could illuminate the part, but did not have a large footprint due to space restrictions in the cell. It also needed to be easily mountable. Because of this it was not feasible to install an off angle lighting source or to use backlighting. The lights were manually aligned to illuminate the bracket and be centred on the beam.

Figure 3.

PLC controlled manufacturing cell with PC controlled vision sytsem.

Figure 4.

Orientation of (a) cameras looking at bracket and (b) radio bracket in holder.

3.2. Software

QVision a custom software system based on MATLAB® has been used to classify the data. This software is capable of loading library images, selecting and extracting features or regions of interest for Eigenimages, training classifiers through to final inspection of new images. It was integrated to the PLC running the manufacturing cell for online inspection of parts. Figure 5 shows an image of the classification results of the program.

Figure 5.

QVision software GUI used to detect the presence of clips.

3.3. Datasets

Two different data sets were used for comparing the classification techniques. They contain pass images and fail images taken from parts on the assembly line. A pass images consists of the clips present and a fail image consists of the clips missing. Figures 6 and 7 illustrate sample cases of these images. Since there are two cameras looking at the top and bottom clips, there are also top and bottom images.

The first data set which will be referred to as the “original set” consisted of 150 pass images and 114 fail images for the top bracket, and 147 pass images and 125 fail images for the bottom bracket.

Figure 6.

Sample “clean” image from manufacturing cell for top bracket: (a) pass and (b) fail.

The second data set (which will be referred to as the orientation set) has been examined to determine the performance of the system under two conditions. Firstly the lighting has changed in this orientation image set. Secondly 3 new failure modes have also been introduced. The clips do not have significant glare, but the background is much more visible in these images. Figures 8 to 11 show the four methods of failure, which are: missing clip, backwards and upside down clip, backwards clips and upside down clip respectively.

Figure 7.

Sample “clean” image from cell for bottom bracket: (a) pass and (b) fail.

Figure 8.

Sample fail (missing clip) for the orientation image set.

Figure 9.

Sample fail (backwards and upside down clip) for the orientation image set.

Figure 10.

Sample fail (backwards) for the orientation image set.

Figure 11.

Sample fail (upside down) for the orientation image set.

3.4. Performance criteria

The industrial partner has specified a performance goal in terms of false positives (FPs) and false negatives (FNs). An FP is a defective part classified as good and hence shipped to the customer. This is a safety hazard and hence can’t be tolerated. There must be no FPs. An FN is a good part classified as bad. The industrial partner has set this at a maximum rate of 2%. It is a measure of the scrap rate. It should be noted that Receiver Operating Characteristic graphs (Fawcett, 2005) can also be generated as a measure of performance.

For the purposes of this study, the root-mean-squared (RMS) error E rms for a set of images is used as a performance measure. The root-mean-square of the output error defined as:


where, Zidis the desired (correct) classification for the i th image, Z i is the output of the classifier algorithm (ZinforZith) and n is the total number of images. Z = 1 is an unconditional pass (clip present) and Z = 0 is an unconditional fail (clip missing).

4. Results

The six classifiers were trained using the original data set. The system was trained on 40% of the original data set and these results were then checked on 20% of the original data set. These images were chosen randomly. A final 40% was reserved for additional testing purposes. The results of this classification are shown in Table 2.

RankingClassifierClipFalse PositivesFalse Negatives% False NegativesRMS Error Erms
Feature based with PCA and NNClip 10000.0001
1stClip 20000.0115
Clip 30000
Clip 4010.40.0547
Feature based with NNClip 10000.0879
2ndClip 2020.80.0899
Clip 30000.0710
Clip 4020.80.0871
Eigenimage based with NNClip 1072.60.1853
3rdClip 2041.50.1166
Clip 3031.10.1304
Clip 4010.40.1309
Feature based with PCA and ANFISClip 1062.30.0721
4thClip 20000.1102
Clip 30114.20.1748
Clip 4093.40.1509
Eigenimage based with ANFISClip 1276290.3932
5thClip 2640150.3319
Clip 3062.30.1512
Clip 4041.50.1908
Feature based with ANFISClip 1129110.9587
6thClip 2293.40.3230
Clip 310124.50.5226
Clip 4662.30.2805

Table 2.

Results with original data set (264 images per clip).

To improve performance a new set of features was used including the PCA colour technique and Rectangular Hough transform technique. These are shown in Table 3. The results of the Eigenimage backpropagation were not included in this table because they are same as those found in Table 2.

The final set of images that was examined was the new orientation set. There are four additional modes of failure and the lighting has changed. These additional modes of failure can be caught by other methods and the industrial partner does not need the vision system to inspect for these defects. However it does present an excellent set of data to test the robustness of the algorithm. As in the previous cases features were defined on the images. However these were not found relative to the centre of the beam because of difficulty finding the beam with the new lighting. The feature-based results are presented in Table 4.

RankingClassifierClipFalse PositivesFalse Negatives% False NegativesRMS Error Erms
Feature based with PCA and NNClip 10000.0047
1stClip 20000
Clip 3020.80.0798
Clip 40000.0005
Feature based with NNClip 10000.0488
2ndClip 2010.40.0647
Clip 3020.80.1043
Clip 4010.40.0908
Feature based with PCA and ANFISClip 1020.80.0621
3rdClip 20000.0688
Clip 30124.50.1416
Clip 4041.50.1284
Feature based with ANFISClip 1162.30.5047
4thClip 2131.10.1874
Clip 3051.90.1458
Clip 4126100.3767

Table 3.

Results with original data set and additional features (264 images per clip).

For the feature based results from Table 2 features were extracted from an image. These were a combination of holes, lines, circles and colours. These features are illustrated for Clip 1 in Figure 12.

Figure 12.

Features defined on Clip 1 for original data set.

RankingClassifierClipFalse PositivesFalse Negatives% False NegativesRMS Error Erms
Feature based with PCA and NNClip 10000.0047
1stClip 20000
Clip 3020.70.0798
Clip 40000.0005
Feature based with NNClip 10000.0488
2ndClip 2010.40.0647
Clip 3020.70.1043
Clip 4010.40.0908
Feature based with PCA and ANFISClip 1020.80.0621
3rdClip 20000.0688
Clip 301240.1416
Clip 4041.50.1284
Feature based with ANFIS
Clip 11620.5047
4thClip 21310.1874
Clip 30520.1458
Clip 4126100.3767
5thEigenimage based with NNClip 19000.1835
Clip 20000.0662
Clip 30000.0646
Clip 40000.0548
Eigenimage based with ANFISClip 11220.90.2253
6thClip 2252.290.1626
Clip 3110.50.1228
Clip 4120.90.0963

Table 4.

Results with orientation data set (286 images per clip).

Again the best results were found with the PCA Neural Network, followed by the Neural Network. The feature-based results did better than the Eigenimage results and the Neural Network results were better than the ANFIS results.

One interesting result is the Eigenimage backpropagation results trained with a Neural Network. If Clip 1 is ignored the results are perfect.

5. Discussion

Six different classifiers were compared to the classification of clips on an automotive assembly. Tests were done on two images sets and with two different feature sets. Consistently it was seen that a Neural Network classifier whether used on feature data, on feature data with PCA applied or on Eigenimage coefficients performed better than the ANFIS system. The NN results always ranked higher on all the tests. It is reasonable to say that the Neural Network performs better than the ANFIS system. It has also been consistently seen that applying PCA on input data improves the results of classification. The results with feature extraction and PCA ranked higher that the results with feature extraction and no PCA on the majority of the tests.

For the original data set the performance of feature based techniques were better than region of interest techniques. In the case of the orientation data set in general the feature based techniques had similar perform to the Eigenimage techniques. However ignoring Clip 1 the Eigenimage Neural Network technique worked better than any other technique on this set of data. This shows that the Eigenimage technique is better able to distinguish multiple types of faults with brighter lighting than a feature based technique.

Applying PCA to a dataset eliminates the need to perform feature selection improving the result in a systematic way. The Eigenimage technique has the benefit of not needing to extract features. A region of interest is selected and the calculations can then proceed. The greatest benefit of these techniques is their speed of training, which makes the system more flexible.

6. Conclusion

The performance of six different classifiers has been compared as applied to the detection of missing fasteners. Traditional feature based classifiers were first used to train Neural Networks (NN) and Neuro-Fuzzy (ANFIS) systems, with and without Principle Component Analysis (PCA). As an alternate, a non-feature based Eigenimage classifier was used to generate the inputs for the classifiers. It was found that when there was only one type of defect, both the NN and Eigenimage based classifiers, but not the ANFIS based classifier, could achieve the required performance. On the other hand, when there was more than one type of defect, only the NN and ANFIS based classifiers could maintain the required level of performance. Finally, given that the Eigenimage based classifier takes much less time to set up and train, it is considered superior to the NN based classifier for practical applications.


The Authors would like to thank the generous support of AUTO21 and OCE, which has allowed this research to be carried out. They would also like to thank Queen’s University for their support in this research.

© 2011 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike-3.0 License, which permits use, distribution and reproduction for non-commercial purposes, provided the original is properly cited and derivative works building on this content are distributed under the same license.

How to cite and reference

Link to this chapter Copy to clipboard

Cite this chapter Copy to clipboard

Brandon Miles and Brian Surgenor (August 17th 2011). Machine Vision for Inspection: A Case Study, Assembly Line - Theory and Practice, Waldemar Grzechca, IntechOpen, DOI: 10.5772/21546. Available from:

chapter statistics

2484total chapter downloads

More statistics for editors and authors

Login to your personal dashboard for more detailed statistics on your publications.

Access personal reporting

Related Content

This Book

Next chapter

Final Results of Assembly Line Balancing Problem

By Waldemar Grzechca

Related Book

First chapter

Multi-Agent Based Distributed Manufacturing

By J. Li, J.Y. H. Fuh, Y.F. Zhang and A.Y.C. Nee

We are IntechOpen, the world's leading publisher of Open Access books. Built by scientists, for scientists. Our readership spans scientists, professors, researchers, librarians, and students, as well as business professionals. We share our knowledge and peer-reveiwed research papers with libraries, scientific and engineering societies, and also work with corporate R&D departments and government entities.

More about us