The classroom specification.
There are two main problems in the present style of education in which there are one teacher and many students in a class. The first problem is to improve communication between one teacher and many students in classes. The second problem is to realize personal education using IT systems. To solve the above two problems, we propose color image processing applications for education in this chapter. To improve communication between one teacher and many students, we realized automatic response analyzer that counts automatically students’ answers using color cards raised by students. The system is an easy-to-use simple system, which only consists of a PC, web cameras, and color cards. The recognition rate was 98% in brighter class and 93% in darker class. To realize automatic personal education, we realized two interactive IT systems using color image recognition. First, we realized color learning system with color analysis functions. Using the system, students can learn color distribution in 3D color space for their selected images. Second, we realized visual programming system that judges automatically the correctness of the graphical image output by student program. If the student’s output image is not correct, the system can point out the wrong part in the output image.
- color image processing
- interactive communication
- interactive learning
- automatic response analyzer
- color learning
- programming learning
Education is the fundamental and important activity to maintain the society. Ideal style of education is personal one teacher to one student education. However, main present style of education is one teacher to many students education in classes. Therefore, there are two problems in education.
First problem is to improve communication between one teacher and many students in classes. In the large classroom, one teacher needs to progress the lesson systematically for many students. In addition, it is necessary to grasp the degree of comprehension of the lesson for all students. For example, the most common way to grasp the status of many students is to ask questions or comments by raising hands. However, there are few students who raised hands and actively speak in the large classrooms. There is a need for a way to grasp the state of each student. To improve communication between one teacher and many students in classes, there are some conventional communication systems, which consist of a teacher’s terminal and students’ terminals. A teacher asks students using a teacher’s terminal, and students answer to the teacher using students’ terminals. However, conventional communication systems containing the student IT terminals disturb face-to-face communication, which is necessary for education in classes, and the systems are expensive in both initial cost and running cost.
Second problem is to realize personal education using IT systems. To solve the second problem, there are conventional e-learning systems. However, in conventional e-learning systems, automatic judging true or false questions are multiple-choice questions or perfect text-matching questions. Conventional system can treat only simple knowledge problems of which answer styles are multiple choice or perfect text matching, for example, not including free-style programming input, which is necessary for students to learn how to write their original programs using programming languages. Therefore, using conventional e-learning systems, students can cultivate basic abilities, but students cannot cultivate application abilities.
2. The research methods
The function to recognize each student status exists in personal one teacher to one student education, but the function doesn’t in conventional systems. We decided to realize the function to recognize each student status in the IT system, because the students can learn as well as in personal one teacher to one student education using the IT system with the function in one teacher to many students education in classes or with no teacher. The representative function of recognizing each student status is the visual recognition function. Color images are important for communication and understanding in education area. To solve the above two conventional problems, we proposed the visual recognition function in IT system using color image recognition technologies. We proposed and realized the color image processing applications for education to solve conventional problems, and we verified the applications by experiments.
We describe the color image processing applications for education in Sections 3–5. With respect to the first problem to improve communication between one teacher and many students in classes, we describe the automatic response analyzer in classroom using image processing and cards in Section 3. With respect to second problem to realize personal education using IT systems, we focus on color learning area and graphics programming learning area, in which image processing is useful. We describe the color learning system with color analysis function in Section 4, and we describe the visual interactive programming learning system using image processing in Section 5. The system can judge students’ programs output graphics answers right or wrong automatically.
3. The automatic response analyzer in classroom using image processing and cards
There are some conventional communication systems that consist of a teacher’s terminal and students’ terminals. As students’ terminals, the dedicated miniature terminals [1–3], the dedicated digital pens , the student’s mobile phones , and the student PC terminals [6, 7] are used. Although dedicated miniature terminals and dedicated digital pens have the advantage of not restricting the work space of students, management costs arise due to the risk of loss. For students’ mobile phones, it is necessary for the students to pay communication packet cost. In a system using a PC terminal for students, it is necessary to make an investment to convert the whole classroom into IT classroom. Conventional communication systems containing the student IT terminals are expensive in both initial cost and running cost, and the systems disturb face-to-face communication, which is necessary for education in classes. Also, there was an attempt to grasp the status of many students by using qCards: a sheet of paper that contains a printed code, similar to a QR code, encoding their student IDs . However, qCards are big. So it is inconvenient to carry, and we cannot do it easily.
3.1. The system architecture
We developed a simple system consisting of a PC and web cameras . It is based on the method  of distributing multiple colored cards to students and letting the teacher to question the color card corresponding to the answer. The PC automatically compiles the card corresponding to the answer that the student cited. Figure 1 shows the configuration of the system. We have mounted web cameras on the tripod and developed the connecting clasp. Since the two web cameras on the tripod are adjacent and high, there is no uncaptured area in front of area between two web cameras, and rear students’ cards can be captured by the web cameras and students can be asked to raise color cards for questions from teachers. We show the color card’s layout with A5 size in Figure 2 . The color cards are made of two irreflexive papers that are fluorescent colored paper and black paper.
The process of the system consists of image acquisition and image processing. The image processing is divided into five stages. First stage is that we change the input image into the grayscale image. Second stage is that we change it into the binarized image. Third stage is that we extracted the area of the color portion from the binarized image. Forth stage is that we performed rectangle determination from the extracted area. Final stage is that we performed color determination after the rectangle determination. According to this system, compared to the conventional system, there is an advantage that the introduction cost is low, the maintenance is easy, and the usage method is simple.
3.2. Evaluation of the system
We experimented using eight candidate fluorescent colored papers shown in Figure 3 . We show the specification of the classroom used for the experiment in Table 1 . There are two lighting conditions. They are the bright lighting condition and the dark lighting condition. The bright lighting condition is that only the teaching side lights are turned off. The dark lighting condition is that the front side lights were also turned off.
|Width (m)||Depth (m)||Number of desks in width direction||Number of desks in depth direction||Number of seats in a desk|
At the first, we experimented using cards put on the desks and two web cameras. We show experimental results in Table 2 .
|Card name||Hue||Bright lightning||Dark lightning|
|Illumina color #80 yellow orange||46||100%||100%|
|Illumina color #80 blue||197||100%||88%|
|Illumina color #80 peony||325||100%||89%|
We selected five color cards shown in Figure 4 using Table 2 . The layout of the classroom and classroom areas captured using three web cameras is shown in Figure 5 . Image input device using three web cameras is shown in Figure 6 . Card recognition rate depending number of web cameras is shown in Table 3 . Questionnaire about usefulness of the system is shown in Table 4 .
|Web camera||Bright lightning||Dark lightning|
|Two web cameras||97%||99%||81%||92%|
|Three web cameras||98%||99%||93%||92%|
|Be useful||Neither||Not useful|
4. The color learning system with color analysis function
Since colors are used in various scenes of everyday life, effective learnings about colors are required. Conventional color learning uses analog methods such as books, color samples, and color charts, and efficient learning has not been done [11–13]. Therefore, it is necessary to develop a system that can efficiently learn colors on computers.
4.1. The system architecture
We developed a color learning system with color analysis function that enables efficient color learning digitally in previous research [14–20]. As shown in Figure 7 , the system can be classified into three systems: a color learning system, an illusion simulation system, and a color analysis function.
The learning items of the color learning system are shown in Table 5 . Among them, the most characteristic learning item is (9) color space (three dimensions). In the learning item (9), hue, saturation, and lightness can be three dimensionally simulated. The screen of the color space is shown in Figure 8 . The students can move the color space three dimensionally with the mouse. By pressing the button on the left side of the screen, it is possible to switch display and nondisplay of hue, saturation, and brightness. Through this simulation, it is possible to visually understand the relationship between hue, saturation, and lightness.
|(1)||Color information by color chart|
|(4)||PCCS tone diagram|
|(6)||PCCS hue circle|
|(9)||Color space (three dimensions)|
The illusion simulation system is a system that simulates an illusion phenomenon by freely changing parameters such as color and motion . The system can simulate 15 kinds of illusion phenomena from five categories of geometrical illusion, light illusion, illusion of color, illusion of shape, and motion illusion.
An example of a simulation screen of Ebbinghaus illusion is shown in Figure 9 . The size, the number, and the position of the white ball around the yellow size ball having a constant size are displayed in two ways with different left and right. Ebbinghaus illusion is the illusion that a ball surrounded by a large sphere is small, and a ball surrounded by a small sphere looks big. In Figure 9 , the ball on the left is larger than the ball on the right.
The color learning system including illusion simulation is capable of interactive learning, but it is passive learning in terms of answering the presented problem. In addition, there is no function that can easily check and reproduce any color you enter. Therefore, a learning system that makes active learning by visually converting input arbitrary color information, confirms and reproduces color information by learners, and leads to deep understanding of colors is required.
We developed a color analysis function as a system that visualizes color information and enables active learning. The color analysis function is a function that can visually understand the color information of the color by visualizing the hue, saturation, and brightness of the pixel three dimensionally. When you click the mouse on the displayed image, you get the hue, saturation, and brightness of the pixel at the clicked position. Then, a sphere is displayed in the coordinates corresponding to the hue, saturation, and brightness acquired in the 3D model of Figure 10 , where the circumference is hue, the x axis and y axis are saturation, and the z axis is lightness. Every time the mouse is clicked, a sphere is added, and multiple spheres are displayed. Also, at the bottom of the displayed image, the hue, saturation, and brightness of the pixel selected with the mouse are displayed.
An application example of the color analysis function is shown in Figure 11 . Figure 11(a) show a test image. The middle oblong rectangle is the same gray from the left end to the right end. On the other hand, the background continuously changes from black at the left end to white at the right end. The mouse pointer is placed almost at the left end of the middle oblong rectangle. The HSV value of the pixel at this position is displayed at the bottom. In Figure 11(b) , the HSV value of the pixel is shown on the color analysis function 3D model. Even if you move the mouse pointer rightward in a horizontally long rectangle, the points on the 3D model do not move. This confirms that the middle oblong rectangle is a single color. For human eyes, the horizontally elongated rectangle in the test image appears bright at the left end and dark at the right end, but it can be realized that this is an effect of light contrast against the background.
4.2. Evaluation of the system
The subjects were given a color learning system and an illusion simulation system, the purpose and usage of the color analysis function and a questionnaire were carried out. In the questionnaire, the comprehensive evaluation of each system was made to answer by a five grade evaluation for each question item. The questionnaire result is shown in Figure 12 .
5. The visual interactive programming learning system using image processing
Nowadays, since many software must be made constantly to maintain IT society, we must bring up many programmers constantly. Therefore, there are many persons learning programming. However, there are fewer persons to master programming skills, because it is difficult for persons to keep high interest and high motivation to write complicate and long programs by themselves. Most of conventional learning programming processes are the processes of learning program language grammar, which are necessary but are not efficient for students to acquire the skill to write complicate and long programs. There are two different types of conventional programming educations reports [21, 22]. In the first report , “a self-study supporting system which is an automatic fill-in-the-blank problem generator” is proposed. However, the system is not suitable for educations to write complicate and long programs. In the second report , “programming experiments” is proposed. In the education, game or other interesting program functions are implemented by students. However, the students are educated only in the classes at school. There are no conventional programming educations or learning systems for students to keep high interest and motivation to write complicate and long programs by themselves.
5.1. The system architecture
To solve the program, we propose the visual interactive programming learning system using image processing. The system requires the students to write the programs that output the indicated figures, and the system compares the correct figures with the student’s program output figures using image processing, and the system judges that students’ programs are correct or not. The system concept is shown in Figure 13 . First, question is displayed to the student to complete the program that outputs the indicated figure. Second, the student inputs the program to complete the whole program. Third, the system outputs the image file of the figure that the student program displays, and the system outputs the image file of the figure that the correct program displays. Fourth, the system compares the two images using image processing, and the system judges that students’ programs are correct or not. Furthermore, the system may point out the different parts of two images. The system may be expected to be valid for students to keep high interest and motivation to write complicate and long programs by themselves.
The programs may display the figures by multiple steps. The programs may display moving images, too. Correspondingly, the system compares the multiple pair of images or a pair of moving images using image processing, and the system judges that students’ programs are correct or not. We will describe the three system functions.
Most simply, the system compares a pair of still images that student’s program and the correct program display, and the system judges that students’ programs are correct or not. The judgment using a pair of still images is shown in Figure 14 . To answer the question, the student inputs the student’s program into the input field at the input form. In Figure 14 , there is the mistake about a circle position. The different parts between the two images that student’s program and the correct program output are displayed in red. Example of a judgment using a pair of still images is shown in Figure 15 . In this case, the correct image is “×”, but the student’s image is “+”. In each images, the different part is displayed in red.
The system may find the wrong parts of student’s programs using plural pairs of output images. The judgment process using plural pair of images is shown in Figure 16 . There are plural input fields in the input form. The students input programs in the plural input fields. The completed program outputs the displayed images at the end of each input field. The system compares the student’s program output image and the correct program output image at the end of each input field. Example of a judgment using plural pair of images is shown in Figure 17 . In these examples, two images output at the end of the first input field are same, but two images output at the end of the second input field are different. Therefore, the student’s program in second input field is wrong. The different parts in the two images are displayed in red.
Furthermore, the system can judge moving output images. The judgment using a pair of moving images is shown in Figure 18 . The input field in the input form is in the program loop, and at the end of the each program loop, the displayed image is output. The system compares the student’s program output image and the correct program output image at the end of each loop. Example of a judgment using a pair of moving images is shown in Figure 19 . Different parts of two images are displayed in red.
We implemented the function describing in this Section 5.1, and we developed the programming problems of which are about drawing point, line, color, figure, and animation, etc.
5.2. Evaluation of the system
We developed the system, and we evaluated the system using questionnaires. We requested the examinees to answer questionnaires that contained the following three questions.
Q1. Is it easy to use the system?
Q2. Do you feel that you can learn programming using the system?
Q3. Can you keep motivation for learning programming by the system?
There are five kinds of answers, “strongly agree” with five points, “agree a little” with four points, “neither agree nor disagree” with three points, “disagree a little” with two points, “strongly disagree” with one point. The result of the first experiment is shown in Table 6 . The examinees are 10 students of Kanazawa Institute of Technology. Average points of the questionnaires are 3.7, 4.1, and 3.8. Based on the questionnaires result, we think that the system concept is useful, but the improvement of the system is necessary for practical uses.
|Evaluation level (points)||Q1||Q2||Q3|
|Strongly agree (5)||2||2||1|
|Agree a little (4)||5||7||6|
|Neither agree nor disagree (3)||1||1||3|
|Disagree a little (2)||2||0||0|
|Strongly disagree (1)||0||0||0|
We proposed the three education applications using color image processing. We verified the applications by experiments. About the automatic response analyzer, we have already used the application in our classes, and we have confirmed that the system helps teachers to manage classes more smoothly. Next study is to develop various functions that help class management to achieve more educational effects.
We have developed prototypes of the color learning system and the programming learning system, and we verified the new functions. However, we have not used the two applications in actual educational classes, because we do not have developed educational courses using the two applications. Next study is to develop educational courses using the two applications.
To improve communication between one teacher and many students, we realized automatic response analyzer that counts automatically students’ answers using color cards raised by students. The system is an easy-to-use simple system, which only consists of a PC, two or three web cameras, and five kinds of color cards. The recognition rate was 98% in brighter class and 93% in darker class. The system made the teacher’s educational management easier. To realize automatic personal education, we realized two interactive IT systems using color image processing. First, we realized color learning system with color analysis functions. Using the system, students can learn color distribution in 3D color space for his or her selected images. Second, we realized visual programming system that judges automatically the correctness of the graphical image output by student program. If the student’s output image is not correct, the system can point out wrong part in the output image. We proposed above the three systems, and we verified the systems by experiments.
This work was supported by JSPS KAKENHI grant number 15 K01041. We thank students and former students who have cooperated with researches in this paper.