Optical Touch Screen and Its Application as a Next Generation Classroom Response System

applications of the to for Virtual Reality is clearly interdisciplinary research. It has, not only Information Technology importance but social, educational, economical importance too. It combines multiple disciplines for the development of virtual reality systems in which the user has the immersive feeling of being in the real world. Virtual reality has several applications in almost all fields of real life. The most typical fields for the application of virtual reality are health-care, engineering and game industry. This book may be a solid basis for the novice and advanced engineers who would like to develop user friendly Virtual Environments for education, rehabilitation and other applications of Virtual Reality. Our book provides a resource for wide variety of people including academicians, designers, developers, educators, engineers, practitioners, researchers, and graduate students.


Introduction
The way we interact with computers strongly influences how we use the technology. The traditional input methods such as using a keyboard or a computer mouse (Forlines et al, 2007) are still in dominance. Meanwhile, new methods have been explored and developed to provide more diversified or more intuitive user experiences. These include touch screen (Albinsson & Zhai, 2003), voice recognition (Krishnaraj et al, 2010) and brain wave detection (Li et al, 2010;Kaul, 2008). Inspired by the market success of Palm Pilot, and more recently Apple iPhone and iPad, touch screen based devices are becoming increasingly popular (Ostashewski & Reid, 2010) and are expanding their applications to many traditionally noncomputing intensive fields, such as health care (Astell et al, 2010;Clark et al, 2009), driving (Lenné et al, 2011) and education (Willis & Miertschin, 2004;Zurn & Frolik, 2004).
A traditional touch screen covers the entire displaying surface with a matrix of resistors or capacitors (Ritchie & Turner, 1975;IEEE Software, 1991). Special circuits are used to capture the changes of the resistances or capacities of the matrix due to the user's touch. The changes are then converted to cursor positions accordingly. The resistive or capacitive technology requires the use of special materials such as indium tin oxide to be both transparent and conductive. However, the supplies of such materials are dwindling fast and the costs are increasing dramatically. Meanwhile, the amount of materials used on such touch screens is almost proportional to their sizes. In compact equipment such as PDAs and cell phones, or in special applications such as public information kiosks, costs of such touch screens can be justified. In regular office or household use, price is often a hurdle that prevents them from being widely adopted.
Different approaches of touch screen technology are developed to overcome the difficulties associated with the resistive or capacitive technologies. For example, an ultrasonic method was proposed to take advantage of surface acoustic waves (Katsuki et al, 2003). Meanwhile, optical based touch screen technology gained renewed interest. It was introduced early but did not gain momentum due to the cost of the digital cameras decades ago. More recently, the prices of single chip digital cameras have dropped significantly. At the level around $1 a piece, it becomes feasible to take cameras as the building blocks of today's input technology.
Several algorithms have been introduced within the optical input technology. Some examples include using frustrated total internal reflection (Han, 2005) or using two cameras www.intechopen.com Virtual Reality and Environments 118 to transform an acrylic plastic to a touch screen (Wilson, 2004). However, these two methods require the cameras to be placed on the opposite side of the screen from the user. They inevitably increased the thickness of the display and are not suitable for household, office, or classroom use.
Another popular approach is to put the camera at the user's side and use it to track a laser pointer or a fingertip (Cheng & Takatsuka, 2006). Meanwhile, a popular do-it-yourself project hacking the infrared camera of a WII remote to track the motion of an infrared pointer (Lee, 2007) have won applauses in the Internet. Nevertheless, the view of the user side cameras can be easily blocked by the body of the users and needs calibrations whenever the display is moved. Except for the fixed large projector screens, these methods are difficult to be adopted on regular desktops or laptops.
Meanwhile, stereovision has long been used in robot navigation to control manipulation or maneuver (Hager, 1995). A virtual touch screen using stereovision and see-through head mounted display was also proposed (Koh et al, 2008). However, the set up of the approaches is not suitable for daily application and the image processing is complicate due to the unstructured and noisy background.
In this article, we will describe the concept of using stereo-or mono-vision from the corners of the displays and track the motions of the pointer from sides. As we will explain later, this approach is simple to implement, inexpensive to equip, and not sensitive to the wear and tear of the screen. Comparing to existing touch screen technologies, it is also easy to scale up or down for different sizes or height/width ratios. Further, we can even generate virtual forces in the active input space and provide more vivid and intuitive user experiences. What's more, one of the most outstanding capabilities of the optical method is that it will superimpose but not obstruct the existing surface. This makes it ideal to be used on nontraditional displaying applications, such as a whiteboard, desktop, or even a regular writing pad, which is a staple stationary in a student's backpack.
Further, we will briefly introduce our application of touch screen in the next generation Classroom Response System (CRS) (Langman & Fies, 2010;Suchman et al, 2006). It takes advantage of the superimposing capability of the optical touch screen on a regular writing surface and can obtain instantaneous feedback from the students beyond multi-choice questions provided by traditional Clickers (Nicol et al, 2003;Siau et al, 2006). That is, the students can write or sketch their answers using touch screen devices or touch screen modified writing pads employing the technology described in this chapter.
The structure of this article is as follows. After this introduction, we will provide two approaches of optical touch screens, one based on stereovision and the other one on pseudostereovision. Then we will introduce the idea of virtual force to be used in touch screen input and further the superimposed optical touch screen as a next generation CRS. After that, a quick conclusion and discussion will be followed in the final section.

Pointer Locator Using Stereovision (PLUS)
The first metheod is to use two digital cameras simultaneously. It is called Pointer Locator Using Sterovision (PLUS) system.

www.intechopen.com
Optical Touch Screen and Its Application as a Next Generation Classroom Response System 119

System set up
In a PLUS system, each of the two cameras comes with a viewing angle that is equal to or greater than 90 degrees. As shown in Figure 1, a supporting structure encloses the flat screen display. At two adjacent corners, two cameras are mounted just above the surface of the display. They are positioned toward the center of the display such that the overlapped viewing field covers the entire screen surface. For convenience of illustration, we assume the cameras are located at the lower left and lower right corners as depicted in Figure 2. In the real-life applications as we will show later in this artical, the cameras are generally put on the top two corners to prevent occlusions by hand and to keep the camera lenses from collecting dust. For convenience of analysis we first build a coordinate system. As seen in Figure 2, the origin of the system is at the focal point of the lower left camera. Its X and Y axes are parallel to the horizontal and vertical edges of the screen respectively. At this section, we only analyze the pointer positions. That is, we will only study the planar (X-Y) movement of the pointer projected at the surface of the display. Therefore, we assume a linear camera method www.intechopen.com Virtual Reality and Environments at this point and ignore the Z-coordinate. The focal point of the camera is offset by -s 1x and -s 1y at horizontal and vertical directions respectively from the lower left corner of the screen. Then the position of the corner is s 1 =[s 1x , s 1y ] T . The lower right corner is s 2 =[s 2x , s 2y ] T =[L H +s 1x , s 1y ] T where L H is the width of the screen. Likewise, the upper right corner is s 3 =[s 3x , s 3y ] T =[L H +s 1x , L V +s 1y ] T where L V is the height of the screen.

Pointer location using stereovision
Assuming the optical axis of the left camera is rotated from X-axis by angle θ 1 , we can obtain the rotation matrix of the camera from screen-based coordinate system to camera-based coordinate system as Please note that we implied the use of homogeneous coordinates in Eq. 2. We will change back and forth between [x, y] T and [x, y, 1] T in calculations and in expressions whenever necessary. Therefore any point P=[P x , P y ] T in the screen can be transformed to at camera 2. We assume a pinpoint camera model with the focal length as 1 .
For camera 1, the projection of the point P on the image plane can be easily measured from the image as N 1 =n 1 *c px1 where n 1 is the pixel position at the image plane and c px1 is the size of each pixel for camera 1. Hence, we can easily obtain the following relationship using similar triangles:

  
The corresponding cursor position of the pointer is obtained as P s =[P sx , P sy ] T =[(P x -s 1x )/s px , (P y -s 1y )/s py ] T where s px and s py are pixel sizes of the display unit at both X and Y directions. Please note that the values of s 1x , s 1y , d 2x , d 2y can be obtained from initial calibration. However, they will not change even if the display is moved since the cameras and the screen are both fixed to the same supporting structure.

Pseudo-stereovision
In normal stereovision, the surroundings of the interested features is often full of image noises. However, the scenes observed by the point-detecting cameras are simple and constructed. Therefore, we can use one camera instead of two to achieve the same result. www.intechopen.com Virtual Reality and Environments 122 As seen in Figure 3, the camera on the right is removed. Instead, a mirror is put on the edge of the screen. For the convenience of illustration, we cut the width of the screen to half of that is shown in Figure 2. The mirror is configured to be parallel to the left edge of the screen, or the Y-axis of the screen-based coordinate system. Therefore we can have a mirror image of the camera (virtual camera) with the exact same physical characters as the real one. By combining the physical camera and its reflection (the virtual camera) together, we can obtain a pseudo-stereovision.
From the set up shown in Figure 3, it is obvious that The focal point or origin of the virtual camera frame is at [d 2x , d 2y ] T =[2L H +2s 1x , 0] T . Meanwhile, as seen from Figure 3, the mirror image of the pointer P=[P x , P y ] T is P 1= [P 1x , P 1y ] T . Its projection to the image plane of the physical camera is N 2 . As depicted from Figure  3, the mirror image of the projection of P 1 on the physical camera is the projection of the point P on the virtual camera. It is clear that N' 2 =-N 2 .
If we substitute the above values to Equation 5, we can easily find the position of pointer P=[P x , P y ] T and its corresponding cursor position P s =[P sx , P sy ] T .

Monovision with mirror image
Alternatively, we can use a different approach. That is, instead of using the virtual camera and convert back to the case of stereovision, we can process the mirror image of the pointer at the physical camera directly.
where is the focal length of the physical camera, θ is the angle of the camera, and N 1 and N 2 are the projected positions of P and P 1 on the image plane.
Solve for P x and P y , we have Note that the result is simpler compare to the true stereovision method due to the equality of focal length and rotation angle θ between the physical and virtual cameras. Meanwhile, we need to cover the combined surface of both the physical screen and the reflected one, so we can calibrate the angle θ to be π/4. In this case, the result can be further simplified as 12 2 12

Virtual boundary
Sometime, the pointer is away from the vicinity of the screen but is still close enough to be picked up by the cameras. This can easily happen when a user is pondering for the next action or is talking in front of the screen. As shown in Figure 5A, in true stereovision, a virtual boundary can be set up in the algorithm to filter out the unwanted movements. Any point P within the virtual boundary is considered legitimate and is processed accordingly. Its motion will be reflected as the movement of the cursor on the screen. If the pointer is outside the virtual boundary, like the position P' in Figure 5A, it can be considered idle and an update is not necessary. However, in the background, the video stream is constantly processed to extract the positions of P at both cameras. The distance of the pointer from the screen surface needed to be constantly calculated and compared with the virtual boundary.
On the other hand, with the method of monovision using mirror, we can set up a virtual boundary by combining the height of the mirror and the viewing angle of the camera. As shown in Figure 5B, when the pointer is above the virtual boundary, the image of the pointer P' 1 will not be picked up by the camera even the pointer P' itself is within the range. That is, only one image feature will be obtained instead of two. A quick on-off logic can be easily used to dismiss this single point without calculating the position and distance first. Since the duration of the pointer in use is often far less than when it is not, demand on computing power will be greatly reduced.

Action space and virtual force
In previous methods, only the X-direction projection (parallel to the screen) on the camera image plane is used. It only provides information on the position of the pointer. Move one step forward, we can take advantage of the Z-direction projection (perpendicular to the screen) of the image and the time derivatives of the projections for more advanced features. One such application is to set up an action space immediately outside the touch screen. In the action space, we can construct virtual forces and use them to achieve the results such as push, pull, stroke, or other similar motions.
First, let us expand the aforementioned model to a 3-dimensional space with the Z-axis perpendicular to the display screen and pointing away from it. This 3D action space is enclosed by the display screen on bottom and the virtual boundary on top. As seen in Figure  6, the pointer can freely move in this space. The projection of the pointer gives a planar trajectory on the camera image plane.
where K s and K p are to be obtained from experiment and to be further adapted using self learning algorithms, a x and a y are accelerations of pointer at x and y directions, while a z is its acceleration at the z direction. The variables a x , a y and a z can all be calculated through second order finite differentiation from P x , P y , and P z .
We should point out that a virtual force can be considered as a function of any subset of the position variables and their derivatives. For example, a simple position based force feedback can be obtained as while a more sophiscated one may look like where K pi , P i , a i , K vi , V i , and b i should be determined by the specific program needs.

www.intechopen.com
Virtual Comparing with the force sensing of traditional touch screens, the virtual force can give a much greater force range. The depth of the action space can be easily adjusted by the virtual boundary. By using cameras with wide viewing angles, the action space can be large enough to yield many interesting 3D motions, which can be important for active computer games or on screen calligraphy or painting. Currently, the pull, push, or similar motions are absent in many applications. If used, they can greatly improve the user experiences.
Further, the users do not even need to physically touch the screen. As long as the pointer or finger is in the action space, its movement will be monitored and reflected on the movement of the corresponding cursor or object in the program.

Touch screen in the next generation Classroom Response System (CRS)
As we explained earlier, optical touch screen described in this article is simple to implement, inexpensive to equip, and not sensitive to the wear and tear due to the pointers scratching on the touch screen. A commercial version of this device can be used to superimpose but not obstruct the existing writing surface. This makes it ideal to be used on non-traditional displaying applications, such as a whiteboard, desktop, or even a regular writing pad. Here we will briefly introduce our application of touch screen in the next generation Classroom Response System (CRS).

Need of the next generation CRS
Actively engaging students in classrooms is always a challenge to the teachers (Smith, 1996;Zhang, 1993). People have tried various ways to improve the interactions even from the early days of chalk and board (Bransford et al, 2000). Choosing a volunteer from raised hands and picking a name from the roll book of the class are two most common traditional methods. However, these methods only give teachers feedback from a small fraction of the class. The sample is often not typical or representative. Many students will not volunteer for fear of public mistake and embarassment. As the result, a small vocal minority will skew the view of the teacher of how the entire class understand the topic (Caldwell, 2007). To increase the sample size, people also tried methods such as shouting the answers, applause, or response cards (Karen et al, 2001), all with their pros and cons. More recently, computer based technologies are introduced to classrooms. They enabled teachers to explore more options (Patten, 2006;Richard et al, 2007;Roschelle, 2003).
The first technology used is computer-based visuals such as Powerpoint (Anderson, 2004;Bannan-Ritland, 2002;Liang et al, 2005). It has become ubiquitous today, but with mixed results. The slide shows help to provide strong graphical impact as "one picture worth a thousand words". However, the interactions and feedbacks from the students are still greatly limited. Sometime, the situation is even worse when the light is dimmed to improve the visual contrast.
Recently, a small gadget called Clicker, or the first generation CRS, is gaining momentum among teachers throughout the K-12 and higher education, and to anywhere with a group of audience (Nicol et al, 2003;Siau et al, 2006). A clicker is a remote-like device that comes with a number of buttons and a wireless transmitter. After the instructor giving out a multichoice or true/false question, the students punch the buttons on their Clickers to select their www.intechopen.com Optical Touch Screen and Its Application as a Next Generation Classroom Response System 127 answers. The answers then are instantly transmitted to the host computer operated by the teacher. A program will collect and summarize the answers and present the aggregated result to the teacher using a chart or graphic.
Both teachers and students welcomed the use of CRS. The engagement is bi-directional. The teacher can continuously assess the level of student understanding or mis-understanding. They can then give targeted clarifications or adjust the course material or pace to address the problems arisen. At the same time, the students can quickly apply the knowledge they just learned and answer the questions without raising hands, which can be intimidating to many. Since the answers are transmitted anonymously, the students will be free from the fear of peer pressure and public embarrassment.
However, still more features are desired for the Clickers. One major drawback of the current Clickers is its dependence on multi-choice or true/false questions (Siau et al, 2006). The teachers make up generally three or four possible wrong answers and mix them with the right one. As a legacy from the early computer-based standardized test, it is easy to grade. However, multi-choice or true/false questions are not the best or the only assessing tools. The number of choices is limited; the result can be skewed by test-taking techniques; and the teachers cannot directly know what the mistakes the students make from the answers.
Meanwhile, since the students can only choose from a limited list of options, their creativities are often ignored or discouraged. In any subject, there are always multiple solutions to a question, especially an open-ended question. In Science, Technology, Engineering, Art & Mathematics (STEAM) and many other fields, we often encourage students to focus on problem-solving progress instead of the final answer. Unfortunately, the multi-choice questions provide no room for the students to derive their own answers. It would be great if we can have a better tool that will encourage open-ended questions and promote problem-solving skills.
One attempt of improvement over clicker is to use touch sensitive tablets (Berque et al, 2004) or touch screen tablets. A touch sensitive tablet digitizes the strokes of the stylus on its surface and send the data stream to the host computer. However, due to the seperation of writing and displaying, the use of this kind device is unnatural and did not gain wide acceptance in the classrooms.
The touch screen pads include Tablet PCs (Koile & Singer, 2006;Willis & Miertschin, 2004;Zurn & Frolik, 2004) or iPad and its competitors (we denote them as xPads). A user can write on the touch screen of the tablet with a stylus or a finger. Then the hardware and software will convert the writing to either the text or graphics, and then display them back on the screen. Once networked together, tablets can be a great tool for a teacher to get instant feedback from the students. The feedback will not be limited to multiple choices, but on every part of the knowledge assessment that can be conducted with a regular paper test.
However, there are still several major drawbacks of these touch screen tablets. First is its cost. For example, the price of a tablet PC starts from $1k and costs much more with rugged design. xPads are generally more affordable, but still costs hundred dollars each. The prices can easily rule out most classroom use, especially when they are under the stress of young students at their active years. Second is the different platforms of the tablet devices in the market. It is able to recommend but difficult to require all the students using the same operating system in one classroom unless they are provided for free. Third is the distraction of the other software, especially the communication and entertainment software installed in the tablets. When a student is having an electronic device in hand, it is not easy to resist the temptation of returning a quick text message or playing a small game while the teacher is not around.

Set up of a CRS using optical touch screen devices
With the above needs in mind, we applied the optical touch screen to the development of the next generation CRS, or Z-Writers. The major part of the Z-Writer is a clipboard style writing surface. Students can put a regular notepad or a stack of test paper on it. They can write on the paper just like what they are doing in the regular test. The difference of the Z-Writer from the regular clipboard is that a sensor bar is installed at the top of the pad. The sensor bar can take either the PLUS or PLUM configuration as shwon in Figure 7A or Figure 7B. We can also network Z-Writers in a CRS setting. Each Bluetooth master can only handle up to 7 devices with today's technology. In Linux set up, we can simply add extra Bluetooth dongles to increase the number of devices the computer can handle. For Windows or Mac OS, the operating systems can not handle more than one Bluetooth master. Therefore, we developed a special Scatternet (Mockel et al, 2007) to expand the capacity of the Z-Writers.

www.intechopen.com
As seen in Figure 8, we divide all the Z-Writers in a classroom to groups of 6. One of them equips a sub-server with an enhanced antenna. It acts as both a receiver and a relay. It collects the data from the group and relay them to the Bluetooth master in the host computer. In this network, each computer can handle up to 7 groups or 42 Z-Writers. That is enough for most classes and more than the maximum capacity that a teacher can handle at the individual level without a teaching assistant. For the larger classes with the help of teaching assistants, a local network of multiple computers or an outright change to using 802.11 can be employed. However, these are beyond the topic of this discussion. The host computer on the teacher's side collects all the data from the sub-servers and save them into a database. A preview program converts the data into pixel points and display them in a preview window like the left part of Figure 9. With a quick glance, the teachers can have an instant view of how the students are performing. When the teachers need to annotate an individual student's work, they can select and magnify the one of interest and display it as seen in the right half of Figure 9 for in-class review and analysis.

www.intechopen.com
Virtual Reality and Environments 130 Figure 10 is a sample course material common to Physics or Engineering majors. One main topic of such majors is load analysis with Free Body Diagrams (FBD). Figure 10 gives a typical example of the force analysis. On left is a block resting on a slanted surface. As shown on right, students need to set up a proper coordinate system, draw the forces, and then write down the governing equations on each direction to find out the magnitudes of the forces.
Before the class, the teacher prepares questions like the one shown in the left part of Figure  10 and makes copies on regular paper. In the class time, the teacher first explains the concepts of FBD and provides examples. Then the handouts with the questions will distributed to the class. The students put the handouts on the writing surfaces of the Z-Writers and then write their answers on the paper just like any other classes. The Z-Writers transfer the results to the host computer for the teacher to review in real time. The teacher is then able to adjust the next step accordingly, such as revisiting the subject, moving on to the next topic, or giving annotations to selected answers. At the end of the class, the students will carry their own paper home while the teacher keeps a soft copy of all answers for future reference.

Conclusion
In this article, we introduced a novel method of using stereovision (PLUS) and monovision (PLUM) to locate the position of a pointer on a display unit.

Benefits of the optical touch screen with virtual force
There are many advantages of the algorithms proposed in this article.
1. Low cost. It is relatively easy and inexpensive to implement. Thanks to the ubiquitous camera phones and webcams, low-cost high-resolution single-chip digital cameras are widely available today. 2. Durable. There is no physical contact between the camera and the pointer. Therefore, the functionality of the screen will not deteriorate due to wear and tear. Even when the screen is scratched or broken, the touch function will still work. 3. Easy to scale up. The complexity and cost of existing touch screens increase quickly when the screens become larger. For example, when the size of a screen doubles, its θ g www.intechopen.com Optical Touch Screen and Its Application as a Next Generation Classroom Response System 131 viewable area quadruples, so is the material used and the computing power needed. In contrast, in the methods introduced in this article, the same two cameras or camera and mirror combination can be easily scaled up or down to screens with different sizes and height-width ratios. 4. 3D input. As mentioned earlier in this article, the action space and virtual force can be used to achieve the effect of 3D input and to obtain virtual force feedback. 5. Multi-touch. Existing computer software can easily detect and follow multiple image features. It will be easy to include multi-touch functions. Combined with virtual force, the multi-touch function can make inputting more versatile and powerful. 6. Compact and light weight. Comparing to the existing optical touch screen using behind screen or user side camera, this side camera method is compact and can save precious space. The cameras are generally small and will not increase the weight or size of the computer display like many existing touch screens.

Benefit of the next generation CRS using touch screen writers
1. Real-time feedback. All educators are facing the perpetual challenge of assessing how well the students understand each course, each example, or even each sentence. Homework, quizzes and exams evaluate each individuals but with latency. Group discussion, flash cards, and the original Classroom Response Systems (Clickers) provide instant but collective feedbacks. The Z-Writer Group combines the benefits of the two by providing instant individual feedbacks to the teachers. 2. Progress monitor. From one computer display, teachers can monitor not only the answers but how the students proceed from start to finish at both collective and individual levels. 3. After-class evaluation. Teachers can replay the writings of every answer after the class. Therefore, they can analyze each student and address their specific needs with higher accuracy. 4. Open-ended questions. Many in-class assessment tools rely on multi-choice questions, which are not enough for STEAM subjects. The Z-Writer Group will enable teachers to give open-ended questions in classes and obtain real-time feedbacks. This will greatly encourage creativities from the students.

Future work
Currently we are working on improving the stability and noise reduction on the optical touch screen algorithm and its use on classroom response system described in this paper. We are also working on expanding the applications of the optical touch screen to art and calligraphy field for paperless painting with realistic strokes. We are also exploring the use of 802.11 for the auditorium size CRS settings.