Finger Braille Recognition System

1. Introduction

The Ministry of Health, Labour and Welfare of Japan estimates that there are nearly 22,000 deafblind people in Japan (2006). Communication is one of their largest barriers to independent living and participation. Deafblind people use many different communication media, depending on the age of onset of deafness and blindness and the available resources. “Yubi-Tenji” (Finger Braille) is one of the tactual communication media utilized by deafblind individuals (see Fig. 1). In two-handed Finger Braille, the sender’s index finger, middle finger and ring finger of both hands function like the keys of a Braille typewriter. The sender dots Braille code on the fingers of the receiver. The receiver is assumed to recognize the Braille code. In one-handed Finger Braille, the sender dots the left part of Braille code on the distal interphalangeal (DIP) joints of the three fingers of the receiver, and then the sender dots the right part of Braille code on the proximal interphalangeal (PIP) joints. Deafblind people who are skilled in Finger Braille can communicate words and express various emotions because of the prosody (intonation) of Finger Braille (Fukushima, 1997). Because there is such a small number of non-disabled people who are skilled in Finger Braille, deafblind people communicate only through an interpreter. Thus, the participation of deafblind people is greatly restricted.

Various Finger Braille input devices, including a wearable input device, have been developed. (Uehara et al., 2000) developed a Finger Braille glove system with accelerometers mounted on the fingertips. (Fukumoto et al., 1997) developed a wearable input device with accelerometers mounted on the top of rings. (Hoshino et al., 2002) developed a Finger Braille input system that mounted accelerometers on the middle phalanges. In addition, (Ochi et al., 2003) developed a bracelet-type Finger Braille input device with eighteen mounted accelerometers. These devices require deafblind people to wear gloves, rings or bracelets to input Finger Braille. With these support devices, deafblind people are not only burdened with wearing the sensors, but also they must master a new communication system using such support devices.

Figure 1.

Two-handed Finger Braille (left) and one-handed Finger Braille (right)

Based on "Development of Finger Braille Recognition System", by Yasuhiro Matsuda, Ischiro Sakuma, Etsuko Kobayashi, Yasuhiko Jimbo, Tatsuhiko Arafune and Tsuneshi Isomura, which appeared in Journal of Biomechanical Science and Engineering, Vol.5, No.1. © 2010 JSME.

The objective of this study is the development of a Finger Braille support device which employs communication through skin contact, because skin contact is the only non-verbal communication possible for deafblind people. The concept of the proposed Finger Braille support device is shown in Fig. 2. The advantages of this support device are as follows: both deafblind people and non-disabled people who are not skilled in Finger Braille can communicate using conventional Finger Braille, and all sensors are worn by the non-disabled people. This support device consists of a Finger Braille teaching system and a Finger Braille recognition system. The teaching system recognizes the speech of non-disabled people and displays the associated dot pattern of Finger Braille. Non-disabled people can then dot Finger Braille on the fingers of deafblind people by observing the displayed dot pattern (Matsuda et al., 2010a). The recognition system recognizes the dotting of Finger Braille by the deafblind people and synthesizes the speech for non-disabled people. Thus, deaf-blind people can communicate without being encumbered by the support device.

Figure 2.

Concept of Finger Braille support device

To develop the recognition system, some requirements must be fulfilled.

In Finger Braille, the sender dots Braille codes directly on the fingers of the receiver. A rule of Finger Braille is that the sender constantly touches the fingers of the receiver even when not dotting, because the receivers feel uncomfortable in the absence of touching or tactile cues. Therefore, sensors must not hinder the skin contact between deafblind people and non-disabled people. Because of the lack of visual and audio information, deafblind people experience difficulty in mastering a new communication system. Thus, the sensors must be worn by the receiver (non-disabled people). In this study, we adopted small accelerometers mounted on the top of finger rings.

In our concept of assistance, deafblind people are equipped with the recognition system, which non-disabled people can also use. The non-disabled people are unspecified. Thus, the recognition system must be independent of the receiver.

For prosody of Finger Braille, the sender dots long and strong at the end of clauses and sentences. The sender can dot strongly with anger, or weakly with sadness (Matsuda et al., 2010b). Thus, the recognition system must be independent of dotted strength.

To develop the recognition system, we adopted one-handed Finger Braille. Here, the recognition system requires independence of the dotted position and recognition of the dotted positions.

In this chapter we describe the Finger Braille recognition system and present experimental results. We first describe the algorithms for the recognition of dotted fingers and positions. Then, an evaluation experiment was carried out.

2. Development of the recognition system

2.1. Configuration of the recognition system

Fig. 3 shows the configuration of the Finger Braille recognition system. The non-disabled people wore rings with small piezoelectric accelerometers (yamco 10SW, Yamaichi Electronics) on the index finger, middle finger and ring finger. Each accelerometer was mounted on the top of the ring. The accelerometers were connected to a tablet PC (TC1100, HP) through charge amplifiers (yamco 4101, Yamaichi Electronics) and an A/D converter (USB-9215A-BNC, National Instruments). The sampling frequency was 10 kHz, the measurement range was ±250 m/s², and the sensibility was 0.2 m/s². The input voltage of the A/D converter 1V is equal to the acceleration 100 m/s².

First, the accelerometers detected the accelerations of the dotting, and acceleration data were acquired. Second, the recognition system recognized the dotted fingers and positions. Third, by parsing the recognized Braille codes, the recognition system converted the Braille codes to Japanese text. Finally, the recognition system synthesized the speech of the Japanese text.

The operating system (OS) was Microsoft Windows XP. The programs of recognition of the dotted fingers and positions were programmed in LabVIEW 8.0 (National Instruments). The Braille code parser was programmed in Win-Prolog 4.500 (Logic Programming Associates). The integrated program was programmed in Microsoft Visual Basic 6. The speech synthesizer was VoiceText (Pentax). Fig. 4 shows an appearance of communication supported by the recognition system.

Figure 3.

Configuration of the recognition system

Figure 4.

Communication supported by the recognition system

2.2. Detection of the shock accelerations by dotting

The acceleration data of three fingers were measured continuously and the differential of the sum of the accelerations of three fingers were calculated. If the differential was greater than or equal to 10V/s, the differential data for 100 ms (pre-trigger 20 ms and post-trigger 80 ms) were acquired. If the differential data stayed in between the upper and lower limits (see Fig. 5), these accelerations were recognized as the shock accelerations by dotting.

Figure 5.

Upper and lower limits of the differential of the sum of the accelerations of three fingers to detect the shock acceleration by dotting.

2.3. Recognition of dotted fingers

2.3.1. Features of shock acceleration by dotting

Fig. 6 shows features of the accelerations by dotting. When the sender dotted Finger Braille on the fingers of the receiver, the accelerometers detected shock accelerations by the dotting of the mounted finger (self dotting) and shock accelerations by the dotting of the other fingers (cross talk). Fig. 7 shows the shock accelerations by self dotting and cross talk. The accelerations by cross talk were less than the accelerations by self dotting. AI1, AM1 and AR1 indicate the amplitudes of accelerations of the index, middle and ring fingers, respectively. The range of the amplitude of acceleration by self dotting and the range of the amplitude of acceleration by cross talk overlap each other. Thus, it is difficult to recognize the acceleration by self dotting by using a constant threshold for the amplitude (Uehara et al., 2000; Matsuda et al., 2010c).

Figure 6.

Features of the accelerations by dotting

Figure 7.

Shock accelerations by self dotting and cross talk (dotting on DIP joints: hard impact)

Fig. 8 shows the frequency spectrums of the accelerations by self dotting and cross talk. The acceleration data for 100 ms (pre-trigger 20 ms and post-trigger 80 ms) were recorded. The window function was the Hanning window. The difference of power between self dotting and cross talk was greater at approximately 100 Hz (Fukumoto et al., 1997; Hoshino et al., 2002). PI, PM and PR indicate the powers at 100 Hz of the index, middle and ring fingers, respectively. The range of the power at 100 Hz by self dotting and the range of the power at 100 Hz by cross talk overlap each other. Thus, it is also difficult to recognize the acceleration by self dotting using a constant threshold of power at 100 Hz (Matsuda et al., 2010c).

Figure 8.

Frequency spectrums of accelerations by self dotting and cross talk

Because the accelerations by cross talk must have a delay (adjacent fingers: 5.0 ms, index finger and ring finger: 8.9 ms), the first detected acceleration must be the acceleration by self dotting. In the case of Fig. 6, the acceleration of the index finger is the first detected acceleration. Then by setting the acceleration of the index finger as the dynamic threshold, the recognition system can recognize the acceleration of the middle finger and ring finger. We noted two parameters related to the index finger, the amplitude of acceleration (AI1) and the power at 100 Hz (PI).

2.3.2. Algorithm for the recognition of the dotted fingers

We derived an algorithm for the recognition of dotted fingers. For example, the variables and equations in the case of Figs. 7 and 8 are identified.

1. Acquire the acceleration data for 100 ms (pre-trigger 20 ms and post-trigger 80 ms) when the shock accelerations by dotting are detected.

2. Set the amplitude and power at 100 Hz of the first detected acceleration as the dynamic thresholds (index finger: AI1, PI).

3. If the amplitude of the second detected acceleration is greater than half of the amplitude of the first detected acceleration (middle finger: AM1>AI1/2) or the power at 100 Hz of the second detected acceleration is greater than the power at 100 Hz of the first detected acceleration minus 10 dB Vrms (middle finger: PM>PI-10), the second detected acceleration is recognized as the acceleration by self dotting.

4. If the amplitude of the second detected acceleration is less than or equal to half of the amplitude of the first detected acceleration (middle finger: AM1<AI1/2) and the power at 100 Hz of the second detected acceleration is less than or equal to the power at 100 Hz of the first detected acceleration minus 10 dB Vrms (middle finger: PM<PI-10), the second detected acceleration is recognized as the acceleration by cross talk.

5. If the power at 100 Hz of the second detected acceleration is less than -58 dB Vrms (middle finger: PM<-58), the second detected acceleration is recognized as the acceleration by cross talk.

6. Steps 3~5 apply to the third detected acceleration (ring finger: AR1, PR).

2.4. Recognition of dotted positions

2.4.1. Features of shock acceleration by dotted positions

Fig. 9 shows the features of the accelerations by the dotted positions. When the receiver’s hand forms a natural longitudinal arch on the desk, the DIP joints are close to the desk, and space exists under the PIP joints. Dotting on the DIP joints causes a hard impact, and dotting on the PIP joints causes a soft impact. Fig. 7 shows the shock accelerations by dotting on the DIP joints. Fig. 10 shows the shock accelerations by dotting on the PIP joints. AI1 indicates the amplitude of acceleration of the index finger. AI2 indicates the damping amplitude of the index finger. The difference of each impact is indicated by its damping amplitude. The damping amplitude ratio of accelerations by dotting on the DIP joints (AI2/AI1) is greater than the damping amplitude ratio of accelerations by dotting on the PIP joints.

2.4.2. Algorithm for the recognition of the dotted positions

We derived the following algorithm for the recognition of the dotted positions. For this example, the variables and equations for the cases of Figs. 7 and 10 were calculated.

1. Calculate the damping amplitude ratio (AI2/AI1) of the acceleration by self dotting.

2. If the damping amplitude ratio is greater than 0.5 (AI2/AI1>0.5), the acceleration is recognized as the accelerations by dotting on the DIP joints.

3. If the damping amplitude ratio is less than or equal to 0.5 (AI2/AI1<0.5), the acceleration is recognized as the accelerations by dotting on the PIP joints.

4. If the amplitude of the acceleration is greater than 150 m/s² (AI1>150), the acceleration is recognized as the accelerations by dotting on the PIP joints.

5. If two or three fingers are dotted at the same time (self dotting), the mean of the damping amplitude ratios is calculated and Steps 2~4 are applied.

After the recognition, the dotted fingers and positions are represented by Braille code. Table 1 lists the Braille code of dotted fingers and positions.

Figure 9.

Features of the accelerations by dotted positions

Figure 10.

Shock accelerations by self dotting and cross talk (dotting on PIP joints: soft impact)

Dotted fingers	Dotting on DIP joints	Dotting on PIP joints
Index finger	11	12
Middle finger	21	22
Index + middle fingers	31	32
Ring finger	41	42
Index + ring fingers	51	52
Middle + ring fingers	61	62
Index + middle + ring fingers	71	72

Table 1.

Braille code of dotted fingers and positions

2.5. Braille code parser and integrated program

We developed a Braille code parser by applying the technology of natural language processing (BUP system) (Matsumoto et al., 1983). The Braille code parser parsed the Braille code and converted it into Japanese text. The Braille code parser consisted of a dictionary, grammar, BUP translator and control program. The dictionary and the grammar were described in definite clause grammar (DCG). The BUP translator translated the dictionary and grammar into a Prolog program. The control program controlled the execution of parsing.

When the programs for the recognition of dotted fingers and positions recognize a dotting, the integrated program sends a list of recognized Braille code to the Braille code parser. Then the Braille code parser parses the list of Braille code. If the list of Braille code is grammatically correct, the Braille code parser sends the converted Japanese text to the integrated program. If the list of Braille code is grammatically incorrect, the Braille code parser sends a "no" to the integrated program.

Finally, when the integrated program receives the Japanese text from the Braille code parser, the integrated program allows the speech synthesizer to synthesize the Japanese text. Fig. 11 shows a screenshot of the recognition system.

3. Evaluation experiment

3.1. Method

To evaluate the recognition of sentences dotted by the Finger Braille interpreter, an evaluation experiment of sentence recognition was carried out.

The subject (sender) was a non-disabled Finger Braille interpreter (experiment: 22 years). The subject gave informed consent after hearing a description of the study.

The dialogues (total: 51 sentences, 143 clauses, 288 words, 686 characters) comprised four daily conversations in a Japanese textbook for foreign beginners (3A Corporation, 1998). The numbers of the dottings of the dialogues are listed in Table 2. In Finger Braille, some characters are dotted on both the DIP joints and PIP joints and some characters are dotted only on the DIP joints or the PIP joints. The average of dotted times per character was 1.75.

Figure 11.

Screenshot of the recognition system. Upper window is the integrated program and lower window is the programs of the recognition of dotted fingers and positions.

Dotted fingers	Dotting on DIP joints	Dotting on PIP joints
Index finger	152	71
Middle finger	77	161
Ring finger	70	66
Index + middle fingers	110	84
Middle + ring fingers	54	126
Index + ring fingers	72	39
Index + middle + ring fingers	70	72

Table 2.

Numbers of the dottings of the dialogues

The experimental flow is shown in Fig. 12. The experiment included one practice session and four experimental sessions (conversations 1 to 4). In the experiment, a tester and the subject sat face to face. The tester wore the accelerometers. The subject spoke one sentence of the dialogues and then dotted the sentence on the tester’s fingers clearly. The tester’s hand set on the desk in each conversation and formed the natural longitudinal arch. If the recognition system synthesized the misrecognized speech, the subject would stop dotting or re-dot the dialogues. To prevent unnecessary pause or re-dotting by the subject, the speech synthesizer was turned off during the experiment. The lists of the recognized Braille code were recorded in the hard disk drive of the recognition system.

Figure 12.

Experimental flow

3.2. Results

3.2.1. Accuracies of recognition of dotted fingers

The mean of the dotting speed was 37.0 characters/min. This was almost 1/3 of the normal dotting speed.

To evaluate the accuracy of recognition, we checked the lists of the recognized Braille code and calculated the accuracies of the recognition by dotting (each dotting on DIP joints or PIP joints) and by character (one or two dottings). Fig. 13 shows the accuracies of the recognition of dotted fingers as a function of conversation and as a function of the calculation unit. Fig. 14 shows the accuracies of the recognition of dotted fingers by dotting as a function of the dotted fingers and as a function of the dotted positions.

The overall accuracy of the recognition of dotted fingers by dotting was 89.7%. In the experiment of conversation 3, the power at 100 Hz of the middle finger was less 5 dB Vrms. The accuracy of conversation 3 was 77.2%. The accuracy without conversation 3 was 94.3%. The accuracies of the middle finger and middle + ring fingers of the dotting on the PIP joints were less than the other accuracies; the accuracies of the index + middle fingers and middle + ring fingers of the dotting on the PIP joints were also less than the other accuracies.

The overall accuracy of the recognition of dotted fingers by character was 82.6%. The accuracy without conversation 3 was 90.0%.

Figure 13.

Accuracies of the recognition of dotted fingers as a function of conversation and as a function of the calculation unit

Figure 14.

Accuracies of the recognition of dotted fingers by dotting as a function of the dotted fingers and as a function of the dotted positions

3.2.2. Accuracies of dotting and recognition

Fig. 15 shows the accuracies of the recognition of dotted positions as a function of conversation and as a function of the calculation unit. Fig. 16 shows the accuracies of the recognition of dotted positions by dotting as a function of the dotted fingers and as a function of the dotted positions.

The overall accuracy of the recognition of dotted positions by dotting was 92.3%. The accuracy of conversation 3 was 88.8%. The accuracy without conversation 3 was 94.9%.The accuracies of the dotting on the PIP joints of the index finger and middle finger were less than the other accuracies.

The overall accuracy of the recognition of dotted positions by character was 88.3%. The accuracy without conversation 3 was 91.2%.

Figure 15.

Accuracies of the recognition of dotted positions as a function of conversation and as a function of the calculation unit

Figure 16.

Accuracies of the recognition of dotted positions by dotting as a function of the dotted fingers and as a function of the dotted positions

3.3. Discussion

3.3.1. Accuracies of recognition

The accuracy of the recognition of dotted fingers by dotting without conversation 3 was 94.3%, and the accuracy of the recognition of dotted positions by dotting without conversation 3 was 94.9%. The accuracy of the recognition of dotted fingers by character without conversation 3 was 90.0%, and the accuracy of the recognition of dotted positions by character without conversation 3 was 91.2%.

In the experiment of conversation 3, the power at 100 Hz of the middle finger was less than 5 dB Vrms, although the power improved in the experiment of conversation 4. This phenomenon was the same as the phenomenon that occurred in the previous experiment (Matsuda et al., 2010c). As real communication using the recognition system, non-disabled people (receiver) can re-set their hand on the desk when they notice a decreased accuracy of recognition. The re-setting of the receiver’s hand should be allowed in the communication.

As previously mentioned, (Uehara et al., 2000) developed a Finger Braille glove system with accelerometers mounted on the fingertips. Three Finger Braille interpreters wore the glove system and dotted Finger Braille. The accuracy of recognition was 73.0%. The dialogues that they used were the number of characters; the dotting speed and range of amplitude of acceleration were not clear. But the accuracy of recognition by our recognition system was greater than or equal to the accuracy of recognition by the glove system.

(Hoshino et al., 2002) developed the Finger Braille input system that mounted accelerometers on the middle phalanges. Three visually impaired people and two non-disabled people who were skilled in Finger Braille wore the input system and dotted 100 randomized characters. They reported that the accuracy of recognition was 99.3%. Because the characters did not form sentences, the subjects might not express the prosody of Finger Braille.

To compare our study with these previous studies, our recognition system could recognize the sentences accurately when the interpreter dotted clearly. Although the accuracy of the recognition is high, the Braille code parser can not convert the list of the Braille code which was grammatically incorrect into the Japanese text. Then the recognition system can not synthesize the misrecognized clauses. We have been improving the Braille code parser.

3.3.2. Feedback to the deafblind people

In this experiment, the subject pointed out the importance of feedback from the non-disabled person (receiver) to the deafblind person (sender) as to whether the non-disabled person recognized the Finger Braille correctly. Because of their deafness and blindness, deafblind people cannot confirm the synthesized speech and displayed text of the recognition system. Therefore, deafblind people are uneasy without feedback, and they will find it difficult to keep up with the conversation. Therefore, we have been developing the combination of a recognition system and teaching system to display the dot pattern of the recognized sentence, so that receivers can offer feedback to senders.

4. Future plans

4.1. Improvement of the mounts of accelerometers

In the previous study, the accuracies of the recognition of the dotted fingers and positions of some subjects are low, when the bottoms of the rings have contacted the desk by dotting, especially the ring finger. Fig. 17 shows the shock accelerations by contact between the bottom of ring and desk. The contact causes different shock accelerations and influences the accuracy of recognition of dotted fingers and positions.

To avoid the shock acceleration by the contact between the bottom of ring and desk, we have been improving the mounts of the accelerometers by two methods (Matsuda et al., 2012). We adopt a cloth band and half-cut ring covered by cloth instead of the previous ring (see Fig. 18). Both the cloth band and half cut ring will not cause the shock acceleration by the contact between the bottom of the mounts and desk.

Figure 17.

Shock accelerations by contact between the bottom of ring and desk

Figure 18.

Previous ring (left), cloth band (middle) and half-cut ring covered by cloth (right)

4.2. Emotion recognition of Finger Braille

In Finger Braille, the sender can express various emotions by changing the duration and strength of dotting (Fukushima, 1997; Matsuda et al, 2010b). The intent of our support device is to assist not only verbal communication but also non-verbal (emotional) communication. To assist in emotional communication, we have been developing an emotion teaching system (Matsuda et al., 2010d) and an emotion recognition system (Matsuda et al, 2010e).

The emotion recognition system is based on the Finger Braille recognition system and recognizes four emotions (joy, sadness, anger and neutral) expressed by the deafblind person. The algorithm of emotion recognition is as follows. First, the emotion recognition system recognizes the dotting by the deafblind person and calculates the duration of dotting and amplitude of acceleration by dotting. Second, the probabilities of four emotions about each dotting are calculated. Third, the means probabilities about a sentence are calculated. The sentence is recognized as the emotion which the mean probability is highest. Regardless of the accuracy of emotion recognition of dotting is not very high, the emotion recognition system can recognize the emotions of sentence accurately.

5. Conclusion

In this chapter, we developed a Finger Braille recognition system and derived the algorithms for the recognition of dotted fingers and positions. Next, an evaluation experiment was carried out. The results of the evaluation experiment showed that the accuracy of the recognition of dotted fingers by dotting was 89.7% (94.3% without conversation 3), and the accuracy of the recognition of dotted positions by dotting was 92.3% (94.9% without conversation 3). Therefore, the recognition system could recognize sentences accurately when the interpreter dotted clearly. We confirmed that non-disabled people (receiver) should re-set their hand on the desk when they notice a decrease of the accuracy of recognition.

Acknowledgement

We greatly thank Ms. Satoko Mishina (Finger Braille interpreter) for her support in the evaluation experiment.

This study was supported by the Japan Society for the Promotion of Science under a Grant-in-Aid for Scientific Research (No. 21500522) and the Ministry of Education, Culture, Sports, Science and Technology of Japan under a Grant-in-Aid for Scientific Research (No. 16700430).