The Use of Leap Motion in Manual Dexterity Testing by the Box and Blocks Test: A Review Study

Abstract

This chapter aims to analyze works in the literature that show the therapeutic effects of using the Leap Motion Controller (LMC) sensor to assess hand fine motor dexterity, especially those involving the Box and Blocks Test. Besides the introduction, we will describe: (a) the LMC device and its forms of interaction in a Virtual Reality environment (immersive and non-immersive); (b) aspects of manual function assessment; (c) the functioning of the traditional Box and Blocks Test (BBT) and its virtual version (VBBT) developed with Virtual Reality technologies; (d) discussion about the VBBT integrated with the LMC, in physical therapy practice.

1. Introduction

The Leap Motion Controller (LMC) is a small, portable, relatively low-cost device compared with other motion capture devices such as Doctor Kinetic®. It accurately detects all the hand and finger joints [1]. It is an interactive technology in Virtual Reality (VR) environments. LMC can be used in non-immersive mode, plugged into the computer via USB (desktop version), or attached to VR glasses (headset version) for immersive interaction. It has been currently being investigated as a technological resource to support upper limb motor rehabilitation interventions, as it allows capturing finer movements of the hands and fingers, which are essential for the rehabilitation of manual dysfunctions found in different conditions [2, 3, 4, 5, 6].

Several models of VR headsets are on the market, such as Gear VR, Rift glasses, and HTC Vive [2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. Gear VR is a more cost-effective solution because it uses the screen of a Samsung smartphone as a viewing display [18]. The trend is that these technologies are available in people’s homes, offering services in the most diverse areas of entertainment, education, and health.

Research with LMC points to its potential use by people with difficulties in fine and gross motor skills, explicitly concerning pincer grasp and power grip strength movements, extension and flexion of fists and fingers, forearm supination and pronation. Due to this extensive repertoire of gestures that LMC can detect, it is possible to find several studies with groups of people with stroke, older people with manual motor dysfunctions caused by aging [13], parkinsonians [8], children with developmental psychomotor disabilities, including cerebral palsy [20], Down syndrome [21], and autism [6].

The LMC allows measuring the motor performance of these people, such as reaction time, bimanual coordination, and the sequence of movements performed with the hands and fingers. For this reason, this remote sensing technology has shown promise for the rehabilitation field as it does not require the patient to wear motion detection devices (e.g., gloves with force and feedback sensors). Therefore, it provides a new interaction between the user and the computer, allowing for more natural and touch-free interaction. Hand dexterity in patients with upper limb motor disorders can be assessed from programmed tasks with graphic objects added to the virtual world [14].

Research involving the LMC integrated with the Box and Block Test (BBT) has emerged recently. Physical and occupational therapists use this test to verify manual function to assess and quantify the unilateral gross manual dexterity in children and adults. The BBT is made up of a wooden box with colored cubes, whose objective is to transport the cubes from one compartment of the box to the other in 1 minute. In the end, the number of cubes transferred per minute is counted. In its VR version, the LMC is used as an interaction device to transport blocks from one compartment to the other of the Box.

This chapter presents the works investigated using the LMC sensor integrated into the BBT. We are developing a version of BBT in VR that can be used on Desktop computers, both with video monitors and with VR glasses (HTC Vive). Then, we present details of this virtual BBT version in this chapter.

In addition to this introductory section, we divided this article into five sections: Section 2 provides details on how the LMC device works; Section 3 briefly presents the main tools for assessing manual function; Section 4 presents the functioning of the Box and Blocks Test and its virtual version in Virtual Reality developed by the researchers of our Game Therapy and Virtual Reality Laboratory; Section 5 shows scientific studies (performed with the Virtual BBT) found in the literature; Section 6 presents the conclusions.

The method adopted for the theoretical review was the query in indexed databases seeking information about the advantages and disadvantages of using the LMC device in immersive and non-immersive virtual reality situations. We sought to identify the errors and inaccuracies most commonly in the use of this device associated with the box and blocks test and the results found regarding the performance of the manual function.

2. Leap motion controller

Leap Motion Controller (LMC) is a small, portable device that accurately detects all the hand and finger joints [1]. It is a compact device, 8 cm wide by 3 cm high. The top of the device is made of smoked glass to hide the two image sensors and infrared LEDs that work together to track the user’s hand movements (Figure 1).

Figure 1.

Leap motion controller (LMC). Source: Leap motion, 2019 https://www.leapmotion.com/.

It is possible to use the LMC connected directly to the computer in non-immersive experiences (Figure 2a). In this case, the video monitor is used as a viewing device. In immersive experiences, the LMC is coupled to VR glasses, such as a Gear VR. (Figure 2b). The simplicity of the LMC could facilitate the approach to technology, increasing the feeling of immersion, imagination, and interaction with the virtual environment [4].

Figure 2.

(a) LMC with non-immersive interaction (desktop). (b) LMC with immersive interaction (VR headset). Source: Leap Motion, 2019 https://www.leapmotion.com/.

The LMC works with accuracy down to 1/100 mm without visible latency in its visual field. The viewing range is 80 cm above and around the device. This limit occurs due to the propagation of the LED light through space since it is difficult to infer the hand’s position from a certain distance.

The LMC is postulated as a playful tool that can favor the inclusion of participants in a different environment, in which they can face new challenges and achieve new goals by interacting in real time with hand and finger movements, functioning as an active therapy that requires high commitment and motivation [9].

In a study by [9], the authors conclude that the LMC is the primary haptic VR sensor for upper limb mobility recovery compared with other non-immersive VR devices, such as the Doctor Kinetic® and Nintendo® Wii game systems, which are more specialized in posture and balance. The results suggest that the LMC can be considered a valuable and effective haptic VR device to improve different aspects of upper limb motor function in neurological patients [9].

3. Manual function assessment

The functional assessment of the hands is part of the therapeutic routine of several professionals who work with strategies for functional recovery of the upper limbs. For this, several tests evaluate the precision and speed of movements, proposing different tasks in standardized tests and validated in other countries. Reference [22] analyzes, through a critical review, several tools for assessing manual dexterity, referring to their psychometric properties. Among the tools reviewed are tests commonly used for upper limb assessment, such as the Minnesota Dexterity Test, the Nine Hole Peg Test (9HPT), the Jebsen-Taylor Hand Function Test (JTHFT), and the Box and Blocks Test (BBT).

These conventional tests for assessing manual function have several limitations, such as the long time to be done, the need for a highly trained professional to determine the patient’s outcome, the patient’s displacement, and the reliability of the performance evaluation, which is low [23]. The use of technology associated with these tests is a way of fulfilling these principles since it can provide exercises in a controlled, repetitive, intensive, interactive, and motivating way [22].

4. Box and blocks test

A. Jean Ayres and Patricia Holser Buehler created the first Box and Blocks Test (BBT). They used a bowl and blocks to assess gross manual dexterity in adults with cerebral palsy [24]. Later, Patricia Holser Buehler and Elizabeth Fuchs changed the shape of the test to a gift box, obtaining copyright in 1957 [16].

As described by [16, 17], the BBT (Figure 3) is used to verify manual function to assess and quantify unilateral gross manual dexterity in children and adults. It consists of a wooden box, 53.7 cm long, with a partition, also made of wood, higher than the edges of the box, separating it into two compartments of equal dimensions. The blocks, also made of wood and in the form of colored cubes (primary colors), are 150 in number, measuring 2.5 cm on a side divided equally by color. The box’s long sides are 53.7 cm by 8.5 cm and are nailed 1 cm thick from the base. The short ends are 7.5 cm by 25.4 cm and have been fixed to the top of the bottom between the long sides.

Figure 3.

Box and blocks test (BBT). Source: MENDES et al., 2001 [17].

As a prerequisite for the application of the test, a quiet environment is required, with the examinee seated in a chair suitable for his/her height. The wooden box should be placed horizontally in front of him so he can fully view the area and equipment in question. It consists of moving as many cubes as possible from one compartment to another for 1 minute [25]. The BBT was proposed [16] with parameters between 20 and 94 years old. In Brazil, the BBT was used for the age group between 15 and 86 years, with typical groups and multiple sclerosis [17]. For the lower ranges, from 7 to 14 years old, the test parameters for Brazilians were mentioned by [12].

These functional measures of dexterity contrast with neuroscientific investigations that show that skill is multicomponent, including the ability to control force, control the timing of movements, execute independent finger movements, and execute motor sequence [26].

4.1 The virtual box and blocks test

Our researchers from the Game Therapy and Virtual Reality Laboratory (Lab GameVR) created a virtual application of BBT, which we call Virtual BBT (desktop version and HTC Vive). The main requirements raised were the following:

• Availability of two forms of use:

  • Conventional: with the countdown starting at 60 seconds.

  • Training: with progressive time without time counting.

• On-screen display of the score to the user.

• Storage of collected data.

• Help feature with usage instructions.

• Bilateral hand training.

Based on these requirements, a game prototype was conceived in VR using the Unity 3D game engine integrated into the LMC. When executing the game file, the Virtual BBT start screen is presented to the user with the following configuration and selection options (Figure 4) “Right or left hand,” “Start,” “Training,” and “? [Help].” At that moment, the user must plug the LMC into the computer where the application is running.

Figure 4.

Virtual BBT home screen. Source: Author.

When selecting the “Start” option, the Virtual BBT screen is shown to the user (Figure 5). The screen shows the Wooden Box and the blocks modeled with Unity 3D. The user can observe the virtual hand moving around the scenario by placing the hands on the LMC. The screen displays the timer counting down. In this game scenario, the user has 60 seconds to transfer the blocks from one box compartment to the other. The same game scenario is displayed by selecting the “Training” option but without the countdown. In this scenario, the user can stay as long as he wants to interact with the Virtual BBT. An immersive version of BBT Virtual is being developed for use with HTC Vive.

Figure 5.

Virtual BBT screen in run mode. Source: Author.

At the end of use, the data referring to the player’s score, date, time, patient’s name, time, and laterality score of the executed hand are stored locally on the user’s computer in a file in the csv format.

5. Discussion

Scientific studies involving box and blocks test in its virtual version are emerging in the context of rehabilitation in patients with neurological diseases, such as Parkinson’s Disease (PD), Cerebral Vascular Accident (CVA), and Multiple Sclerosis (MS), among others, intending to study the validity, feasibility, and psychometric properties of the test [3, 10, 15]. However, few studies have developed the virtual BBT (VBBT) [10].

The interest in developing virtual tests is multiple [10]. It can reduce the inter-observer subjectivity in the classic assessment, providing a complete virtual rehabilitation at home where patients can assess their cognitive and motor improvements using validated virtual evaluation.

Table 1 shows some studies that addressed the topic, including author, study population, research country, goals, method, and the main results.

Author	Study population	Research country	Goals and method	Main results
Onã et al., 2020.	Participants with Parkinson’s Disease (PD)	Spain	Developed the VBBT using an immersive headset and the LMC to evaluate unilateral manual dexterity. Participants performed the physical BBT (once) and the virtual BBT (twice, one immersive and one non-immersive) separately.	3D depth perception allowed the movement of more cubes in the immersive virtual BBT regarding BBT.
Giersen et al., 2016.	Typical Individuals	United States	Present their version of the VBBT and the results of a pilot study in which participants completed the BBT and VBBT, comparing their scores and opinions.	The number of video games and VR experience was positively correlated with task performance.
Alvarez-Rodriguez et al., 2020.	Neurological Diseases	Spain	Developed a form of application of the VBBT using the LMC. The sample consisted of 24 individuals, divided into two groups: the typical group (n = 12) and the group with neurological diseases (n = 12). The study is conducted in a single experimental session by performing the physical and virtual BBT with the dominant hand.	BBT and VBBT high-performance showed corresponding final results, with a high tendency between the two tests.
Teruel et al., 2019.	Patients who suffer a spinal cord injury at the cervical level and affects the function of; the upper limb.	Spain	Present a tool developed from the BBT for its virtual version, using the LMC and Unity 3d to supervise the therapy execution.	VBBT enables the therapists to customize therapy according to each patient’s specific needs.

Table 1.

Summary of studies.

Source: Author.

Researchers [3] assessed the validity of BBT and VBBT in participants with Parkinson’s Disease (PD). They developed the VBBT using an immersive headset (Rift eyewear) and the LMC to evaluate unilateral manual dexterity in this audience. Participants were instructed to perform the physical BBT (once) and the virtual BBT (twice, one immersive and one non-immersive) separately. The results indicated a moderate correlation between the physical BBT and the VBBT scores.

The results show that the 3D depth perception allowed the movement of more cubes in the immersive virtual BBT regarding BBT. This study presented three relevant findings: the gap in the number of blocks transported in the physical and virtual systems seems to tend to be a constant, the correlation between the obtained result between the physical and virtual systems is statistically significant, and the test-retest analysis shows an excellent statistically considerable correlation between attempts with the virtual system. In this sense, the relationship between physical and virtual systems can be improved using fully immersive VR due to better depth perception.

In a study [11], researchers present their version of the VBBT and the results of a pilot study in which participants completed the BBT and VBBT, comparing their scores and opinions. The authors also compared how the participants handled time during both versions running. The Unity 3d platform and the LMC device were used for the virtual version. For gameplay, timers were implemented for the participants’ 15-second practice and the 60 seconds of the entire session. During the whole session, the LMC data were recorded for future analysis. These data include participant position, palm position, fingertip position, and joint angles.

The results compared the scores and opinions of participants who took both versions of the test, showing that the number of video games and VR experience was positively correlated with task performance. The results also showed that participants with less knowledge of video games and VR performed in a slower time than those with more experience, who reported greater enjoyment than those who did not have VR contact. Finally, the authors conclude that the participants found the virtual version more frustrating. However, they still preferred to deal with this version rather than the physical one to have one more chance at performance. On average, healthy participants moved more 35 blocks in the BBT than in the VBBT.

In another study by [27], they developed a form of application of the VBBT using the LMC. The sample in the research consisted of 24 individuals, divided into two groups: the typical group (n = 12) and the group with neurological diseases (n = 12) and, consequently, impairment of upper limb motor function. The study is conducted in a single experimental session by performing the physical and virtual BBT with the dominant hand. Firstly, the physical BBT was performed, starting with a practice test lasting 15 seconds. Once the test was completed, the participants had a 5-minute rest before running the VBBT. Before beginning this version, there was also a practical period in which the participant performed two 1-minute tests to make him comfortable and familiar with the virtual environment. The 15-second test period started with the dominant or less affected hand, followed by 60 seconds with the automatic counting of correctly transported blocks.

This research presented some limitations reported by the authors, such as difficulties with the technology used (LMC), whose performance was strongly conditioned by the environmental conditions and the computer’s performance. However, this study showed that considering the physical BBT, upper limb motor skill was statistically higher in the typical population than in the people with neurological diseases. The same behavior was observed for the VBBT. Besides, within each analyzed group, the performance in the BBT was superior to that of the VBBT. However, BBT and VBBT high performance showed corresponding final results, with a high tendency between the two tests.

The VBBT version developed in this work showed high consistency in its application in a sample of typical individuals and patients with neurological diseases. According to the authors, the next step is that the VBBT, integrated with LMC, serves as an element to assess motor dysfunctions, allowing manual dexterity training in the population with neurological diseases.

Other researchers [28] present a tool developed from the BBT for its virtual version, using the LMC and Unity 3d. The authors included some feedback in this version (VBBT) to increase the patients’ motivation, with the intention that they become aware of their competence while performing exercises for rehabilitation. The developed game can be configured according to the needs of each one. It offers facilities to define the dominant hand, the number of blocks the patients will have to pick up and their size. After that, the game starts so that the player can interact with the blocks, lasting until all the blocks have been placed on the non-dominant side area of the screen or until the participant gives up playing.

During the VBBT execution, patients receive information about the current game and their progression concerning previous moves. While the participant performs the task, the game simultaneously shows the number of moved blocks (score), the number of blocks they need to move to finish the game correctly (goals), the elapsed time, and the current speed related to the number of blocks moved per minute. In addition, to complete this information, a dynamic status bar is also shown to describe the relationship between the patient’s current speed and the speed of their previous best and worst performance. The value of this status bar changes in real time, decreasing if the patient does not move new blocks or increasing when a new block is moved. When finished, additional information is presented to the patient on a result screen, showing the player’s final score, goal, elapsed time, and speed.

The authors emphasize that upper limb rehabilitation has become a critical need due to the number of people affected by this condition when they have a neurological disease, for example. They also point out that the existing treatments for these conditions are demanding and expensive. Therefore, the VBBT enables the therapists to customize therapy according to each patient’s specific needs, exploring an important feature, which is the introduction of different motivation facilities to attract this audience to perform a repetitive task, which, on the other hand, could be tedious.

From the studies discussed, it can be noticed some of the advantages and disadvantages of using the LMC [27], which can be a device dependent on the conditions presented by the computer to which it will be connected. Some results [11] showed that participants with less knowledge of video games and VR performed in a slower time than those with more experience, who reported greater enjoyment than those who did not have VR contact. Therefore, the use of LMC may also be related to the individual’s previous contact with technology.

When used with the BBT, the LMC proved to be more acceptable in its immersive version [3], mainly because users are able to transfer more blocks when inserted in the 3d view, which consequently generates less fatigue in the manual function and an increase in the motivation. Another advantage of using this technology is that together with the virtual BBT, health professionals have the possibility to meet the needs of each patient individually, through time, the number of blocks, feedbacks, among others [28].

It is important consider that the use of MC associated with the virtual BBT allows the execution of the task of transferring blocks in a situation of evaluation and training for the manual function without the need to the physical blocks, reducing the chance of any type of contamination, BBT the manual dexterity, promoting the development and health of users.

Therefore, there is a need for expansion in studies that encompass the rehabilitation of manual dexterity, technology, and its devices and the tests involved in this context, such as BBT in its virtual version.

6. Conclusions

This chapter aimed to present an overview of a research development using the Virtual BBT. It is noteworthy that the development of BBT, in its immersive and non-immersive virtual version integrated with the LCM device, presents itself as a possibility for testing and assessing patients with or without changes in manual dexterity. This technology emerges as an innovative and motivating way for use in various areas, such as health. Then, more research is needed with different audiences to investigate its usability and effectiveness of both (virtual BBT and the LMC device) and their forms of use so that technical issues can be increasingly improved in the use of these equipment.

Conflict of interest

The authors declare no conflict of interest.

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brazil (CAPES).