A Personal Robot as an Improvement to the Customers’ In- Store Experience

Abstract

Robotics is a growing industry with applications in numerous markets, including retail, transportation, manufacturing, and even as personal assistants. Consumers have evolved to expect more from the buying experience, and retailers are looking at technology to keep consumers engaged. In today’s highly competitive business climate, being able to attract, serve, and satisfy more customers is a key to success. It is our belief that smart robots will play a significant role in physical retail in the future. One successful example is wGO, a robotic shopping assistant developed by Follow Inspiration. The wGO is an autonomous and self-driven shopping cart, designed to follow people with reduced mobility (the elderly, people in wheelchair, pregnant women, those with temporary reduced mobility, etc.) in commercial environments. With the Retail Robot, the user can control the shopping cart without the need to push it. This brings numerous advantages and a higher level of comfort since the user does not need to worry about carrying the groceries or pushing the shopping cart. The wGO operates under a vision-guided approach based on user-following with no need for any external device. Its integrated architecture of control, navigation, perception, planning, and awareness is designed to enable the robot to successfully perform personal assistance, while the user is shopping.

1. Introduction

In recent years, a high concern with user satisfaction has been observed in the retail industry. This is particularly accentuated with the rise in online shopping which pushes retailers to provide a better in-person shopping experience to attract customers. Among customers in the public, one of the main groups of interest is people with disabilities. This is visible not only in the marketing strategies but also at the political level, where accessibility for disabled people is becoming the topic of regulation and legislation.

It is estimated that in Portugal about 8–10% of the population has some form of disability [1] and that in Europe alone there are about 50 million people with disabilities and 134 million people with reduced mobility. Apart from people using wheelchairs, there are other cases in which people are temporarily or permanently disabled, and these include: an elderly person using a cane, or someone with a foot or leg injury who requires the use of crutches, pregnant ladies, and parents with prams.

In fact, if we add the disabled, the elderly, pregnant women, and couples with children, we find that between 30 and 40% of all Europeans could benefit from improved accessibility. In addition to those people with reduced mobility due to disability or injury, there are many people without mobility issues who could benefit from assistance in carrying heavy bags.

Shopping environments are highly heterogeneous and give rise to a high frequency of dynamic interactions that trigger various senses and emotions in humans. This often causes a high level of stress in people and those with mobility limitations.

Some of the identified difficulties include [2]:

• People who use wheelchairs

  • no adequate forward reach at basins, counters, and tables; and

  • surfaces that do not provide sufficient traction (e.g., polished surfaces).

• People who have trouble walking

  • no seating in waiting areas, at counters and along lengthy walkways;

  • access hazards associated with doors, including the need to manipulate a handle while using a walking aid; and

  • surface finishes that are not slip-resistant or are unevenly laid.

Besides the difficulties brought by the shopping environment itself, conventional shopping carts, which can carry many products and which are provided with wheels so that the shoppers can push them, also have serious drawbacks, one of them being their considerable size. This is simultaneously an important asset and a significant drawback, as although shopping carts can hold large and bulky products, the increased mass complicates maneuverability and handling. Maneuverability is particularly compromised when making turns in supermarket aisles or when avoiding other carts, shelves, and indeed other shoppers [3]. Smaller baskets appeared on the market to overcome the traditional shopping carts’ drawbacks. These baskets were developed to hold a set of items while at the same time being easy to move. They contain wheels or rolling elements incorporated into the bases which allow them to be moved when parallel to the floor or when inclined. However, even though these baskets improve maneuverability due to their reduced size and capacity, they also have drawbacks typical of their morphology, such as the need for the user to bend down for placing or removing items, among others. Furthermore, such baskets can have drawbacks typical of the way they are stored, since stacking them vertically can entail a problem for elderly shoppers or shoppers with any type of physical limitation [3].

With these described difficulties in mind, this paper presents a new robotic concept to help and assist people (giving special emphasis to people with reduced mobility) in these types of environments, through a user-following scenario. The wGO (Figure 1) is an autonomous and self-driven shopping cart, designed to follow people with reduced mobility (the elderly, people in wheelchairs, pregnant women, temporary reduced mobility, etc.) in commercial environments. With the robot, the user can control the shopping cart without the need to push it. This brings numerous advantages and a higher level of comfort, since the user does not need to worry about carrying the groceries or pushing the shopping cart.

Figure 1.

wGO.

In this chapter, the wGO is introduced for the first time and we present its behavior and main features. Preliminary technical results obtained in real scenarios are also given. Finally, a user satisfaction survey is presented. The structure of the chapter is as follows: the current available solutions and their limitations are discussed in Section 2. In Section 3, the wGO’s behavior is described, while its architecture is given in Section 4. Section 5 illustrates the wGO behavior in a real scenario (a supermarket during regular opening hours) and analyses a user study. Conclusions and directions for future work are given in Section 6.

2. Existing solutions

Looking at the commercial market, the most obvious existing solutions are those provided by shopping cart producers.² These providers typically have products targeted for customers in wheelchair, but not products for other types of users with reduced mobility (e.g., pregnant women). A different type of solution is the adapted system. Some examples are the “amigo mobility” scooter³ and the adapted wheelchairs,⁴ promoted by Egiro.⁵ These products are, however, not particularly user-friendly. The user needs to first move into the mobility auxiliary device and then to learn how to use it (which may be particularly hard for the scooter case). In the case of wheelchair users, the user also needs to leave their own personal chair, which may cause discomfort and unnecessary stress. Another problem with these solutions is that the user is visibly distinguishable from the other supermarket clients, which may discourage some people from using it [4].

While this topic of assisted shopping using robotics has received very little attention in the academic research community, several systems exist where robots are used to help people with reduced mobility.

In [5], an anticipative shared control for robotic wheelchairs, targeted at people with disabilities, is presented. The same idea, of intelligent wheelchairs, is also the focus of the work in [6] where a data analysis system which provides an adapted command language is presented. The work in [7] presents an analysis of the implementation of a system for navigating a wheelchair with automation, based on facial expressions, especially eyes closed using a Haar cascade classifier, aimed at people with locomotor disability of the upper and lower limbs.

A smart companion robot for the elderly people, capable of carrying out surveillance and telepresence tasks, is described in [8]. Also, with the aim of helping elderly people through telepresence, a low-cost platform capable of providing augmented reality for pill dose management was developed in [9]. In [10], an approach based on the Dynamical System Approach for obstacle avoidance of a Smart Walker device to help navigation of elderly people is presented.

Perhaps, the closest application to the focus of this chapter is presented in [11] where a product locator application is proposed. The application runs on heterogeneous personal mobile devices, keeping the user private information safe and locating the desired products over each supermarket’s map. We believe that such a system could be complementary to the wGO and could be used in combination to further improve customers’ shopping experiences.

3. wGO: Specifications and behavior descriptions

The wGO is designed to have an ergonomic shape, friendly both to the target users (people with reduced mobility) and to the environment (commercial retail environment). An illustration of the robot’s hardware is shown in Figure 2. Its main internal sensors are as follows: ultrasound sensors, a Laser Range Finder (LRF), and active vision sensors. This combination was selected due to their complementary features. While the ultrasonic sensor detects any type of material that is not sound absorbing, it has the wide beam-width and echo problems as main drawbacks. LRF provides 270° information and its precision is high. It is, however, sensitive to dust. Active vision provides very rich information (image + 3D), but it has a relatively small field of view and low precision.

Figure 2.

wGO hardware illustration.

The ultrasound sensors have a minimum and maximum range of 3 cm and 4 m, respectively, and an estimated field of view of 60°. The LRF has a maximum range of 6 m and performs 270° laser scanning. The specifications of the active vision systems are as follows:

• Range: 0.6–8 m (optimal 0.6–5.0 m)

• Color camera: 1280x960 @ 10 FPS

• Depth camera: 640x480 (VGA) 16 bit @ 30 FPS

• Horizontal field of view: 60°

• Vertical field of view: 49.5°

The system is initialized when the user presses the start button. At this moment, the user is typically facing the wGO (Figures 3 and 4). After initialization, the user starts shopping and the wGO follows him or her.

Figure 3.

wGO initialization: wheelchair typical case.

Figure 4.

wGO initialization: non-wheelchair typical case.

In cases where the person goes out of the image sensors’ field of view (Figure 5), there is a 270° laser scanner that aids the tracking process.

Figure 5.

Illustration of a case where the user is out of the field of view of the RGB and depth sensors, but visible by the 270° laser scanner.

4. wGO: System architecture

Figure 6 depicts the functional diagram of the application that is embedded in the robot and is responsible for gathering information from the sensors, for example, the encoders and the RGB and depth cameras, as well as controlling the movement of the robot. Therefore, this figure depicts a high-level representation of how perceptual data can be combined and used to enable a robot to follow a user in a realistic environment.

Figure 6.

wGO software flowchart.

Internally, the application is divided into several modules: Vision, Sensors, Behavior, Executing, and Control system. The Vision module grabs and processes RGB and depth information. In addition, the same module performs people detection [12], false-positive reduction, and identification tasks. The Sensors module grabs data received from the sensors and verifies the existence of obstacles. The Behavior module includes the tracking [13] of the detected person and the generation of the path [14] for the robot to follow. In path generation, a local localization method based on odometry is used to retrieve the location of the robot and the fusion of the vision and laser tracking results is made [1, 15–17]. The Executing system receives the generated path and the obstacle detection information and, according to the behavior and the desired action for the robot, sends commands to the Control [18–20] module that moves the robot along the predefined path. Some of the low-level navigation procedures, which ensure that the robot is always in a safe state, include hardware fault identification, obstacle detection, and maximum velocity limitation.

5. Results

In the first part of this section (subsection 5.1), some real scenario results are shown. A formal, quantitative, real-world evaluation is highly complicated due to many complex factors, such as the need to test in multiple different environments, testing with several user groups (including those with reduced mobility), the lack of any accepted standard evaluation protocol for the objective measurement of robotic assistance in a retail environment, etc. Therefore, only initial qualitative results based on realistic experimentation are shown in this paper. A formal evaluation which addresses each of these issues will be performed in future work.

The second part (subsection 5.2) describes a user satisfaction inquiry made on a real retail scenario on a population of 78 volunteers and its results.

5.1. wGO: Technical results

Starting with the detection process, the top left part of Figure 7 shows the original RGB capture from a typical user following scenario in a commercial shopping environment. In the bottom left, the initial detection gives rise to two false positives, corresponding to the two ladies in the back, while the intended target is the men with his back to the robot. By using the RGB and depth information, shown in the top, these false positives can be removed, with the result shown in the bottom right.

Figure 7.

People detection example. Top left: RGB information; top right: depth information; bottom left: original detections; bottom right: result after removal of false positives.

The tracking process is illustrated next. In Figure 8, the person is visible both by the vision and by the 270° laser scanner, while in Figure 9, the person is only visible by the 270° laser scanner. In both cases, the tracking is not lost and the wGO can follow the person.

Figure 8.

Tracking example where the person is visible both by the vision and by the 270° laser scanner. Purple dot in the map image corresponds to the person as localized by the image module, while the blue dot corresponds to the person as localized by the 270° laser scanner.

Figure 9.

Tracking example where the person is only visible by the 270° laser scanner. Blue dot in the map image corresponds to the person as localized by the image module.

Path generation is used to decide about the navigation strategy of the wGO. Since the tracking module can in general return results from multiple sources of information (e.g., vision and laser), it is necessary to merge (fuse) them into one. An example of this result combination is provided in Figure 10. This fusion step makes the system more robust to errors in either one of the sensors and helps in producing more stable trajectories.

Figure 10.

Fusion examples. On the left is a case where the person is visible both by the vision and by the 270° laser scanner, while on the right the person is only visible by the 270° laser scanner. Cross symbol in the map image of the left corresponds to the fusion result of the person as localized by the image module with the person as localized by the 270° laser scanner. Triangle symbol in the map image on the right corresponds to the person as localized by the image module and is also the final result.

Having one estimation of the person’s localization, it is now necessary to decide where to send the robot (path generation). Moreover, it is important to keep some consistency in the results. Inaccuracies produced by the sensors can lead to highly unstable paths, which is not desirable. An example of a path made by the wGO is shown in the left part of Figure 11. An increase in the smoothness of the final route, when compared to a traditional approach, is observed.

Figure 11.

Path generation and effective path illustrations. In these figures, the purple dots are the effective path done by the wGO, blue circle represents the wGO, large green circle the target destination, white circles the waypoints generated by a traditional path generation approach, smaller green circles the waypoints given by the technique present in the wGO, red dashed lines the trajectory given by the traditional algorithm, and purple dashed lines the final trajectory produced by the wGO’s system. In the background map, yellow and cyan areas are obstacles, red areas are security zones (although it is not advisable, the wGO can still use them if strictly necessary) and gray areas are free zones.

Finally, a sample of the control results is given in the right part of Figure 11. It can be observed that the trajectory is stable while avoiding all the present obstacles.

5.2. wGO: User satisfaction survey

To provide some qualitative feedback on the operation of the wGO, a user survey was conducted. The questionnaire was divided into six main blocks. The three first blocks contained statements designed as a Likert scale (with four levels in the first two blocks and 10 levels in the third one), the fourth block had an open-ended question, the fifth block had only one question concerning the shopping frequency, and the last block considered some demographic information. The survey is as follows:

1. Block 1: What is your satisfaction level with the following wGO aspects? (four levels Likert scale: ''Highly unsatisfied'', ''Unsatisfied'', ''Satisfied'' and ''Very satisfied'')

   • Available space to move in the store

   • Use in comparison with the other alternatives (e.g. scooters)

   • Speed of the shopping process

   • Commodity during the shopping process

2. Block 2: Future Use (four levels Likert scale ranging from ''Unlikely'' to ''Very likely'')

   • Would you use the wGO again?

3. Block 3: General satisfaction (ten levels Likert scale ranging from ''Very bad'' to ''Very good'')

   • How do you evaluate the wGO?

4. Block 4: Open-ended question

   • What would you change in the wGO?

5. Block 5: How often do you go to the supermarket?

   1. Once a month

   2. Twice a month

   3. Once a week

   4. Twice a week

6. Block 6: Demographics

   • Sex

   • Age

The survey was answered by 78 people, with significantly more females (49) than males (29). The age distributions are as illustrated in the left side of Figure 12. Note that there are significantly more volunteers on the older age class (older than 45) and significantly less of the younger class (between 18 and 24). Also, in the age class of 25 to 34, most of the volunteers are females while in the class of people higher than 45 there is a significantly higher number of male users.

Figure 12.

Age and shopping frequency distributions.

Considering shopping frequency, most of the survey participants go to the supermarket as often as once a month or twice week (right side of Figure 12). As a curiosity, all the “once a month” shoppers are male.

Results for question of Block 1 are shown in Figure 13. No volunteer attributed score 1 to any aspect, and people rated highly all the wGO aspects (with average scores always above 3.3). While its commodity was the highest rated aspect (being the amount of “Very satisfied” answers significantly high), its space in store was the lowest rated one (being the amount of “Very satisfied” answers significantly low and the amount of “Satisfied” significantly high), pointing to the fact that there are still some slight improvements to be made. People in the age class of 18 to 24 are significantly less happy with the space within store than other age groups. When looking at commodity, people in the same age group of 18 to 24 are mostly “Satisfied” (being this rate significantly higher).

Figure 13.

Results for Block 1: satisfaction level with some of the wGO aspects.

When looking at the relationship of the shopping frequency with the speed, no “once a month shopper” was “Very satisfied” with the wGO speed. Conversely, “twice a week” shoppers were significantly “Very satisfied” with the speed, as illustrated in Figure 14.

Figure 14.

Shopping frequency versus wGO speed.

Similarly, when looking at the relationship of the shopping frequency with the commodity, most of the “once a month” shoppers are “Unsatisfied,” while “twice a week” shoppers were significantly “Very satisfied” (Figure 15).

Figure 15.

Shopping frequency versus wGO commodity.

Results for question of Blocks 2 and 3 are shown in Figure 16. People are very likely to use the wGO again (being the amount of “Very likely” answers significantly high). Within the “Very likely” future users, a significantly high number belongs to the higher than 45 age group.

Figure 16.

Results for Block 2: future use and Block 3: general satisfaction.

A significantly high number of people attribute the highest possible score on the general satisfaction scale.

When analyzing relationships between the several variables and the general satisfaction, it can be concluded that:

• User gender has no effect on general satisfaction.

• All the users in the age range of “18-24” gave a score of 8 to the general satisfaction question.

• “Once a month shoppers” attribute a value of 6 or 7 to the general satisfaction question.

• Among the “very satisfied” users with the Space in the store, 65% have “General satisfaction” of “10” (this is higher than the average for “10”). Conversely, among the “Unsatisfied” users with the Space in store factor, 29% have “General satisfaction” of “5” (this is higher than the average for “5”).

• Users that believe wGO compare poorly with other alternatives and also give low scores in the general satisfaction question. The value of the Pearson Correlation Coefficient for this relationship is 0.6472, which translates into a moderate-positive correlation.

• Among the users that are “Very satisfied” with the speed, 24% have “General satisfaction” of “9” (this is higher than the average for “9”).

• Less satisfied users with the wGO commodity also score lower on general satisfaction. The value of the Pearson Correlation Coefficient for this relationship is 0.6096, which translates into a moderate-positive correlation.

• Future use has a moderate-positive correlation with general satisfaction (Pearson Correlation Coefficient = 0.6014). Moreover, among the users who are “Unlikely” to use the wGO again, all of them have “General satisfaction” of “5” or less.

Concerning the open-ended question of Block 4, some limitations were pointed out, namely:

• limited space for the shopping items;

• low velocity;

• obstacle avoidance should be improved; and

• increase the robustness (“lose” the operator less times).

6. Conclusion

The wGO, an autonomous shopping cart, has been introduced. Experiments made in real scenarios are very encouraging, and a high-user satisfaction was observed. The participants on the user study demonstrated a comfortable behavior during the experiments as well as a very easy understanding of the robot’s operating system (especially, related with the perception and navigation). Comments like “My shopping was very fast!”, “In fact, it was a precious help!”, and “I think it is awesome, I will certainly use and recommend it!” were made by the volunteers.

Some problems, however, remain to be solved. One of them is the limited space for the shopping items. Concerning the robot’s behavior, low velocity and identification errors were mentioned.

Although the short-term application for the wGO is for commercial environment usage scenarios, several other applications are foreseen, for instance, at the shop floor of the manufacturing industry and logistics.

Acknowledgments

The authors would like to thank all the supporters, including TIC RISCO, Portugal Ventures, PME investiments, Finova, Compete, QREN, União Europeia Fundo Social Europeu, Mais Centro, Compete 2020, União Europeia Fundos Europeus Estruturais e de Investimento, sifIDe, inovaRIA, CEiiA, Fundão, UBI, Tente. We would also like to thank Portugal 2020, with the projects POCI-01-0247-FEDER-00.6398.