Perspective Chapter: European Robotics League – Benchmarking through Smart City Robot Competitions

Abstract

The SciRoc project, started in 2018, is an EU-H2020 funded project supporting the European Robotics League (ERL) and builds on the success of the EU-FP7/H2020 projects RoCKIn, euRathlon, EuRoC and ROCKEU2. The ERL is a framework for robot competitions currently consisting of three challenges: ERL Consumer, ERL Professional and ERL Emergency. These three challenge scenarios are set up in urban environments and converge every two years under one major tournament: the ERL Smart Cities Challenge. Smart cities are a new urban innovation paradigm promoting the use of advanced technologies to improve citizens’ quality of life. A key novelty of the SciRoc project is the ERL Smart Cities Challenge, which aims to show how robots will integrate into the cities of the future as physical agents. The SciRoc Project ran two such ERL Smart Cities Challenges, the first in Milton Keynes, UK (2019) and the second in Bologna, Italy (2021). In this chapter we evaluate the three challenges of the ERL, explain why the SciRoc project introduced a fourth challenge to bring robot benchmarking to Smart Cities and outline the process in conducting a Smart City event under the ERL umbrella. These innovations may pave the way for easier robotic benchmarking in the future.

1. Introduction

Technology has an increasing impact on all our lives, and as the population of the world becomes more predominantly urban, it is in cities that most of us will feel this impact. Two technologies which seem destined to shape our experience are Smart Cities and Robotics. Of course, for robots to be useful we need to trust that they will do their job properly and reliably, and this requires that we should be able to test them against standards, and see their performance with our own eyes.

Resulting from a series of initiatives funded by the European Union’s Framework 7 and Horizon 2020 programmes, the European Robotics League (ERL) has been organising robot competitions for robots throughout Europe. Competitions are based around benchmarking, and aim to give robot developers data to enable the comparison of different systems, algorithms and approaches to solving complex tasks [1]. The ERL originally consisted of three challenges, which were suitable for Industrial, Service, and Emergency Robots (the latter challenge required that robots would work together in air, land and sea to solve a simulated disaster similar to that experienced in Fukushima). We have renamed the Industrial challenge as ‘Professional’ and the Service challenge as ‘Consumer’ to recognise the fact these sectors are converging around human and robot co-working with a focus on human-robot interaction.

One factor which defines the ERL’s competition events is that they are most often in locations that are hidden away from the public such as robotics labs, test facilities, or disused ports. This distancing from the public is not deliberate and indeed not desired. Most often the competition events take place in locations which are determined by the need for accessible and safe indoor and outdoor environments that meet the needs of the benchmarking tasks at hand.

Surveys such as the Eurobarometer [2] have shown us that public attitudes towards robots are mixed, viewing them often with suspicion. It is unclear whether these attitudes are based on robotic fact or fiction; most people may not have had a direct experience of a robot. Since we wish to engage the public to raise issues around acceptance and desirability, and we wish to empower the people of Europe to direct robotic development, it seems clear that we should move our demonstrations of robots into venues that are easily accessible and where future use cases might involve people and robots working alongside each other.

It was for these reasons that the SciRoc project extended the European Robotics League to include the Robotics in Smart Cities Challenge. This allows teams to benchmark their robots and test their skills on a variety of tasks that one might find in a Smart City environment. Furthermore, teams need to interact with the Smart City data infrastructure while performing these tasks in order to demonstrate the value of the city host’s investment and its readiness for future technology.

2. ERL processes for benchmarking and competition

We follow the process of designing competitions that has been outlined in the RoCKIn project and visualised in Figure 1. One of the driving forces behind a competition is the formulation of a challenge. In the SciRoc context those challenges build the foundation of the three individual leagues:

Figure 1.

Competition design process.

ERL Professional Service Robots (ERL Professional): How can mobile manipulation robots support low-volume, high-variety manufacturing?

ERL Consumer Service Robots (ERL Consumer): How can mobile manipulation robots support (elderly) citizens in their domestic environment?

ERL Emergency Robots (ERL Emergency): How can multi-domain robotic systems support rescue teams in emergency situations?

In all of these leagues, we follow the same methodology as outlined in the RoCKIn project;

• Functionality benchmarks (FBMs), which evaluate the performance of robot modules dedicated to specific functionalities, in the context of experiments focused on such functionalities.

• Task benchmarks (TBMs), which assess the performance of integrated robot systems facing complex tasks that require the interaction of different functionalities.

SciRoc integrates all three leagues in a common scenario of a Smart City. This particular challenge revolves around the question of how autonomous robots can support citizens in various smart shopping tasks.

While the challenges frame the overall benchmarks and competitions they are too complex to be solved within a reasonable time frame like a single EU-funded project and too generic to be solved by individual researchers. Thus, as a next step, interesting scenarios are derived from the challenge. A scenario consists of the task to be solved, the environment in which this task will be executed and the robot that will solve the task. Each of those three categories is defined in terms of their features, that is, a specification of the possible or allowed variability.

From the perspective of a scientific experiment the features are free or fixed parameters. The definition of a scenario also gives rise to a list of (abstract) functionalities that a robot is required to have. Examples of such functionalities may include task planning, manipulation, grasping, visual object recognition, speech understanding or speech generation. It should be noted that the scenario definition purely describes an application domain, but, by design, is not meant to provide insights into a robot’s performance in solving a particular task in that domain. This latter aspect is addressed in the benchmark (competition) definition phase where the goal is to define tests that evaluate a robot’s performance. The first step is the definition of questions or hypotheses that should be addressed by the benchmark. Only the right questions allow constraint of the overall design space of a scenario to a competition scenario where few, explicitly chosen features can be varied. The feature variation characterises each benchmark test by a choice or instantiation of all relevant features. Three major aspects govern those choices:

• The specification aspect determines how the feature is described. For some features concrete properties and attributes can be asserted, for instance, in terms of values or enumerations (“imperative” specification). One example is the size of a room measured by width, length and height. In contrast, other features are better described via exemplifying representatives (“declarative” specification). An example is the specification of an object as a “coffee mug”.

• The temporal aspect determines at which point in time the feature is instantiated. The choices vary from the scenario specification (e.g. the types of areas), over the testbed setup (e.g. the concrete size of areas), before the benchmark experiment execution (e.g. the location of manipulable objects) to during the benchmark experiment execution (e.g. outdoor light conditions).

• The control aspect determines the manner in which the parameters are specified or fixed. The choices vary from concrete values (e.g. runtime of an experiment) over ranges (e.g. allowed minimum and maximum size of areas) to unspecified conditions (e.g. outdoor weather).

Given different feature variations their impact on the robot’s performance can be measured and compared via well-defined metrics. Similarly, benchmarks for the functionalities are defined. The steps to conduct the benchmark are described in the benchmark procedures. One artefact of particular interest that is compiled during this phase is the rulebook which communicates and clarifies the taken decisions and procedures to various stakeholders.

The next phase is the benchmark experiment execution where the tests are instantiated and robots solve the tasks in the real-world environment. The execution consists of the three activities (i) scenario setup, (ii) experiment control; and (iii) data collection. In the final phase, the resulting experimental data, also from multiple instantiations of benchmark experiments, is analysed to provide answers to the questions underlying the benchmark definition.

We now examine how the three European Robotics Leagues implement the same underlying methodology, and the ways in which this was expanded upon to create the Smart City Challenge.

3. ERL consumer robots

The European Robotics League Consumer Service Robots (ERL Consumer) is a research competition that aims at bringing together the benefits of scientific benchmarking with the attraction of scientific competitions in the realm of consumer service robotics. The objectives are to foster research in consumer service robotics for home applications and to raise public awareness of the current and future capabilities of such robot systems to meet societal challenges like healthy ageing and longer independent living.

Currently, ERL Consumer raises challenges in domestic environments that resemble similar challenges to be posed in smart city environments (e.g., shopping malls), such as interacting with humans, recognising and picking objects from shelves, bringing the objects to the human who requested it, and/or moving outside the home to populated areas, so as to prepare teams for the Smart City Tournaments. But ERL Consumer objectives can be adapted to other challenges, still within domestic environments or in other environments, e.g., hospitals.

All the ERL Consumer TBMs and FBMs stem from a User’s Story developed during the pioneer EU FP7 RoCKIn project [3]:

Granny Annie is an elderly person, who lives in an ordinary apartment. Granny Annie is suffering from typical problems of ageing people: she has some mobility constraints. She tires fast. She needs to have some physical exercise, though. She needs to take her medicine regularly. She must drink enough. She must obey her diet. She needs to observe her blood pressure and blood sugar regularly. She needs to take care of her pets. She wants to have a vivid social life and welcome friends in her apartment occasionally, but regularly. Sometimes she has days not feeling so well and needs to stay in bed. She still enjoys intellectual challenges and reads books, solves puzzles, and socialises a lot with friends.

For all these activities, ERL Consumer is looking into ways to support Granny Annie to live a full and independent life.

3.1 Environment description and testbeds

The ERL Consumer environment reflects an ordinary European apartment with all its environmental aspects, like walls, windows, doors, or blinds, as well as common household items, furniture and decoration. The apartment depicted in Figure 2 serves as a guideline. It has

• Rooms (accessible to the robot): living room (with windows, couch, two armchairs, one coffee table, and one TV table), dining room (with one glass top dining table and two dining chairs), kitchen (with one kitchen table and two chairs, kitchen cabinet with multiple drawers and wash sink, two wall mounted kitchen shelves), inside hallway (with one coat rack), bedroom (with one window, a double bed, two side tables, two table lamps and one large wardrobe with mirror).

• Spatial areas (inaccessible to the robot): outside hallway, bathroom, patio.

• Well levelled floor, uniform all over the testbed, but including carpets.

• Wooden walls, most of them 50 cm high, but including one, behind the kitchen, 200 cm high.

Figure 2.

ERL consumer testbed: On the left a real testbed during a RoCKIn@home tournament in 2015; on the right: The reference testbed as a simulated environment.

The furniture and available objects (e.g., glasses, forks, knifes, pillows, cups) were chosen to set up a long-term research program with challenges for robot navigation (e.g., mirror in the wardrobe, tables with metallic reflective legs) and perception (e.g., glass-top table, natural backgrounds).

In addition to furniture and decoration, the apartment is equipped with a computer network infrastructure:

• Server: a computer used to manage the network.

• Switch: an Ethernet switch used to connect all the networked devices.

• AP: An Access Point the mobile robot wirelessly connects to. Acts as a bridge between WLAN and LAN and provides access to a network of Ethernet cameras that provide perspectives of the home and of the outside hallway. Remote image acquisition is possible, and the camera parameters (frame rate, resolution, colour gains) can be changed over Ethernet.

Home automation embedded devices may be installed and are accessible within the apartment’s WLAN, e.g.,

• the lamps in the bedroom (e.g., on the bed stand) are accessible and controllable via network;

• the shutters on the bedroom or living room window are accessible and controllable via network.

Eight testbeds, certified to follow the standard specifications of this environment, are currently part of the ERL testbed network:

1. ISRoboNet@Home Test Bed, at the Institute for Systems and Robotics in Instituto Superior Técnico, U. Lisboa, Portugal.

2. ECHORD++‘s RIF @Peccioli, at The BioRobotics Institute, Scuola Superiore Sant’Anna, Peccioli, Italy.

3. Leon@Home Test Bed, at the Universidad de León, León, Spain.

4. BRL Anchor Personalised Assisted Living Studio, at the Bristol Robotics Laboratory, University of the West of England, Bristol, UK.

5. PAL Robotics Assisted House, at PAL Robotics S.L., Barcelona, Spain.

6. Heriot-Watt@Home Test Bed, at Heriot-Watt University, Edinburgh, UK.

7. IDEAAL Living Lab, at the OFFIS Institute for Information Technology, Oldenburg, Germany.

8. Cobot Maker Space Living Space, at the University of Nottingham, UK.

ERL Consumer tournaments take place in these testbeds. Some of the testbeds include a Motion Capture (MoCap) System, which enables tracking with high accuracy robots, people and object locations. MoCaps are mostly used in the FBMs.

3.2 Task benchmarks

Currently, ERL Consumer includes 4 Task Benchmarks:

• Getting to know my home: The robot must detect new changes in the environment, and update a semantic map of the apartment, within a limited time frame. The task is performed by the robot autonomously, though it may include moments of symbiotic interaction with a user in the apartment, e.g., to learn more about an object or a location. At the end of the knowledge acquisition phase, the robot must show the understanding of the new environment, namely by addressing changed objects and furniture locations, handling one of the changed objects between two furniture locations, one of them with its original location changed.

• Welcoming visitors: The robot needs to handle a set of known and unknown visitors, who arrive individually at the home entrance in random sequence. The robot must correctly recognise (using the networked camera over the outside hallway) the known visitors and interact with the unknown visitors to identify them and understand their purpose of visit. The robot must then perform a set of visitor-specific behaviours that could range from manipulating and delivering of objects to guiding and following the visitors.

• Catering for Granny Annie’s comfort: This task aims at providing general purpose requests of Granny Annie inside the apartment. It focuses on the integration of different robot abilities such as human-robot interaction, navigation, and robot-object interaction. The robot is required to understand the actions requested by speech (such as finding, picking, and bringing an object to Granny Annie) and execute them accordingly.

• Visiting my home: This TBM focuses on safe navigation in dynamic environments, people perception, obstacle avoidance and tracking and following a human. In this task, the robot should visit a set of predefined rooms, to perceive and count the number of people in each room, while avoiding and/or interacting with different obstacles based on the nature of the obstacle. Furthermore, the robot must interact and follow a previously unknown person outside the arena through a small crowd and then guide that person back to the arena.

3.3 Functionality benchmarks

Currently, ERL Consumer includes 6 Functionality Benchmarks [4, 5]:

• Object Perception: Evaluates the ability of a robot to recognise and localise a wide range of objects. A set of objects, selected from the list of ERL Consumer items, is positioned, one at the time, on a table located directly in front of the robot. For each object presented, the robot must perform the following activities: i) Object detection: perception of the presence of an object on the table and association between the perceived object and one of the object classes. ii) Object recognition: association between the perceived object and one of the object instances belonging to the selected class. iii) Object localization: estimation of the 3D pose of the perceived object with respect to the surface of the table (ground-truth provided by the MoCap).

• Navigation: Evaluates the ability of a robot to correctly, safely, and autonomously, navigate in an ordinary apartment, including: the navigation within furniture, walls, and doors, in a previously mapped area; avoiding collisions with different types of unknown obstacles, in unknown positions (not previously mapped); and navigating in the presence of people in the arena.

• Speech Understanding: Evaluates the ability of a robot to understand speech commands that a user gives to a robot in a consumer environment, including all the related issues, such as background noise caused by other ongoing activities and by the robot motion. A set of spoken sentences, recorded in different environments, is broadcasted through a speaker. The robot needs to interpret the commands and produce an output according to a defined representation.

• People Perception: Evaluates the ability of a robot to locate and recognise humans. Similarly to object perception, the robot must recognise a person who is standing inside a target area and to estimate the location of this person.

• Person Following: Evaluates the ability of a robot to effectively follow a human target around and through obstacles and a crowd of walking people. The benchmark requires that the robot accompanies a human target and always maintain a desired distance with this person.

• Grasping and Manipulation: Evaluates the ability of a robot to correctly grasp and manipulate objects. In particular, it assesses the object picking and placing capabilities of robots suitable for many consumer applications such as setting up a dining table in domestic environments.

4. ERL professional service robots

ERL Professional Service Robots (ERL-PSR) League is focused on the major challenges addressed by H2020: industrial robots addressing the flexible factories of the future and modern automation issues.

Greater automation in broader application domains than today is essential to ensure European industry remains competitive, production processes are flexible to custom demands and factories can operate safely in harsh or dangerous environments. In the ERL-PSR competition, robots will assist in filling stocks in a department store. The task includes locating, picking, transporting and placing them in the proper shelfs. The combination of human versatility and reliability of mobile robots will optimise the entire process. The ERL-PSR competition is looking to make these innovative and flexible manufacturing systems, such as that required by the smart factory, a reality. This is the inspiration behind the challenge and the following scenario description.

In this version of the ERL-PSR we focus on the problem of picking products from a shelf and placing them in the shopping basket. The domain is a common department store, where the robot helps to arrange the inventory of the department store on the shelves. The main functionality tested in this episode is mobile manipulation using an autonomous robot. In recent years, mobile manipulation has become a problem of interest among researchers due to the variety and complexity of challenges and robot capabilities that are involved. For instance, in a grocery store setting, there are so many complexities that a robotic hand or gripper should take into account, such as handling objects of different weight, sizes, shapes, texture and compliance. Picking up a bag of pet food is very different from picking up bananas or cucumbers. Different robotic competitions [6, 7, 8, 9, 10, 11] have picking and packing problem, aiming at 1) measuring the performance of these complex systems, 2) defining metrics and assessing benchmarking criteria, 3) fostering research and development of new approaches and technologies, and 4) creating awareness and drawing more attention from the general public to robotics.

4.1 Environment description and testbeds

The ERL-PSR testbed consists of different elements which include the arena, networked devices, department store shelves and the products in the store. The robot can communicate with the department store inventory management system and to other networked devices. The robot receives tasks to perform from the inventory management system.

The following set of scenario specifications must be met by the ERL-PSR environment.

The environment can consists of various numbers of spatial areas:

1. rows of shelves

2. rows of workstations

Figure 3-a shows an example of these areas in the ERL-PSR environment. The spatial areas extend beyond the space occupied by the respective workstations or objects and include the surrounding area as well.

Figure 3.

ERL PSR different testbeds. a) ERL PSR testbed, Sankt Augustin, Germany. b) Testbed with objects and shelves.

All spatial areas are located on the same level, except where specified otherwise. There are no stairs in the environment. The environment is a replica of department store with aisles of shelves containing different objects. The environment has a boundary. The precise dimensions and the arrangement of the spatial areas are not predefined, but estimated sizes are given. The estimated sizes of the spatial areas are as follows: workstations 2 m × 2 m and shelves 5 m × 0, 5 m. The bounding box of the environment has a minimum area of 16m² and a maximum area of 100m². More space is used, when areas and workstations are doubled for teams working in parallel. Workstations are used as storage areas for objects. They may be accessible from different locations, i.e. it might be possible to reach a workstation from two or more sides. Additionally, there are workstations of different heights present in the environment, ranging from 0 cm up to 15 cm. If a workstation has a height of 0 cm, a tape will mark the area (see Figure 3). The tape will be taped on the floor and is blue/white striped. This tape may be crossed by the robot and does not count as a collision.

4.1.1 Objects in the environment

The objects to be manipulated will be selected by the organiser for each tournament. The following lists describe the different categories of objects that can be used by the organiser during a tournament depending upon the scenario being chosen. The different categories of objects are (Figure 4):

1. RoboCup objects

2. Ocado objects

3. Chocolate objects

Figure 4.

Objects in environment.

4.1.2 Robots and teams

A competing team can use single or multiple robots. The robots are not required to be certified for industrial usage. The robot should have at least the following capabilities:

• autonomous navigation.

• grasping capability wireless communication capability(802.11 version 5Ghz).

• safe for public usage.

The different subsystems of the robot should work in the environment and manipulate the objects specified in the rulebook. The teams can use any sensor available in the market provided they are safe to use with humans and have the corresponding certification. The teams are not allowed to modify the arena which is finalised by the organiser. The teams are allowed to use any internal communication protocol. The robot shall pass through a safety test before it is admitted in the arena.

4.2 Task benchmarks

4.2.1 Fill a box with parts

This task is one of the primary task in a department store where customers fill their basket with products from shelves. The robots have to deal with flexible task specifications, especially concerning information about object constellations in source and target locations, and task constraints such as limits on the number of objects allowed to be carried simultaneously, etc. The robot has to pick up several parts from different source locations and deliver them to several destination locations.

4.3 Functionality benchmarks

The task benchmark was subdivided into multiple Functionality Benchmarks. The list of benchmarks executed are:

• Object Detection: this functionality benchmark evaluates robot capabilities of locating an object at a given location. One of the common tasks for service and industrial robots is to locate the object which can possibly be placed at a particular location. In addition to that, several secondary objects and decoys may be present at the location too. The robot is required to find particular objects among a set of objects and decoys. The target object is either included or not depending on the variation for every trial.

• Manipulation Pick: this functionality benchmark evaluates the grasping capability of the robot. The robot has to identify the object in front of it and then attempt to grasp it and pick it up. Once the object has been picked up the robot has to notify.

• Manipulation Place: this functionality benchmark evaluates the placing capability of the robot. Based on the task different objects have to be placed in different orientation. The robot based on the task has to identify how to place the object and then notify about the status of the placement.

• Exploration: this benchmark evaluates the robot exploration and navigation capability. The test benchmarks the navigation capability of the robot to simultaneously explore the environment and perceive the environment for a particular object present in the environment.

5. ERL emergency robots

ERL Emergency tournaments challenge multi-domain teams of marine, land and aerial robots with search and rescue tasks, inspired by a disaster response scenario [6]. Following the experience of the euRathlon 2015 Grand Challenge [7], the ERL Emergency competition started in 2016 and culminated in the ERL 2017 Major Tournament [8] held at the Tor del Sale power plant site of Piombino, on the coast of Tuscany (Italy). The inspiration came from the 2011 Fukushima accident [9]. Land, marine and aerial robots cooperated in a disaster scenario organised at a real power plant, and were required to survey the accident area, to identify and help missing workers (mannequins) and to intervene to stem a leak by closing valves both inside the building containing the machine room of the plant and underwater.

Similar concepts were followed in the SciRoc project, and in 2018 and 2019, ERL Emergency moved to a model of local tournaments with two-domain competitions: land + underwater and land + aerial robots competitions. The Centre for Maritime Research and Experimentation (CMRE) organised two editions of the land + underwater robot competition at its premises in La Spezia (Italy) in 2018 and 2019. The land + aerial competition was hosted by the Advanced Centre for Aerospace Technologies (CATEC) at Seville (Spain) in 2019. These events were a preparation for the Smart City events, where only the tasks of the ERL Emergency involving the aerial robots were present.

Over the years, ERL Emergency has proposed team tasks which required advanced perception skills (to locate numbers positioned underwater, underwater pipes, or persons for first-aid kit deliveries using aerial robots), intertwined with autonomy and cooperation capabilities in realistic scenarios. Multi-domain cooperation has been specifically searched and pushed. Robots are required to be able to accomplish adaptive missions, for instance executing different actions on the basis of different sensed conditions. The increased attention to autonomy and cooperation has also been accompanied by an increasingly metrological attention to benchmarking the robot performance. This has been achieved by separating the Task Benchmarks from the Functionality Benchmarks, following the general trend of the ERL evaluation framework [10].

5.1 Land+underwater robots events held at CMRE sea water basin

In the SciRoc project, CMRE conducted two ERL Emergency local tournaments, in 2018 and 2019, challenging multi-domain teams composed of an underwater and a land robot with a scenario of emergency response to a simulated explosion in a harbour. The areas of operations included a building with the surrounding outdoor space for land robots, and the CMRE sea water basin for underwater robots (see Figure 5). The marine robots were challenged with the realistic conditions typical of at-sea operations (water salinity, changing light conditions, waves and tides).

Figure 5.

(left) the Feelhippo AUV from the UNIFI robotics team in action. (Centre) the yellow pipe structure with the manipulation console used in the ERL emergency competitions. (right) a multi-beam mosaic by team Tomkyle of the competition area. The different objects of potential interest (buoys, pipeline assembly structures, the gate are visible).

The scenario represented an accident of a yacht in a harbour. The vessel clashed on the dock causing an explosion. The accident affected the area around the harbour, both outdoor and a building. The land robots (unmanned ground vehicles – UGVs) were tasked to survey the outdoor area, localise pipes and find a missing worker (represented by a mannequin). The robots were requested to deliver a first-aid kit in proximity to the mannequin. Successively, inside the building, the robots were required to localise and close a valve and to find a canister. The specific valve to be closed was communicated to the UGV by the underwater robot of the same team. UGVs needed to transport the canister from inside the building to a simulated outdoor fire location. Autonomous underwater vehicles (AUVs) had to pass through a validation gate - composed of two submerged buoys, survey an area to detect a missing person underwater (a realistic mannequin) and localise an emitting pinger (see Figure 5). They also were required to inspect a pipeline structure and localise a damage (represented by a marker), at the same time reporting its shape and size. Underwater robots had to be autonomous, which means that all navigation and perception tasks were needed to be accomplished autonomously as well. Buoys of different colours were to be identified, localised and their colour recognised. The robots had to perform a different action, depending on the buoy colour: as an example, turning in a clockwise circle around the buoy or stopping for 30 seconds increasing the depth. The objective was to push teams to integrate perception with adaptive and reactive mission planning in a realistic scenario such as presented in the CMRE water basin. Here the changing and real conditions, such as the limited visibility underwater, created severe difficulties for object recognition by robots, even in the case the buoys were bright orange or red in colour.

Our experience suggests that one of the important aspects in real-world competitions is the possibility to guarantee teams sufficient time for practicing and tuning their systems to the changing and complex environmental conditions. This is especially true for marine robots, since teams may have difficulties to have access to sea waters for testing before the event. To allow teams to test, a practice arena was prepared.

As a way to increase the level of the challenge, the size of the objects to be detected was reduced with respect to previous competitions. Some of the tasks and functionalities were common to the ERL Emergency Local Tournament in Seville (land + aerial robots), such as the object recognition or map building functionalities. This allowed for a similar comparison of results in both competitions for a given Functionality Benchmark (FBM) (e.g. object detection). The same kind of objects of potential interest were used so the Object Detection FBM could be evaluated in both competitions. Some of the tasks were required to be completed with robots operating in a fully autonomous way, while others could be accomplished remotely controlled (e.g. with the assistance of an operator). However, no manual perception tasks were allowed. All object detections had to be either autonomous (real-time by the robot) or automatic (offline by a computer).

In both events, a mix of well-experienced teams and new teams attended the competitions. In 2018, five teams participated (three for marine domain, one for land and one deploying robots for both domains), while in 2019 [11], the number of teams increased with the participation of seven teams (four for marine domain, two for land domain and one with both segments).

Results highlight the improvement of team performance over the years, especially for teams which succeeded in participating in multiple editions of our events. Our policy to increase the task difficulty year after year led in fact the teams to increase their technical and, above all, management skills, thanks to the gained experience from their unavoidable errors and from the interactions with other entries. For these reasons, it is important to guarantee a continuity in the organisation of the competitions, especially when robots have to handle real-world scenarios in non-controllable conditions, which dramatically increase the task difficulties.

5.2 Land+aerial robots events held at CATEC

CATEC organised an ERL Emergency local tournament in February 2019, targeting aerial and land robots working in an outdoor/indoor environment [12]. The scenario was an earthquake in an industrial area near a factory building, where a robotic team composed of land (UGV) and air robots (UAV) had to intervene. The priorities are to discover and assist missing people, and determine if the building has suffered any serious damage. The robots have to search for missing workers (one outdoors and one indoors), find them as soon as possible and deploy an emergency kit to both of them. Besides, the robots must check the state of the building after the earthquake, for which a detailed map is required to assess the safety of the area (see Figure 6).

Figure 6.

(left) the UAV and UGV from team LARICS in action, reaching the outdoor missing worker. (Centre) the rover UGV and the UAV from team raptors looking for the indoor missing worker. (right) top view of a 3D map of the competition area by team LARICS. The trajectory followed by the robot is shown in blue.

As mentioned before, some of the functionalities were shared with the land + underwater robot competitions. In this case, the same kind of markers were used as objects to be recognised, so the Object Detection FBM could be properly evaluated in both competitions. From the starting points, land and aerial robots must inspect the area, autonomously detecting and avoiding obstacles while traversing the competition arena. Then, they needed to find a suitable entrance that could be used to enter the building, using similar markers as the ones used for object recognition. While accessing the building, robot navigation must deal with transitioning from an outdoor to an indoor environment. Once inside, robots had to build a detailed map of the scenario, either 2D or 3D according to their sensor suite. During the whole mission execution, as soon as a missing worker was found (represented by a mannequin), the robot provided a first-aid kit as soon as possible.

The setup for the competition was an area of approximately 200 m x 30 m free of obstacles, in an area where operating UAVs in Visual Line Of Sight (VLOS) is not forbidden according to regulations in Spain. The building was represented with a 20 m x 20 m marquee where the aerial and ground robots need to access. Static obstacles (e.g. stones, holes, vegetation...) and dynamic obstacles (e.g. birds...) were expected in the outdoor area, as well as loss of Wi-Fi signal. As with any outdoor competition, there was also the possibility of rain, wind, dust and muddy areas, but weather was generally fine during that week. A practice area was also set up to provide teams enough space for testing their systems before deploying them in the competition arena.

5.3 Task benchmarks

TBMs in ERL Emergency require the cooperation of robots in different domains. This often requires teams to work together.

5.3.1 Yacht accident in the harbour (land+sea)

This two-domain task benchmark is focused on acquiring knowledge about the environment and its explicit representation; and to cooperate between domains to search for the missing workers and give them assistance. The ground and underwater robots are required to understand the changes in the environment and interact with it either through cooperation between them (autonomous robot-robot) or their operators (human-robot interaction) or with a mixed approach. A minimum of one land robot and one underwater robot is required to participate in this task.

The motivating scenario is as follows;

An accident occurs in the harbour when a yacht arriving at the pier damages a gas pipeline which leaks and causes an explosion. This also affects the building containing the pipeline section on land and people that were in the docks area are dispersed. The emergency response team arrives soon but members must maintain a safe distance from the fire. For this reason, the use of robotic vehicles is essential. A robotics team composed of land (UGV) and underwater robots (AUV) is ready to intervene.

Three missions can be undertaken.

• Mission-A: Search for missing workers. Locate and help missing workers, removing rubble trapping them.

• Mission-B: Reconnaissance and environmental survey. Provide situational information to the emergency team, exploring the damaged building and underwater parts of the damaged pier.

• Mission-C: Pipe inspection and stopping the fire. The correct valves must be turned to stop a simulated fire, both underwater and in the damaged building.

5.3.2 Emergency in a building (land+air)

This two-domain task benchmark is focused on acquiring knowledge about the environment and its explicit representation; and to cooperate between domains to search for the missing workers and give them assistance. The ground and aerial robots are required to understand the changes in the environment and interact with it either through cooperation between them (autonomous robot-robot) or their operators (human-robot interaction) or with a mixed approach. A minimum of one land robot and one aerial robot is required to participate in this task.

The motivating scenario is as follows…

An earthquake has occurred in an industrial area near a factory building, and two workers are missing (one inside, and one outside). The emergency response team arrives soon but members of the response team must maintain a safe distance from the building. For this reason, the use of robotic vehicles is essential. A robotic team composed of land (UGV) and air robots (UAV) must discover any missing people and whether the building has suffered any serious damage. Robots have to find wrokers as soon as possible and deploy an emergency kit to each. Robots must check the state of the building after the earthquake and create a detailed map for the emergency team.

This task benchmark comprises two missions’ goals:

• Mission-A: Delivery of emergency kits to missing workers. The two workers must be found as quickly as possible, and first aid kits deployed.

• Mission-B: Mapping for safety assessment of the building. Both land and air robots must collaborate in creating a 2D or 3D map of the space inside.

5.4 Functionality benchmarks

• 2D Mapping Functionality (Land): This measures a land robot’s ability to explore a 2D area while visiting a number of waypoints, and is scored according to map coverage and accuracy of the waypoint locations.

• Mapping Functionality (Air): This measures an aerial robot’s ability to explore the competition area while visiting a number of waypoints, and is scored according to map coverage and accuracy of the waypoint locations.

• Vertical Wall Mapping Functionality (Sea): This assesses the capabilities of marine robots in extracting information about a specific wall of the damaged pier. The identity of the wall to be explored will be communicated to the underwater robot by the ground robot. After the exploration, teams must provide a 2D or 3D map of the designated wall along with several measurements calculated from the map.

• Object Recognition Functionality (Land): This assesses the capabilities of ground robots to recognise objects that might be found in an outdoor and indoor disaster response environment. The benchmark requires that robots detect Objects of Potential Interest (OPIs) and identify the type of each object found, and is scored according to the precision and accuracy of identifying and locating the objects.

• Object Recognition Functionality (Sea): This assesses the capabilities of underwater robots in extracting information about observed objects. The objects to be recognised in this FBM are the orange buoys that act as obstacles. Each obstacle buoy is identified by a black number, from 1 to 4, and scores depend on the number of buoys correctly detected, identified, and accurately located.

• Object Recognition Functionality (Air): This assesses the capabilities of aerial robots to recognise objects that might be found in an outdoor and indoor disaster response environment. The benchmark requires that robots detect Objects of Potential Interest (OPIs) and identify the type of each object found, and is scored according to the precision and accuracy of identifying and locating the objects.

6. Smart cities competitions

The label ‘Smart Cities’ is used with increasing frequency with a main focus on the use of engineering approaches within the built environment to improve citizens’ quality of life. These visions present the city as an entity, which gathers and processes data to decision makers and infrastructure, and which offers data services to enable and improve services offered by other stakeholders such as private companies. Since the city senses its environment, processes this data, and takes action through effectors like traffic signals, the Smart City resembles a Robot. We might expect to find that Smart Cities and Robots gain mutual benefits from their integration, and also find some common problems which both must overcome.

The world’s population is becoming more urban, so it is likely that most human-robot interactions will take place in Smart Cities. Therefore, it seems that any ethical challenges that arise from the mutualistic interactions of Robots and Smart Cities will have a significant impact for these future citizens. How can robots act best to sustainably promote our happiness and prosperity?

When we first stated our intention to bring a scientific robot competition to the heart of a major public space, our ambition was to address this question by stimulating public discussions based on robotic facts about the roles which robots might play in the future in their smart city. Our motivation was to demonstrate that robots and smart cities are natural partners which add mutual value, and in doing so to showcase the state of the art in European Robotics to benefit the teams, sponsors, and wider community. We were able to achieve our aims through the successful delivery of two ERL Smart Cities Challenges to date. The first in Milton Keynes, UK in September 2019 and the second in Bologna, Italy in September 2021.

Our intention was to provide for the sustainable future of the ERL beyond the H2020 funded SciRoc project, and as part of this our first competition was led by a project partner working with the city, and with some funding from the project budget, while the second was led by a third party and entirely self-funded. This demonstrates that this mode of public engagement with benchmarking through competition can be repeated using this model in the future.

In the following sections we explain how and why the TBMs and FBMs of the ERl were adapted and extended to create the Smart City Episodes. We also explain how the critical elements which need to be considered for a city host looking to implement a future Smart City event, or similar urban robotic benchmarking challenge.

6.1 Milton Keynes, 2019

The first Smart City event took place at the Milton Keynes Centre: MK shopping mall over five days, providing teams the opportunity to compete in five different episode scenarios. This public space attracts on average over half a million people each week, and many of these people paused to watch the robots as they competed. The event’s sponsors included PAL Robotics and Ocado, and attracted 11 international robotics teams from different Universities throughout Europe.

Allied events included a Symposium to debate the risks and opportunities associated with the emergence of a “hybrid society”, and a Workshop on the Evaluation and Benchmarking of Human-Centered AI Systems. This public engagement helped foster public debate and equip citizens to be engaged active stakeholders in their city’s future.

6.2 Bologna, 2021

The second Smart City event took place at Palazzo Re Enzo, in the heart of Bologna. In contrast to the first Smart City event in Milton Keynes, this competition was organised by the host city, which provided all the resources to run the event, assisted by technical and scientific support from the SciRoc Consortium to set up and run the five episodes.

The implementation of the competition had to face significant uncertainty caused by the Covid-19 pandemic. Challenges arising through unforeseen travel restrictions were addressed with innovative technical solutions in terms of the arrangements of the competition, to allow teams and robots to compete both locally and remotely.

Such measures included the introduction of a simulation environment for a service robots scenario; remote participation on a physical robot made available on site; introduction of a novel challenge related to sign language that had outstanding success within the deaf community; synergy with the H2020 Eurobench project aiming at implementing benchmarks for robots; design of a new challenge for emergency robots; and finally distributed execution of the competition with both on site and remote participants performing the tests at their own labs.

6.3 Hosting a Smart City event

In this section we briefly describe the process of hosting a SciRoc Smart City competition in the hope that this assists readers interested in running a similar benchmarking event, or hosting a future Smart City Event.

6.3.1 The selection process

Smart City hosts are determined through an open call process. Interested candidates can register their interest in response to the call and are required to submit a proposal for the event, and complete an interview with the selection committee. The selection committee, comprising members selected to remove conflicts of interest, determined the host through a final vote.

6.3.2 Competition design

The competition is divided into a series of robotic challenges, referred to as episodes. This represents an innovation of the structure of the competition, based on Functionality Benchmarks (FBMs) and Task Benchmarks (TBMs). Episodes are intended to be an intermediate challenge between the local tournaments’ FBMs and the TBMs. Even though the methodology and the approach to the competition are de facto unchanged, the proposed format based on episodes is designed to make a step forward towards their application in a realistic scenario, to allow for a better communication towards the general public and to lower the entry level for the competing teams.

An episode places a given functionality, tested during a specific FBM in one of the other ERL challenges, into a more social and operational context, while limiting the amount of effort needed for their development.

Episodes are organised into categories depending on the task to achieve and the type of robots involved. We group them into three categories:

1. HRI & Mobility: all episodes involving robots able to show social behaviours, such as verbally interacting with (human) customers or navigating respecting proxemics, in line with the current ERL-Consumer. These are meant for any robot (wheeled, or legged) with navigational and verbal communication.

2. Manipulation: this category includes all episodes requiring robots to achieve manipulation tasks, applying ERL-Professional services to the smart city context. Episodes here are meant for any robot able to navigate and equipped with arms and effectors for the manipulation of objects.

3. Emergency: the last category comprehends tasks and challenges addressed by small VTOL aerial robots, along the lines of the ERL-Emergency. Any VTOL aerial platform able to carry and deliver items in specific locations, navigate in outdoor and indoor spaces, detect and avoid obstacles can take part in episodes of this category.

6.3.3 Funding

The event is primarily funded through sponsorship funding, although it is expected that the host city will make their own contribution. For example, at the 2021 event in Bologna, the Municipality allowed the use of the competition venue Palazzo Re Enzo at no cost. This contribution was worth 65,000 EURO in kind. The University of Bologna also contributed 36,000 EURO in cash sponsorship towards the event.

6.3.4 Logistics

The host city is responsible for organising the venue for the event. When selecting the venue, organisers should consider that the location must be able to accommodate a number of factors, including: infrastructure for a minimum of five episodes, accessibility for teams to move robots around the venue, the potential use of drones within the venue, secure storage areas for robots and equipment, to name a few.

In addition to the competition venue, the host city should consider the following requirements for the functioning of the event: robot transportation, accommodation, financial support for teams, co-located events, the organising team and roles required within and the data hub.

6.3.5 The data hub

Central to emerging Smart Cities are online platforms for data sharing and reuse, which are normally called Data Hubs. Such Data Hubs have two purposes. They provide static datasets, such as demographic data which changes slowly, and dynamic live data gathered from sensors deployed within the city. The latter could include data about traffic flows, the environment, and the movement of people and robots, etc.

The MK Data Hub is a state of the art computational infrastructure, which supports the acquisition and management of city data. Specifically, it provides both a catalogue of several hundred data sources, as well as a development environment to facilitate the creation of data-intensive applications. The MK Data Hub played a key role in the 2019 SciRoc event in Milton Keynes by allowing us to simulate the diversity of systems with which robots were asked to interact throughout the competition.

Going forward, this approach could contribute a valuable additional output of robot competitions. First, by enabling the dynamic simulation of the environment in which the robots operate, the MK Data Hub introduced a contextual element in the benchmarks.

Second, a Data Hub environment provides the opportunity to record a large amount of heterogeneous information about the robots’ behaviour. Such information can be reused (a) to enable a deeper analysis of the shortcomings of robots, (b) to compare the performance of the same robot in different trials, and in different competitions among the years, as well as (c) for archival purposes. Ultimately, recording robot messages and analysing their behaviour may constitute the first step towards benchmarking elements such as self-awareness and deliberation capabilities.

6.3.6 Teams

Teams are recruited through an open call process, which is disseminated via the SciRoc website and social platforms, mailing lists, and by the SciRoc Consortium within academic institutions and the Robotics community. All previous year’s teams are also contacted directly to notify them of the open call.

6.4 Scientific contributions

SciRoc had the ambition to provide a testbed for experimental evaluation of Human-Robot Interaction (HRI) approaches. In fact, competitions offer a unique opportunity to create interaction scenarios for evaluating the performances of different approaches to human-robot interaction. In this respect, the work carried out in SciRoc had an impact on the HRI research community: the results of the experimental evaluation carried out in the SciRoc episodes, specifically designed for this purpose, have been published in the top venues for HRI research. Specifically, the Elevator episode of SciRoc 1, where the robot had to interact with users in taking an elevator, was accompanied by the creation of an ad hoc questionnaire [13] and the data collected during the competition have been used to demonstrate the potential for carrying out experimental studies within robotic competitions [14, 15]. Moreover, the Sign Language episode of SciRoc 2, where the robot was supposed to interpret and produce phrases of the Italian sign language, was accomplished with an unprecedented collaboration and impact with the deaf people community [16].

6.5 The continuation of Smart City events in the future

Results from teams are encouraging, and successful attempts at challenges of annually increasing difficulty suggest that teams have grown in capability through their repeated engagement with the benchmarking process. We have underlined the importance of continuing the organisation of such events for supporting the growth of teams and research groups, providing them with an annual real-world ground where to test their systems. This is especially true after the lockdown caused by the COVID19 pandemic, which increased the difficulties for teams to access real-world training areas (in particular marine sites).

Although the SciRoc project will finish in 2022, we are working to continue the ERL and the Smart City Events after the project’s close. The euRobotics international non-profit association will adopt the Smart City Events and continue to run them through an open call process, as with our previous Bologna event. It is intended that the design of the SciRoc events are responsive to the needs of the city stakeholders and, to some extent, the events are designed in order to support the communications and ambitions of the host city. We are therefore confident that city partners will continue to respond to our open call, to host Smart City events in the future.

The SciRoc Smart City Events saw collaboration with the H2020 funded EUROBENCH project to extend its work on benchmarking humanoids, and the H2020 METRICS project has been adopted within the ERL brand, following ERL benchmarking methodology. We hope that this will be repeated in the future, and anticipate new ERLs. Future Smart City competitions will provide a venue for more European Robotics projects to showcase their outputs to the public.

7. Conclusions

In this chapter we presented the SciRoc project which continued the work of the ERL and extended it to Smart City Events. We showed how the common TBM & FBM approach of the ERL was adapted and extended to enable these events to increase the value of the benchmarking endeavour through a massive increase in public engagement. The Smart City events delivered value to multiple stakeholder groups;

• Municipalities showed their city to be a future-ready venue to attract technological innovation and investment, and demonstrated the value of their smart city infrastructure

• Citizens and End Users saw robots performing relatable and believable tasks in accessible environments, which enabled fact-based discussions of fears and ambitions

• Host Institutions demonstrated their value in the political and economic ecosystems of their region, and showcased their capabilities to a wide and diverse audience

• Sponsors promoted their brand regionally and nationally. The value they gained beyond this depended on the nature of their business, for example, by building their network of top-quality robotics researchers

• Teams tested their robots, demonstrated their skills, and had fun while they did it

• Researchers extended the reach of their public engagement, developed and tested new benchmarking techniques and ways of working, and strengthened their networks

Bringing Robotics Benchmarking into the public arena in this way is important, we feel. For robots to be useful, they must be trusted, and for this trust to grow, real robots must be seen and their capabilities assessed.

Due to the restrictions placed upon us by the Covid-19 pandemic and the associated restrictions in travel and public assembly, we had to find new ways of working which would allow the teams flexibility in the ways they interacted with the Smart City Episodes. These methods may provide a basis for future development of benchmarking, and point the way towards a mixture of physical and virtual assessment.

As the project comes to an end, we are happy that we have surpassed our objectives, overcome unforeseen challenges, and that we have prepared the way for the continuation of the ERL through the support of the European Scientific Community and other stakeholders in the years to come. We hope that the trend to harmonise benchmarking under the ERL umbrella continues, and that the Smart City Events provide an arena for more European Projects to meet the public and therefore advance the successful and safe use of robotics within Europe and beyond.