Pedro U. Lima

By Giulio Fontana, Matteo Matteucci, Francesco Amigoni, Viola Schiaffonati, Andrea Bonarini and Pedro U. Lima

The main innovation brought forth by the European Project RoCKIn is the definition, implementation and application to an actual robot competition of the novel paradigm of benchmarking through competitions. By doing so, RoCKIn set in motion an evolutionary process to transform robot competitions from successful showcases with limited scientific impact into benchmarking tools for the consistent and objective evaluation of the performance of autonomous robot systems. Our work began by revisiting, in the light of the features and limitations of a competition setting, the very foundations of the scientific method; then we built on these by designing a novel type of competitions where the concepts of benchmark and objective performance metrics are the key points; finally, we arrived to the implementation of such concepts in the form of a real-world robot competition. This chapter describes the above process, explaining how each of its several aspects (theoretical, technical, procedural) has been tackled by RoCKIn. Special attention will be devoted to the problems of defining performance metrics and of capturing the ground truth needed to reliably assess robot perceptions and actions.

Part of the book: RoCKIn

By Matthew Studley, Sarah Carter, Francisco J. Perez-Grau, Antidio Viguria Jiménez, Gabriele Ferri, Fausto Ferreira, Deebul Nair, Sven Schneider, Paul G. Plöger, Pedro U. Lima, Meysam Basiri, Gerhard K. Kraetzschmar, Daniele Nardi, Lun Wang, Emanuele Antonioni, Vincenzo Suriani and Luca Iocchi

The SciRoc project, started in 2018, is an EU-H2020 funded project supporting the European Robotics League (ERL) and builds on the success of the EU-FP7/H2020 projects RoCKIn, euRathlon, EuRoC and ROCKEU2. The ERL is a framework for robot competitions currently consisting of three challenges: ERL Consumer, ERL Professional and ERL Emergency. These three challenge scenarios are set up in urban environments and converge every two years under one major tournament: the ERL Smart Cities Challenge. Smart cities are a new urban innovation paradigm promoting the use of advanced technologies to improve citizens’ quality of life. A key novelty of the SciRoc project is the ERL Smart Cities Challenge, which aims to show how robots will integrate into the cities of the future as physical agents. The SciRoc Project ran two such ERL Smart Cities Challenges, the first in Milton Keynes, UK (2019) and the second in Bologna, Italy (2021). In this chapter we evaluate the three challenges of the ERL, explain why the SciRoc project introduced a fourth challenge to bring robot benchmarking to Smart Cities and outline the process in conducting a Smart City event under the ERL umbrella. These innovations may pave the way for easier robotic benchmarking in the future.

Part of the book: Human-Robot Interaction