RoCKIn: Impact on Future Markets for Robotics

Abstract

The goal of the project “Robot Competitions Kick Innovation in Cognitive Systems and Robotics” (RoCKIn), funded by the European Commission under its 7th Framework Program, has been to speed up the progress toward smarter robots through scientific competitions. Two challenges have been selected for the competitions due to their high relevance and impact on Europe’s societal and industrial needs: domestic service robots (“RoCKIn@Home”) and innovative robot applications in industry (“RoCKIn@Work”). The history and reasoning behind the chosen task and functionality benchmarks in RoCKIn are explained by providing an insight from the International Federation of Robotics and an analysis on RoCKIn’s impact on the industrial robot market domain is carried out. To paint a broad picture, RoCKIn is compared to other robot competitions and similarities, differences and challenges those competitions share are pointed out. Some industrial robot market requirements and the way RoCKIn addressed them are explained. Strength and weaknesses of the project in regard to their market impact are emphasized and it is shown how these were continued and addressed by RoCKIn’s successor European Robotics League (ERL).

1. Introduction

This chapter gives a brief overview on the current situation of the robot market. It discusses the potential impact robot competitions could have on this market by not only focusing on RoCKIn’s (Robot Competitions Kick Innovation in Cognitive Systems and Robotics) contributions, but also on its successor, the European Robotics League (ERL), and robot competitions and benchmarking in general. It is divided into two main sections:

• Competitions in industrial market domains

• Analyses on market impact

This chapter focuses mainly on industrial robot competitions and their impact on the industrial robot market, though some of the information provided is also applicable for service robot competitions and their impact on the service robot market.

The following sections briefly discuss the current situation of the robotics market, the general requirements of end users and the way RoCKIn addressed them. It is further shown how the industrial and service robot markets are potentially influenced by robot competitions and some of their long-term benefits. Some success stories beginning in robot competitions are introduced before concluding with an outlook on upcoming robot competitions and benchmarking efforts throughout Europe.

2. Competitions in industrial market domains

According to the International Federation of Robotics (IFR), the robot market can be divided into the two major areas: industrial and service robotics. Classification within those areas follows and extends the Standard Industrial Classification of All Economic Activities (ISIC) revision 4. For industrial robotics alone over a dozen different fields are analyzed. The most interesting in the context of RoCKIn@Work are Manufacturing, Electrical/electronics, and Automotive. From an application perspective, Handling operations / Machine tending, and Assembling and disassembling inspired the RoCKIn@Work benchmarks. In RoCKIn@Home, the service robot, application areas Robots for domestic tasks and Elderly and handicap assistance were represented through the benchmarks.

With an increasing need for robot-based automation, solutions come from the demand to objectively compare different solutions. As part of a solution development or purchase process, end users usually compare the data sheets of different robot makers/vendors and sometimes even create their own benchmarks to be able to find the most promising solution. In particular, the automotive industry creates benchmarks that are known to the robot makers, but not to the general public. They compare and judge the robot performance in secrecy and share the results with the robot makers. Typical benchmarking criteria are measurable quantities, such as cycle time, path accuracy, pose repeatability, mean-time between failures, cost, etc. Soft factors such as perceived product quality, brand image, long-term customer relationships, etc. are also playing a role here.

The big question is how to objectively compare different solutions. There are a lot of different robot challenges, benchmarks, and competitions set in industrial market domains. Often they intend to address one specific problem. Only very few of them try to solve different tasks of varying complexity. Most of them have in common that a winner will be determined by one or more referees, but they lack an objective scoring and comparability of different functionalities. This is a key difference to the idea of RoCKIn@Work, where the single functionality of a robotic system, as well as the overall performance on a task level are scored through objective metrics that rely on data gathered throughout tests.

This section briefly describes a few industrial end-user problems and outlines the importance and impact of robot competitions have had so far and how they might even get a stronger influence.

2.1. Robot market challenges

As outlined in the introduction, the worldwide robot market is growing fast and companies that intend to utilize robots in their factories to automate their production processes, especially SMEs that have not yet started to automate their processes, are confronted with an increasing amount of possibilities to do so. The number of robot companies is growing rapidly and they often provide very specialized solutions to common problems. It is very hard for companies to select a solution that fits best their needs because there is no objective way to compare the solutions offered in the market. This is where industrial robot competitions start to become attractive for industry and where they set themselves apart from other well established competition formats. They no longer focus only on the playful character of a competition, but also intend to provide solutions for a problem occurring in the real world. This helps SMEs, with only little experience in robotics so far, to get an idea of the state of the art in robotics and about possible solutions for a production challenge they face in their own factories.

2.2. Different industrial robot competitions

This section outlines some of the robot competitions set in the industrial domain with a high impact in the robotics community.

They have in common that the quality of the participating teams is very high. It is apparent that the competitions are not taken lightly and a lot of effort is spent on their preparation. One more indication for this is that a lot of Ph.D students participate in them and make them part of their daily work. They are often accompanied by younger undergraduate students, thus sharing knowledge and providing them with hands-on experience outside the usual university curriculum.

What is missing in most of them is that they do not provide a setup for mobile manipulation, something which is becoming of increasing importance in Factory of the Future scenarios [1]. Instead they focus on a very complex setup to challenge participants, either with a lot of different objects to manipulate, or with a very complex task as subset of a real world application.

2.2.1. Airbus Shopfloor Challenge

This challenge was held for the first time at ICRA 2016 in Stockholm. Teams had to solve several rounds of a simplified drilling task on an artefact representing part of the aircraft fuselage. Success was measured based on the number of holes drilled within a specified time and accuracy. Apart from a cash prize for the winning team, much more important was the possibility for the winners to develop their idea for commercial application within Airbus. Further details are provided in Ref. [2].

2.2.2. Amazon Robotics Challenge

The goal of the Amazon Robotics Challenge [3] is to strengthen the ties between the industrial and academic communities and promote shared and open solutions to some of the big problems in unstructured automation of so far logistics/handling use cases. The challenge consists of three different tasks. In the Pick task, teams have to remove target items from storage and place them into boxes. During the Stow task, teams have to pick target items from totes and place them into storage. This is followed by a final round, where all items are first stowed and then selected items are picked into boxes.

2.2.3. Bayer Robotics Competition

The year 2016 marked the start of another robotics competition that is organized and funded through Grants4Tech, an initiative of Bayer AG, aiming at improving production processes in the life science industry [4]. Very similar to the Amazon Robotics Challenge, the competition aims at strengthening the bond between industry and academic robotic research communities and enthusiasts. The task at hand is derived from a real world problem. At Bayer, everyday sampling of powder for quality control from a drum of incoming raw materials is done hundreds of time. To find a solution how this process could be automated, the robotics competition was founded. The task is broken down into subtasks, some of which are a complex task in itself. At the time of writing, the competition is ongoing, therefore no conclusion can be drawn yet.

2.2.4. European Robotics Challenge (EuRoC)

The European robotics challenge is a project funded through the European Union’s Seventh Framework Programme. It aims at sharpening the focus of European manufacturing through a number of application experiments, while adopting an innovative approach that ensures comparative performance evaluation. It consists of three industry-relevant challenges:

• Reconfigurable interactive manufacturing cell

• Shop floor logistics, and manipulation

• Plant servicing and inspection

Out of the before mentioned competitions, EuRoC is the only one that, like RoCKIn, utilizes benchmarking as performance measurement. Another key difference is that this competition involves mobile manipulation and runs in different stages over a period of four years, whereas the other competitions are yearly events. More information can be found at [5].

2.3. RoCKIn@Work

The RoCKIn competition, as described in detail in previous chapters, sets itself apart from the other industrial robot competitions described above. Its main focus is not only the competition part, but also benchmarking. The project is, to the best of our knowledge, the first robot competition that allows performance assessment of a robot functionalities and abilities relying on objective metrics through comparison of recorded data with a ground-truth system. The idea is to use competitions as a tool for scientific experiments to foster scientific progress and innovation which has been very well perceived in the research and industrial community.

Although the direct impact of RoCKIn to industry is not measurable, it is important to have a competition that engages students to have a closer look at the hardware and software they develop or use. Building a modular system where single functionalities can be tested without interference of other parts of the robotic system is still, as far as robot competitions go, unique to RoCKIn. RoCKIn therefore was an ideal entry format to competitions and to benchmarking. Younger students, maybe in their second or third year at university, had a chance to join a team and be able to contribute, either through a software/hardware module of their own, or through modifying existing functionality.

Those skills learned and the hands-on experience is very valuable on a job application. Personal contact to a representative of industry, in RoCKIn’s case, the KUKA Roboter GmbH, opened up possibilities for participants to apply for internships or to write their final thesis in one of the major robotics companies worldwide because hands-on experience significantly increases the success rate of applicants. It can be observed that students with a background in robotics competitions, either RoCKIn or others, are in general more resilient and committed to their assigned task. Their work, be it their final assignment from university or work done during an internship, is often of higher quality than the work of those who do not have this experience. During their time in the company, they need less support from their supporting employee. Often they are able to work better on their own and in teams.

2.4. European Robotics League: Industrial Robots

The European Robotics League (ERL) is the successor to RoCKIn. It is part of the larger project RockEU2, funded through the Horizon 2020 framework program of the European Commission. The ERL Industrial Robots (ERL-IR) league is successor to RoCKIn@Work. As explained in the first chapter of this book, the main difference between the ERL and RoCKIn is distinction between major and local tournaments.

This concept was introduced to tackle problems that were grounded in the competition format itself. In RoCKIn, and also other robot competitions, often fast solutions, or hacking some new functionality, was common during competitions and consequently led to misbehavior of the system. The tight schedules during competitions simply did not allow for any major changes in hardware or software. By introducing local tournaments teams should be able to spend enough time preparing for the benchmarks to avoid any major errors in their system. They should also get the time to fix minor problems between benchmark execution the “right” way, avoiding hacking wherever possible. In the end, this should lead to much more robust systems, able to face the challenges set by the benchmarks repeatedly.

At the moment the ERL addresses rather undergraduate students than PhDs. For this reason, the local tournaments provide another advantage over usual competitions: training and direct interaction between experts and participants. Compared to the ERL Service Robots (ERL-SR) league, ERL-IR has even younger participants, not accompanied by many PhD students. This might be the case because in ERL-SR, there is no standardized platform unlike in ERL-IR where until now all teams participated with a KUKA youBot, making it the de-facto standard platform for the league. The advantage of local tournaments in ERL-IR now presents itself through the possibility of direct sharing of software or knowledge between participants because they use the same hardware platform. Even helping out with parts, tools or reducing time spend on searching for software errors because the error is already known by someone else which is common among the teams. This exchange is very fruitful and helps fostering progress toward smarter robots through knowledge sharing and community building.

3. Analyses on market impact

3.1. Industrial requirements

Industrial applications have a set of requirements that are common among most of them. These requirements include:

• Setup time

• Cycle time

• Dependability

• Technology readiness

• Cost

RoCKIn addressed those topics through the design of the RoCKIn@Work testbed and the definition of the task and functionality benchmarks. To make sure that setup time is kept to a minimum the time allowed for each team to prepare their robot for a task was fixed by providing them with a time slot before each benchmark execution. Cycle time was addressed by imposing a time limit on the benchmark run (typically 10 min). The time for a single step in a task Benchmark, for example recognizing an object, has not been fixed, but was thought of as possibility by the consortium. It has not been implemented because the performance of a single functionality was assessed during the functional benchmarks.

In the case of dependability, the teams had to command their robots to execute every benchmark multiple times at several pre-fixed starting times. This ensured that teams could spend time between the benchmarking runs on improving the robot setup and fix some smaller errors, leading to a more dependable system at the end of the competition. Although programming (“hacking”) during competitions can be seen negatively, it at least teaches the team members valuable lessons about keeping deadlines, performance requirements, and quality issues.

The RoCKIn project had only little resources compared to other competitions. The teams participating in the competition also had only few resources. Consequently, the solutions found to tackle the challenges had to be low-cost. For example, hardware changes most often came through 3D-printed additions to the main robot platform used. This is a very cost-effective way to change a hardware setup compared to milled metal parts which are still standard in industry.

3.2. Analyses of current situation

The strengths of RoCKIn were manifold. One that stood out was the engagement teams showed toward the scientific and benchmarking aspects of the competition. Working in a common problem domain and sharing knowledge toward the goal of solving the task, more than once prevailed over “winning” the competition. To experience and test their robot system in a well-defined manner outside their own lab and in competition with others was seen as very valuable, both for participants as well as for the project.

What was missing before RoCKIn and has not yet been solved to a satisfying degree is specification of benchmarking challenges. The benchmarks defined by RoCKIn are a step in the right direction, but do not yet reach the impact that would be necessary to engage more partners from industry. Being very complicated up to a point where most teams were not able to execute a complete benchmark is noteworthy. Expectations by the consortium were maybe too high in the beginning, but the rules were also not flexible enough to take the participants skill level into account for a specific benchmark. One more thing still lacking is a process of tech transfer to industry. In RoCKIn, as well as in other robot competitions, there is still no clearly defined process as how software modules could be transferred to a real world application, something that industry would be very interested in. Further the possibilities to evaluate performance of single modules or functionalities are still limited. The infrastructure to roll out benchmarking in a broader context does not yet exist.

To approach this problem, RoCKIn laid a very good foundation on which the ERL is now building on. Having a clear specification of a testbed for benchmarking is utilized by the ERL to replicate them in laboratories all over Europe. A certification process ensures that the new testbed is able to execute specific task or functionality benchmarks. Rule evolution during the competition years is going to address different participant skill levels and will allow for more possible combinations of functionality into task benchmarks. The scientific part of the competition is going to become considerably more prominent and the amount of benchmarking data useful for other researcher will increase.

The growing number of robot competitions is both blessing and curse. Too many competitions specializing on a particular problem might lead to more specialized solutions instead of generalized solutions. Specialized solutions are something companies are already very proficient in. To advance the state of the art, those solutions need to become applicable to a wider range of applications to keep up with market expectations. Production has to be much more flexible in the future, which is why applications should not be tailored to one specific use case anymore.

4. Success stories

This section shows two success stories that had their beginning in robot competitions.

The first is the Amazon Robotics Challenge, which is explained in Section 2. It started out as the Amazon Picking Challenge as part of the ICRA Robot Challenges in 2015. This was preceded by Amazon acquiring Kiva Systems, which itself became Amazon Robotics. Kiva Systems was a company realizing efficient warehouse logistics, founded by one of the participants of the RoboCup small-size league.

The second success story is the one of the RoboCup Standard Platform League. The league started out using the Sony AIBO as standard platform. After discontinuation of AIBO, a new company, Aldebaran Robotics, was founded by one of the league’s participants. They developed the small humanoid robot NAO which replaced AIBO as standard platform. After acquisition of Aldebaran Robotics by SoftBank, it was announced that in 2017 their robot Pepper will become the new standard platform for the newly introduced RoboCup@Home Standard Platform League.

These stories show that robot competitions are worth much more than one would expect on first sight. It is the team spirit and the ever growing community that pushes talented people to pursue something extraordinary which might not be possible without it.

5. Conclusion and outlook

Throughout Europe, robot competitions gain momentum and start to attract more researchers every year. It is expected that new competitions, which focus more closely on end-user needs and which provide solutions with a higher Technology Readiness Level (TRL) than current ones, will appear. The possibility to transfer developed technologies to a market ready application more easily will raise interest of more companies. The increasing complexity will lead to more (semi-)professional teams competing in competitions. It is very likely that more teams will be supported by companies through collaboration or financial backup.

The professionalization of robot competitions, as can be seen by other big players like Amazon, Airbus, or Bayer creating their own competitions, is going to result in more success stories like the ones pointed out in this chapter.

Most of the current industrial robot competitions share the thought of an open community and highly flexible and exchangeable software and hardware. At the point where they find together and start to truly interact with each other, maybe even through a co-organized robot competition, it can be expected that a lot of innovative products will become available on the market. This will further increase the competitiveness of Europe as one of the major players in the Fourth Industrial Revolution [6].