TORNADO: foundaTion mOdels for Robots that haNdle smAll, soft and Deformable Objects

TORNADO will develop an innovative, multifunctional and adaptive cloud robotics platform, supporting advanced navigation of an autonomous mobile robot (AMR) within complex, time-varying, real-world, human-populated indoor environments. The TORNADO AMR will be able to manipulate small, soft or deformable objects (SSDs) to an unprecedented degree of success, as well as to naturally interact with humans via hand gestures or verbal conversation, by exploiting the zero-shot generalization abilities of deep neural Foundation Models (FMs) for robotics. The AMR’s intelligence will rely on a pool of pretrained cloud-hosted FMs, which shall be further adjusted on-the-fly to the current situation via Out-of-Distribution Detection, Test-Time Adaptation and Few-Shot Adaptation subsystems. These will exploit human feedback if available, but will also support autonomous and dynamic cognitive adaptation. Additionally, the TORNADO system will be able to automatically select and set-up on-the-fly the most suitable combination of FMs and non-neural robotics algorithms during deployment, depending on the current situation. In cases of failure, on-the-fly skill acquisition will be supported via integrated, novel Learning-from-Demonstration methods facilitated by an innovative Augmented Reality (AR) interface and eXplainable AI (XAI) algorithms. The adaptive TORNADO system will allow the robot to perform difficult, non-repetitive manipulation tasks on previously unseen SSDs that may change shape during handling, as well as to flexibly adjust to SSDs of different sizes during operation. Measurement of human trust to interactive robots and human behavioral modeling will aid optimal integration/acceptance of TORNADO into society. Validation will take place at TRL-5 in 3 different industrial Use-Cases: flexible small gears manipulation and deformable ply-sheets handling (gears factory), palliative patient care (hospital) and product quality sampling/waste collection (dairy processing plant).