RoboVacation! A well-deserved break for our Robot House team

This article was co-written by Dr Patrick Holthaus, Mohamad Reza Shahabian, Dr Vignesh Velmurugan, Sílvia Moros and Lewis Riches, and was originally published in the September issue of New Electronics magazine.

This summer, a team of robotics researchers' summer vacation has been supported with the help of their own assistive robot technology. This is the story of RoboVacation! - A well deserved break, a demonstration that researchers at the University of Hertfordshire’s Robot House prepared as one of eight simultaneous robotics live demos participating in Robot Lab Live on YouTube. Here, we want to investigate the demo’s iterative conception process, the final demo design, and how the team prepared for and dealt with problems during the streaming.

Demo design

The Robot House team designed RoboVacation as a follow-up from last year’s demonstration where they already invited viewers to join a virtual tour of the house and see how the robots can interact with human visitors. A core focus in this year was to expand this experience and offer some additional value to the audience. For that reason, the fun and engaging vacation theme was selected, which almost everyone would be able to relate to.

At the same time, the vacation theme would also allow to better showcase how the robots could use their capabilities to assist people in their daily activities, for example when providing beverages or reminding to pack a racket for the beach. While the team are preparing to go, robots would offer their help on cognitive tasks (reminding them to book flights or hotels), physical tasks (prepare or carry food/drinks), social support (entertainment, relaxation), and coaching (advising on exercise or nutrients).

The team identified some potential for improvement compared to the previous year in providing a stronger audience involvement than before, which is why they decided to allow viewers to have an influence on the robot’s behaviour. This inspired the idea of splitting the whole demonstration into four individual scenarios that can work independently of each other. These scenarios could then be voted by the audience so that they, instead of the researchers, would determine what would happen next.

A final major consideration for the demo design was to emphasise that each team member’s individual research interests aim to directly contribute to making assistive robots a reality and that the Robot House provides an ideal environment for working towards that goal. Therefore, the team engaged in an iterative scenario selection process starting from a pool of several initial ideas where robots could be helpful within the theme, which would then be refined with each team member’s input to potentially include topics relevant to their research interests. This process would yield the following four scenarios: (1) Helping to pack, (2) Provide food and drinks, (3) Inform family and friends, and (4) Perform a security check. As a result, these scenarios would therefore subliminally and explicitly to varying degrees contain research problems around human-robot interaction.

Dr Holthaus is interested in understanding how assistive home robots’ behaviours can be created in a socially acceptable and trustworthy way. Such considerations are relevant in all four scenarios, where it is, for example, important to know how humans would want a robot to position itself around them or when it would be adequate to interrupt a conversation. Only when robots, as social beings, adapt appropriately to our personal preferences will they be accepted and trusted by potential future users.

Vignesh Velmurugan’s research focuses on design of user-friendly and socially acceptable home-based rehabilitation robotic glove for people recovering from a neurological injury. Scenario 1 is thus designed to portray how such rehabilitation robots can integrate within a domestic setting, alongside other robots. A part of this scenario involves one of the robots recognising when a team member neglects to include the glove while packing and then reminding them about it, ensuring that their training is uninterrupted.

This and other scenarios are also related to Lewis Riches’ work who focuses on developing trustworthy and reliable companion robots that can improve people's lives without compromising personal data. All scenarios thus include aspects that show how within the home, the context of a situation is dynamic with different people and numbers of people present, significantly influencing how someone would want their companion robot to share personal data.

Multiple scenarios are shaped by Mohamad Reza Shahabian's research, which is to enable human daily activity recognition in domestic environments by seamlessly integrating multiple affordable, potentially low-resolution cameras from different perspectives. With his work, he develops powerful perception capabilities for robots that enables delivering situation-aware functionalities. These were, for example, important in Scenario 3 when someone had moved up the stairs and the robots would wait for them when starting a phone call.

Sílvia Moros is interested in several aspects of social robotics, among others in the use of multiple specialized robots fit for different purposes and their use and collaboration in day-to-day scenarios around the house. Most of the scenarios therefore were designed to involve two kinds of robots, each fit for a different purpose and with different capabilities.

Preparation phase

After agreeing on the scenarios, the demonstration was realised along two parallel lines: Implementation of robot behaviours and interactive testing as well as setting up the stream, including multiple audio and video sources.

For implementation, the team agreed to mimic the approach chosen in the previous year where all robots would act semi-autonomously, i.e. the overall sequence of behaviours would be predetermined and remote controlled by an operator and sensing would be avoided as much as possible. In this configuration, the scenarios would be mostly deterministic, repeatable and reliable by omitting error-prone components such as speech detection or face detection. However, robots would navigate autonomously between rooms and locations within the house and execute longer subroutines independently for a smooth user experience. Frequent team meetings and interactive scenario testing involving humans and robots were therefore required to ensure congruence between robot control sequence and scenario script.

Live stream

The nature of last year’s demonstration, where the team introduced the Robot House - amongst others - from one of the robot’s perspectives, required a total of five camera perspectives, of which up to four were broadcast at the same time in a split screen. In contrast, the team aimed at a more sequential order of shots this year, reducing the amount of parallel content to improve focus for the audience. The resulting setup of a maximum of two parallel perspectives and three main locations of dining area, sitting area, and kitchen also reduced requirements on the number of workstations, cameras, microphones, and network bandwidth compared to the previous year while at the same time increasing image quality.

Consequently, the team has opted to only use a single streaming software where the different cameras were integrated as individual guests as opposed to last year where multiple streaming software were used in sequence. This change further reduced bandwidth requirements while retaining a user-friendly interface and layout-flexibility while streaming.

Since a primary goal of this year’s demo was viewer engagement, the audience was given an active role in steering the demo via the YouTube live chat. With that, they could determine the next of the four scenarios, which would then be initiated by one of the robots. This way, the researchers were unaware what the next scenario would be, inducing a diversity in their behaviour. To allow for direct interaction between the audience and the team, the demo concluded with a Q&A session where viewers could ask questions to be answered live by the researchers.

Challenges and solutions

Since interactive robots are error-prone, the demonstration was designed to be robust against potential failures during the live event incorporating multiple redundancies and fallback solutions. Indeed, the day of the live stream presented with some challenges that were accounted for as well as others that led to the formulation of new solutions to tackle them in subsequent demonstrations, for example when devices ran out of battery despite being fully charged.

A unique challenge on the day of the live stream was a hardware issue with one of the interactive robots, which had a prominent speaking role in almost all scenarios. Probably due to the warmth on the day, it unexpectedly refused to accept commands during the demo, forcing the research team to try to control its speech by typing each utterance. This had never happened in any of the trials, so the team had to overcome this issue in real time, which posed challenges for the people interacting with the robot as well as those controlling it. This challenge provides a perfect example of why it is difficult to deploy interactive robots in people’s homes. It draws attention to the still imperfect ways robots function and highlights the need for research into devices that can interact robustly with humans.

Conclusion

In summary, RoboVacation! - A well deserved break proved to be a highly successful outreach event that attracted one of the largest audiences of Robot Lab Live and a unique experience for the team. Brilliant questions from the audience verify that the demonstration was able to make several highly complex research topics relatable using real world examples and showcasing how robots could help people in their daily life. The demonstration has been made possible as a team effort of researchers with different skill sets and experience working together on a unique public-facing opportunity. Shaping the scenarios together, preparing for the live event each in an individual role, preparing for and overcoming problems, and discussing future mitigations has initiated new conversations between people on future research collaboration and novel ideas for demonstrations.

Visit the Robot House website to find out more about human-robot interaction research at Herts.