TELEPRESENCE AND TRAINING

The client was a large manufacturer that was just setting up their simulation environment. Their initial requirements were fairly modest. They wanted to create a simulation environment that they could use to train their robotic systems in. The initial task for this robot was to train the arm and grabber. They wanted to create virtual objectives comprised of various objects, ie cylinders, boxes of various masses. Allow a human operator to manipulate the arm, record the movements and data, then use that to start training the virtualized robotic system.

The requirements grew from there, the next task was to create a digital twin of a complete robotic system and it's sensors. They wanted to use this system not just for training but also for telepresence of the robot. We started with importing the robots data from a URDF file then constructing the robot automatically from the file definition and supplied assets. We then insured that the robot had proper IK and control system implemented as well as simulated the various sensors. We put this virtualized robot into a simulated environment for training operators in various situations (living rooms, kitchens, institutional environments…)

We then took this system and applied it to Augmented Reality with both the Magic Leap and the Hololens. This allowed us to place the virtual robot in a real-world environment. This was a fairly straightforward step since the latest AR systems incorporate robust SLAM and dynamic Spatial Mapping. The virtual robot could easily maneuver in the real-world, with collision detection, obstacle avoidance, and route finding. We simply had to add a new UI.