RAI Moves Whole Body Manipulation Outside the Lab With Spot Robot Rolling Heavy Tires

Boston Dynamics’ Spot spent years demonstrating careful, coordinated movements inside instrumented rooms, yet the real test has always been whether those same skills could survive once the ceiling cameras and external computers disappeared. Researchers at the RAI Institute have now answered that question by teaching the robot to upright, roll, and drag a 15-kilogram car tire across ordinary outdoor surfaces using nothing more than the depth cameras already mounted on its body and a compact computer riding onboard. The tire still outweighs what the arm alone can lift, so the legs, torso, and arm continue to share the load, only this time the entire decision process runs locally and continuously.

Earlier versions of this task relied on a large amount of lab-specific infrastructure. Motion capture cameras watched every movement the tire made, a powerful desktop workstation processed the numbers to determine what to do next, and you had to create a script by hand to tell the robot when to go from walking to rolling to uprighting and back again. To be honest, those crutches needed to go. A reinforcement learning locomotion policy maintains the object upright and moving, but independent skill policies now make modest modifications to base speed and arm joint targets approximately 50 times per second.

Those changes add up, so if the output is zero, they simply maintain their position, but if the output is high for a while, they begin to develop some force. All of these talents use the same fundamental reward system: how close we are to the final objective, how effectively the gripper is in contact with the tire, approximate body location, and a penalty for any jerky movement from the tire. They put a number of random variables at it, such tire mass, friction, and size, over tens of thousands of simulated robots all running on a single GPU, and we can complete each skill in a few hours.

However, it’s the perception side that is the most difficult to break. In the simulation, the policies are always fully aware of the tire’s position. Real depth photos are shaky, noisy, spotty, and occasionally totally obscured by the robot’s own arm. Rather than attempting to impose a finalized policy on the real world and hoping for the best, we kept the reinforcement learning going while gradually substituting less trustworthy signals for pose information.

First, the network learns to compress the position data into a compact form. Next, we began feeding it bits and bobs, acquiring the genuine stance when the camera could see it but nothing else. We didn’t start reading actual depth images until everything was steady. To make it even more realistic, we included simulated sensor noise, missing pixels, and other details to ensure that the final policies were not taken off guard by the real cameras. The robot’s behavior is noticeably different. With perfect expertise, it takes off like a house on fire, spinning the tire and keeps it moving. When it comes to depth photos, it is far more cautious, circling around to maintain the tire in view and making minor changes to ensure it never loses it.

A tiny small network manages the sequencing, which previously required an external script. It takes in depth images from Spots 5 cameras (fronts at 20 frames per second, sides and backs at 7 frames per second), the robot’s own sensors and body position, and a goal location calculated using visual odometry. Then it determines which skill should be active at any given time and which one to call up.

The network was trained by first watching a scripted rule set that had access to the true tire state, then learning to make the same choices from camera data alone. Because the individual skills remain separate networks, the gating network simply routes the appropriate outputs. When a person deliberately knocks the tire over mid-roll on a basketball court, the gating network switches to the uprighting skill, recovers the tire, and resumes rolling without any external intervention.

All of this computation now fits on a Mac Mini carried by the robot and completes each forward pass in roughly one millisecond. The same outdoor sequence was repeated in four uninstrumented locations: a skatepark, a basketball court, a garage, and the floor of the institute’s own building. In each case the tire begins lying on its side. Spot walks over, stands it up, rolls it toward a goal the operator sets once, and drags it the final distance. Additional tests showed the robot hooking a tire dropped from a van, walking backward to pull it over an edge, and tipping a standing tire flat onto a pallet. None of these environments contained motion-capture markers or external computers.