Training drones to fly fast, around even the simplest obstacles, is a crash-prone exercise. The hope is that this system, dubbed 'Flight Goggles', will significantly reduce the number of crashes drones experience during training.
MIT adds that the system will also serve as a virtual testbed for any number of environments and conditions in which researchers might want to train fast-flying drones.
“We think this is a game-changer in the development of drone technology, for drones that go fast,” says Associate Professor Sertac Karaman of MIT. “If anything, the system can make autonomous vehicles more responsive, faster, and more efficient.
“In the next two or three years, we want to enter a drone racing competition with an autonomous drone, and beat the best human player."
To do so, the team will have to develop an entirely new training regimen.
Currently, training autonomous drones is a physical task: Researchers fly drones in large, enclosed testing grounds, in which they often hang large nets to catch any careening vehicles. They also set up props, such as windows and doors, through which a drone can learn to fly. When vehicles crash, they must be repaired or replaced, which delays development and adds to a project’s cost.
Assoc Prof. Karaman says testing drones in this way can work for vehicles that are not meant to fly fast, such as drones that are programmed to slowly map their surroundings. But for fast-flying vehicles that need to process visual information quickly as they fly through an environment, a new training system is necessary.
“The moment you want to do high-throughput computing and go fast, even the slightest changes you make to its environment will cause the drone to crash,” Assoc Prof. Karaman says. “You can’t learn in that environment. If you want to push boundaries on how fast you can go and compute, you need some sort of virtual-reality environment.”
The team’s virtual training system comprises a motion capture system, an image rendering program, and electronics that enable the team to quickly process images and transmit them to the drone.
The actual test space — a hangar-like gymnasium in MIT’s new drone-testing facility in Building 31 — is lined with motion-capture cameras that track the orientation of the drone as it’s flying.
With the image-rendering system, Assoc Prof. Karaman and his team can draw up photorealistic scenes, such as a loft apartment or a living room, and beam these virtual images to the drone as it’s flying through the empty facility.
“The drone will be flying in an empty room, but will be ‘hallucinating’ a completely different environment, and will learn in that environment,” Assoc Prof. Karaman explains.
The virtual images can be processed by the drone at a rate of about 90 frames per second — around three times as fast as the human eye can see and process images. To enable this, the team custom-built circuit boards that integrate a powerful embedded supercomputer, along with an inertial measurement unit and a camera. They fit all this hardware into a small, 3D-printed nylon and carbon-fibre-reinforced drone frame.
The researchers carried out a set of experiments, including one in which the drone learned to fly through a virtual window about twice its size. The window was set within a virtual living room. As the drone flew in the actual, empty testing facility, the researchers beamed images of the living room scene, from the drone’s perspective, back to the vehicle. As the drone flew through this virtual room, the researchers tuned a navigation algorithm, enabling the drone to learn on the fly.
Over 10 flights, the drone, flying at around 2.3 meters per second, successfully flew through the virtual window 361 times, only “crashing” into the window three times, according to positioning information provided by the facility’s motion-capture cameras. Assoc Prof. Karaman points out that, even if the drone crashed thousands of times, it wouldn’t make much of an impact on the cost or time of development, as it’s crashing in a virtual environment and not making any physical contact with the real world.
In a final test, the team set up an actual window in the test facility, and turned on the drone’s onboard camera to enable it to see and process its actual surroundings. Using the navigation algorithm that the researchers tuned in the virtual system, the drone, over eight flights, was able to fly through the real window 119 times, only crashing or requiring human intervention six times.
“It does the same thing in reality,” Assoc Prof. Karaman says. “It’s something we programmed it to do in the virtual environment, by making mistakes, falling apart, and learning. But we didn’t break any actual windows in this process.”
He says the virtual training system is highly malleable. For instance, researchers can pipe in their own scenes or layouts in which to train drones, including detailed, drone-mapped replicas of actual buildings — something the team is considering doing with MIT’s Stata Center. The training system may also be used to test out new sensors, or specifications for existing sensors, to see how they might handle on a fast-flying drone.
“We could try different specs in this virtual environment and say, ‘If you build a sensor with these specs, how would it help a drone in this environment?’’ Assoc Prof. Karaman says.
The system can also be used to train drones to fly safely around humans. For instance, Assoc Prof. Karaman envisions splitting the actual test facility in two, with a drone flying in one half, while a human, wearing a motion-capture suit, walks in the other half. The drone would “see” the human in VR as it flies around its own space. If it crashes into the person, the result is virtual, and harmless.
“One day, when you’re really confident, you can do it in reality, and have a drone flying around a person as they’re running, in a safe way,” Assoc Prof. Karaman says. “There are a lot of mind-bending experiments you can do in this whole VR thing. Over time, we will showcase all the things you can do."
