Harvestworks
Resident Artist
The Creativity+Technology=Enterprise Program at Harvestworks was a residency for artists to develop an existing creative technology work of theirs towards commercialization. My interactive video technology for Flight18 was selected for support and was one of only two projects that received additional support for a second year.
During the first year of the Residency, my interactive video environment was developed for use in commercial installations. In the second year, the project evolved into a novel piloting platform for drones.
The Starting Point
Flight18 (2012) was a spaceflight dance party performance that consisted of a large video environment (built in Unity3D) that immersed the audience as passengers of the Spaceship Kaleidoscope. The audience interacted with the environment by flying the ship throughout the solar system using an iPad, allowing them to explore freely during the experience.
The first year of the Residency was used to develop a number of features to support quick customization of the 3D environment and interface, and to easily change the output configuration to accommodate an arbitrary number of displays and projectors for different installation spaces.
Parrot CEO Henry Seydoux wearing the paper sphere
The First Iteration
In 2013, the first iteration of the new piloting concept was explored during a hackathon at the Drones and Aerial Robotics Conference at the NYU School of Law. Parrot sponsored the event and provided their AR Drone as a development platform. A makeshift video environment was made by projecting the video from a small 360 camera attached to the drone onto a paper sphere worn over the head of the user.
It became clear that by leveraging the unique perspective offered by drones, the risk to first responders in dangerous environments could be mitigated to a large degree. Coupling the immersive environments that had been developed during the Residency with an efficient camera system could be the basis for a control platform with real world value. This exploration became the focus of the Residency.
The Concept
The live camera feeds used on drones give pilots a limited first person view and provide poor situational awareness.
My concept places the pilot in an immersive video environment displaying a live camera feed in 360 degrees, allowing them to simply turn their head to see their surroundings.
This dramatically increased situational awareness, allowing for high precision flying in tighter spaces and greater ability to avoid obstacles.
The Prototype
A semi-heavy lift quad-rotor drone was equipped with a bespoke 360 degree camera payload. An augmented GoPro with custom firmware (to allow full frame capture) was fitted with a catadioptric lens. The live camera feed was transmitted to a ground control station where the video was processed and projected into a virtual environment. Wearing a VR headset immersed the drone pilot in the virtual environment, giving them an unobstructed first person view in any direction from the aircraft.
The video to the left shows the working prototype.
The residency was hosted by Harvestworks who provided administration and educational resources, funding from the Rockefeller Foundation, and professional advisory from ITAC (Industrial and Technology Assistance Corporation).
The commercialization strategy was further supported through the FastTrac TechVenture entrepreneurship program at the State University of New York, Levin Institute, made possible by a partnership between the City of New York and the Kauffman Foundation.
