Using San Francisco’s Embarcadero waterfront as a laboratory, Swordfish creative director Matt Silverman tested VR video “geotour” capture and authoring

Building an interactive VR video app with “Hyperport” Hotspots

Swordfish is a San Francisco-based motion design studio that specializes in marketing and user experience design. Director Matt Silverman wanted to try to build an interactive VR video experience that would connect a series of videos through interactive “hyperports.” These hyperports would teleport the user from shot to shot throughout a contiguous space. He had four main goals for the test:

1) to experiment with live-action VR video capture and learn which points of view were interesting for the viewer,

2) to fine-tune the post-production process of stitching, rotoscoping, compositing and rendering,

3) to design various interactions for the hyperports themselves (gaze, click, or other gestures), and

4) to work with Unity 3D to build a VR app.

Swordfish is based in downtown San Francisco, so Matt decided to stroll around the Embarcadero waterfront, which stretches from AT&T Park (home to the Giants baseball team) to Aquatic Park (home of the famous Ghirardelli chocolate factory). Stopping at 20 scenic locations along the way, Matt shot VR video clips with a duration of four minutes each. He then shot one timelapse establishing shot from across the bay on Treasure Island facing towards the city, and later placed markers and name labels for each Embarcadero location in Unity.

Using storyboards to organize the hotspots, and designing the hyperports

Matt took stills from each shot and created storyboards with Boardfish (a storyboard page layout tool he developed), and inserted a visual pin/marker that would help him correlate the different hyperports between shots. As a local, Matt was very familiar with the images and locations. If he had been working in an unfamiliar location, he realized that more formal markers, GPS coordinates, or other indicators would have been helpful to reconstruct the spatial relationship between the shots once he was back in post-production.

Matt designed a hotspot marker and animation that is triggered by the viewer’s gaze. The gaze will initiate a 3-2-1 countdown, and then a sound and animation pop triggers the hyperport “teleportation” to the next video scene. Play the video to see this in action.

Step 1: Storyboarding the hotspots

This PDF (pictured) contains stills from each shot, with indicators dropped in as pins/markers, showing the different connecting hyperports throughout the different scenes.

Step 2: Positioning hyperports

Here we can see the hyperport hotspots positioned on a storyboard of one of the shots, indicated with simple “map pin” icons.

Step 3: Hotspot design

Different-sized pins, with a smaller size and slight elevation above the horizon for pins that were farther from the camera, create a sense of 3D depth that suggest that these pins are “farther away” from the viewer.

Using the SPIN Play SDK to build the custom VR App

Swordfish used Unity 3D to develop a custom app to play this VR experience. They experimented with full configurations and interactions for the environment, testing different graphics, animations, gestures (both gaze and point-click with controllers), sound effects, et cetera. They even used the WRLD Unity SDK, which allows the user to fly around the city like Superman, or pull back to a bird’s-eye view of a computer-generated San Francisco to quickly jump to any placemark in the project.

They added Pixvana’s SPIN Play SDK to their Unity project, and for each scene used the SDK to play videos hosted on Pixvana SPIN Studio. In Swordfish’s SPIN Studio account, each of the videos was encoded into various formats that were ideal for streaming under different circumstances to the various VR headsets.


The finished VR Geotour of San Francisco Waterfront

Swordfish uses their custom VR app as a client demo, and uses it as a reference for future VR work they are planning.