VR Geotour of the San Francisco Waterfront

Creative Director Matt Silverman from Swordfish wanted to test VR video “geotour” capture and authoring using the San Francisco Bay Boardwalk as a playground.

Building an interactive VR video app with “Hyperport” Hotspots

Swordfish  is a San Francisco based motion design studio that specializes in marketing and user experience design.  Director Matt Silverman wanted to experiment with building an interactive VR video experience that connects a series of videos through interactive “hyperports” that allow the user to teleport from shot to shot throughout a contiguous space.  His main goals for the test were (a) to experiment with live-action VR video capture to find what kinds of POVs were interesting for the viewer, (b) to fine tune their post-production process including stitching, rotoscoping, compositing and rendering, © to design various interactions for the hyperports themselves (gaze, click, other gestures), (d) to work with Unity 3D to build a VR app.

Swordfish is based in downtown San Francisco so Matt decided to stroll around the Embarcadero waterfront stretching from AT&T Park (home to the Giants baseball club) to Aquatic Park (home of the famous Ghirardelli chocolate factory), stopping at 20 scenic locations along the way. He shot VR video clips with a duration of 4 minutes each,  He then shot one timelapse establishing shot from across the bay on Treasure Island facing towards the city, and later placed markers and name labels for each Embarcadero location in Unity.

Using Storyboards to organize the Hotspots, and Designing the Hyperports

Matt took stills from each shot and created storyboards with Boardfish (a storyboard page layout tool he developed), and inserted a visual pin/marker that would help him correlate the different hyperports between shots.  As a local, Matt was very familiar with the images and locations–if he had been working in unfamiliar location he realized a more formal marker, GPS coordinates, or other indicators would have been helpful to help him reconstruct the spatial relationship between the shots once he was back in post-production.

Matt designed a hotspot marker and animation that when gazed upon by the viewer, animates a 3-2-1 countdown and then with a sound and animation pop, triggers the hyperport “teleportation” to the next video scene (play video to see).

Step 1: Storyboarding the Hotspots

This PDF (pictured) contains stills from each shot, with indicators dropped in as pins/markers, showing the different connecting hyperports throughout the different scenes.

Step 2: Positioning Hyperports

Here we can see the hyperport hotspots positioned on a storyboard of one of the shots, indicated with simple “map pin” icons.

Step 3: Hotpost Design

Different size pins, with smaller size and slight elevation above the horizon for pins that were farther from camera, create a sense of 3D depth that suggest “farther away” from viewer.

Using the SPIN Play SDK to build the custom VR App

Swordfish used Unity 3D to develop a custom app to play back this VR experience.  They experimented with full configurations and interactions for the environment, including testing different graphics, animations, gestures (gaze and point-click with controllers), sound effects, etc.  They even utilized the WRLD Unity SDK which allows the user to pull back to a birdseye view of a CG San Francisco to quickly jump to any placemark in their project as well as fly around the city (or the entire world) like Superman.

They added Pixvana’ SPIN PLAY SDK to their Unity Project and for each scene pointed to SDK to play videos hosted on Pixvana SPIN Studio.  In Swordfish’s SPIN Studio account each of the videos was encoded into various formats that were ideal for streaming under different circumstances to the various VR headsets.


The finished VR Geotour of San Francisco Bay Boardwalk

Swordfish shows their custom VR app to clients and uses it as a reference for future VR work they are planning.  They use Pixvana SPIN Studio to “VR Cast” the videos themselves for reviewing the project content during video stitching and editing.