Virtual Reality video image quality is a breadth-and-depth subject matter that frequently comes up when our team is chatting with our customers and partners. We are all-things-vr-video geeks and in particular we love playing with cameras and are fans of websites that do side-by-side image comparisons from various cameras. This inspired our Creative Director Scott Squires to spend an afternoon configuring a conference room at our Seattle office with a series of test-charts and sprinkled the room with playing cards, varied posters, and some of our collection of Ultraman figurines. This room was then declared a “hot-set” and the only changes we made between shooting with each of the test cameras, was changing the cameras themselves.
Pixvana’s Creative Director and co-founder Scott Squires set out to test various VR video cameras by dressing one of our conference rooms with a mixture of charts and fine detail objects (such as playing cards and Ultraman figurines), which he then filmed using a variety of cameras.
First some observations about the subject and challenges:
- The only way to really evaluate VR video image quality is inside of a VR headset. Images here on a webpage are illustrative and indicative of general quality issues. Ultimately, putting on a headset and looking at video is the only real test.
- VR Cameras have many important attributes including their size, ease of use, portability, control and post-production software… in this article we are only looking at a limited element of these cameras–what the image sensors produce in terms of a comparative resolution/quality of image.
- Because the cameras are of different sizes, the captured images of our test scene are very similar but not *exactly* matching. This is enough for us to get a general comparative view.
Here’s what the overall scene looks like after being captured:
The Cameras in our test shoot
We used a mix of cameras that we had easily on-hand and that represent a good gamut of generally available cost/quality cameras in the market as of writing in June 2018:GoPro Fusion, GoPro Omni, Insta360 Pro, and a DSLR based capture using a Canon 5D.
Principal Camera Image shows the relative image sizes/shapes captured using the primary camera in each of the VR video camera rigs tested. Note the varying degrees of optical curvature from the cameras with more aggressive fisheye lenses, and the larger surface area / size of the image captured in the higher end DSLR Canon 5D.
We aimed to place each camera at the same height/location, but slight differences in the camera’s sizes leads to a slight misalignment amongst the images. A few notes on the cameras used:
- GoPro Fusion This is a good representative camera of the dozen or so “2 lens 4K cameras” on the market. These cameras are small, portable, cost $500 or less, and produce a stitched 360 degree video that is 5.2K in resolution.
- GoPro Omni This camera is near-and-dear to our hearts at Pixvana, as these “GoPro rigs” were among the first practical VR cameras available in 2015-2016, and the Omni is a delicious solution (albeit one that is quickly becoming obsolete because of lack of ongoing development from GoPro). Six (6) GoPros each capturing a 2.7K @30fps image, produces a much higher resolution of total image coverage that can be stitched to a 8000×4000 resolution (“8K”).
- Insta360 This camera is among the first of the high-quality at reasonable priced “pro vr cameras” that have been purpose-built for VR video, with a “pro” resolution of 8K or above in mind.
- Canon 5D This is not a VR camera! In fact, producing a VR video with this camera body would actually be impractical. However we chose this camera for this test because the lens on the camera, the excellent Canon 8-15m 2.8, is often the choice for building VR video rigs using DSLR or Cinema camera systems such as the Sony A7 series, or the RED cameras. This test image is indicative of what a “full frame sensor” high quality cinema or pro-photography camera rig can attain. Here at Pixvana we’ve made several short films using this level of camera, such as this shoot. This image is essentially the benchmark in this series of tests, showing what higher quality sensor and optics can capture. However because in this test-shoot we used only one (1) camera, we shot still images to compose the spherical coverage of the scene (by using a nodal tripod head to align the images). For that reason Scott does not appear seated in the scene as he does with the other images (he was behind the camera snapping the shutter).
8K Comparison Images
The stitched images above all aligned, white-balanced, and cropped to emphasize the 360*100 degrees in the center horizon. GoPro Fusion (upper-left), GoPro Omni (upper-right), Insta360 Pro (lower-left), and Canon 5D (lower-right).
The stitching of each image was generated by the accompanying software for each camera system (we did not worry about stitching quality or optimizing edges, as we are more interested in the overall image quality in areas not affected by stitching boundaries). We then normalized all of the images to a standard resolution of 8000×4000 (actual “8K” is generally 7680 x 3840, but we used a simple 8000×4000 size in this study) and to a common white-balance (both of which we did as image edits in Photoshop). If you want to look at these images at full resolution, you can download them here:
- Photoshop file with all layers at 8000 resolution for turning on/off layers and comparing images
- Photoshop file with all layers at 4000 resolution (same as above, but at 4k = smaller download)
- Just Camera A – GoPro Fusion, 8k jpeg, or RAW
- Just Camera B – GoPro Omni, 8k jpeg, or RAW
- Just Camera C – Insta360, 8k jpeg, or RAW
- Just Camera D – Canon 5d, 8k jpeg, or RAW
All of the post-production steps we’ve taken (stitching, scaling in photoshop, white-balance) introduce slight interpolation artifacts to all images–we are never looking at the “raw” file. But these “normalized” images allow us to compare/contrast their relative sharpness, color, image detail, etc. in a more direct side-by-side fashion.
Comparing the Results – 1:1 Image Strips
First let’s take a look at a series of cropped areas where we can see strips of the different images at 1:1 pixel resolution, side-by-side. We intend to build a interactive widget whereby you can see multiple cropped areas in a widget–but for now they are just placed below sequentially.
Notice the higher resolved detail in the G horizontal and vertical stripes, which are discernible with the 5D, but variously muddy with the other cameras. The sharpness of the Fusion is actually pretty amazing for being 1/4 of the resolution of the other cameras.
The range of detail and color are very clear in this set of crops.
Comparing the Results – 1:1 A/B Slider View
And now let’s do side-over-side “A/B” comparisons of the results by comparing the different cameras. The slider in this widget allows for a interactive wipe from left-to-right that reveals the cropped areas of any two camera results. Very noticeable resolution and image fidelity differences are evident when comparing the 5.2k GoPro Fusion images to the Canon 5D. Less obvious, but still very different image differences, can be seen comparing the GoPro Omni and the Insta360 Pro images.
In the interactive widget click on the different camera-pair-icons to load any two images, then use the slider to swipe between them:
A/B Side:Side Swipe of Baby Poster
A/B Side:Side Swipe of Chart Detail
A/B Side:Side Swipe of Circle Grid
A/B Side:Side Swipe of Flower Detail
A/B Side:Side Swipe of Neko Case Poster
A/B Side:Side Swipe of Ultraman Figurines
Comparing the Results – Commentary
Image Quality encompasses a number of different variables and each step in the process can cause a loss in quality. The tests illustrate how even with the same number of pixels, two different cameras can produce different apparent resolutions.
We could take a deeper dive into things like Modulation Transfer Function (MTF) of the imaging process or noise levels of the different sensors, but what we have outlined here provides a good basis for comparison and further discussion. One thing we skipped over in this test is the additional compression and loss of quality that occurs when these images are recorded as video by each of the cameras. In some cases camera vendors additionally filter and compress the videos aggressively to allow them to be written faster and with less storage demands to on-board memory cards.
The difference in resolving power from the lowest end camera (GoPro Fusion) to the high-end camera (Canon 5D) was significant but not as dramatic as we had expected. The smaller sensor in a less expensive package (electronics, lens, etc) was bound to have compromises compared to a higher end camera system. If we compared it to a high end RED cinema camera shooting at 8k the differences would be even greater. It’s likely more differences would be apparent if we were comparing systems with standard lenses instead of fisheyes. While the Canon 8-15mm lens is a professional still lens, there are no fisheye lenses in the same class as high end cinema lenses. Also keep in mind that the GoPro Fusion is a higher end camera in the consumer space; there are numerous even lower-cost and lower-quality cameras in the market.
The technical challenges of the 360 video pipeline and limitations of today’s VR Headsets make it very challenging to try to match the fidelity that viewers have come to expect from their viewing of video in modern movie theaters across huge screens, at home on 70” HD and 4K televisions, and on ultra-high-pixel-density displays on their laptops and modern smartphones. Every step that causes a loss in quality takes it even further away from the viewers expectations.
Scott and Russell from @teampixvana on location with the GoPro Omni camera, which combines a nice portable/light form factor, with good high-quality 8k VR video resolution. Outdoor/well-lit environments render much better than in more challenging indoor and low-light settings, where the camera struggles to resolve details.
Recommendations for choosing a VR video camera
We consider the GoPro Fusion and other consumer grade cameras in its class to be great for experimentation and testing, but beneath the minimal threshold of quality for any general VR video production, even against today’s headsets which already are capable of displaying higher resolutions.
As the final quality can never exceed the quality of the initial capture, always try to shoot with the best camera system at the highest resolution practical for your particular production. Budget, close parallax, mono/stereo, size/weight, etc. will also impact the choice of camera given your creative needs.
Aim to maintain the least loss of image quality throughout your post-production process by selecting file formats and settings carefully as you move to stitching and then final edit. Do not stitch to a higher resolution than the captured system can actually produce, as that will needlessly add time and complexity with no benefit to quality.
Our Plans for Additional Tests of VR Cameras
The VR camera world is continually evolving. Several new 180 cameras are available or announced that significantly increase the quality of stereo 180 degree video production like the Zcam K1 and the announced partnership between Lucid and RED Cinema Camera for a new 180 camera. There are also high-end dedicated 360 camera systems from Zcam and Kando that use larger four thirds sensors and more cameras (up to 9) that significantly increase resolution. And one of the best stereo cameras is the Yi Halo combined with the Google Jump stitching service that produces excellent 8k stereo output from 17 cameras. We have conducted tests with all of these and will be writing about them soon.