Light Field Capture — A Paradigm Shift for Cinematography, Visual effects and Live-Action Virtual reality?
The Lytro Immerge has been hailed as the holy grail for cinematic VR production — the camera that could change everything for live-action capture. There is wide consensus among VR experts and enthusiasts that light field imaging is the future of cinematic content in immersive mediums like VR and AR, eventually superseding 360 videos. Let’s see what the hype is all about.
360 videos — a transitional medium
Google and Facebook have been making a hard push into spherical 360 content. However, many smart folks in VR scoff at the current breed of 2D and 3D 360 video (Google Jump and Jaunt included), some even going so far as to say it’s not VR at all.
Why? Because spherical video doesn’t offer positional tracking i.e. you can only change the direction you’re looking at, but not your position. Meaning, you cannot lean your head around and look behind that object in front of you, experiencing parallax as you would in real life.
These VR purists believe that you need head orientation and head position aka 6-DOF positional tracking, otherwise it’s not “real” VR and that 360 video could poison the well for VR as a whole. Many, myself included, would respectfully disagree, citing that 360 content is more like a gateway drug or movie trailer for high-end VR, but that’s a conversation for another day.
Point is, the current variety of 360 video is awesome when done right, but clearly has its limitations and that’s a point well taken.
Enter light field capture
Light field imaging solves the problems 360 video faces, giving you head tracking, parallax and the full 6 degrees of motion, thus providing a far more immersive experience. But how exactly?
Without getting too deep into it, most 360 video capture rigs involve traditional cameras with wide lenses, place in a spherical configuration looking outwards. As Ryan Damm quips, it’s usually “a big ball o’ cameras (BBOC).”
In contrast, true light field solutions like the Lytro Immerge use hundreds of cameras that capture every ray of light and its direction from different viewpoints — in essence capturing a hologram of the environment as it would be viewed from a certain volume of 3D space. This is called a light field volume. However, this technology is not without it’s own limitations.
Presently, the storage, bandwidth and processing requirements for professionals to capture and work with light field volumes is astronomical. The camera itself and the infrastructure to support it is costly — we’re talking “hundreds of thousands” of dollars. Consider that the entire storage array you’ll get with the Lytro Immerge stores about an hour of footage.
It’s tough enough for professionals to work with this stuff, let alone getting consumer hardware to play it back without a lot of fancy compression, which no doubt folks like OTOY are actively working on.
Does this means the near-term benefits of using a light field camera are non-existent? Not quite.
In the talk below, Jon Karafin, Head of Light Field Video @ Lytro, sheds light (pun intended) on their Immerge camera and talks about the various cinematographic and VFX possibilities light field imaging opens up:
Key takeaway: in the near-term, playback of light field volumes on end-user hardware clearly isn’t feasible. Instead, the immediate benefit of light field capture is to get more flexibility in post-production and eventually output great looking “traditional” stereo (i.e. 2 flat images), thereby leveraging existing playback devices and distribution platforms.
Unlike traditional imaging, where what you capture is more or less what you get, true light field imaging promises an order of magnitude increase in what you can do in post-production. Things that would make VFX and cinematography folks do cartwheels of joy:
Let’s say you wanted to pull focus in a very complex manner on objects flying around, or reframe a shot to move the camera closer to the actors, or do a 600 fps slow-mo sequence, or remove an actor or object completely from the scene — with light field imaging you can do this and more will little effort. This fundamentally gives content creators a flexibility in post-production than ever before.
Lytro views light field post-production as having 3 phases:
- Cinematography: wherein the director, DOP etc. would sample the light field volume and dial in on settings they like — leveraging infinite refocus, ability to change camera position, exposure, frame rates, shutter angle etc, and then choose a flat stereo output we all know very well.
- VFX: here, despite the final output being flat, all the light field data is retained and utilized to unlock greater possibilities like accurate CG object integration, depth-based keying, scene relighting, automatic 3D tracking and semi-automatic rotoscoping. This will save time, money and means far less grunt work for VFX artists, letting them focus on achieving the artistic vision.
- Distribution: for final output you can automatically re-interpolate things like stereo disparity, FPS and shutter angle to target specific platforms and devices — a game changer when trying to deliver an optimal experience of the same content to myriad of devices (from TV to Cardboard to GearVR to Oculus/Vive). You can even create 180 degree stereo or flat 16:9 content.
Does that mean light field cameras are the best option?
Not quite. The natural question is how does the Lytro Immerge camera stack up against the myraid of GoPro rigs, Nokia’s Ozo, or “psuedo” light field cameras like the Google Jump or Jaunt Neo?
Sure we’d have to wait for head-to-head comparisons, but the short answer is all approaches have trade-offs and it’s still about choosing the right tool for the job.
Some situations would benefit from the all-in-one Nokia Ozo or Jaunt Neo type setup. Other times a 3D printed GoPro rig or Sphericam style solution might suffice. Consider that the camera you choose for a journalism piece in Syria vs. a POV surf shoot in Hawaii vs. a music video shoot in a Hollywood studio, would naturally vary. Is one camera better than the other? Well sure, but it completely depends on your criteria. There’s no magic bullet and it’s still about choosing the right tool for the job.
So what does this all mean for content creators?
What light field imaging does offers is the idealized dream of futuristic production, where almost everything can be changed in post without any hair pulling. It’s a godsend for creative professionals and will undoubtedly unlock new possibilities for storytelling.
But technical constraints mean that in the near-term, we are at best sampling this light field to eventually deliver traditional 2D or 3D imagery we all know and love (well most of us anyway :P).
Alternatively, we’re restricted to static scenes or capturing performance of actors and objects versus the entire scene.
I think Mark Bolas, Director for Mixed Reality Research at USC ICT, said it best:
Light fields suck because there’s just too much data. So somehow you have to winnow down the data, compress it and do things. To me the cinematographer and director are a real-time data compression algorithm. They’re saying we don’t need that, we don’t need that, we need this… If I know that’s the shot I want, give me a gorgeous lens [on a traditional camera] and i’ll get that shot. On one hand you have the complete 4D representation of everything [and on the other hand you have traditional capture]… In the end it’s going to come down to content in a piece. — Mark Bolas, MxR Lab, USC ICT
Wrapping it all up — expect unexpected upgrades as we move along
With light-field capture, content creators get an order of magnitude increase in flexibility when it comes to post-production and delivery.
We can say goodbye to light-probe photography on set, tedious green screen setups, digital set reconstructions, camera tracking drift and painful rotoscoping work to name a few. Automatic distribution means the content gets viewed exactly the way the creator intended, regardless of the platform or device the user views it on.
The old trope of “fix it in post” actually becomes feasible, even desirable with light field imaging. Very exciting stuff, but clearly a work in progress.
Did you like what you read?
Recommend this post (by clicking the❤ button) and share this.
Drop a comment below or share your thoughts on Twitter @bilawalsidhu
Want to dive deeper into light field imaging? Start here:
Suggested Reading
- What Is A Light Field Camera, And What Isn’t? An In-Depth Look
- So What Exactly is a Light Field Volume?
- Why the Lytro Immerge is a big deal
- Light fields — the future of VR-AR-MR
Suggested Viewing
