Virtual Reality (VR) is going to transform how stories are told and how they are consumed. The 1880’s saw the introduction of the foundational technologies behind moving pictures and sound. These foundational technologies led to the world we know today. These technologies were world changing.
VR is that technology today. Being in the story, not on the outside looking in opens the door to innovation.
Chris Milk, Founder and CEO of Within has said this about VR,
“VR represents a technology communicating to us using the same language in which our consciousness experiences the world around us: the language of human experience.”
What does this mean for the entertainment industry? Opportunity. This is an opportunity to create something entirely new, based on a foundation that was created over one-hundred and twenty five years ago.
VR Observer was very fortunate to sit down with Jonathan Whittaker and Phoebe Elefante of the New York Film Academy (NYFA) to discuss the NYFA’s VR workshops and the future of storytelling. To anyone interested in this emerging realm of storytelling, this is a great place to start.
1. Tell us a little about yourself, what is your background and what brought you to Virtual Reality (VR)?
Jonathan Whittaker: I started out professionally as an electrician and then worked my way up the ladder to cinematographer, eventually realizing my ultimate goal of getting paid as a director. My very first directing gig was at the helm of a 3D special for Sony Pictures, and part of my prep included time on the Sony backlot learning the immersive imaging technology. Prior to that experience I had always considered myself a bit of Luddite, but when my eyes were opened to the potential of the medium and the added tools it provided for the storyteller I was quickly converted. Since that revelation I have been apt to explore new avenues for narratives and VR seemed like the most logical next step to me.
Phoebe Elefante: I’m on this unconscious schedule of falling into a deep infatuation with some form of media every five years or so. I published a magazine, which got me into trans-media storytelling, which got me into MMORPGs and virtual worlds, which got me into games, which got me into M/A/VR. I’ve held many job titles, but my talents are definitely in producing/creative management/org design (whatever you want to call facilitating the creation of art), and making up story worlds. So the scope of MAVR is exciting to me from a creative standpoint because I can realize so many more layers of obsessive world construction. But what really excites me about the spectrum of immersive media is what it can do for more pragmatic or traditional applications, especially education. The closer we can get to the holodeck or a pacifist version of “Ender’s Game,” the more excited I get about the future.
2. Why does a film school want to tackle storytelling in VR?
JW: Even though it is in the name, the New York Film Academy is more than a film school. We offer many disciplines from acting to animation, and at the heart of every program what we are really teaching is storytelling. Knowing that, why would we not be interested in this incredibly immersive platform?
PE: We’re a LOT more than a film school (says the chair of animation and game design)! We’re really a storytelling school, and we utilize the whole range of audio-visual media to tell stories and create emotional journeys for our students and players or audiences. Exploring the same objectives in a new medium is just logical … but perhaps more specifically as storytellers and creators of imaginary worlds, we’re excited to explore an evolution from flat screen media to create immersion, or at least, the sense of immersion. With VR, the tools are making that a lot easier … and a lot harder.
3. What’s different about your VR programming?
JW: I think it comes back to what makes NYFA different from most institutions of higher education, and that is the hands-on approach. We firmly believe you learn most from doing, and we’ve applied that pedagogical methodology in the virtual reality programs.
PE: Immersion. If immersive storytelling is a jump up from flat-screen storytelling, then full-time immersion in a creative process is a giant leap towards a more effective learning environment for students looking to make that conceptual and creative transition. If you really understand what this medium is, and what it can do, there’s no reason to hold back.
4. What do you see as the evolution of your program as the medium develops? Both short-term and long-term?
PE: In the short-term, we’ve built a lot of flexibility into the courses, so that as the technological landscape changes, we can teach what’s most up-to-date, and most promising. For instance, when we started to research and design these workshops a year ago, we couldn’t have included 360 sound — the workflow was just too technical to attempt in such a short timeframe. Now we can.
On the interactive side, we’re now considering teaching some basic compositing, so that we can incorporate 360 video or stills into CGI-enhanced environments. And of course, the overlap of skill-sets we teach in our other courses create opportunities for synthesis across departments (like game design and animation, for example).
In the long-term, we’d like to see these programs expand to cover the entire spectrum of Mixed, Augmented, and Virtual Realities, as well as offering a range of workshops focused on individual aspects of VR, just like we have with film. So, one day, maybe we’ll offer VR screenwriting, or acting for immersive media, in addition to these foundational workshops.
5. How might the programs eventually merge?
JW: Through the incorporation of live video assets (360 video) in a game engine platform, combining the interactivity inherent in gaming with the high fidelity realism of video. The end result would be narratives that offer a deeper level of interactivity through the presentation of choices that could determine the outcome or at least the path of the experience.
PE: The line between interactivity and live action in M/A/VR is already blurry, and it’s our belief that the technological landscape will support the merging of these forms, just like they have in film and digital gaming. So it’s possible that in the next couple years, we may want to offer comprehensive instruction in all aspects of VR in a one-year conservatory setting.
6. What do you see as the future of VR/AR and MR in storytelling?
PE: All storytelling has evolved from a group of humans sitting around a fire, sharing their experiences. But in a lot of ways, all the forms of capturing stories — print, radio, and moving pictures — are just approximating the intimacy and emotional engagement of that core human experience. So MAVR will definitely become more social, more personal, and more intimate (in an emotional as well as physical way). And it will also become more accessible, so that it’s used for telling the kinds of stories that perhaps “don’t matter” from some high-minded artistic standpoint, but are necessary for interpersonal communication and relationships.
7. What marketable skill sets would the students learn in the programs? Key takeaways?
JW: The biggest takeaway in regards to the creative/theoretical side would be learning how to construct stories in this new space, answering the question of what makes for a great VR experience. From a practical standpoint, students will learn camera and post-production skills, the “how-to” technical knowledge.
PE: Beyond the conceptual how-to-think-in-immersive-spaces instruction, students will walk through an entire project workflow: brainstorming and concept development, pre-viz and prototyping, and then building their project through Maya and Unity (in addition to whatever skills they bring to their projects). When they leave, they’ll have created a playable prototype of an interactive VR project, gotten feedback and industry insights from a panel of VR industry professionals, and begun to establish a presence in the NYC MAVR community.
8. Can you discuss the content that will be produced during a short-term program?
JW: In the 8-Week Narrative VR Filmmaking Workshop, students will be producing four projects over the course of eight weeks:
1. 360° Photo: Through this exercise, students will capture a story with a single 360 degree image.
2. 360° Documentary Video: Building on the fundamentals learned in the 360° photo project, students will shoot a short documentary (one shot in total) on location in NYC.
3. 360° Scripted Narrative Project: Students will write a short script to be filmed in 360, blending spatial audio and 360° video to create an immersive environment told with multiple shots.
4. 360° Final Narrative Film: Using all of the skills and techniques learned to this point in the workshop, students will craft either a narrative short, documentary or music video of up to five minutes in length.
PE: Students in the 8-Week Intro to Interactive VR Workshop will document and produce a working prototype of an interactive, immersive experience for the HTC Vive. With this course, we want participants to bring their ideas and expertise with them when they decide what sort of experience they’ll create. I’ve spoken to prospective students who want to use VR to teach dance to kids with sensory challenges, build interactive marketing apps for retail clients, and demonstrate cellular reactions to different drugs for the med tech community. Their primary objective in the workshop is to design an interaction or series of interactions, inside an evocative setting, that provides feedback to the user — their prototype can serve as both demo and proof of concept as they find ways to develop full-scale versions of these ideas.
In the 8-Week VR Game Design Workshop, we’ll focus on concept development and workflow practices specific to games. The projects will be games, with a clear goal, set of rules, and win/lose states. Participants in the VR game design workshop will also learn more complex inputs, and — since this is the more advanced of the two interactive workshops — may be a bit bigger in scope than the one-room limitation of the intro projects.
9. What has been your single greatest challenge? Tell us how you addressed it, and what you learned as a result.
JW: For me, with filmmaking being my background, the biggest challenge has been letting go of control and working without a “frame.” Once I got past my desire to assert my own point of view, I was able to embrace the medium and trust the audience, while also realizing that any two given experiences could be different … and that’s not just okay, it’s great!
PE: For game designers, the need to think and compose outside of the frame hasn’t been the issue — the agency of the player is a factor we’re used to working with. For me, the biggest challenge has been convincing people outside of the gaming community that VR is a viable creative and professional path, no matter what their field. Mainstream, consumer VR makes possible experiences that cannot be had any other way, so now that we have the means, what are the stories and experiences we’ve been waiting to tell?