Virtual Reality in Android

Mario BodemannAndroid and Java Evangelist@mariobodemann@contentful

Virtual Reality in Android

API driven CMS, emphasis on content creation and delivery.

Virtual Reality in AndroidUsing Cardboard

What is Cardboard?

● Easy to produce● Available from

different companies and in different shapes

● can have a button/lever/hole for interaction

● only needs a smart phone (Android/ios) and an App

How does it work?

● Use 3D data and movement to draw two images: left and right eye

● Distort images for lenses

● Combine image to scene in brain

Programming Cardboard

How do I program Cardboard?

https://developers.google.com/cardboard/android/

● SDKs available for ● Java

○ OpenGL○ Pure Android

● Unity Plugin○ Unity Engine (C# /JS)○ Platform independent○ No reliable linux development available

● Demo app○ Find a box and

press the button○ On github

● Library○ jar distributed

with demo repository

○ no maven repository

How To: Android Demo

https://github.com/googlesamples/cardboard-java

Setup of the demo

Activity

Model Repository

OpenGL

Interaction

● All code in one Activity○ OpenGL code ○ Model creation

● Hard to expand on

Expanding on the demo

● App using above code as base

● Expanding it to new use cases

● Having fun exploring

Contentful Cardboard Idea

https://github.com/contentful-labs/contentful-cardboardhttps://play.google.com/apps/testing/com.contentful.cardboard

MVPR

Presenter

Model Repository

OpenGL

Interaction

Interactor

Activity (View)

VR Renderer

● Split Activity into several parts

● Improve modularity● Improve testability● Improve readability● Improve line number

count ;)

Interactor

● Repository abstraction

● Uses registered Listener to inform Presenter

● Live time of the Presenter

public interface Interactor { interface Listener { void onModelReceived(ProtoModel model); void onError(Throwable throwable); }

void setListener(Listener listener); void requestModels();}

Presenter

● Coordinates Interactor and View

● Enriches data from Interactor

● Tells View to update its models

public class Presenter implements Interactor.Listener {

// … setup

private int received = 0;

@Override public void onModelReceived(ProtoModel model) { model.position[0] += received * 4.0f; received++; view.addProtoModel(model); }

View

● Connects Renderer and Presenter

● Takes models and enqueues them in the renderer

● Startup point for Android app

● Deals with interaction

public class MainActivity extends CardboardActivity implements CardboardView.StereoRenderer, View {

@Override public void onCreate(…) {

renderer = new CardboardRenderer( getApplicationContext()); presenter = new Presenter( new ContentfulInteractor(this)); presenter.bind(this); }

@Override public void onCardboardTrigger() { vibrator.vibrate(25); userInteracted(); }

View (cont.)

● Listens to Cardboard Library callbacks

● For setup of a frame● And rendering every

eye, redirecting it to the renderer

@Overridepublic void onSurfaceCreated(…) { renderer.surfaceCreated();}

@Overridepublic void onNewFrame( HeadTransform headTransform) { renderer.prepare();}

@Overridepublic void onDrawEye(Eye eye) { float[] perspective = eye.getPerspective(Z_NEAR, Z_FAR); float[] eyeView = eye.getEyeView(); renderer.render(perspective, eyeView);}

Renderer public class CardboardRenderer implements Renderer { @Override public void surfaceCreated() { addFloorAndCeiling(); }

@Override public void prepare() { addPendingModels(); glEnable(GL_DEPTH_TEST); glEnable(GL_BLEND); // … More gl* calls }

● Abstracts from OpenGL rendering

● Generates OpenGL GPU Models

● Listens to View

@Overridepublic void render(float[] perspective, float[] eyeView) { multiplyMM(view, 0, eyeView, 0, camera, 0);

for (final Model model : backgroundModels) { renderModel(perspective, model); }

for (final Model model : foregroundModels) { renderModel(perspective, model); }}

Renderer (cont.)

● Render all models added

● Starting with background objects, since foreground objects contain transparency

Advantage of new system

Sample Challenge: How to change a texture?

● Change file● Rerun tooling● Re releasing new version in store● Convince user to update

● Need technical person● Potentially through several iterations

Purposal:Using A CMS

● Change a asset online

● Restart app to resync data (done at startup?)

● User only needs network

● Can be done by non technical person

Using A CMS

● Everything except for the grid comes from CMS

● Changing it with drag and drop

● Changing VR content from outside

Future

● Unit tests?● Provide different Renderer VR/classic/engine code?● CMS for Shader?● A/B testing of in assets?● Your ideas?

Future

Recap

Recap

● Cardboard is relatively easy to program and use● Expanding from Google Demo is the key to success● Using a CMS for VR

Questions and Answers