Final Project: VR Panorama

Many of the scripts I use -- particularly for shading -- are borrowed from community wikis or Unity documentation examples. My main job was designing a way to fit all these components together.

Overview

The user should be surrounded by a 3D panorama of an image. The 3D panorama should shade certain interactable objects in the world. For instance, if you are standing in an open field and holding a round mirror ball, the ball should show the field in its reflection. My goal here is to create a visual experience that maybe makes you think the scene could be taken from a real camera.

Part 1: VR Development in Unity

This is not the main focus of the project, but it's interesting and it did take a decent amount of time to rid logs of bugs and setup issues.

It would be good to get most of the headaches of setup and hardware issues out of the way first. I would like to be developing in the VR environment so I can test in VR all the way along for myself. I am going into this with no knowledge of VR development -- however, I do own a HTC Vive, I've coded projects in Unity a decent amount so I understand the Asset hierarchy structure (objects in the scene are represented by an "asset tree" data structure), and I know my C family. Combine these to get a yummy VR recipe.

Most tutorials for starting VR development lack proper instructions. The API itself is dense and doesn't explain how to get started. On the other hand, I managed to find a very good tutorial by Eric Kerckhove, but much to my initial dismay, Valve appears to have updated the input interface for Vive controllers as of a few months ago.

After much finagling and both physical and virtual setup, I finally managed to understand and set up the scene for a VR experience using the surprisingly intuitive brand-new input system. Aside from normal physical models, my Unity scene now contains...

1. a Player.prefab object, which holds the camera and the player's two controllers/hands
2. a Hand.cs script that allows the player's "hands" to interact with objects in the virtual world
3. a Throwable.cs script that makes its objects uniquely interactable by allowing the player to "throw" them

The wording is important, as the new input system (which is pretty awesome now that I know how it works) tries to abstract away the act of pressing a controller button. The action of grabbing ("pinching", as it's called when the user squeezes the controller trigger) enables the user to manipulate the physical world.

Now that I have the VR part working, I can start getting into its applications to image processing and hopefully produce an interesting scene. Really, the VR just examines the scene more intimately -- the actual panorama presents a separate challenge.

Part 2: Building the Panorama Scene

There are a few ways to approach putting you "inside" the panorama.

One classic idea is to simply display the panorama in 2D and rotate it with the camera. This works well but (1) it limits you from translational movement and (2) the sphere idea becomes a little more complicated. Of course, you could overlay the sphere as a "infinite background" in 3D translational space, which I might do if I have extra time, but for now I'm not going to use this panorama approach.

My attempt will be to create a sphere just large enough to encapsulate the user and map a spherical panorama onto the inside of the sphere. Let's see how this approach turns out.

First, I deconstructed the test scene except for the floor and added a sphere at coordinate (0, 0, 0), making it large enough that you should not be able to step outside of it. The sphere is invisible from inside, because Unity "culls" the inside of meshes for efficiency. Per a suggestion I found on Stack Exchange that linked to this wiki post, it's possible to flip the sphere inside-out by simply reversing the normals. You could apply this to any mesh -- the beloved Stanford dragon mesh, for example.

Unity uses a shading language similar to GLSL called HLSL -- it's a little tough to get a handle on much like GLSL because you really need to be precise on your vector calculus. To get started on this language I used a sample tutorial by Verónica Valls, as well as the sample image from the tutorial. The tutorial is actually pretty close to what I was hoping to achieve, though it will need some tweaking. Below is the original image from the site as well as the image mapped onto a sphere using some careful HLSL.

Unfortunately you will notice the mapping is a little too bright for some reason, probably a precision error happening somewhere? I'll deal with this later. Combine this with the normals reversal and the VR player position, and you are standing in front of the Chicago bean.

Hmmm, what could be causing the image to get so bright? In my past experience with graphics, brightness issues were caused by floating point precision errors or scalar mistakes.

This tutorial explains how HLSL shaders work in Unity. Upon closer inspection, the SurfaceOutput.Albedo value of the surf function, which calculates surface lighting properties, represents the intensity of light reflecting from a surface. I calculate this function using the classic tex2 (also found in GLSL), which samples from the texture at UV (scalar pair between 0 and 1) coordinate.

More importantly, lighting itself is determined by a function called half4 Lighting[LIGHTINGTYPE] (SurfaceOutput o, half3 dir, ...) that states that the lighting of an object is dependent on the viewing direction. This book was an excellent updated reference for how to understand surface shading models. In fact, the light here was getting automatically adjusted towards an ambient color of white. I was able to reduce the intensity of the ambient color to get a more natural reflectance. Again, a scalar problem.

With that fixed, you'll see I brushed over the equirectangular part. This is exactly the topic covered by my project on image mosaics and projections!! This is turning out to be a powerful application of this technique. Since the above image belongs to the panorama-mapping tutorial and I'd like to use my own data, I'll need either (1) access to a database with these types of images or (2) research on how to produce equirectangular images, which are special in that at height = 0 and height = max_h, the images warp horizontally to infinity much like flat maps of the Earth, for instance. Later, I will probably investigate how to generate more of these, and maybe develop a UI for switching the environment texture.

Part 3: Shading the Spheres

The final outline of my original general goal was getting a interactable object to be shaded by the surrounding environment, like a mirror. This will likely require me to delve back into HLSL, and (hopefully) this task won't be made trivial by Unity's rendering engine... Unsurprisingly enough, it is.

Recently, Unity replaced their standard shader asset library with... a "Standard" shader that has customizable bars for light specularity. This comes with a tool called the ReflectionProbe. The probe samples its surroundings spherically and then stores the data as a CubeMap (these maps will be important later). You can just tell the reflection probe to sample from its immediate environment, and it will do so whenever specified. Just once? Sure. Every frame? Sure. Custom script? Sure. As expected, the reflection probe trivializes this entire endeavor.

The scene is beautiful and runs well due to Unity's powerful engine. However, several questionable things are happening here.

1. The probe collects data every frame. This is accurate but highly inefficient.
2. The collection data is inaccurate to begin with, as there is only one probe and it is attached to one of the spheres.
3. Adding more probes -- one for each sphere -- is not an option because it will become computationally expensive and what with Unity's prefab structure.

Upon closer observation you'll notice the user never leaves the inside of the panoramic sphere and that there are no other salient objects to render. The user does not even exist as a visible object themselves. Therefore, I'd propose that it is only necessary to collect a single cubemap. All you need to do is generate a CubeMap like the probe does and then write a script to use it.

The CubeMap is basically an abstraction for an image. Conveniently (and coincidentally) it can be stored as a 2:1 equirectangular image. Given you already have the equirectangular image, you can import a "CubeMap texture" directly into Unity. I've done that now.

Remember how I want to make this a reflection. I'm concerned that if I rotate the image in the wrong way, I will lose the orientation of the sphere. That is, what you see in the sphere should be exactly what is behind you.

To combat this, instead of directly converting the equirectangular image to a sphere and needing to worry about positioning, I am going to take a snapshot of its interior as soon as the scene gets initialized using a camera's RenderToCubemap method (basically creating a temporary probe).

In terms the HLSL itself, Unity's Writing Vertex and Fragment Shaders was useful for this, but in the end, I went with a simple application from the community wiki. I will detail the HLSL methods further in my paper (at bottom of page).

Great! Now that all the components are working I can focus on my data.

Part 4: Acquiring Equirectangular Images

Now that I've composed a VR tool by combining a few nifty Unity tutorials together, I'd like to have my own data (as much as I like Verónica's Chicago Bean sample). There is actually a great website dedicated to equirectangular picture samples. I took this image:

And I used it as an input for my panorama sphere:

And I think that concludes the panorama exploration for here. If I have time, I'd like to add a GUI for switching panorama photos and maybe the ability to adjust some settings with the same GUI. I'd also like to shoot my own equirectangular photos of Berkeley, which requires some careful cylindrical positioning. On the other hand, I'm pretty happy how my idea got translated into reality (well, virtual reality, haha).

My final paper for the class can be found here.