What is this project?

Virtual Reality has a great potential to provide good aid for the designers to visualize their ideas. It is already being used in architecture and interior design.

Besides that, I am also very interested in lighting design, both CG and physical. What if we can learn lighting design in a more interactive way? Or we can brainstorm lighting ideas very quickly in VR?

Combining both together, I made a tool for VR Interactive Lighting Design. You can put in any scene and do lighting pre-visualization with different types of light. As a technical artist, I also provided an interface that you can link to any shader you made, and change the value of the parameter to debug the shader.

What it can be used for

  • Interactively pre-visualize lighting design in any scene you want to put into your game/film/experience.
  • Debug shaders you made, especially  the ones for VR
  • Lighting Training

This project is made

With Vive and Unreal Engine 4, programmed with Blueprint, all by myself.

Demo

Interesting problems to solve

Making the Best Use of the Space

You always need to deal with this constraints on play area for virtual reality. 

In my case, the playable area is 1.6m by 2.9m. It is not very big and even if it is bigger, it always has a limitation, so you can not fit in every scene that you want to design lighting for. 

Teleportation is a common way to deal with mobility problem. However, it is not used in this project for the following reasons: 

  1. It works fine if you just want to be on the ground, but in the case of lighting design, you also need to reach high. How can you determine how high you want to get with teleportation? If you set some teleportation point in the air that you can teleport to, it will gave less freedom and also it may create motion sickness since you are in the air.
  2. If you need to adjust the location of the light, you need to go back and forth between the place you view the lighting and the place you put the light. 
  3. I personally don’t enjoy teleportation very much because it usually create a feeling of disoriented

If you add a control to move yourself up and down, it still can not solve problem number 2 (go back and forth between locations). Also, it doesn’t feel natural.

The way I solve this problem is to change the size of the model. Each model need to have two size: toy size and real size. You need to light the scene with the toy size model, and experience it with the real size one. It is like making a stop motion film: you adjust the little toy-size models, and when you finally make the film, you are experiencing it in its real size.

 

Then designing the playable area is easier. The picture on the left shows the toy-size mode and on the right shows the real size mode. 

When you are designing the lighting in toy-size mode, you have the scene to be lit in front of you, and you can get the tools you need behind you (the creation tools area). When you switch to real size mode, the model will scale up and fill up the area.

The reason there’s only two scales for now is to make it easier to control for the users. Scaling freely will bring new challenges of making intuitive and natural interactions for it, which I avoided here for this first version of the prototype. 

Design the Controls and the Interface

The UI can be divided into two kinds: creation and control. As you can see, everything that brings/destroys things in the scene have a physical representation. Since it is virtual REALITY, everything should come and go with a good reason. 

So I created this machine that spawn the light when you push down the create button on the panel.

For the controls that you use to adjust the light, it’s similar to a controller for the television. Since we are using a Vive controller, it feels more like “you are holding a controller” instead of “you have a hand”, so it make sense that if you push down a button, a menu will open. 

For the light control, the lights’ parameters are broken down into intensity, color temperature and size. The intensity controls the power of the light and actually it will also scale according to the scale of the scene. In order to mimic real physical lighting design, color temperature is used instead of color. I am thinking of adding color with physical representations like the lighting gels. The size here is only changing the size of the light’s UI to help you position it in the place you want easier. I need to iterate on this size control since during playtest, I find it confusing for the users especially when they are using the point light. (They assume that it is a mesh light instead of a point light and will change intensity according to its size)

The spotlight itself has an additional control, which is the cone size. 

Another control I really want to add is attenuation, which I will in the next version.

Expand this tool to cover shader debugging

It was in the middle of the development process that I found it would be great to add some function that you can use to debug the shaders.

The first advantage, of course, is that you can see the shader from 360 angles and you can easily find problems about this shader. If you can debug a certain parameter, you can just plug in that parameter to the control. Another advantage is that you can check its reaction to different lighting conditions and the performance. 

During development I made three shaders and tested out their behavior in VR. 

The snow shader has displacement at the bottom of the model. The moss shader is a material function and you can lay it on top of the material of any model. It is based on world position and you can adjust the amount of moss (and a lot more). The rainbow shader calculate the color according to the view direction and is still being improved.

I learnt a lot testing these shaders in the tool. Take the snow shader as an example, it will break at UV seam, and also it has more aliasing than the others. I am currently looking at how to better solve this problem. 

Playtest

I playtested with my friends and faculty at school. They really had fun in it and here are some screenshots of the lighting they created. 

One thing I find during the playtest is that people spend more time on the Batman scene, where you can observe the lighting on the person very well, and the snowman scene, where you can go crazy with the shaders.

It makes me think that one feature I can add to the experience is to place human model with the scale 1:1 into the scene and you can move it around and see the lighting on the model. 

Next step for this project…

I will keep developing this project and make it more fun. There are several things that I am considering adding:

  • More parameters for the lights
  • Create better UI to indicate the values of the lights. I am thinking of putting the value of the parameters next to the light and it will detect whether you are looking at that light to show that info.
  • Putting in characters that you can scale up/down and move around. 
  • Try it on animated scenes.
  • Take test shots with fake cameras

There are two different directions I can go for this project. One is more CG-artist centered, which will bring more questions about how to improve the lighting and export it to be used in the art pipeline. Which will involve:

  • How to improve the lighting quality. Since all the lights right now are movable lights, the quality of lighting is not desirable. How can will use tricks to make it better? How to bring more aspects of CG lighting into the tool (e.g. HDRi)?  
  • How to export the lighting setup

The other way is to wow the photographers, theatrical lighting designers, etc. This will need me to:

  • Model the real light and replace point light, directional light and spot light. I need to talk more with the people in this industry and figure out how to.
  • Think about real use case that people want to use this tool to better help their design

One thing I would also like to try out is to connect Virtual Reality with Augmented Reality. I will keep exploring this problem 🙂