Home About Us Links Sandbox Contact
sandbox / dynamic focus study
December 19th, 2004
Dynamic Focus Study

Dynamic User Focus through focus bracketing and automatically generated depth map.

My objective with this study was to address a basic problem with layering focus bracketed shots. As shown when you play the movie below, as you focus a lens the field of view of the lens changes. In the example below I used a Canon 50mm 1.8 lens at 1.8 and took 15 shots. The FOV changed by about 10% from one end to the other.

So why do I want to layer focus bracketed shots anyway? Well, there could be a number of uses but what I am after is a more interactive experience for the viewer. Much of what we do as photographers takes control away from the viewer. We choose where to view the scene from, how it is composed and the lighting. We also choose the depth of field and the depth for the focus plane. My goal is to give the focus plane back to the viewer.

Once I have layered the shots then it is just a small step for me to allow the user to control the focus plane by having them move the cursor on screen over different points on the image. The area at the cursor becomes in focus as does anything else at that depth in the captured scene. A better solution than the cursor may be a touch screen, or even better how about a camera looking back at the viewer that looks at eye movement to detect where the viewer is looking. May sound far fetched but technology is already used in a similar fashion in consumer cameras.

One step I left out was how to use cursor location to control the focus plane. Read on below to learn about how I automate the generation of this as well as more of my ideas about giving control back to the viewer.


As described earlier the movie above shows shots as taken out of the camera. You can see that the field of view captured changes as the focus plane moves.


This movie has been corrected for the change in field of view. I used the yellow crosses as reference points to generate a profile for this lens. There was about a 10% change in FOV through the focus range.


This shows the start of the automated process to generate a depth map from focus bracket shots. This shows the difference masks between adjacent shots. Read below for more detail.

So how to I have the computer interpret the cursor location to come up with the correct focus plane? Well I make a depth map. I could try and do this by hand based on my observations of the scene, but it occurred to me there may be a better way. I don't understand the technology behind modern auto focus systems, but it is my understanding that as the camera searches for the correct focus distance to the subject that it looking for changes in localizes contrast at focus point on the subject. So I thought; I have a set of shots, what if I look at changes in the images from shot to shot. Any significant changes from one shot to the next should indicate that it is near the focus plane. So that is just what I did, the results of which you can see in the movie above.



As there is not detail at every point in the scene to help determine depth I used a series of blurs to try to get a better result for the entire scene. As you play the movie above time and change in color show an increase in distance from the camera. Below is a still image based on the source images of the movie above.


Final depth map from test shots. Red is closer to the camera and Violet is furthest away. Areas without color indicate locations where there was not adequate image detail to determine depth.

So as the caption above says this is my final depth map. Although I initially wanted to make it grayscale so I could use it with 3d software extrude the depth of the image, for now I decided that color was easier to read. Red is closest to the camera while violet is the furthest away. White (or gray) areas are where there was not enough image detail for any localized contrast changes to occur. The biggest example of this is the floor, however you may note that where there was a reflection on the floor, that it did pick up some depth data. Other areas lacking detail are the white wall at the middle left and the neck of the bear on left. Similarly to white areas not having depth data, the level of saturation is an indicator of how much localized contrast was found at the location. So it is possible that higher saturation areas have a higher level of accuracy then very faint colors.

Notes:
  • What software can be used to view this type of content. There may be a number of options, but my first thought is the SPI-V engine. This is mainly intended for panoramic content but if support is added for dynamic changes based on the navigation layer then this would work; although I'd have to think about how to have it crossfade through 10-20 images instead of just two images.
  • In addition to just using this for still images it could have other applications. For starters it could be used for interactive panoramic images. Using SPI-V engine in similar fashion to how it can show adaptive dynamic range.
  • So, what is next after dealing with giving control of focus back to the viewer? How about field of view. Here is what I'm thinking. Imagine a looking out a window in a room towards a distant landscape. Lets say you are standing 15 feet away from the window. What happens as you move closer? Well you see more of the scene, that is to say your field of view framed by the window has widened. So what if we mount something on a photo that can sense the viewers distance. At that point all we need to do as change the field of view of the image and we are set; you end up with more realistic representation of a scene that I imagine would be a little eerie to see in action.
  • To take this a step further you would also want to deal with changes in the viewers viewing angle. As they go from side to side or move up or down the view should change. At this point we are beyond just changing depth of field and would want to use a panoramic image as the source in stead of a traditional still image. I see tablet PC's as ideal for this. Imagine a full panoramic where you can spin around to change the direction you look and can zoom in and out by moving closer to or further away from the scene. It could be as if your tablet screen is a window into another world (not trying to be poetic, speaking more of a literal window).
  • I'm sure I'm not the first person to use this approach. For all I know it is used in astronomy and fields, but I'm not always good at research and I just like playing. If I'm missing some obvious way to make a good depth map of a scene; like maybe using a fish finder?, let me know, I don't want to re-invent the wheel. I would think that down the road it would be possible for the camera to focus bracket just as it does exposure bracket. This would make it an easier and more accurate process.
  • One thing I have not addressed here is depth of field. Although my technique would let the user set the focal plane, it does not give control of the depth of field. I guess the best approach would be to set a depth of field similar to what the human eye has.
  • I won't say that capturing focus bracketed video or on the fly depth maps would be impossible, but it would be daunting. Maybe would capture more traditional two lens 3d video and build depth data from that and then add in the blur later? CGI for video games on the other hand would seem pretty easy.
  • OK, last note, if the text here does not make sense or jumps from place to place, I apologize, I'm just writing a draft out of thin air before I forget my thoughts about it.

page last edited on 12.19.04
Home | About Us | Our Work | Links | Sandbox | Contact Us

Copyright © 2004 Inertia, LLC | Boulder, Colorado
www.inertia-llc.com | info@inertia-llc.com