Computational Photography

Computational Illumination

by dr. Francho Melendez

today's schedule

Recap
coded illumination

Flash-no flash
Temporal coding
Directional coding
Spatial coding
Some SIGGRAPH

Lab 4

context

computational photography

plenoptic function

plenoptic imaging
coded photography
computational displays
capturing light transport

computational photography

plenoptic imaging (epsilon photography)

Dynamic range

HDR Imaging

Spectral imaging

Asorted Pixels

Angular Information (light fields)

Light Fields

Field of view

Panorama Stitching

Depth of field

Flexible DOF Photography

Space

Super-resolution, Gigapixel imaging

coded photography

Coded Illumination

Flash/No-flash photography

Coded Exposure

Flutter Shutter

Coded Apertures

Mask on lens

overview

coded illumination

Flash-no flash
Temporal coding
Directional coding
Spatial coding
Some SIGGRAPH

flash no-flash

denoising

Cross-Bilateral Filter based Approach

removing flash artifacts

[Agrawal, Raskar, Nayar, Li; Siggraph05]

removing flash artifacts

[Agrawal, Raskar, Nayar, Li; Siggraph05]

removing flash artifacts

[Agrawal, Raskar, Nayar, Li; Siggraph05]

removing flash artifacts

[Agrawal, Raskar, Nayar, Li; Siggraph05]

removing flash artifacts

dark buildings
unkown shapes
reflections

removing flash artifacts

well-lit buildings
tree, street shapes
reflections on windows

removing flash artifacts

combination

removing flash artifacts

The algorithm consists of the following steps: First, we compute the gradient fields of the daytime and nighttime input images by using simple forward differencing in the x and y direction. Thresholding the gradient field images allows us to compute the locally-important areas. These are areas of high variance in the nighttime image, shown here in white. The pixels of the importance image are white in the locally-important areas that are taken from the nighttime image and black in the context areas that are taken from the daytime image. We use prost-processing (eroding, fattening and feathering) to consolidate the selected areas and ensure smooth transitions. A mixed gradient field is computed as a weighted mean of the input gradient fields, using the pixel values in the importance image as weights. The final result is obtained by integrating the mixed gradient field.

removing flash artifacts

The algorithm for videos is a generalization of the one for dynamic scenes, using video frames instead of multiple snapshots taken at nighttime. The algorithm consists of the following steps: First, we compute the gradient fields of the daytime input image by using simple forward differencing in the x and y direction. Then, for each video frame, we start by also computing the gradient fields. The next step is to use the previous frames compute a mask image that highlights the areas that depict the events. In this case, the event is the movement of the cars on the highway. The mask is computed using thresholded pixel differences between successive frames, followed by prost-processing (eroding, fattening and feathering) to consolidate the selected areas and ensure smooth transitions. Thresholding the gradient field of the current frame allows us to compute the locally-important areas. We combine the areas that depict the events with locally-important areas into an importance image. Again, areas in white are to be taken from the nighttime image, and areas in black from the daytime image. To use temporal hysteresis, we average the importance images for the last K frames. We then compute the mixed gradient field as a weighted mean of the input gradient fields, using the pixel values in the importance image as weights. The final result is obtained by integrating the mixed gradient field.

several llumination coding strategies

temporal coding

motion tracking

motion capture, unique temporal code for LEDs
e.g. phase space:

960 fps
< 10 ms latency
visible or IR LEDs

time of flight

pulse systems: LIDAR

Light Detection And Raging

Typically requires scanning

$t = \frac{2d}{c}$

$c \approx 3 * 10^8 \frac{m}{s}$

pulse systems: LIDAR

small, usually need scanning

pulse systems: LIDAR

continues wave

comercial devices

continues wave

kinect fusion

directional coding

additive nature of light

light stage

light stage 1

light stage: used in Spiderman 2

light stage 3

light stage 6

light waving

Estimating Light Positions From Photographs Alone

image-based relighting

image-based relighting: lab

photometric stereo

diffuse (Lambertian) surfaces are viewpoint independent

$I = \rho\begin{bmatrix}L_x & L_y & L_z\end{bmatrix}\begin{bmatrix}N_x\\ N_y \\ N_z\end{bmatrix}$

photometric stereo

$\begin{bmatrix}I^{(1)} \\ I^{(2)} \\ I^{(3)} \end{bmatrix} = \rho \begin{bmatrix} L_x^{(1)} & L_y^{(1)} & L_z^{(1)} \\ L_x^{(2)} & L_y^{(2)} & L_z^{(2)} \\ L_x^{(3)} & L_y^{(3)} & L_z^{(3)} \end{bmatrix} \begin{bmatrix} N_x\\ N_y \\ N_z \end{bmatrix}$

$I = L \cdot N$

diffuse (Lambertian) surfaces are viewpoint independent

assume albedo is constant, invert matrix $N = L^{-1} \cdot I$

photometric stereo

spatial coding

kinect pattern

structure lighting

some SIGGRAPH papers

Separation of Global and Direct Illumination

Doppler Time-of-Flight Imaging

Relighting with 4D incident light fields: relighting

Relighting with 4D incident light fields: capture

Relighting with 4D incident light fields: relighting

Confocal Imaging

Lab 4:poisson compositing

compositing in the gradient domain

paste vector field

Paste source gradient into target image inside a selected region
Make the new gradient as close as possible to the source gradient while respecting pixel values at the boundary

discrete 1D minimization

big quadratic

derivatives

set derivative to 0

recap

Interpolation

discrete 1D minimization

intuition

in 2D: membrane interpolation

what when v is not null?

today's lab

Due: 30 November

Poisson Reconstruction and Blending

announcements

http://franchomelendez.com/Uwr/teaching/COMPHO/_LECTURES/L6/computational_illumination.html

http://franchomelendez.com/Uwr/teaching/COMPHO/Labs/Lab5.zip

franchomelendez@cs.uni.wroc.pl

credits and references and aditional readings

These slides have been prepared with materials, slides, and discussions from the following.

Ramesh Raskar

Paul Debevec

Richard Szeliski

Computational Photography

Computational Illumination

today's schedule

context

computational photography

plenoptic function

plenoptic imaging

coded photography

computational displays

capturing light transport

computational photography

plenoptic imaging (epsilon photography)

coded photography

overview

coded illumination

flash no-flash

denoising

denoising

removing flash artifacts

removing flash artifacts

removing flash artifacts

removing flash artifacts

removing flash artifacts

removing flash artifacts

removing flash artifacts

removing flash artifacts

removing flash artifacts

several llumination coding strategies

temporal coding

temporal coding

motion tracking

time of flight

pulse systems: LIDAR

pulse systems: LIDAR

pulse systems: LIDAR

pulse systems: LIDAR

continues wave

continues wave

continues wave

continues wave

continues wave

continues wave

continues wave

continues wave

continues wave

kinect fusion

kinect fusion

directional coding

directional coding

additive nature of light

light stage

light stage 1

light stage: used in Spiderman 2

light stage 3

light stage 6

light waving

image-based relighting

image-based relighting

image-based relighting: lab

photometric stereo

photometric stereo

photometric stereo

photometric stereo

photometric stereo

photometric stereo

spatial coding

spatial coding

spatial coding

spatial coding

spatial coding

spatial coding

kinect pattern

kinect pattern

structure lighting

structure lighting

structure lighting

some SIGGRAPH papers

Separation of Global and Direct Illumination

Separation of Global and Direct Illumination

Doppler Time-of-Flight Imaging