# Computational Photography

## Computational Light Transport

by dr. Francho Melendez

## today

• computational light transport
• quick summary of the course

# computational light transport

capture light transport for relighting

manipulate light transport

control what light we record

indirect light

long range trasnport

scene geometry, normals...

## capturing T

camera and projector aligned to be in the same place. T is simetric. Retriprocity of LT.

how to capture T

# burst photography

## burst photography

idea: take 2 or more images (bursts of images) and combine them

• super-resolution
• focal stack
• aperture stack - confocal stereo
• blurry/noisy
• flash/no flash
• HDR

## expand resolution

idea: capture multiple low-res (LR) images and fuse them into a single super-resolved (SR) image

## depth from focus

Confocal Stereo [Hasinoff and Kutulakos 2007]

## flash / no-flash photography

[Pettschnigg et al., 2004]

## flash / no-flash photography

[Eisemann, Durand 2004]

## tone mapping

[Durand and Dorsey, 2002]

# mosaics and warping

## expanding the field of view

• Compact Camera FOV = 50 x 35°
• Human FOV = 200 x 135°
• Panoramic Mosaic = 360 x 180°

## homography

$\begin{bmatrix}wx'\\wy'\\w\end{bmatrix} = \begin{bmatrix}* & * & *\\* & * & *\\* & * & *\end{bmatrix} \begin{bmatrix}x\\y\\1\end{bmatrix}$

$p' = Hp$

## forward VS backwards

Backward Mapping eliminate holes

Needs a invertible wrap function: Not always possible

## non-parametric warping

• Input correspondences at key feature points
• Define a triangular mesh over the points
• Same mesh in both images!
• Warp each triangle separately from source to destination (affine)

# internet data

Using Internet Billions of Images to...

• Improve our photographs
• tag
• describe
• fill holes
• colorize
• explore places
• recognition
• computer intelligence
• learning to see

# coded illumination

## motion tracking

• motion capture, unique temporal code for LEDs
• e.g. phase space:
• 960 fps
• < 10 ms latency
• visible or IR LEDs

## continues wave

comercial devices

assisted stereo

# coded apertures

## deconvolution: 2 types

ill-posed problem

# focus blur

## aperture and PSF

• Depth-dependent PSF
• Engineer PSF to be depth invariant -> deconvolution is easier

• Circular apperture is not (well) invertible
• Engineer PSF to be broadband (flat Fourier magnitudes) -> well posed (Coded Appertures)
• Coded apperture for depth recovery
• Coded apperture based on perception

# computational displays

## dimensions of the plenoptic function

• contrast and luminance
• color gamut
• spatial resolution
• temporal resolution
• angular resolution
• other designs

## color gamut

• more saturated primaries
• larger number of primaries

## temporal upsampling

• black data insertion: limits hold-type blur
• on-off backlight: possilble with LED backlights
• interpolation along motion: needs optical flow estimation
• warp between frames

## layered 3D display

generalize paralax barriers

transparencies

what to put in the layers? computation

simulation

# Light Transport

## light trasnport matrix

camera and projector aligned to be in the same place. T is simetric. Retriprocity of LT.

# summary

## plenoptic function

• $L = L(x,y,z, \theta, \phi, \lambda, t)$

### computational displays

• 3D Displays
• Light Field Displays
• HDR Displays
• Volumetric Displays

### capturing light transport

• Image Relighting
• Separate direct from indirect illumination
• Separate relflectance components
• Femtophotography

# objectives

## knowledge

• Get an overview of the field
• Learn concepts of Digital Photography
• Understand how different C.P. techniques work at high-level
• Understand the fundamentals of this techniques

## skills

• Experience with implementing image processing algorithms
• Experience with the practicalities of Digital Photography
• Review a technical paper
• and present it

## attitudes

• Reflect about the future of Photography
• Reflect about future applications of Photography
• Reflect about the importance of cameras at present and future
• Look at photography with a different mindset

• what did you like?
• what didn't you like?
• too high level?
• more mathematical tools?
• more optimization tools?
• more freedom in labs/projects?

### anouncements

• marks for the labs should be in USOS today
• that will be your mark for the course,...
• ...unless you are missing some labs
• they should sum up more than 50%
• late labs have a 20% penalization...
• unless justified...
• but can't make you fail
• extras and optimizaitons were taken into account