Computational Photography

Computational Light Transport

by dr. Francho Melendez

today

  • computational light transport
  • quick summary of the course

computational light transport

capture light transport for relighting

manipulate light transport

control what light we record

indirect light

long range trasnport

scene geometry, normals...

  • light trasnport matrix
  • examples of light trasnport matrices
  • light trasnport calculations
  • optical computing

light trasnport matrix

light trasnport matrix

light trasnport matrix

light trasnport matrix

light trasnport matrix

capturing T

camera and projector aligned to be in the same place. T is simetric. Retriprocity of LT.

how to capture T

relighting

scanning

digital photography

pinhole camera

dimensionality reduction

aperture

depth of field

exposure

field of view - zoom

how to capture color

photons to RAW

demosaicing


white balance

gamma correction


compresion JPEG

RAW to JPEG

Instead of independent steps, all as one optimization problem

burst photography

burst photography

idea: take 2 or more images (bursts of images) and combine them

  • super-resolution
  • focal stack
  • aperture stack - confocal stereo
  • blurry/noisy
  • flash/no flash
  • HDR

expand resolution

idea: capture multiple low-res (LR) images and fuse them into a single super-resolved (SR) image

expand DOF

use highest gradient

depth from focus

Confocal Stereo [Hasinoff and Kutulakos 2007]

high + low resolution pairs

burry & noisy pairs

flash / no-flash photography

[Pettschnigg et al., 2004]

flash / no-flash photography

[Eisemann, Durand 2004]

non-photorealistic camera

high dynamic range imaging

tone mapping

[Durand and Dorsey, 2002]

mosaics and warping

expanding the field of view

  • Compact Camera FOV = 50 x 35°
  • Human FOV = 200 x 135°
  • Panoramic Mosaic = 360 x 180°

how does it work?

homography

$\begin{bmatrix}wx'\\wy'\\w\end{bmatrix} = \begin{bmatrix}* & * & *\\* & * & *\\* & * & *\end{bmatrix} \begin{bmatrix}x\\y\\1\end{bmatrix}$

$p' = Hp$

parametric (global) warping

forward VS backwards

Backward Mapping eliminate holes

Needs a invertible wrap function: Not always possible

non-parametric warping

  • Input correspondences at key feature points
  • Define a triangular mesh over the points
  • Same mesh in both images!
  • Warp each triangle separately from source to destination (affine)

internet data

Using Internet Billions of Images to...

  • Improve our photographs
  • tag
  • describe
  • fill holes
  • colorize
  • explore places
  • recognition
  • computer intelligence
  • learning to see

53,464 nouns, 79 million images. Very low resolution (32x32)

painting2GPS

search using sketches

coded photography

coded illumination

several llumination coding strategies

motion tracking

  • motion capture, unique temporal code for LEDs
  • e.g. phase space:
    • 960 fps
    • < 10 ms latency
    • visible or IR LEDs

pulse systems: LIDAR

continues wave

comercial devices

light stage 3

photometric stereo

spatial coding

assisted stereo

coded apertures

source of blur

image Formation

Deblurring is a deconvolution

deconvolution: 2 types

ill-posed problem

focus blur

aperture and PSF

aperture and PSF

  • Depth-dependent PSF
    • Engineer PSF to be depth invariant -> deconvolution is easier

  • Circular apperture is not (well) invertible
    • Engineer PSF to be broadband (flat Fourier magnitudes) -> well posed (Coded Appertures)
    • Coded apperture for depth recovery
    • Coded apperture based on perception

focal sweep

coded apertures

coded apertures

coded apertures

engineered PSF

coded exposures

coded exposures

coded exposures

motion invatiant blur

parabolic sweep

coded rolling shutter

light fields

light field

taxonomi

spatial multiplexing

other applications

computational displays

dimensions of the plenoptic function

  • contrast and luminance
  • color gamut
  • spatial resolution
  • temporal resolution
  • angular resolution
  • other designs

modern displays

perceptual considerations

apparent brightness

perceptual considerations

afterimages

dual modulation

color gamut

  • more saturated primaries
  • larger number of primaries

radiometric correction

wobbing display

temporal upsampling

  • black data insertion: limits hold-type blur
  • on-off backlight: possilble with LED backlights
  • interpolation along motion: needs optical flow estimation
  • warp between frames

perceptually-motivated temporal upsampling

angular resolution

layered 3D display

generalize paralax barriers

transparencies

what to put in the layers? computation

polarization fields

dual layered display

takes advantage of temporal integration

dual layered display

simulation

Light Transport

light trasnport matrix

light trasnport matrix

light trasnport matrix

camera and projector aligned to be in the same place. T is simetric. Retriprocity of LT.

relighting

scanning

femto-photography

what is femto-photography?

using time of flight cameras

Transient Light Transport

summary

plenoptic function

  • $L = L(x,y,z, \theta, \phi, \lambda, t)$
  • plenoptic imaging

  • coded photography

  • computational displays

  • capturing light transport

plenoptic imaging (epsilon photography)

coded photography

computational displays

  • 3D Displays
  • Light Field Displays
  • HDR Displays
  • Volumetric Displays

capturing light transport

  • Image Relighting
  • Separate direct from indirect illumination
  • Separate relflectance components
  • Femtophotography

objectives

knowledge

  • Get an overview of the field
  • Learn concepts of Digital Photography
  • Understand how different C.P. techniques work at high-level
  • Understand the fundamentals of this techniques

skills

  • Experience with implementing image processing algorithms
  • Experience with the practicalities of Digital Photography
  • Review a technical paper
  • and present it

attitudes

  • Reflect about the future of Photography
  • Reflect about future applications of Photography
  • Reflect about the importance of cameras at present and future
  • Look at photography with a different mindset

your opinion

  • what did you like?
  • what didn't you like?
  • too high level?
  • more mathematical tools?
  • more optimization tools?
  • more freedom in labs/projects?
  • comments?

anouncements

  • marks for the labs should be in USOS today
  • that will be your mark for the course,...
  • ...unless you are missing some labs
  • they should sum up more than 50%
  • late labs have a 20% penalization...
  • unless justified...
  • but can't make you fail
  • extras and optimizaitons were taken into account

SIGGRAPH Volunteers

9 February 2016: Application Deadline