Computational Photography

Computational Imaging II

by dr. Francho Melendez

lab 2

difficulties?
questions?
Did you try other averaging methods?
Bilateral Filter?

today's schedule

Recap
Panoramas,Mosaics
Some cool papers
Warping
Homographies
Lab 3

recap

photons to RAW

RAW to JPEG

take 2 or more images and combine them to get a "better one"

expand resolution

idea: capture multiple low-res (LR) images and fuse them into a single super-resolved (SR) image

expand DOF

use highest gradient

depth from focus

Confocal Stereo [Hasinoff and Kutulakos 2007]

high + low resolution pairs

burry & noisy pairs

flash / no-flash photography

[Pettschnigg et al., 2004]

flash / no-flash photography

[Eisemann, Durand 2004]

non-photorealistic camera

high dynamic range imaging

tone mapping

[Durand and Dorsey, 2002]

Mosaics

expanding the field of view

Compact Camera FOV = 50 x 35°

Human FOV = 200 x 135°

Panoramic Mosaic = 360 x 180

expanding the field of view

Compact Camera FOV = 50 x 35°
Human FOV = 200 x 135°
Panoramic Mosaic = 360 x 180°

old idea

how does it work?

simple overview

4 correspondances
reproject one on the other

more complete overview

Automatic correspondances: Feature detector
Projection in a different space
Blending

a pencil of rays contains all views

same center of projection!

image reprojection

do we need geometry of the two planes in respect to the eye?

2D transformation

but what kind of transformation?

2D transformation

but what kind of transformation?

Projective – mapping between any two projection planes with the same center of projection

rectangle should map to arbitrary quadrilateral
parallel lines aren’t
but must preserve straight lines
same as: project, rotate, reproject

called Homography

homography

$\begin{bmatrix}wx'\\wy'\\w\end{bmatrix} = \begin{bmatrix}* & * & *\\* & * & *\\* & * & *\end{bmatrix} \begin{bmatrix}x\\y\\1\end{bmatrix}$

$p' = Hp$

more later...

Some cool things with mosaics

Interactive Digital Photomontage [Agarwala et al. 2004]

Interactive Photomontage

Let there be color - Large scale Texturing of 3D reconstructions [Michael Waechter et al. 2014]

Large scale Texturing of 3D reconstructions

Capturing and Viewing Gigapixel Images [Kopf et al. 2007]

Locally Adapted Projections to Reduce Panorama Distortions [Kopf et al. 2009]

Panorama Distortions

Panoramic Video from Unstructured Camera Arrays [Perazzi et al. 2015]

Transfusive Image Manipulation [Kaan Yücer et .al 2013]

Transfusive Image Manipulation

Initialize with sift images, compute Wrap between images

Transfusive Image Manipulation

Content-Preserving Warps for 3D Video Stabilization [Feng Liu el al. 2009]

Unwrap mosaics [Rav Acha el al. 2008]

Unwrap Mosaics

Dynamic Video Narratives [C. Correa el al. 2010]

Warping

image warping

parametric (global) warping

2D linear transformations

as 2D matrices (translation cannot be expressed as a 2D matrix)

homogeneous coordinates

affine transformation

projective transformation

2D image transformations

These transformations are a nested set of groups Closed under composition and inverse is a member

warping

Given a coordinate transform (x’,y’) = T(x,y) and a source image f(x,y), how do we compute a transformed image g(x’,y’) = f(T(x,y))?

forward warping

inverse warping

forward VS backwards

Backward Mapping eliminate holes

Needs a invertible wrap function: Not always possible

non-parametric warping

Input correspondences at key feature points
Define a triangular mesh over the points
Same mesh in both images!
Warp each triangle separately from source to destination (affine)

morphing sequence

Create an intermediate shape (by interpolation)
Warp both images towards it
Cross-dissolve the colors in the newly warped images
$Image_{halfway} = (1-t)*Image_1 + t*image_2$

homographies

a pencil of rays contains all views

same center of projection!

homography

$\begin{bmatrix}wx'\\wy'\\w\end{bmatrix} = \begin{bmatrix}* & * & *\\* & * & *\\* & * & *\end{bmatrix} \begin{bmatrix}x\\y\\1\end{bmatrix}$

$p' = Hp$

2 centers: does it work?

planar scene (or far away)

PP3 is a projection plane of both centers of projection, so we are OK

for a general scene?

getting H

Find the homography H given a set of p and p’ pairs
How many correspondences are needed?
Tricky to write H analytically, but we can solve for it!

getting H

$\begin{bmatrix}wx'\\wy'\\w\end{bmatrix} = \begin{bmatrix}a & b & c\\d & e & f\\g & h & i\end{bmatrix} \begin{bmatrix}x\\y\\1\end{bmatrix}$

9 unknowns and $w$

$w$ is easy: $w = gy + hx + i$

Set up a system of linear equations:

$Ah = b$

where vector of unknowns $h = [a,b,c,d,e,f,g,h]^T$

Need at least 8 eqs, (setting $i=1$)

Solve for h. If overconstrained, solve using least-squares

$min\|Ah-b\|^2$

blending

Blending Problem

Smooth Blending

Easier to compute in the original space

Result of the weighted average

Smooth blending Problem

Two bands blending

Gaussian filter to separate frequencies

Two bands blending

Low frequencies with smooth blending

Two bands blending

High frequencies use the pixel with biggest weight

Two bands blending

today's lab

Due: 16th November

Warping with Homographies
Compute homographies
Manual Mosaic
Blending

announcements

http://franchomelendez.com/Uwr/teaching/COMPHO/_LECTURES/L4/computational_imaging_II.html

http://franchomelendez.com/Uwr/teaching/COMPHO/Labs/Lab3.zip

franchomelendez@cs.uni.wroc.pl

credits and references and aditional readings

These slides have been prepared with materials, slides, and discussions from the following.

Fredo Durand, Alexei Efros, Videos and papers from the authors