Composing

Projection

Having determined the absolute rotation matrix for every single source image, and given the angle of view (which can be derived from the focal length, which is assumed to be known), we basically have a pixel mapping onto a sphere around the viewpoint. We can then use an arbitrary projection to visualize the sphere. We implemented three different projections, a planar one, a spherical one (limited to 180 degrees views), and an angular one (good for 360 degree panoramas, but distorts for large vertical span). All of the images we show here are in angular projection.

Blending

In the ideal case, once the source images are registered, they can be overlaid to get the resulting panorama. In practice though there are small errors in the registration, perspective distortions, and nonlinear barrel distortions. Therefore the images will not fit exactly at their respective edges.

Hence there is a need to smoothly blend over between images. When the source images are "faded" in towards their center, then in practice the seams that are visible in simpler overlay solutions are virtually gone. Sometimes this blending also reduces ghosts resulting from moving objects in the scene.

Figure 1 shows a possible arrangement of source images. It can clearly be seen that multiple (more than two) images can overlap at the same time. The blending algorithm has to weigh each image's pixels in the overlap region with a factor such that the edge pixel's weight is zero. As soon as there is no overlap any more, the weight has to approach one. In between a smooth transition has to occur. Adding the irregular geometry of the arrangement, it is obvious that an exact solution to this problem will be rather complicated [Burt83].

Figure 1: Several registrated images

We therefore resorted to a simple heuristic that is very easy in it's implementation but still yields very satisfactory results. The algorithm simply measures the distance of a given point P (see Figure 1) to the closest edge in a given source image. The pixel weight is then proportional to this distance to some power n:

where M is the number of overlapping images and d the distance to a source image's closest edge. In practice, a value for n between 3 and 4 has proven to yield the best results. The higher this parameter n is, the sharper the transition between the images, and the less blurred the transition regions appear. However, in the presence of large registration errors a smaller n can at least partly disguise the misalignments. Approaches similar to this are widely used in image mosaicing and have been described e.g. by Bao et al. [Bao98].

Examples

Figures 2 and 3 show an example of a case of gross misregistration. In addition, the leaves of the tree are moved by the wind between the shots, so the tree looks extremely blurred in the un-blended image. This can be compensated to a certain extent by the blending. Some artifacts and shadows still remain in the enhanced version, but it looks a lot better then the original version.

[bingnosmooth.jpg]
Figure 2: Composing without blending

[bingsmooth.jpg]
Figure 3: Composing with blending

Exposure Compensation

One problem that arised as we were rendering our first panoramas was the presence of contrast discontinuities in the final image, that, beside making the panorama pretty unrealistic, made seams visible unless a very heavy averaging of the images was done. It actually comes from the fact that digital cameras have automatic gain control, which leads to different exposure levels in adjacent images. To compensate for this, we proceed approximately as we do for spatial registration, except that we now work in the pixel value space instead of pixel coordonnates space. Similarly we first need to model the pixel value transformations between images. We should ideally account for gamma correction, exposure level differences, and any non-linear operation achieved by the camera.

We first decided to model this transformation by a linear mapping in the intensity (gamma-corrected) space. The registration between images consisted in a linear regression on a subsampling of the pixels in the overlapping area:

However we realised that this method was yielding incorrect results. Indeed a slight spatial misregistration of the images introduces some non-correlated noise in the linear regression, which has the effect to flatten the correlation slope and increase accordingly the constant factor of the linear mapping. Which lead to situations where the derived transform from A to B was not the inverse from the the transform from B to A.
Consequently, we considered the constant factor as irrelevant and restricted to a relative gain between images. We also simplified the exposure registration to make it rougthly independent from the quality of the spatial registration. We actually computed the gain as the ratio of the mean luminances for each images in the overlap area.

Global registration consists in running all the relative gains through the least-squares global optimizer and derive an unique absolute gain for each image. That gain is used in the blending operation to adjust the image brightness according to the mapping Image/valueMapping.GIF

Exposure Correction Examples

The following images show the kind of artefacts that appear due to different exposure levels in adjacent images and the compensated result : the images of the back of the room are taken against the light, so they seem very dark compared to the wall on the right. In the right image, the gain discontinuity is corrected by darkening the wall.

[noexpcomp.jpg]
Figure 3: without compensation

[.jpg]
Figure 4: with compensation

Previous (Global Registration) · Up (Main) · Next (Results)