Overview

Browsing is a fundamental tool for multimedia systems. A good browsing tool should retain novel information and discard redundant information. During camera rotation, there is significant overlap of scene information between successive frames.Then a scene synopsis can be constructed by stitching or mosaicing'' individual frames. The resulting mosaic is an efficient representation of a sequence of frames because the redundant information in the frame sequence produced by the camera rotation has been discarded.

We have developed a Java application for generating mosaics. The user interface displays the successive frames at the top of the window, and the generated mosaic at the bottom. The two main operations are (1) the frame-to-frame correspondence detection, which must detect matching features or correlated groups of pixels, and (2) the image transform operation, which computes what the frames would look like in the frame of reference of a still camera with a very large field of view.

So far, our frame-to-frame correspondence has been defined by mouse clicks on matching point features on the successive frames. This approach allowed us to focus on the code for the second operation, the image transform, which is completed and produces satisfactory mosaics. The image transform operation is approximated by an affine transformation that combines a translation, a rotation, and a scaling. The 4 unknown parameters that describe the transformation are found by solving an over-determined system, in which each equation describes the information generated by a single frame-to-frame feature matching.

Focusing back to the first operation, we are now taking advantage of the fact that video sequences in the MPEG format already contain significant correspondence information. Indeed MPEG encoders generate the matchings between blocks of 16x16 pixels for most pairs of successive frames when they calculate the so-called motion vectors. We have developed tools that allow us to visualize the fields of motion vectors over frames, and images produced by these tools show that the fields of motion vectors are reliable and consistent with the observed camera motions. Our goal for the coming months is to complete a Java mosaicing application that takes an MPEG video sequence as an input and produces a summary sheet of this sequence that includes single key-frames for still shots and mosaiced key-frames for shots with significant camera motion.