ADVANCED MULTIMEDIA IMAGING

Professor Reinhard Klette


This course is on various subjects in Multimedia Imaging, such as camera calibration, video recording, video processing, computer vision (stereo and motion analysis), 3D video visualization, computer graphics (animated visualization of results), and performance evaluation. These are taught in the context of one multi-facetted project (seee .enpeda..) on driver assistance.

Prerequisites: Experience in programming (in particular, C/C++ and OpenCV; either Windows or Linux is fine) and in digital imaging.

The course assignments count 30% of the total mark, and the exam 70%. There are two individual assignments:

(1) implement and test one stereo algorithm for recorded stereo sequences, and
(2) implement and test one motion analysis algorithm.

For each assignment, a report needs to be submitted by the due date (as announced in lectures), and solutions and data need to be explained in lab meetings or short seminars.


Students will collect (as arranged by lecturer) their own test data by using stereo cameras in a test vehicle.

Left to right: HAKA1, stereo cameras behind windscreen, scene flow analysis, 3D visualization for ETRI visitors, computer graphics for generating synthetic stereo sequences, camera calibration
(1) Course intro, driver assistance, digital imaging (rehearsal)
  1. Course Intro; Driver Assistance at Daimler A.G. (MI00_AMI_2008, with contributions by Uwe Franke et al.)
  2. Introducing HAKA1 (MI65)
  3. Tutorials: Rehearsal: Digital Imaging (MI55)
    Reading material (by Tobi Vaudrey on OpenCV): Handout OpenCV
    Tutorial 1: Basic image analysis
    Tutorial 2: Sequence analysis and optical flow
(2) Static and dynamic imaging, calibration, stereo analysis, stereo viewing
  1. Static or Dynamic Imaging (MI49, with contributions by Stefan Gehrig and Tobi Vaudrey)
  2. Basics of Camera Calibration (MI38, with contributions by Tobi Vaudrey)
    See also the Camera Calibration Toolbox for Matlab (by J.-Y. Bouguet) and references as given on this website
  3. Binocular Stereo (MI19)
    See also engineered image pairs with ground truth (backup of map, sawtooth, tsukuba, and venus)
    and real-world sequences with approximate ground truth for evaluating stereo algorithms
  4. Lab meeting about recorded video sequences, with
    a brief intro into stereo visualization for 3D video presentations and
    the mapping of image sequences into avi-clips
  5. Correspondence Analysis (Simple Methods) (MI54)
  6. Belief Propagation (MI66, with contributions by Shushi Guan)
    For calculating the lower envelope of a quadratic cost function, see Euclidean Distance Transform (MI30, by Gisela Klette)
  7. Dynamic Programming (MI68, with contributions by Zhifeng Liu)
  8. Lab meeting about Stereo Analysis (with brief presentations of all students)

    SUBMISSION OF FIRST ASSIGNMENT ON STEREO SEQUENCE ANALYSIS
    Latex template for report (provided by Gisela Klette, for optional use)

(3) Optic flow, motion analysis, object tracking
  1. Optic Flow: Basics; Horn-Schunck Algorithm (MI20)
    see also Derivatives for Optic Flow (MI59) (by John Barron)
  2. Lucas-Kanade Optic Flow Algorithm (MI50, with contributions by John Barron)
  3. BBPW Optic Flow Algorithm (MI50, by James Milburn)
  4. Comparing stereo and motion analysis techniques on various samples of data
  5. Lab meeting about Motion Analysis (with brief presentations of all students)

(4) Kalman Filtering
  1. Kalman Filter (MI37, with contributions by Karsten Knoeppel)
  2. Kalman Filtering: Water Tank Example (MI63, by Tobi Vaudrey)
    (File KalmanExample.xls for interactive parameter testing, see Coursework 63.1)
  3. Kalman Filtering: Disparity Integration Example (MI64, by Tobi Vaudrey)
(5) Performance evaluation, ground truth

    SUBMISSION OF SECOND ASSIGNMENT ON MOTION ANALYSIS

(6) Further subjects
  1. Structured Light
  2. Photometric Stereo
  3. Wireless Video (with contributions by Ji Sun)
  4. 3D TV

Copyright for material on this website (if not otherwise specified): Reinhard Klette
SOFTWARE and MATERIALS

(1) There are free software downloads available for multimedia imaging, such as OpenCV, OpenGL, GIMP, and ImageJ. See also the software-link on The Computer Vision Homepage.

(2) On OpenCV, see materials prepared by Tobi Vaudrey for COMPSCI 375 in 2008: Handout OpenCV, Basic image analysis, and Sequence analysis and optical flow. See also the Intro into OpenCV by Gady Agam.


A FEW LINKS

Collections of links
Overview on driver assistance systems
Automatic parking
Web site on 3D shape recovery technologies

University research
The Drivsco Project
Vision-based driver assistance system
Self-driving bus at UCB
Urban 3D Modelling from Video

Company R&D
Inro - Advanced vhicle automation, New Zealand
Blackhawk Tracking Systems, New Zealand
Driver assistance systems at Hella Aglaia, Germany
Driver assistance systems at Hella KGaA Hueck and Co, Germany
Advanced transport systems by ULTra, UK, see this 2007 info
Connected cars 'promise safer roads', 6 July 2007 (printable version: here)

Conferences
Pacific-Rim Symposium on Image and Video Technology
3DTCV-CON
Int. Conf. Arts & Technology
Robot Vision
IEEE International Conference on Robotics and Automation (in 2008, over 1330 attendees and 661 papers presented)
Int. Symposium on Robotics
IEEE Intelligent Vehicles Symposium

Contests
Middlebury Stereo Evaluation
Middlebury Optic Flow Evaluation
DARPA Urban Challenge
DARPA Grand Challenge


A FEW REFERENCES

B. Leibe et a.: Coupled Object Detection and Tracking from Static Cameras and Moving Vehicles (2008)
H.M.Ozaktas and L. Onural (eds.): Three-Dimensional Television (2007)
Ernst D. Dickmanns: Dynamic Vision for Perception and Control of Motion (2007)
Klaus et al.: Segment-based stereo matching using belief propagation and a self-adapting dissimilarity measure (2006)
Zhao et al.: Local-global stereo matching algorithm (2006)
Franke et al.: 6D-vision - fusion of stereo and motion for robust environment perception (2005)
T. Dang et al.: Fusing optical flow and stereo disparity for object tracking (2002)
Barron and Klette: Quantitative color optical flow (2002)
Barron and Klette: Experience with optical flow in colour video image sequences (2001)
G.Egnal: Mutual information as a stereo correspondence measure (2000)
J. Miura et al.: An active vision system for real-time traffic sign recognition (2000)
Adelson et al.: Pyramid methods in image processing (1984)