Click for bigger pictures

As part of my thesis work, I am working on a new statistical approach for view synthesis. It is particular effective for texture-less regions and specular highlights, two major problems that most existing reconstruction techniques would have difficulty with. We are preparing to report our work to ICCV 2003. Some initial results are presented on the left, the top row shows several input images while the bottom row shows the reconstructed point cloud. 

A multi-resolution stereo algorithm that can be implemented on commodity graphics hardware. A paper will appear in CVPR 2003.  We will bring a live demo to CVPR as well. 

We present a novel use of commodity graphics hardware that effectively combines a plane-sweeping algorithm with view synthesis for real-time, on-line 3D scene acquisition and view synthesis. The heart of our method is to use programmable Pixel Shader technology to square intensity differences between reference image pixels, and then to choose final colors that correspond to the minimum difference, i.e. the most consistent color. We filed an invention disclosure with UNC.

My internship at Microsoft Research (MSR) during summer 2001 has focused on maintaining eye-contact for desktop video teleconferencing. we took a model-based approach that incorporates a detailed individualized three-dimensional head model with stereoscopic analysis. This approach is very effective; we probably achieved the most realistic results in published literature for eye gaze correction (see the images on the left; the middle is the synthesized view that preserves eye-contact, the other two are the input images.). In the process, we can also get very accurate 3D tracking results of the head pose. The images below show the face model projected on the tracked head. MSR has filed two patent applications for our algorithms and systems.

We want to design a system that facilitate many-to-many teleconferencing. Instead of providing a perceptively correct view for every single user, we strive to provide the best approximating view for the entire group as a whole. We demonstrate two real-time acquisition-through-rendering algorithms: one is based on view dependent texture mapping with automatically acquired approximating geometry, and the other uses an array of cameras to perform Light Field style rendering. 

The goal of Tele-Immersion is to enable users at geographically distributed sites to collaborate in real time in a shared, simulated environment as if they were in the same physical room. While the entire project was a interdisciplinary, multi-site collaboration, I was mainly invovled in in real-time data capture and distribution.

We worked on improving the field of view and resolution for 2D video teleconferencing. The result is a simple, yet effective technique for producing geometrically correct imagery for teleconferencing environments. The necessary image transformations are derived by finding a direct one-to-one mapping between a capture device and a display device for a fixed viewer location, thus completely avoiding the need for any intermediate, complex representations of screen geometry, capture and display distortions, and viewer location. Using this technique, we can easily build an immersive teleconferencing system using multiple projectors and cameras. 

The PixelFlex system is composed of ceiling-mounted projectors, each with
computer-controlled pan, tilt, zoom and focus; and a camera for closed-loop calibration. Working collectively, these controllable projectors function as a single logical display capable of being easily modified into a variety of spatial formats. The left image shows a stacked configuration that can be used for stereo display.