Project: 3D Object Transfer

Description: 

Given two video sequences of different scenes acquired with moving cameras, it is interesting to seamlessly transfer a 3D object from one sequence to the other. In this paper, we present a video-based augmented reality approach to geometry-correctly transfer realistic 3D objects between two or more non-overlapping, moving video sequences. Our framework builds upon techniques in camera pose estimation, 3D spatiotemporal video alignment, depth recovery, key-frame editing, natural video matting, and image-based rendering. In our approach, we first estimate the camera pose for both source and target video sequences. Then, a global 3D alignment process is designed to minimize the camera pose difference and align the camera trajectories of the source and target videos in 3D space. After exploring the depth information over the whole video sequence, we explicitly apply it on key-frame segmentation to relieve the burdens of key-frame editing and also efficiently improve the quality of video matting. During the object transfer, our approach not only correctly restores the geometric deformation of the 3D object due to the different camera trajectories, but also effectively retains the soft shadow and environmental lighting properties of the object to ensure that the augmenting object is in harmony with the target scene.

Keywords: Video transfer, Video Cut-and-Paste, Video-based augmented reality.

Publications: 

  • Jiangjian Xiao, Xiaochun Cao, and Hassan Foroosh, "3D Object Transfer Between Non-overlapping Videos", IEEE Virtual Reality Conference, 2006. (PDF)
  • Jiangjian Xiao, Xiaochun Cao, and Hassan Foroosh, "A New Framework for Video Cut and Paste", International MultiMedia Modeling Conference, Beijing, China, Jan 4-6, 2006.

How does it work? 

Transfer of a 3D 'sit-talking' person between two non-overlapping videos captured by moving cameras: (a) Two original frames from the input source video. (b) Two result frames by a naive cut-paste process without the 3D alignment of the camera trajectories. It is obvious that the transfer is geometrically incorrect, where the pose, sitting location, and size of the inserted person are not consistent with the target scene since the source and target cameras have different motion. (c) Homography-based transfer by 2D reference plane alignment (the floor in (a) and desk in (b)), where the transferred person does "sit" in the same location on the desk throughout the whole target video but the 3D points out of these registered planes appear distorted. (d) shows the final results of our method, which are both geometrically correct and visually realistic.

Videos:

Source & Tranfer
Source, Transfer of two & three objects
More Examples