Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections
Supporting Beyond-Surface Interaction for Tabletop Display Systems by Integrating IR Projections Hui-Shan Kao Advisor : Dr. Yi-Ping Hung Outline 2 Introduction Related Work System Design Interaction Techniques Applications Conclusion Outline 3 Introduction Related Work System Design Interaction Techniques Applications Conclusion Introduction 4 Interaction with surfaces mainly support Multi-touch Tangible input Interaction beyond surfaces Combine with mobile displays iPad iPhone Pico Projector Tablet PC Introduction 5 Two scenarios Type 1: Using Type Multi-resolution presentation Augmenting personal information 2: Using pico projector tablet PC Interactive 3D viewer Type2 Type1 Challenge 6 How to know the 3D spatial relationship with surface in real-time ? Type2 Type1 ? Outline 7 Introduction Related Work System Design Interaction Techniques Applications Conclusion Localization of Device 8 Enable 3D interaction on tabletop display Need to recognize the 6DOF of device The way to know 6DOF Magnetic tracker Penlight H.Song, T.Grossman, G. Fitzmaurice, F. Guimbretiere, A. Khan, R. Attar, and G. Kurtenbach. Penlight: combining a mobile projector and a digital pen for dynamic visual overlay. In Proc. CHI ’09, 2009 Localization of Device 9 Vision based tracker Markers with known structure Interactive handheld projector ARToolkit Visible Marker ! Xiang Cao, Clifton Forlines, and Ravin Balakrishnan. Multi-user interaction using handheld projectors. In Proc. UIST ’07, 2007. D. Wagner and D. Schmalstieg. ARToolKitPlus for Pose Tracking on Mobile Devices. In Proc CVWW’07: Proceedings of 12th Computer Vision Winter Workshop, 2007. Outline 10 Introduction Related Work System Design Hardware Configuration Interaction Techniques Applications Conclusion Hardware Configuration 11 IR Camera IR Projection invisible markers Color Projection visible content Color Projector IR Camera IR Camera IR Projector Mirror Hardware Configuration 12 Multi-touch Surface layer support touch force Diffuser layer display image Glass Placement Glass of glass layer and diffuser layer on top of diffuser Diffuser on top of glass Hardware Configuration 13 Option 1: Glass on top of diffuser pico-projection reflection • degrade illumination • damage user’s eye Picoprojector touch-glass diffuser Hard to Solve! IR camera IR & color projector IR camera Hardware Configuration 14 Option 2: Diffuser on top of Glass IR-projection reflection • spot effect in IR images Picoprojector diffuser touch-glass spot IR camera spot IR & color projector IR camera Hardware Configuration 15 Removing IR spot by using two cameras IR Camera 1 IR Camera 2 IR Camera(1) Weighting Mask (1) Stitched View IR Camera(2) Weighting Mask (2) Outline 16 Introduction Related Work System Design Interaction Techniques Adaptive Markers for Camera Estimation Dynamic Markers for Multi-Touch Applications Conclusion 3D Position Estimation 17 3D position? Type2 Type1 3D position? 3D Position Estimation 18 ARToolKitPlus for 3D estimation Fiducial marker Self-identify by ID ID = 0 Projecting a grid of markers on tabletop (0,0) Each marker with Unique ID Associated position Compute the camera’s position based on the marker perceived (1,1) Multi-Level Markers 19 Uni-level marker Camera might observe the markers too small or too big Multi-level marker System resizes the IR markers according to camera position Marker Split and Merge 20 Adapting the maker size Markers merge, when camera moves far enough Markers split, when camera moves too close Ensure camera to see at least 4 markers Cooperating with multi-touch 21 Marker on for camera positioning Marker off for finger detection Foreground-ROI detection for marker on/off 22 Cooperating with multi-touch 23 ? Cooperating with multi-touch 24 Background Simulation 25 Offline Save each marker as a patch image and record the position of marker (offline) :: for each marker _ base view = patch patch collection Background Simulation 26 Online As the layout re-arrange, the simulated background can be built by the saving patch in real time (online) :: for each marker in layout ( _ base view ) = patch of marker simulated background Background Simulation 27 ROI Generation 28 1st frame after finger touch 2nd and the other frames Summary of Foreground Detection 29 IR Camera of Mobile Device Applications Real scene Smoothing IR Cameras Color IR Projector Projector Kalman Filtering Observed Image Foregrounds Finger Touches ROI Tangible Objects Layout Manager Simulated Background Prediction Outline 30 Introduction Related Work System Design Interaction Techniques Applications Conclusion Three Applications 31 Three applications provide intuitive and natural manipulation. iLamp iFlashlight iView iLamp 32 iLamp Combine a pico projector and an IR camera Project seamless high-resolution content, bringing more detailed information Act as a desk lamp for personal use iLamp Rpc Tpc 34 [Rp |Tp ] = [Rpc |Tpc ] x [Rc |Tc ] Rp Tp ? Rpc Tpc Real-time Rc Tc Pico projector and camera calibration 35 Off-line : Find transformation between camera and pico projector The projector can be viewed as a dual of camera Rpc Tpc are the Rotation and translation of the attached camera Rpc Tpc Pico projector and camera calibration 36 On-line : Find transformation between pico projector and tabletop Estimate Rc Tc in real-time Compute Rp Tp [Rp |Tp ] = [Rpc |Tpc ] x [Rc |Tc ] Rpc Tpc iFlashlight 37 iFlashlight A mobile version of iLamp, can be moved easily. Multi-user cooperation iView 39 iView An intuitive tool to see 3D content or augmented information of the 2D map from different perspectives. Outline 41 Introduction Related Work System Design Interaction Techniques Applications Conclusion Conclusion 42 A new interactive surface based on the programmable invisible markers. Supporting both on-surface and above-surface interaction for any device outfitted with an IR camera. Bring another level of information on interactive surface. 43 Thank you
© Copyright 2024