1. No category

4/8/2015
Title
Advances in 2D/3D Video Analysis:
using context & 3D depth information
Prof.dr.ir. Peter H.N. de With (TU/e VCA)
With contributions from:
(In cooperation with ViNotion)
Dennis vd Wouw, Willem Sanberg.
Egor Bondarev, Lingni Ma (TU/e VCA)
Slide - 1
EE- SPS / Video Coding & Architectures
• Faculty Electrical Engineering
• Dept. Signal Processing Systems
• Video is a rich area (computing, memory,
speed, bandwidth)
• Healthcare, AV Multimedia, PC systems,
Security, Geo-referencing, etc.
• Video Coding and Architectures
• Research group of about 25-30 researchers
and 3 postdoc staff members (approx. 2/3
PhDs)
• Covers Professional imaging
• Also implementations
• real-time architectures, fast coding
systems, special embedded applications
Slide 2
SPS-VCA Group Overview
Slide - 2
1
4/8/2015
SPS-VCA - (1) /
Video Content Analysis and Applications
Ship & context
detection, Watervisie
R’dam
Video Proc.
Fast & realtime SW
Traffic sign detection &
3D positioning,
CycloMedia Techn.
Content Analysis
Fall incidents & Medical
training for hospitals &
Kempenh. Inst.
Bank robbery
application analysis in
EU ITEA Cantata
Object detection, ViNotion
Scalable Video Coding &
Analysis, VDG Security
Content Anal. & RT prototypes,
spin-off
SPS-VCA Group Overview
Slide - 3
Slide 3
8-4-2015
Contents
1. Use of Context Information: improved ship
detection
2. Robust obstacle detection with 3D depth
3. Fast stixel proc.: depth-based object detection
4. 3D reconstruction of spaces/environments
Slide - 4
2
4/8/2015
Outline
1. Use of Context Information for
improved ship detection
Slide - 5
Ship Detection / Motivation and Goals
• Why detecting ships using video camera?
– Video-based ship detection is promising in port surveillance
Low cost
Easy to manage
Small, non-metal ships
Valuable supplement to radar system
• Challenges
Water, harbor infrastructure,
vegetation, etc.
– Dynamic and complex background
– Large variations of ship appearances in size, shape and color
• Our Goals
– Develop robust ship detection based on water region detection
– Find entire ship for various applications: size estimation, tracking,
etc.
6
Slide - 6
3
4/8/2015
Context: Water Detection（1/2）
False positives
can be reduced
• Why Context
– Ships only travel inside the water region
– Beneficial if water region is a-priori extracted and
provided as contextual information
• Context Extraction: Water Detection
Segmentation + Region-based Classification
Slide - 7
Context: Water Detection（2/2）
contextual information
for ship detection
Original surveillance
images
Results of segmentation
Classification of water
regions
8
Slide - 8
4
4/8/2015
Improved Ship Detection (1/2)
• Improvements:
• Detection of entire body of the ship, not only the cabin
• Merge the redundantly detected cabins for one ship
• Temporal information is added for improving the
detection coherence
• Refinement of the bounding box indicating the ship
9
Slide - 9
10
Slide - 10
Improved Ship Detection (2/2)
Flow Chart of Offline Demo
Ⅱ
5
4/8/2015
Demonstration of Results
• Comparison between 2 Demos
Video of Online Demo
Video of Offline Demo
11
Slide - 11
Conclusions on Ship Detection
• Conclusions
• Robust automatic ship detection for camera-based port
surveillance
• Water detection provides contextual information to facilitate ship
detection
• Temporal information improves the detection
• Enables multiple ship detection for both moving and static ships
• Future work
• Dealing with occlusion problems add robustness where needed
• Develop and exploit a tracking algorithm for combined
detection-tracking strategies to employ intelligent scenarios in
usage
12
Slide - 12
6
4/8/2015
Outline
2. Depth reconstruction
with stereo imaging
Robust collision
avoidance with 3D
Slide - 13
Next step in improved reliability in object detection
Possible extension for higher quality
• Apply local 3D reconstruction
• Overlapping cameras
• Advanced camera calibration
• Homography allows 3D reconstruction
• Enables distance calibration
• More accurate detection, higher robustness road sampling
Slide - 14
7
4/8/2015
Obstacle detection in 3D / distinguish from background
●
Alternative to change detection method
●
Two cameras to estimate depth
–
Similar to humans using two eyes to perceive depth
–
Measure displacement between L/R images → estimate depth
–
Obstacles have different depth than background
Left
Right
Displacement
Tram track
detection
Obstacle detection
in 3D
Object
recognition
Collision
prediction
Warning
Slide - 15
Facilitate object recognition with large variations
●
Automatic detection of most important traffic participants
–
Detect objects with large appearance variation in single image
–
Classes: pedestrians, cyclists, cars
–
Object class informative for risk
Pedestrian
Car
Cyclist
Tram track
detection
Obstacle detection
in 3D
Object
recognition
Collision
prediction
Warning
Slide - 16
8
4/8/2015
Stereo data acquisition / Experimental Setup – (1)
●
Mount camera on tram
●
Depth information desired (obstacle detection in 3D)
–
Two cameras
4/8/2015
Collision avoidance - Phase 1
Slide 17 of 33Slide - 17
Stereo data acquisition / Experimental Setup – (2)
●
●
Custom fabricated stereo-rig
–
Variable baseline
–
High-end 20 MPixel cameras
–
Accurate GPS positioning
–
Flexible: emulate simple system
Computer system for processing
–
Hexa core Intel i7-3960x
–
Dual high-end GPU
–
Large storage
4/8/2015
Collision avoidance - Phase 1
Slide 18 of 33Slide - 18
9
4/8/2015
3D Obstacle detection pipeline
Camera
interface
Depth
reconstruction
Track/ lane
detection
Ground surface
modelling
Obstacle
detection
Exploit depth to find obstacles
See if any collide with the trajectory
Segmentation
& semantics
Collision
prediction
Warn
driver
Slide - 19
Experimental Results – obstacle detection in 3D
●
●
Accurate localization of obstacles
Detect any object
(car, bike, tram, pedestrian, ….)
●
No learning and GPS required
●
Conclusion:
–
Obstacle detection is
straightforward with depth sensing
–
DEMO video
Slide 20 of 33Slide - 20
10
4/8/2015
Conclusions – Benefits for Object recognition
●
Enhance functionality: Can be used to classify an object
(e.g. car, pedestrian, cyclist)
●
Robustness against variations in light and object appearance
●
Conclusion:
–
Can be used to increase system functionality
–
Assign importance based on object class
(e.g. pedestrian more fragile than car, or security risk)
Slide - 21
Outline
3.
Fast stixel processing for
depth-based object detection
Slide - 22
11
4/8/2015
Stixel principle: processing concept
•
Column-wise probabilistic analysis
•
Models the world in rectangular column patches
•
•
•
Assumed geometry: linear parallellax with
superimposed fronto-parallel obstacles
Segment and label into ground or obstacle
•
Ground disparity:
•
Obstacle disparity:
≅
≅
Efficient and optimal solution
using Dynamic Programming
•
Cost function: log-likelihood
Baseline system:
D. Pfeiffer, “The Stixel World”, Ph.D. dissertation, Humboldt-Universitat Berlin, 2011
Slide - 23
Our Stixel proc. system: extension with color
Fuse Color data into core of the algorithm (assume independence)
∗
•
arg max
∈
•
•
~ ∏&'(
$)*
,
•
#$ , %$
Learn
| ,
$
~∏
3, - ,
#$ , %$
,.
12
0
)( ∏,),/
$
$
+, - ,
3, - ,
online and self-supervised
•
Starting with a uniform color-distribution for ground and obstacles,
update that model for each evaluated frame
•
Highly adaptive to new environments
Our system:
W.P. Sanberg, G. Dubbelman, P.H.N. de With,
“Extending the Stixel World with Online Self-supervised Color Modeling for Road-versus obstacle Segmentation”,
IEEE Conf. on Intelligent Transportation Systems (ITSC), Oct. 2014 [IN PRESS]
Slide - 24
12
4/8/2015
Experiments – Our Stixel proc. system
• Experiments with:
•
Color space
•
Number of histogram bins
•
Range of learning window
•
Sample selection
• Results:
•
Indexed RGB (adaptive!)
•
Single 3-seconds-old frame gives enough color information
(low computational constraints!)
•
Increase F-score ‘drivable distance’ from 0.86 to 0.97
Our system:
W.P. Sanberg, G. Dubbelman, P.H.N. de With,
“Extending the Stixel World with Online Self-supervised Color Modeling for Road-versus obstacle Segmentation”,
IEEE Conf. on Intelligent Transportation Systems (ITSC), Oct. 2014 [IN PRESS]
Slide - 25
Future outlook – Our Stixel system
Slide - 26
13
4/8/2015
Future outlook – Our Stixel system
Slide - 27
Outline
4. Multi-modal 3D
reconstruction
processing
Algorithms for full-3D
Segmentation of objects
Slide - 28
14
4/8/2015
Sensing Platforms
Set of 2D/3D Sensing Platforms
• Few laser-scanner platforms (LIDARs)
• RGB-D mapping system (Kinect)
• Stereo Cameras, Panoramic 360 camera
• Autonomous robot
Slide - 29
Multi-modal 3D Reconstruction processing – (1)
Full 3D Reconstruction
Model of environment
Slide - 30
15
4/8/2015
2D/3D Data Fusion Pipeline
Pointclouds
from laser-scan
Point- cloud filtering &
Mesh generation
Generated 3D meshed model
Mapping the textures
on meshes
Enhancement &
Alignment of meshes
3D model with mapped texture
Total volumetric 3D model
Image samples from 360-camera
Slide - 31
Experim. Results: Detailed Reconstruction & Texturing
Slide - 32
16
4/8/2015
Technologies / Significant processing…
Simultaneous Localization and Mapping (SLAM):
For both mono-camera and RGB-D sensors
Real-time
Loop closure and bundle adjustment
Point-cloud filtering and registration
3D Segmentation, decimation and semantic classification
Kinfu Large Scale (Kinect pipeline)
Multi-view texturing of point clouds and meshes
Data fusion:
LIDAR data with stereo/kinect data
Point-cloud data with multi-view textures
Video data into 3D model
PAGE 33
Slide - 33
Outline contributions to 3D processing
Real-time photo-realistic RGB-D SLAM
Efficient mesh reconstruction
16m x 8m office reconstructed in real time with RGB-D camera
Slide - 34
17
4/8/2015
RGB-D SLAM / introduction
Challenges
noval sensing interface, first system in 2011
tracking and mapping at 30fps
consistency vs. noisy measurement
data representation, 9 million points per second
Slide - 35
RGB-D SLAM / Overview of proposed algorithms
Schematic overview
Slide - 36
18
4/8/2015
Mesh Reconstruction / introduction
Triangulation
from discrete points to continuous surface
Applications
3D modeling
rendering / visualization
3D printing
robot reasoning
Slide - 37
Mesh Reconstruction / introduction
Challenges
large-scale point cloud
preserve geometry
redundancy (planar regions)
5.6 million points
11.2 million triangles
Slide - 38
19
4/8/2015
Mesh Reconstruction / Algorithm overview
Hybrid triangulation
Required
- Plane detection in 3D space
- Non-plane segments
Slide - 39
Mesh Reconstruction / Fast triangulation algorithm
Quadtree-based planar triangulation
in
input plane
extract boundary
construct quadtree
Slide - 40
20
4/8/2015
Mesh Reconstruction / result
90% planar points are removed
90% planar triangles are reduced
Slide - 41
Mesh Reconstruction / Demonstration result
DEMO MESHING
DEMO RENDERING
Slide - 42
21
4/8/2015
Conclusion on near future of video content analysis
• Exploiting context information for improved ship detect.
• Exploit sensor sensitivity and full resolution
• Better camera calibration, exploit further sensing modalities
• Extended obstacle detection with local 3D information
• Enhanced knowledge about object, broader viewing angle
• Better situational awareness, better signaling to operator
• Fast stixel processing object detection in depth signals
• 3D space sensing with e.g. Lidar & advanced Im. Proc.
• 3D modeling and reconstruction
• 3D object detection and classification
Slide - 43
7.2 Conclusions – (2)
Thanks for your attention
• Successful participation in October 2012 NATO Trial
• Performing in the top of the participants
• Only one showing a real-time system with live reporting
Slide - 44
22