Welcome to Accurate
Robotic 3D Vision
An educational webinar sponsored by
Universal Robotics and Yaskawa Motoman
Robotics
Speakers
Hob Wubbena, Director of Marketing, Universal Robotics
•
•
•
•
Engineering, technical planning and product marketing for Hewlett-Packard
and Agilent Technologies for 25 years
Aerospace & defense, telecommunications, test & measurement, electronic
manufacturing, and the chemical process industries
3 patents, numerous marketing awards, ~12 published articles
B.S. Civil Engineering, University of Wisconsin; Masters Business, Denver
University
Erik Nieves, Technology Director, Yaskawa Motoman Robotics
•
•
•
•
Management and engineering, for Motoman Robotics for 20 years; currently
focused on corporate strategic technology roadmap and emerging
applications
Standards Development & Education Committees for the Robotics Industries
Association (RIA), “Ask the Experts” forum, RIA
Many published robot technology articles including control and metrology
B.S. Mathematical Physics, Southwestern Adventist University
7/21/2011
Copyright © 2011 Universal Robotics
Page 2
Experts
Aditya Nawab, R&D Manager, Universal Robotics
•
•
•
•
Extensive design experience in computer vision, hardware/software
integration, autonomous vehicle development and industrial robotics
Worked on numerous Department of Defense and NASA robotic projects
His work at Universal focuses on dexterous manipulation, force control,
sensory-motor coordination, forward and inverse kinematics, dynamics
of articulated manipulators, and R&D project management
B.S. Mechanical Engineering, Florida Atlantic University; M. S.
Mechanical Engineering, University of Florida
Greg Garmann, Technology Leader, Yaskawa Motoman
Robotics
•
•
•
Technology Leader at Yaskawa America, Motoman Robotics Division,
and has been involved in automation for more than 25 years.
Developed vision capabilities for robotics guidance using 2D and 3D
technologies
B.S. Computer Engineering, Wright State University
7/21/2011
Copyright © 2011 Universal Robotics
Page 3
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision
• Types of 2D, 2½D & 3D robotic
vision systems
• Elements of Accuracy
• Choosing a 3D vision system
• Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 4
Robotic 3D Vision Introduction
• Beyond scope of webinar:
–
–
–
–
–
–
–
3D vision (movies) anaglyphs for depth perception
Photometric stereo from telecentric camera
Depth From Focus (DFF)
3D Mosaicking
Vision analysis tools (blob analysis, face recognition)
Time of Flight (TOF)
Future webinar
Laser line projector
• Robotic vision delivers real-time data about objects
–
–
Part Inspection
Vision Guidance (X, Y, Z position & Rx, Ry, Rz pose)
• 3D accuracy requires both intrinsic camera calibration
and hand-eye calibration
7/21/2011
Copyright © 2011 Universal Robotics
Page 5
Repeatability vs. Accuracy
• Repeatability is important
for automated tasks where
robot picking and placing
to same locations
• Relative Accuracy is
important for random
tasks where the spatial
location is constantly
changing:
– Random box
depalletization
– Random box moving
– Random part picking
– Random bin picking
7/21/2011
Copyright © 2011 Universal Robotics
Page 6
Intrinsic Camera Calibration
Calibration process
•
•
•
Camera sensor information
Fiducial in full FOV
15 varying fiducial images
Y
Z
Data in 3 coordinate systems
•
Image, Camera, Object
X
Image rectification (ΔY)
•
Removes perspective and/or
lens distortions
If multiple cameras
•
•
Computes disparity, distance,
3D coordinates
Aligns image on common plane
7/21/2011
Camera
Object
Y
Z
Y
Z
X
Image
Copyright © 2011 Universal Robotics
X
Focal
length
Page 7
FOV
Field of View
3D Vision & Robotic 3D Vision
3D Vision: stereo vision with depth (z) perception resulting from the
disparity (Δx) of different images of the same object
3D Accuracy: resolution in the Δz depth of an object viewed from
stereoscopic vision. It is based on quality and geometry of cameras.
Δy distance difference between the cameras (placement and orientation) is corrected
during intrinsic camera calibration, resulting in a rectified image.
The Δx disparity of each camera's image of same pixel point in space is computed through
3D software algorithms.
Robotic 3D Vision
•
3D Vision with vision guidance
for a robot delivers in real-time:
–
–
–
–
7/21/2011
Copyright © 2011 Universal Robotics
Position (X, Y, Z)
Pose/orientation (Rx, Ry, Rz)
Interactively, with tool offsets
Choice of robot tool affects vision
requirements (vacuum and grippers
can work within 2-3 mm)
Page 8
Hand–Eye Calibration
• First, calibrate
cameras
• Then Hand-Eye
calibration enables
data transform
across coordinate
systems
–
Robot “hand” tool
guided by camera “eye”
• RESULTS FOR:
–
–
–
VISION GUIDANCE
Part Inspection
(optional)
3D Part Model creation
(optional)
7/21/2011
Copyright © 2011 Universal Robotics
Page 9
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision
• Types of 2D, 2½D & 3D robotic
vision systems
• Elements of Accuracy
• Choosing a 3D vision system
• Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 10
Types of 2D, 2½D & 3D Vision Systems
2½D
2D
Rz
Rz
X
3D
Z
Rz
#1
#2
Z
X
X
Rx
•Single Camera
•Single camera with known
calibration plate, OR multiple
images to obtain depth, OR add
TOF/laser
•Move camera OR add 2nd camera
with shape-matching (model) OR
TOF/laser with surface-matching
(point cloud)
•Vision data in same plane
•Z data added
•Rx, Ry data added
•X, Y, Rz(angle) only; no Z
•Postion X, Y, Z, Rz (angle) only
•Position X, Y, Z, & Pose Rx, Ry, Rz
•No tilt (Rx, Ry)
•No tilt (Rx, Ry)
•Allows for part tilt
•Camera distance constant
•Camera distance can change
•Camera distance can change
7/21/2011
Copyright © 2011 Universal Robotics
Page 11
Types of 3D Vision
Camera System
Vision
Methodology Required
Comments
Single Stationary
2D,
2½D,
3D
2½: ONLY if height is
known & fixed. Height can
be variable with TOF
3D: Shape-match
Good if plane is fixed
Single Moving
3D
3D: with shape-match or
surface-match
Can measure depth,
Slower than Stereo
Binocular Stereo
3D
Shape-match (model) or
surface-match (point cloud)
Surface-match requires
TOF or laser
3D
Mosaicking
Shape-match (model)
Surface-match requires
TOF or laser,
Slower than binocular
Multiple Stereo
7/21/2011
Copyright © 2011 Universal Robotics
Page 12
Single Stationary Camera
• Using 3D Model
• Shape-Based 3D
Matching
–
Speed up by using only a
region of interest,
eliminate unnecessary
edges
• Surface-Based 3D
matching
7/21/2011
Copyright © 2011 Universal Robotics
Page 13
Single Camera, Multiple Positions
• Parts must be stationary
for measurement (e.g.
Stacked parts)
• May require multiple
images of same object
from different camera
locations
• Good for vision guidance
and part inspection
7/21/2011
Copyright © 2011 Universal Robotics
Page 14
Binocular Cameras
• Only one image required
• Faster than multiple
image approach
• Can find randomly
located objects
• Use of standard off-theshelf cameras
• Can provide full field of
view of robot operating
envelope
7/21/2011
Copyright © 2011 Universal Robotics
Page 15
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision
• Types of 2D, 2½D & 3D robotic
vision systems
• Elements of Accuracy
• Choosing a 3D vision system
• Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 16
3D Vision Accuracy and Robot
Operating Envelope Geometry
3D Accuracy, Δz, is affected
by:
Object
Z = object distance
Δz
Virtual
Image 1
P1 (x1,y1,z1) P2 (x2,y2,z2)
Z
X
Camera 1
Virtual
Image 2
f=focal
length
Camera 2
b=baseline distance
7/21/2011
•Distance, b, between pair of
cameras
•Distance, Z, from object to
cameras
•Disparity, Δd, offset in x between
image points, P1 & P2
•Focal length, (pixels, not mm)
2
Δz = Z
f·b
· Δd
where f (pixels) =
f of Lens (mm) * Camera Horiz. Resolution
(px) / CCD Horiz. sensor width (mm)
Copyright © 2011 Universal Robotics
Page 17
3D Camera Positions for Objects on
Flat Surface
2nd pair orthogonal
~100mm apart @ 45˚
1st pair cameras just
above robot at ~100mm
apart, 45˚ below
horizontal
NOTE: Irregular objects
require about 300px by
300px for determining
position & pose
Robot
±150mm depending
on part & envelope
dimensions
Robot Work
Envelope
Robot Envelope
Height
Robot Envelope
Depth
Parts / Boxes on Flat Surface
7/21/2011
Copyright © 2011 Universal Robotics
Page 18
NOTE: Rectangular
objects require about
200px by 200px for
determining position
& pose
Camera Positions for Objects in Bin
1st pair cameras just above
robot at ~100mm apart, 60˚
below horizontal
2nd pair orthogonal
~100mm apart @ 60˚
NOTE: Irregular objects
require about 300px by
300px for determining
position & pose
Robot
Operating
Envelope
Robot operating
envelope Height
±150mm depending
on part & envelope
dimensions
Robot
Envelope Depth
Parts or Boxes in Bin
7/21/2011
Copyright © 2011 Universal Robotics
Page 19
NOTE: Rectangular
objects require about
200px by 200px for
determining position
& pose
Robot Accuracy & Repeatability
•
•
•
Accuracy: how close a robot can reach a commanded 3D position
Accuracy varies with robot speed, robot reach, and with payload
Accuracy of a robot is determined by three elements of the system:
– Resolution of the control system
– Error operating the robot arm under closed loop servo operation
– Imprecision of mechanical linkages, gears & deflections under load
•
•
Repeatability: the ability to duplicate an action or a result every time
See ISO 9283
7/21/2011
Copyright © 2011 Universal Robotics
Page 20
Typical Robot Repeatability
•
Plotted over 100
robots from 3
companies
–
–
•
Repeatability loosely
a function of:
–
–
–
•
2Kg to 1,200Kg
SCARA, material
handling, welding,
assembly…
Payload (Kg)
Reach (M)
Speed (˚/sec)
Best repeatability
approaches limit line
7/21/2011
Copyright © 2011 Universal Robotics
Page 21
Robot Accuracy
• Accuracy is typically worse than repeatability, not
constant over workspace
• Full robot calibration:
–
Accuracy = 2X – 4X of Repeatability; can approach 1X
• Typical robot calibration:
–
Accuracy = 3X – 5X of Repeatability
Best (2X)
Repeatability Accuracy
0.015mm
0.03mm
0.06mm
0.1mm
0.2mm
0.03mm
0.06mm
0.12mm
0.2mm
0.4mm
7/21/2011
Typ. (4X)
Accuracy
0.06mm
0.12mm
0.24mm
0.4mm
0.8mm
Payload, Reach, Speed
2kg
5kg
35kg
50kg
275kg
Copyright © 2011 Universal Robotics
Page 22
0.6M
0.8M
1.3M
1.6M
2.5M
375 deg/s
270 deg/s
170 deg/s
170 deg/s
90 deg/s
Overall 3D Robotic Vision System
Accuracy
System Accuracy results (go to Universal Robotics 3D Made Easy
Calculator at www.universalrobotics.com/calc
7/21/2011
Copyright © 2011 Universal Robotics
Page 23
Camera Selection
Scalable vision systems enable a wide variety of cameras to fit
the robot operating envelope geometry and accuracy needs
USB Webcams &
GigE Camera
Selection
Medium Size Parts
& Boxes
(workspace < 1.0M wide
x 1.0M deep x 1.0M
high)
Boxes, Parts, &
Pallets
(workspace < 1.5M x
1.5M x 1.5M)
Working Distance from Cameras to Objects
(Camera selection provides <1mm camera accuracy, resulting in a system accuracy of 1-3mm
within the workspace. Consult Universal Robotics for final configuration)
0.3 - 0.5M
1.0M
2.0M
2.5M
USB 1.0 MP, f3.7
USB 1.3 MP, f3.5
GigE 1.4 MP, f3.5
(0.5M workspace)
USB 1.3 MP, f4.5-f6
USB 2.0 MP, f3.5-4.5
GigE 1.2 MP, f3.5- 4.5
GigE 1.4 MP, f4.5 –f6
USB 2.0 MP, f6 – f8
GigE 1.2 MP, f6 - f8
GigE 1.4 MP, f8 -f12
USB 2.0 MP, f8 - f12
GigE 1.2 MP, f8 –f12
GigE 1.4 MP, f12 -f16
Consult with
Universal Robotics
Consult with
Universal Robotics
USB 1.3 MP, f6 - f8
USB 2.0 MP, f4.5 – f6
USB 2.0 MP, f6 –f8
GigE 1.4 MP, f8 – f12
7/21/2011
Copyright © 2011 Universal Robotics
Page 24
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision
• Types of 2D, 2½D & 3D robotic
vision systems
• Elements of Accuracy
• Choosing a 3D vision system
• Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 25
Attributes of Low-Cost 3D Vision
WORKSPACE:
__ Robot operating envelope within 1.5M x 1.5M?
__ Cameras mounted within 2.5M of workspace?
ROBOT:
__ Part-box picked up with vacuum or grippers? (sets 3mm accuracy)
__ Part or box weighs < 50 Kg? (robot size)
__ ≤10 parts-boxes/min or 600/hour? (can USB2.0 cameras)
PARTS OR BOXES:
__ Part has visible edge and/or boundary? (enables shape-matching)
YES: White egg with black background; surface with reflectivity < 75
NO: Dull black molded rubber ball with black background
__ Part/box have CAD model? (shape-matching; no special lighting)
__ Only one part? (scalable to multiple parts, but additional work)
7/21/2011
Copyright © 2011 Universal Robotics
Page 26
3D Vision at 2D Prices
• Standard off-the-shelf components scalable to multiple
parts with world-class software
Vision System $K
3D Vision at 2D Prices
$60
$50
$40
$30
$20
$10
$1985
1990
1995
2000
2005
2D Machine Vision
3D Machine Vision
7/21/2011
Copyright © 2011 Universal Robotics
Page 27
2010
Robotic Vision Options
MotoSight 3D
Spatial Vision
2½ D Surface
Traditional 3D Precise 3D
Machine Vision Inspection
Random Part Pick
Automated Assembly
Machine Vision
Precise Part
Inspection/Guidance
$8-$10k
$8k
$12k - $18k
$20k - $25k
Two Fixed Cameras
One Camera on Robot One – Multiple
Arm
Fixed or on Robot
One Camera on Robot
Arm
3D: complex parts
2½ D: planar parts only 3D: complex parts
2½ D: planar parts only
3D shape match
3D surface match
3D geometric
pattern match
Stationary & Moving
Parts
Stationary Parts
Stationary & Moving
Stationary Parts
Parts
Three – Ten Parts
One Part
One Part
One Part
1 - 5 mm accuracy
@ 0.5 – 2.5M
1 - 5 mm
@ 0.5 - 2.0M
~0.2 - 2 mm
@ 0.1 - 0.5M
~ 0.12 mm
@ 0.08M
7/21/2011
Copyright © 2011 Universal Robotics
Page 28
3D geometric pattern
match
Key Benefits of Robotic 3D Vision
• Locates 3D Objects in 3D Space
–
–
Identifies 6 degrees of freedom X, Y, Z, Rx, Ry, Rz
Not just 2D flat parts in space
• 3D Shape Matching
–
–
–
Identifies complex objects
Handles varying lighting conditions
Identifies partially hidden objects
• Scalable Precision
–
–
Choose just the right cameras for your needs
Only ~3mm accuracy needed for vacuum pick or grippers
• 3D Vision at 2D Prices
–
MotoSight 3D Spatial Vision U.S. List $10,000
7/21/2011
Copyright © 2011 Universal Robotics
Page 29
Robotic 3D Vision Summary
• Use 3D where part complexity, location
randomness, or motion require it
–
–
Vision guidance
Part inspection
• 3D is a mainstream solution; requires choosing
right tool & approach
–
Only pay for what you need; end effectors can manage
from 2-3mm
• Motoman robots are some of the most accurate
robots in the world
• Universal Robotics 3D Made Easy Calculator at
www.universalrobotics.com/calc
7/21/2011
Copyright © 2011 Universal Robotics
Page 30
AGENDA: Accurate Robotic 3D Vision
• Introduction to Robotic 3D Vision
• Types of 2D, 2½D & 3D robotic
vision systems
• Elements of Accuracy
• Choosing a 3D vision system
• Q&A
7/21/2011
Copyright © 2011 Universal Robotics
Page 31
End
7/21/2011
Copyright © 2011 Universal Robotics
Page 32
3D Calculator Inputs
3D Calculator Inputs
1st Pair Cameras
(0,H+Offset,0)
2nd pair parallel cameras
~100mm apart; orthogonal to 1st
1st pair cameras ~100mm
apart; pointed at center
2nd Pair Cameras
(-½W, H+Offset,½D)
Robot Envelope Height
Robot Work Envelope
Robot
Center (0, ½H, ½D)
Object
Height
Obj. Depth
Object
(-½W, 0, D )
Center Edge
(0,0,0)
(½W, 0, D )
Robot Envelope Depth
5/31/201
2
Copyright © 2011 Universal Robotics
Page 33