Travel Support Application to Amazon Picking Challenge at ICRA 2015
Team IntBot, Colorado School of Mines
This document discusses our robotic system and applied techniques to address the Amazon Picking Challenge. The objective is to demonstrate our qualification and progress in order to support our travel grant
application to attend the Challenge at ICRA 2015.
Robot and Workspace Setup
A Baxter research robot is used by our team. The setup
of the robot and workspace is shown in Fig. 1. The robot
is located in front of the shelf and faces to it; the order
bin is placed in the middle of the robot and the shelf. This
workspace setup enables the Baxter robot to simultaneously use two arms to pick the items and enables the robot to reach all bins of the shelf.
To endow our robot with critical 3D perception capabilities (along with 2D perception ability using the Baxter’s
wrist color cameras), two RGB-D cameras (i.e., a Kinect
and an Asus Xtion PRO LIVE) are installed on the robot,
Figure 1. Robot and workspace setup.
as shown in Fig. 2(a). A head mount is 3D printed to hold
the Kinect sensor that observes the top two layers of the shelf, and the Asus Xtion is attached to the front
bar to observe the bottom two layers of the shelf. No other physical modifications are made on the robot.
Perception and Planning Approach
Our approach contains four components: object recognition, grasping planning, motion planning, and error detection and recovery. All components are implemented in the Robot Operating System (ROS) [1].




Object Recognition: Methods based on both 3D and 2D perception are used to detect and classify
the items. 3D model of each item is applied to generate synthetic 3D point clouds from different
perspectives, as shown by the school glue item used in the Challenge (Fig. 2(b)). The synthetic 3D
point clouds and the 3D data in the BigBIRD dataset [2] are used to train an exemplar SVM classifier to detect and localize the items in the color-depth scenes (Fig. 2(c)) acquired by the fixed RGBD cameras. Our 3D object recognition is implemented using the Point Cloud Library (PCL) [3]. 2D
object recognition methods are implemented using OpenCV [4] based on color images acquired
from the Baxter’s wrist cameras (Fig. 2(d)). The 2D methods can further improve the accuracy of
the classification results obtained by 3D object recognition methods.
Grasping Planning: The initial grasping points of each item are computed from RGB-D data using
the Deep Grasping method [5]. Then, when the Baxter robot moves its arms to the correct bin of
the shelf, the grasping points are refined by observing the item from multiple perspectives. The
grasping point refinement is implemented within Moveit! [6] using 2D visual data.
Motion Planning: Currently, the motion trajectory of robot arms from a start position to the center
of each shelf bin is pre-computed. After the robot moves its arms to the correct bin, MoveIt! [6]
is applied to plan a short trajectory between the bin center and the grasping point. A future work
before attending the Challenge is to implement real-time, dynamic motion planning techniques
using MoveIt! [6] to enable the Baxter robot to continuously grasp items.
Error Detection and Recovery: Currently, a simple method based on 3D robot perception is implemented to detect whether an item has been successfully grasped. If not, and the item remains in
a shelf bin, the robot will perform the grasping one more time.
Page 1 of 2
(a) RGB-D cameras
(b) 3D model of school glue
and synthetic 3D point clouds
(c) 3D scene and its color
and depth images
(d) 2D scenes from robot wrist cameras
Figure 2. Object recognition based on 2D and 3D robot perception.
Capabilities and Advantages of the Implemented Robotic System
The current workspace setup and the used perception and planning techniques have the potential to enable our Baxter robot to:




Pick across a wide workspace: the robot is able to pick items from all bins of the shelf;
Simultaneously pick items using two arms to significantly improve efficiency;
Get items past the front lip on the edge of each bin by lifting the items and then moving it out;
Pick items that have a variety of different properties (e.g., boxes, cups, and soft toys).
A video is submitted with this travel support application (available on Youtube: https://youtu.be/EZh2ZuZLoY), which demonstrates the aforementioned capabilities and advantages of our robotic system.
Qualification of the IntBot Team
The IntBot team is led by Professor Hao Zhang in the Department of Electrical Engineering and Computer
Science at Colorado School of Mines (CSM). Dr. Zhang received the Ph.D. degree in Computer Science in
2014 under the supervision of Professor Lynne E. Parker (who serves as the General Chair of ICRA 2015).
Dr. Zhang’s research focuses on 3D perception, robot learning and human-robot teaming for human-centered robotics applications. The team contains 15 graduate students and one undergraduate student, majored in Computer Science, Electrical Engineering and Mechanical Engineering, who can well contribute
to different aspects of the Challenge. A Baxter robot (and several other robotic arms) are available in Dr.
Zhang’s laboratory, which are directly applicable to the Challenge. CSM was awarded with the practice
equipment (although the shelf has not been received yet), which is shared between our IntBot team and
Team Blaster led by Professor John Steele.
For additional information, please visit: http://hcr.mines.edu/AmazonPickingChallenge.html.
References
1.
2.
3.
4.
5.
6.
“The Robot Operating System”, http://www.ros.org/, accessed on 03/14/15.
A. Singh, J. Sha, K. S. Narayan, T. Achim, P. Abbeel, “BigBIRD: A large-scale 3D database of object instances,” in
IEEE International Conference on Robotics and Automation (ICRA), 2014.
R. B. Rusu and S. Cousins, "3D is here: Point Cloud Library (PCL)," in IEEE International Conference on Robotics
and Automation (ICRA), 2011.
“The OpenCV Library”, http://opencv.org/, accessed on 03/14/15.
I. Lenz, H. Lee, A. Saxena, “Deep Learning for Detecting Robotic Grasps”, in International Journal of Robotics
Research (IJRR), to appear, 2015
“MoveIt!”, http://moveit.ros.org/, accessed on 03/14/15.
Page 2 of 2