Data-Driven Grasp Synthesis—A Survey Jeannette Bohg,Member,IEEE,Antonio Morales,Member,IEEE,Tamim Asfour,Member,IEEE,
and Danica Kragic,Senior Member,IEEE
Abstract—We review the work on data-driven grasp synthesis and the methodologies for sampling and ranking candidate grasps. We divide the approaches into three groups bad on whether they synthesize grasps for known,familiar,or unknown objects.This structure allows us to identify common object reprentations and perceptual process that facilitate the employed data-driven grasp synthesis technique.In the ca of known objects,we concentrate on the approaches that are bad on object recognition and po estimation.In the ca of familiar objects,the techniques u some form of a similarity matching to a t of previously encountered objects.Finally,for the approaches dealing with unknown objects, the core part is the extraction of specific features that are indicative of good grasps.Our survey provides an overview of the different methodologies and discuss open problems in the area of robot grasping.We also draw a parallel to the classical approaches that rely on analytic formulations.ps移动快捷键
Index Terms—Grasp planning,grasp synthesis,object grasp-ing and manipulation,object recognition and classification,visual perception,visual reprentations.
I.I NTRODUCTION
G IVEN an object,grasp synthesis refers to the problem of
finding a grasp configuration that satisfies a t of criteria relevant for the grasping task.Finding a suitable grasp among the infinite t of candidates is a challenging problem and has been addresd frequently in the robotics community,resulting in an abundance of approaches.
In the recent review of Sahbani et al.[1],the authors di-vide the methodologies into analytic and empirical.Following Shimoga[2],analytic refers to methods that construct force-closure grasps with a multifingered robotic hand that are dex-terous,in equilibrium,stable,and exhibit a certain dynamic behavior.Grasp synthesis is then usually formulated as a con-strained optimization problem over criteria that measure one or veral of the four properties.In this ca,a grasp is typically defined by the grasp map that transforms the forces exerted
Manuscript received March17,2013;accepted October25,2013.Date of publication November21,2013;date of current version April1,2014.This paper was recommended for publication by Associate Editor J.Dai and Editor D.Fox upon evaluation of the reviewers’comments.This work has been sup-ported by FLEXBOT(FP7-ERC-279933).
J.Bohg is with the Autonomous Motion Department at the MPI for Intelligent Systems,T¨u bingen70569,Germany(e-mail:jbohg@tuebingen.mpg.de).
A.Morales is with the Robotic Intelligence Lab,Universitat Jaume I,Castell´o 12071,Spain(e-mail:Antonio.Morales@uji.es).
T.Asfour is with the Karlsruhe Institute of Technology,Karlsruhe76131, Germany(e-mail:asfour@kit.edu).
D.Kragic is with the Centre for Autonomous Systems,Computational Vision and Active Perception Lab,Royal Institute for Technology,Stockholm10044, Sweden(e-mail:dank@kth.).
Color versions of one or more of thefigures in this paper are available online at ieeexplore.ieee.
Digital Object Identifier10.1109/TRO.2013.2289018at a t of contact points to object wrenches[3].The criteria are bad on geometric,kinematic,or dynamic formulations. Analytic formulations toward grasp synthesis have also been reviewed by Bicchi and Kumar[4].
Empirical or data-driven approaches rely on sampling grasp candidates for an object and ranking the
m according to a specific metric.This process is usually bad on some existing grasp experience that can be a heuristic or is generated in simulation or on a real robot.Kamon et al.[5]refer to this as the comparative and Shimoga[2]as the knowledge-bad approach.Here,a grasp is commonly parameterized in[6]and[7]:
1)the grasping point on the object with which the tool center
point should be aligned;
2)the approach vector which describes the3-D angle with
which the robot hand approaches the grasping point;
3)the wrist orientation of the robotic hand;
4)an initialfinger configuration.
Data-driven approaches differ in how the t of grasp candi-dates is sampled,how the grasp quality is estimated,and how good grasps are reprented for future u.Some methods mea-sure the grasp quality bad on analytic formulations,but more commonly,they ,human demonstrations,perceptual information,or mantics.
A.Brief Overview of Analytic Approaches
梁的成语
Analytic approaches provide guarantees regarding the crite-ria that measure the previously mentioned four grasp properties. However,the are usually bad on assumptions such as sim-plified contact models,Coulomb friction,and rigid body mod-eling[3],[8].Although the assumptions render grasp analysis practical,inconsistencies and ambiguities,especially regarding the analysis of grasp dynamics are usually attributed to their approximate nature.
In this context,Bicchi and Kumar[4]identified the problem offinding an accurate and tractable model of contact compli-ance as particularly relevant.This is needed to analyze stati-cally indeterminate grasps in which not all internal forces can be controlled.This ca ,for underactuated hands or grasp synergies,where the number of the controlled degrees of freedom(DOF)is fewer than the number of contact forces. Prattichizzo et al.[9]model such a system by introducing a t of springs at the contacts and joints and show how its dexter-ity can be analyzed.Rosales et al.[10]adopt the same model of compliance to synthesize feasible and prehensile grasps.In this ca,only statically determinate grasps are considered.The problem offinding a suitable hand configuration is casted as a constrained optimization problem in which a compliance is introduced to simultaneously address the constraints of contact reachability,object restraint,and force controllability.
As is the
1552-3098©2013IEEE.Personal u is permitted,but republication/redistribution requires IEEE permission.
See www.ieee/publications standards/publications/rights/index.html for more information.
ca with many other analytic approaches toward grasp syn-thesis,the propod model is only studied in simulation where accurate models of the hand kinematics,the object,and their relative alignment are available.
敬业福字
In practice,systematic and random errors are inherent to a robotic system and are due to noisy nsors and inaccurate models of the robot’s kinematics and dynamics,nsors,or of the object.The relative position of object and hand can therefore only be known approximately which makes an accurate place-ment of thefingertips difficult.In2000,Bicchi and Kumar[4] identified a lack of approaches toward synthesizing grasps that are robust to positioning errors.One line of rearch in this di-rection explores the concept of independent contact regions as defined by Nguyen[11]:a t of regions on the object in which eachfinger can be independently placed anywhere without the grasp loosing the force-closure property.Several examples to compute them are prented by Roa and Su´
a rez[12]or Krug et al.[13].Another line of rearch toward robustness against inaccurate end-effector positioning makes u of the caging formulation.Rodriguez et al.[14]found that there are caging configurations of a three-fingered manipulator around a planar object that are specifically suited as a waypoint to grasping it. Once the manipulator is in such a configuration,either opening or closing thefingers is guaranteed to result in an equilibrium grasp without the need for accurate positioning of thefingers. Seo et al.[15]exploited the fact that two-fingered immobilizing grasps of an object are always preceded by a caging configu-ration.Full body grasps of planar objects are synthesized by firstfinding a two-contact caging configuration and then us-ing additional contacts to restrain the object.Results have been prented in simulation and demonstrated on a real robot. Another assumption commonly made in analytic approaches is that the preci geometric and physical models of an object are available to the robot,which is not always the ca.In addition, we may not know the surface properties or friction coefficients, weight,center of mass,and weight distribution.Some of the can be retrieved through interaction:Zhang and Trinkle[16] propo to u a particlefilter to simultaneously estimate the physical parameters of an object and track it while it is being pushed.The dynamic model of the object is formulated as a mixed nonlinear complementarity problem.The authors show that even when the object is occluded and the state estimate cannot be updated through visual obrvation,the motion of the object is accurately predicted over time.Althoug
h methods like this relax some of the assumptions,they are still limited to simulation[10],[14]or consider2-D objects[14]–[16].
B.Development of Data-Driven Methods
Up to the year2000,thefield of robotic grasping1was clearly dominated by analytic approaches[2],[4],[11],[17]. Apart ,[5],data-driven grasp synthesis started to be-1Citation counts for the most influential articles in thefield.Extracted le.com in October2013.[11]:733.[4]:490.[17]:477.[2]:405.[5]: 77.[18]:384.[19]:353.[20]:100.[21]:110.[22]:95.[23]:96.[24]:108.[25]: 38.[26]:156.[27]:39.[28]:277.[29]:75.[30]:40.[31]:21.[32]:43.[33]: 77.[34]:26.[35]:191.[36]:58.[37]:75.[38]:39.come popular with the availability of Graspit![18]in2004. Many highly cited approaches have been developed,analyzed, and evaluated in this or other simulators[19]–[24].The ap-proaches differ in how grasp candidates are sampled from the infinite space of possibilities.For grasp ranking,they rely on classical metrics that are bad on analytic formulations such as the widely ud -metric propod in Ferrari and Canny[17]. It constructs the grasp wrench space(GWS)by computing the convex hull over the wrenches at the contact points between the hand and the object. quantifies the quality of a force-closure grasp by the radius of the maximum sphere still fully contained in the GWS.
Developing and evaluating approaches in simulation is attrac-tive becau the environment and its attributes can be completely controlled.A large number of experiments can be efficiently per-formed without having access to expensive robotics hardware that would also add a lot of complexity to the evaluation process. However,it is not clear if the simulated environment rembles the real world well enough to transfer methods easily.Only re-cently,veral works[24],[39],[40]have analyzed this question and came to the conclusion that the classic metrics are not good predictors for grasp success in the real world.They do not em to cope well with the challenges arising in unstructured environ-ments.Diankov[24]claims that in practice grasps synthesized using the metrics tend to be relatively fragile.Balasubrama-nian et al.[39]systematically tested a number of grasps in the real world that were stable according to classical grasp metrics. Compared with grasps planned by humans and transferred to a robot by kinesthetic teaching on the same objects,they under-performed significantly.A similar study has been conducted by Weisz and Allen[40].It focus on the ability of the -metric to predict grasp stability under object po error.The authors found that it performs poorly,especially when grasping large objects.
As pointed out by Bicchi and Kumar[4]and Prattichizzo and Trinkle[8],grasp closure is often wrongly equated with stability.Closure states the existence of equilibrium which is a necessary but not a suffi
cient condition.Stability can only be defined when considering the grasp as a dynamical system and in the context of its behavior when perturbed from an equilibrium. Seen in this light,the results of the aforementioned studies are not surprising.However,they suggest that there is a large gap between reality and the models for grasping that are currently available and tractable.
For this reason,veral rearchers[25]–[27]propod to let the robot learn how to grasp by experience that is gathered dur-ing grasp execution.Although collecting examples is extremely time-consuming,the problem of transferring the learned model to the real robot is nonexistant.A crucial question is how the object to be grasped is reprented and how the experience is generalized to novel objects.
Saxena et al.[28]pushed machine learning approaches for data-driven grasp synthesis even further.A simple logistic re-gressor was trained on large amounts of synthetic,labeled train-ing data to predict good grasping points in a monocular image. The authors demonstrated their method in a houhold scenario in which a robot emptied a dishwasher.None of the classical童装店
BOHG et al.:DATA-DRIVEN GRASP SYNTHESIS—A SURVEY291
principles that are bad on analytic formulations were ud. This paper spawned a lot of rearch[2
9]–[32]in which es-ntially one question is addresd:What are the object fea-tures that are sufficiently discriminative to infer a suitable grasp configuration?
神话简谱From2009,there were further developments in the area of3-D nsing.Projected Texture Stereo was propod by Konolige[41].This technology is built into the nsor head of the PR2[42],a robot that is available to comparatively many robotics rearch labs and running on the OpenSource middle-ware ROS[43].In2010,Microsoft relead the Kinect[44],a highly accurate depth-nsing device that is bad on the tech-nology developed by PrimeSen[45].Due to its low price and simple usage,it became a ubiquitous device within the robotics community.Although the importance of3-D data to grasp has been previously recognized,many new approaches were propod that operate on real world3-D data.They are either heuristics that map structures in this data to grasp config-urations directly[33],[34]or they try to detect and recognize objects and estimate their po[35],[46].
C.Analytic Versus Data-Driven Approaches
恩格斯简介Contrary to analytic approaches,methods following the data-driven paradigm place more weight on the object reprentation and the perceptual ,feature extraction,similar-ity metrics,object recognition or classification,and po esti-mation.The resulting data is then ud to ret
rieve grasps from some knowledge ba or sample and rank them by comparison to existing grasp experience.The parameterization of the grasp is less specifi,an approach vector instead offingertip posi-tions)and,therefore,accommodates for uncertainties in percep-tion and execution.This provides a natural precursor to reactive grasping[33],[47]–[50],which,given a grasp hypothesis,con-siders the problem of robustly acquiring it under uncertainty. Data-driven methods cannot provide guarantees regarding the aforementioned criteria of dexterity,equilibrium,stability,and dynamic behavior[2].They can only be verified empirically. However,they form the basis for studying grasp dynamics and further developing analytic models that better remble reality.
D.Classification of Data-Driven Approaches
Sahbani et al.[1]divide the data-driven methods that are bad on whether they employ object features or obrvation of humans during grasping.We believe that this falls short of capturing the diversity of the approaches especially in terms of the ability to transfer grasp experience between similar objects and the role of perception in this process.In this survey,we propo to group data-driven grasp synthesis approaches bad on what they assume to know a priori about the query object: 1)Known Objects:The approaches assume that the query
object has been encountered before and that grasps have already been generated for it.Commonly,the robot has access to a databa containing geometric object models that are associated with a number of good grasps.This databa is usually built offline and,in the following,will be referred to as an experience databa.Once the object
你好的英文单词
has been recognized,the goal is to estimate its po and retrieve a suitable grasp.
2)Familiar Objects:Instead of exact identity,the approaches
in this group assume that the query object is similar to the previously encountered ones.New objects can be familiar on different levels.Low-level similarity can be defined in terms of shape,color,or texture.High-level similarity can be defined bad on the object category.The approaches assume that new objects similar to old ones can be grasped in a similar way.The challenge is tofind an object rep-rentation and a similarity metric that allows to transfer grasp experience.
3)Unknown Objects:Approaches in this group do not as-
sume to have access to object models or any sort of grasp experience.They focus on identifying the structure or fea-tures in nsory data for generating and ranking grasp candidates.The are usually bad on local or global features of the object as perceived by the nsor.
Wefind the previous classification suitable for surveying the data-driven approaches since the assumed prior object knowl-edge determines the necessary perceptual processing and asso-ciated object reprentations for generating and ranking grasp candidates.For known objects,the problems of recognition and po estimation have to be addresd.The object is usually reprented by a complete geometric3-D object model.For fa-miliar objects,an object reprentation has to be found that is suitable for comparing them to already encountered object in terms of graspability.For unknown objects,heuristics have to be developed for the directly linking structure in the nsory data to candidate grasps.
Only a minority of the approaches discusd in this survey cannot be clearly classified to belong to one of the three groups.Most of the included papers u nsor data from the scene to perform data-driven grasp synthesis and are part of a real robotic system that can execute grasps.
Finally,this classification is well in line with the rearch in thefield of neuroscience,specifically,with the theory of the dorsal and ventral stream in human visual processing[51].The dorsal pathway process immediate action-relevant features, while the ventral pathway extracts context-and scene-relevant information and is related to object recognition.The visual pro-cessing in the ventral and dorsal pathways can be related to the grouping of grasp synthesis for familiar/known and unknown o
bjects,respectively.The details of such links are out of the scope of this paper.Extensive and detailed reviews on the neu-roscience of grasping are offered in[52]–[54].
E.Aspects Influencing the Generation of Grasp Hypothes The number of candidate grasps that can be applied to an object is infinite.To sample some of the candidates and define a quality metric for lecting a good subt of grasp hypothes is the core subject of the approaches reviewed in this survey.In addition to the prior object knowledge,we identified a number of other factors that characterize the metrics.Thereby,they influence which grasp hypothes are lected by a method. Fig.1shows a mind map that structures the aspects.An
292IEEE TRANSACTIONS ON ROBOTICS,VOL.30,NO.2,APRIL
2014
Fig.1.We identified a number of aspects that influence how thefinal t of grasp hypothes is generated for an object.The most important one is the assumed prior object knowledge,as discusd in Section I-D.Numerous different object-grasp reprentations are propod in the literature that are relying on features of different modalities such as2-D or3-D vision or tactile nsors.Either local object parts or the object as a whole are linked to specific grasp configurations.Grasp synthesis can either be analytic or data-driven.The latter is further detailed in Fig.2.Very few approaches explicitly address the task or hand kinematics of the robot.
important one is how the quality of a candidate grasp depends on the ,the object-grasp reprentation.Some ap-proaches extract local object ,curvature,contact area with the hand)around a candidate grasp.Other approaches take global ,center of mass,bounding box) and their relation to a grasp configuration into account.Depen-dent on the nsor device,object features can be bad on2-D or3-D visual data as well as on other modalities.Furthermore, grasp synthesis can be analytic or data-driven.We further cate-gorized the latter in Fig.2;there are methods for learning either from human demonstrations,labeled examples,or trial and er-ror.Other methods rely on various heuristics to directly link the structure in nsory data t
o candidate grasps.There is rel-atively little work on task-dependent grasping.In addition,the applied robotic hand is usually not in the focus of the discusd approaches.We will therefore not examine the two aspects. However,we will indicate whether an approach takes the task into account and whether an approach is developed for a gripper or for the more complex ca of a multifingered hand.Tables I–III list all the methods in this survey.The table columns follow the structure propod in Figs.1and2.
II.G RASPING K NOWN O BJECTS
If the object to be grasped is known and there is already a databa of grasp hypothes for it,then the problem offinding a feasible grasp reduces to estimating the object po and then filtering the hypothes by reachability.Table I summarizes all the approaches discusd in this ction.
红烧羊肉图片BOHG et al.:DATA-DRIVEN GRASP SYNTHESIS—A SURVEY
293
Fig.2.Data-driven grasp synthesis can either be bad on heuristics or on learning from data.The data can either be provided in the form of offline-generated labeled training data,human demonstration,or through trial and error.
TABLE I
D ATA -D RIVEN A PPROACHES FOR G RASPING K NOWN O
BJECTS
TABLE II
D ATA -D RIVEN A PPROACHES FOR G RASPING F AMILIAR O
BJECTS
TABLE III
D ATA -D RIVEN A PPROACHES FOR G RASPING U NKNOWN O
BJECTS
Fig.3.Typical functional flow-chart for a system with offline generation of a grasp databa.In the offline pha,every object model is procesd to generate grasp candidates.Their quality is evaluated for ranking.Finally,the list of grasp hypothes is stored with the corresponding object model.In the online pha,the scene is gmented to arch and recognize object models.If the process succeeds,the associated grasp hypothes are retrieved,and the unreachable ones are discarded.Most of the following approaches can be summarized with this flowchart.Some of them only implement the offline part.[7],[19],[21]–[24],[39],[56],[57],[59],[60],[65].
A.Offline Generation of a Grasp Experience Databa First,we look at approaches for generating the experi-ence databa.Figs.3and 5summarize the typical functional flowchart of the type of approaches.Each box reprents a processing step.Note that the figures are abstractions that summarize the implementations of a number of papers.Most
