編號： 畢業(yè)設計(論文)外文翻譯 （原文） 院 （系）： 機電工程學院 專 業(yè)： 機械設計制造及其自動化 學生姓名： 學 號： 指導教師單位： 姓 名： 職 稱： 年 6 月 3 日 Robotics and Autonomous Systems 54 (2006) 1026–1038 www.elsevier.com/locate/robot Automatic visual guidance of a forklift engaging a pallet Michael Seelingera,?, John-David Yoderb a Yoder Software, Inc., 715 W. Michigan Ave., Urbana, IL 61801, United States b Department of Mechanical Engineering, Ohio Northern University, Ada, OH 45810, United States Received 4 June 2004; received in revised form 11 September 2005; accepted 2 October 2005 Available online 1 August 2006 Abstract This paper presents the development of a prototype vision-guided forklift system for the automatic engagement of pallets. The system is controlled using the visual guidance method of mobile camera-space manipulation, which is capable of achieving a high level of precision in positioning and orienting mobile manipulator robots without relying on camera calibration. The paper contains development of the method, the development of a prototype forklift as well as experimental results in actual pallet engagement tasks. The technology could be added to AGV systems enabling them to engage arbitrarily located pallets. It also could be added to standard forklifts as an operator assist capability. §c 2006 Elsevier B.V. All rights reserved. Keywords: Mobile manipulation; Visual guidance; Machine vision 1. Introduction This paper presents the development of a prototype system for the vision-guided automatic engagement of arbitrarily- positioned standard pallets by a computer-controlled forklift. The method presented here enables a robotic forklift vehicle to engage pallets based on their actual current location by using feedback from vision sensors that are part of the robotic forklift. The development of this technology could advance the current state of the art in material handling in two distinct ways. First, the technology could be added to commercially avail- able AGV (automatically guided vehicle) material handling systems. Typically these systems use floor-embedded wires or laser beacons located throughout their areas of operation to nav- igate the AGVs through the warehouse and to align themselves for pallet engagement. While most AGV systems are used for horizontal transportation of material, leaving the task of storing pallets in racks up to humanly driven standard forklifts, there are some AGV systems capable of stowing and retrieving pal- lets from rack locations. In order to engage a specific pallet, the pallet’s vertical position (or rack height) as well as its po- sition in the reference frame of the map must be known to the ? Corresponding author. Fax: +1 217 344 2987. E-mail address: mseelinger@yodersoftware.com (M. Seelinger). AGV system. AGVs are effective in engaging pallets that have been placed in the racks by the AGVs themselves because the AGVs are capable of placing the pallets to within 1 cm of the desired location [1]. AGV pallet engagement assumes the pallet is positioned to within 1 cm of the nominal desired location. If a pallet is placed in the rack by a human operating a standard forklift, there is no assurance that the pallet will be located pre- cisely enough for an AGV system to engage the pallet. Other factors can cause an AGV to lose the necessary precision for pallet engagement such as disruption of a laser beacon signal or loss of communication with the guiding wires or a degrada- tion of the system’s calibration. (Note: background material on AGV systems can be found in [1] and/or online at [2].) The use of the vision-guided method for pallet engagement presented here would provide AGV systems the flexibility to perform their final positioning and engagement based on the pallet’s actual location relative to the AGV. This would enable human operators to place pallets amidst the previously AGV- only racks, affording much more flexibility and interaction between AGV material handling units and human operators on standard forklifts. Also, the ability to engage pallets based on their actual locations would enable AGV systems to operate in less structured environments. For example, rather than only working in carefully laid-out warehouses, AGVs could unload pallets automatically off tractor-trailers, currently a very labor intensive task. 0921-8890/$ - see front matter §c 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.robot.2005.10.009 Fig. 1. Crown SC 3-wheeled electric forklift prior and after retrofit. The second way that the vision-guided pallet engagement method presented here could advance the current state of the art would be to add it to standard, manually-guided forklifts. The pallet engagement process can be difficult for forklift drivers due to the fact that their line of sight is obstructed by the mast of the forklift itself. This is especially true when loading/unloading very high racks. The vision guidance control method presented here could be added to standard forklifts to automate the pallet-engagement portion of the forklift material handling operation. The driver would have the responsibility of navigating through the warehouse or other building, avoiding obstacles and getting the forklift into view of the desired pallet. Then, the operator could switch into ‘automatic-engagement mode’ which would enable the system to engage the pallet automatically. Such a system could reduce the amount of product damage that occurs in forklift accidents involving pallet engagement and transportation of product. Potentially, it could also improve safety as it would allow the forklift to engage pallets safely avoiding the possibility of knocking products off pallets that are located very high off the ground. The vision-guided method for forklift pallet engagement presented in this paper is called ‘Mobile Camera-Space Manipulation’ or simply ‘MCSM’. It was developed originally for the visual guidance of a planetary exploration rover equipped with a robotic arm [3] and has been adapted and further developed to control the forklift prototype system presented here and shown in Fig. 1. While the method requires that at least two vision sensors be attached to the forklift system, it does not require the system to maintain any sort of strict calibration of the cameras relative to the system nor the cameras relative to each other. Such calibrated relationships would be difficult to maintain in the harsh climate typically encountered by industrial forklifts. Likewise it would be difficult to establish and maintain these calibrated relationships on many forklift systems due to the low-precision (relative to typical industrial robotics) nature of the mechanics of these models. MCSM uses vision-based estimation along with the nominal kinematic model of the forklift and a simple camera model to achieve a high-level of positioning precision in a robust fashion. This precision and robustness has been demonstrated by experimental results of the forklift prototype, which will be presented in Section 6. Following this introduction, the paper is divided into five sections. Section 2 gives background for the reader to understand the current state of the art in vision-guided pallet engagement by forklift vehicles as well some other vision- guided mobile manipulator systems. Section 3 gives some necessary background information on standard CSM, which is a basis for developing MCSM. Section 4 presents the prototype forklift used in the development and testing of the method. Section 5 presents how the method of MCSM is developed to control the forklift. Such topics as target definition, trajectory generation and tracking, and vision-tracking will be discussed. The experimental results are given in Section 6. Section 7 contains the conclusions. 2. Background MCSM is a method for controlling mobile manipulators via vision sensors that are carried with the mobile system. The term mobile manipulator refers to a system consisting of a manipulator with holonomic degrees of freedom (DOFs) mounted on a mobile base which has non-holonomic DOFs. (See [4] for a description of the difference between holonomic and non-holonomic constraints.) A forklift is an example of a mobile manipulator. The forks of the forklift can be thought of as a manipulator. Typical forklifts, including the one discussed in this paper, have three holonomic fork DOFs: vertical, sideshift, and tilt angle. The mobile base of the forklift has two non-holonomic DOFs of control: the angle of the steering wheel as well as the angle of the drive wheel(s). These can be used to position the forklift in three DOFs: the (X, Y) location of the forklift in the plane parallel to ground as well as its angle of orientation. Other examples of mobile manipulators include exploration rovers equipped with robotic arms and construction equipment such as backhoes or front loaders. MCSM is an extension of standard Camera-Space Manipula- tion (CSM), a control strategy that has proven through years of testing to provide highly accurate control of holonomic systems using fixed cameras [5–7]. Standard CSM has even been used to control a mobile manipulator, a miniature ‘simulated’ fork- lift [8,9]. However, standard CSM is ill-suited to control mobile manipulators since it is limited by the need to control the mo- bile system using stationary cameras, which severely limits the workspace of the mobile system. MCSM overcomes this limita- tion of standard CSM since it allows the cameras to move with the mobile manipulator. Page`s et al. present a computer vision system for automatic forklift pallet engagement [10]. Their method incorporates the use of a single calibrated camera. Their work focuses more on the pallet recognition method of the vision system rather than on the overall visual control of the forklift system itself. Currently, their system is configured to engage pallets that are located on the ground level only. The trajectory generation method presented by Page`s is primitive. It incorporates a scheme of turn, drive straight, turn, drive straight. While this method of trajectory generation does allow the forklift to engage the pallet, it is slow and impractical as the system must frequently stop and start. Page`s presents some experimental results, however as with Miller [9], the vehicle used was not an actual forklift but a small mobile robot with forks attached to the manipulator intended to simulate the motion of an actual forklift. The system presented in this paper differs in several respects. MCSM requires the use of at least two cameras; however, it does not require camera calibration. The strategy for pallet recognition employed with MCSM differs substantially from Page`s’ system. Also, the MCSM system is not limited to engaging pallets that are located only on the floor. Unlike the system of Page′s et al. the trajectory generation and tracking method used with MCSM enables our system to engage a pallet much the same way a human operator would drive a forklift without the need for stopping and starting. The only other vision-guided forklift system that the authors are aware of is the Automated Material Transport System which is under development at the National Robotics Engineering Consortium [11]. The development of this system seems to rely heavily on computer vision both for the navigation throughout the warehouse as well as the engagement of pallets. Unfortunately, there is no public information available that describes the methods employed for pallet engagement. Several manufacturers provide forklift-style AGV systems [12–15]. Product information available through company websites focus on load capacity, fork reach, repeatability, and in the case of warehousing applications the aisle width in which the AGV can operate. Currently AGV companies do not offer vision systems as part of their control package for the automatic engagement of as-located pallets. technique guided by a separate vision sensor with an eye- in-hand configuration. MCSM differs from MacKenzie and Arkin’s approach in several respects. For instance, MCSM uses the same vision sensors for navigating the mobile platform as well as positioning of the manipulator. Also, MCSM’s visual control approach is based on an open-loop method rather than the closed-loop visual servoing technique. The advantages of the open-loop approach over closed-loop visual servoing techniques are discussed in [19]. Tsakiris, Rives, and Samson present a vision-guided mobile manipulator also based on visual servoing techniques [20]. An eye-in-hand vision system configuration guides the mobile robot and its onboard multiple DOF manipulator. The goal of their research is to present methods for achieving stable pose control of both the non-holonomic base as well the manipulator mounted camera, which they achieve by the addition of holonomic DOFs for positioning the camera. Aside from the differences in open loop versus closed loop control, their system uses only one camera and is mounted in the eye-in-hand configuration whereas MCSM systems use at least two cameras neither of which is mounted on the manipulator. 3. MCSM as an extension of standard CSM Standard CSM plays an integral role as a subsystem of the MCSM forklift system. While a more complete description of standard CSM is given in [5,6], it is important to discuss certain basic CSM principles in order to understand the MCSM forklift system. The first part of this section develops CSM as it is used to control purely holonomic systems. The second part of this section describes how standard CSM has been used to control a ‘simulated’ forklift mobile manipulator. 3.1. Standard holonomic CSM In typical implementations of standard CSM for controlling holonomic manipulators CSM uses widely separated cameras located remotely from the robot they control. CSM works by estimating a relationship in the reference frame of each camera between the appearance of image-plane visual features located on the manipulator with the internal joint configuration of the robot. This relationship, f , is based on the orthographic camera T model [21] and is described with a set of view parameters given by C = [C1, C2, . . . , C6] : xc = fx (C, ?) = (C 2 2 2 2 Other examples of vision-guided mobile manipulators in the literature include systems proposed by Pissard-Gibollet and Rives, and Swain and Devy [16,17]. The goal of their system is to navigate the mobile robot using a vision sensor 1 + C2 + C3 + C4 )X(?) + 2(C2C3 + C1C4)Y(?) + 2(C2C4 ? C1C3)Z(?) + C5 yc = fy(C, ?) = 2(C2C3 ? C1C4)X(?)  (1) held by a manipulator mounted on the mobile base. They do not address the task of actually engaging an object with the manipulator mounted on the mobile platform. MacKenzie and Arkin describe a vision-guided mobile manipulator for drum sampling [18]. This system servos the mobile base into position using a camera mounted on the manipulator’s waist. Then the manipulator engages a drum via a visual servoing 2 2 2 2 + (C1 ? C2 + C3 ? C4 )Y (?) + 2(C2C4 + C1C3)Z(?) + C6 where (xc, yc) is the position in the reference frame of one of the cameras and ? is the pose of the manipulator (see Fig. 2 for definition of coordinate systems). The position of a point on the manipulator measured relative to the (X, Y, Z) holds in a localized region of joint-space and camera-space operation. Due to the fact that the measurement and positioning are carried out in the same reference frames (those of the participating cameras) CSM compensates for the inadequacies of the orthographic camera model as well as inaccuracies in the kinematic model of the robot. Even if the kinematics of the system were to be altered during a given operation (for instance if a joint is deflected by the load it carries) CSM can compensate by simply acquiring more joint-space/camera-space samples. Experimental results demonstrating this ability are presented in [3]. Additionally, a process called ‘flattening’ has been developed to increase the precision of CSM/MCSM systems by correcting for the inadequacies of the orthographic camera model. Flattening uses orthographic camera model parameters and a rough estimate of the distance between the robot reference frame, (X, Y, Z) in Fig. 2, and camera reference frames, (X 1, Y 1, Z 1) and (X 2, Y 2, Z 2) in Fig. 2, to obtain identical c c c c c c Fig. 2. Coordinate system definitions for robot and cameras. coordinate system is fixed to the base of the manipulator as can be seen in Fig. 2. It is dependent upon the nominal kinematic model of the robot and its corresponding pose, ?. The view parameters are estimated based on sample pairs of robot internal joint configuration and camera-space location of manipulator features. The minimization of results as would have been achieved by using the more accurate ‘pin-hole’ camera model (see [7] for more detail). 3.2. Non-holonomic CSM 2. A form of standard CSM called ‘non-holonomic CSM’ has been used to control a small mobile manipulator, a ‘simulated forklift’, that is capable of engaging a small pallet [8,9]. MCSM was developed, in part, to overcome some of the restrictions and disadvantages of using this form of CSM to control mobile ns .xc x 2 . Γ = . k=1 k ? f (C, ?). + .yck ? fy(C, ?). Wk (2) manipulators. CSM requires a set of cameras that are stationary relative to the mobile manipulator system. This severely limits the workspace of a mobile system and makes the use of over all C provides the necessary conditions to solve for the six view parameters for each camera. In Eq. (2) ns denotes the total number of joint-space/camera-space samples. The Wk of Eq. (2) is a relative weighting factor that enables the system to place a higher relative weight on samples acquired near a particular region of interest. When all samples are weighted equally, for instance in the initialization of the view parameters, Wk = 1. When the manipulator approaches its target incoming samples might be assigned a weight of Wk = 10 or Wk = 20 or some other weight higher than samples acquired more distant from the current pose. This skews the view parameter relationship towards the more heavily weighted data so that the system is more accurate in this heavily weighted region of joint and camera space. While CSM uses the orthographic camera model and the nominal kinematic model of the robot, it still can be stated truthfully that CSM, and therefore MCSM as well, is a calibration-free method. The reasons for this derive from the use of estimation and the fact that the system automatically skews the relationships described by Eq. (1) such that they work well in a local region of operation. In other words, Eq. (1) could describe a calibrated relationship if the view parameters were established and held fixed. However, these view parameters change frequently either through updating with new visual information or simply by skewing the information used to estimate them such that the relationship described by Eq. (1) CSM impractical for the task of pallet engagement by an industrial forklift. CSM’s need for stationary cameras creates other complications such as reduced camera resolution as the system moves further from the cameras, the possibility of the system itself obstructing the view of the camera(s), and maintaining communication between the remote cameras and the mobile vehicle. The relationship between the physical, 3D position of manipulator features and their corresponding camera-space locations as described in Eq. (1) now depends on