COMBINATION OF ROBOT CONTROL AND ASSEMBLY PLANNING FOR A PRECISION MANIPULATOOR Abstract This paper researches how to realize the automatic assembly operation on a two-finger precision manipulator. A multi-layer assembly support system is proposed. At the task-planning layer, based on the computer-aided design (CAD) model, the assembly sequence is first generated, and the information necessary for skill decomposition is also derived. Then, the assembly sequence is decomposed into robot skills at the skill-decomposition layer. These generated skills are managed and executed at the robot control layer. Experimental results show the feasibility and efficiency of the proposed system. Keywords Manipulator Assembly planning Skill decomposition Automated assembly 1 Introduction Owing to the micro-electro-mechanical systems (MEMS) techniques, many products are becoming very small and complex, such as microphones, micro-optical components, and microfluidic biomedical devices, which creates increasing needs for technologies and systems for the automated precision assembly of miniature parts. Many efforts aiming at semi-automated or automated assembly have been focused on microassembly technologies. However, microassembly techniques of high flexibility, efficiency, and reliability still open to further research. Thispaper researches how to realize the automatic assembly operation on a two-finger micromanipulator. A multi-layer assembly support system is proposed. Automatic assembly is a complex problem which may involve many different issues, such as task planning, assembly sequences generation, execution, and control, etc. It can be simply divided into two phases; the assembly planning and the robot control. At the assembly-planning phase, the information necessary for assembly operations, such as the assembly sequence, is generated. At the robot control phase, the robot is driven based on the information generated at the assembly-planning phase, and the assembly operations are conducted. Skill primitives can work as the interface of assembly planning to robot control. Several robot systems based on skill primitives have been reported. The basic idea behind these systems is the robot programming. Robot movements are specified as skill primitives, based on which the assembly task is manually coded into programs. With the programs, the robot is controlled to fulfill assembly tasks automatically. A skill-based micromanipulation system has been developed in the authors’ lab, and it can realize many micromanipulation operations. In the system, the assembly task is manually discomposed into skill sequences and compiled into a file. After importing the file into the system, the system can automatically execute the assembly task. This paper attempts to explore a user-friendly, and at the same time easy, sequence-generation method, to relieve the burden of manually programming the skill sequence. It is an effective method to determine the assembly sequence from geometric computer-aided design (CAD) models. Many approaches have been proposed. This paper applies a simple approach to generate the assembly sequence. It is not involved with the low-level data structure of the CAD model, and can be realized with the application programming interface (API) functions that many commercial CAD software packages provide. In the proposed approach, a relations graph among different components is first constructed by analyzing the assembly model, and then, possible sequences are searched, based onthe graph. According to certain criterion, the optimal sequence is finally obtained. To decompose the assembly sequence into robot skill sequences, some works have been reported. In Nnaji et al.’s work, the assembly task commands are expanded to more detailed commands, which can be seen as robot skills, according to a predefined format. The decomposition approach of Mosemann and Wahl is based on the analysis of hyperarcs of AND/OR graphs representing the automatically generated assembly plans. This paper proposes a method to guide the skill decomposition. The assembly processes of parts are grouped into different phases, and parts are at different states. Specific workflows push forward parts from one state to another state. Each workflow is associated with a skill generator. According to the different start state and target state of the workflow, the skill generator creates a series of skills that can promote the part to its target state. The hierarchy of the system proposed here ,the assembly information on how to assemble a product is transferred to the robot through multiple layers. The top layer is for the assembly-task planning. The information needed for the task planning and skill generation are extracted from the CAD model and are saved in the database. Based on the CAD model, the assembly task sequences are generated. At the skill-decomposition layer, tasks are decomposed into skill sequences. The generated skills are managed and executed at the robot control layer. 2 Task planning Skills are not used directly at the assembly-planning phase. Instead, the concept of a task is used. A task can fulfill a series of assembly operations, for example, from locating a part, through moving the part, to fixing it with another part. In other words, one task includes many functions that may be fulfilled by several different skills. A task is defined as: T =（Base Part; Assembly Part; Operation） Base_Part and Assembly_Part are two parts that are assembled together. Base_Part is fixed on the worktable, while Assembly_Part is handled by robot’s end-effector and assembled onto the Base_Part. Operation describes how the Assembly_Part is assembled with the Base_Part; Operation ∈ {Insertion_T, screw_T, align_T,...}. The structure of microparts is usually uncomplicated, and they can be modeled by the constructive solid geometry (CSG) method. Currently, many commercial CAD software packages can support 3D CSG modeling. The assembly model is represented as an object that consists of two parts with certain assembly relations that define how the parts are to be assembled. In the CAD model, the relations are defined by geometric constraints. The geometric information cannot be used directly to guide the assembly operation—we have to derive the information necessary for assembly operations from the CAD model. Through searching the assembly tree and geometric relations (mates’ relations) defined in the assembly’s CAD model, we can generate a relation graph among parts, for example, In the graph, the nodes represent the parts. If nodes are connected, it means that there are assembly relations among these connected nodes (parts). 2.1 Mating direction In CSG, the relations of two parts, geometric constraints, are finally represented as relations between planes and lines, such as collinear, coplanar, tangential, perpendicular, etc. For example, a shaft is assembled in a hole. The assembly relations between the two parts may consist of such two constraints as collinear between the centerline of shaft Lc_shaft and the centerline of hole Lc_hole and coplanar between the plane P_Shaft and the plane P_Hole. The mating direction is a key issue for an assembly operation. This paper applies the following approach to compute the possible mating direction based on the geometric constraints (the shaft-in-hole operation of Fig. 3 is taken as an example): 1. For a part in the relation graph, calculate its remaining degrees of freedom,also called degrees of separation, of each geometric constraint. For the coplanar constraint, the remaining degrees of freedom are . For the collinear constraint, the remaining degrees of freedom are . and can also be represented as and . Here, 1 means that there is a degree of separation between the two parts. , and so, the degree of freedom around the z axis will be ignored in the following steps. In the case that there is a loop in the relation graph, such as parts Part 5, Part 6, and Part 7 in Fig. 2, the loop has to be broken before the mating direction is calculated. Under the assumption that all parts in the CAD model are fully constrained and not over-constrained, the following simple approach is adopted. For the part t in the loop, calculate the number of 1s in ; where is the remaining degrees of freedom of constraint k by part i. For example, in Fig. 2, given that the number of 1s in and is larger than and , respectively, then it can be regarded that the position of Part 7 is determined by constraints with both Part 5 and Part 6, while Part 5 and Part 6 can be fully constrained by constraints between Part 5 and Part 6.We can unite Part 5 and Part 6 as one node in the relation graph, also called a composite node, as shown in Fig. 2b. The composite node will be regarded as a single part, but it is obvious that the composite node implies an assembly sequence. 2. Calculate mating directions for all nodes in the relation graph. Again, beginning at the state that the shaft and the hole are assembled, separate the part in one degree of separation by a certain distance (larger than the maximum tolerance), and then check if interference occurs. Separation in both ±x axis and ±y axis of R1 causes the interference between the shaft and the hole. Separation in the +z direction raises no interference. Then, select the +z direction as the mating direction, which is represented as a vector M measured in the coordinate system of the assembly. It should be noted that, in some cases, there may be several possible mating directions for a part. The condition for assembly operation in the mating direction to be ended should be given. When contact occurs between parts in the mating direction at the assembled state, which can be checked simply with geometric constraints, the end condition is measured by force sensory information, whereas position information is used as an end condition. 3. Calculate the grasping position. In this paper, parts are handled and manipulated with two separate probes, which will be discussed in the Sect. 4, and planes or edges are considered for grasping. In the case that there are several mating directions, the grasping planes are selected as G1∩G2∩...∩Gi, where Gi is possible grasping plane/edge set for the ith mating direction when the part is at its free state. For example, in Fig. 4, the pair planes P1/P1′, P2/P2′, and P3/P3′ can serve as possible grasping planes, and then the grasping planes are The approaching direction of the end-effector is selected as the normal vector of the grasping planes. It is obvious that not all points on the grasping plane can be grasped. The following method is used to determine the grasping area. The end-effector, which is modeled as a cuboid, is first added in the CAD model, with the constraint of coplanar or tangential with the grasping plane. Beginning at the edge that is far away from the Base_Part in the mating direction, move the end-effector in the mating direction along the grasping plane until the end-effector is fully in contact with the part, the grasping plane is fully in contact with the end-effector, or a collision occurs. Record the edge and the distance, both of which are measured in the part’s coordinate system. 4. Separate gradually the two parts along the mating direction, while checking interference in the other degrees of separation, until no interference occurs in all of the other degrees of separation. There is obviously a separation distance that assures interference not to occur in every degree of separation. It is called the safe length in that direction. This length is used for the collision-free path calculation, which will be discussed in the following section. 2.2 Assembly sequence Some criteria can be used to search the optimal assembly sequence, such as the mechanical stability of subassemblies, the degree of parallel execution, types of fixtures, etc. But for microassembly, we should pay more attention to one of its most important features, the limited workspace, when selecting the assembly sequence. Microassembly operations are usually conducted and monitored under microscopy, and the workspace for microassembly is very small. The assembly sequence brings much influence on the assembly efficiency. For example, a simple assembly with three parts. In sequence a, part A is first fixed onto part B. In the case that part C cannot be mounted in the workspace at the same time with component AB because of the small workspace, in order to assemble part C with AB, component AB has to be unmounted from the workspace. Then, component C is transported and fixed into the workspace. After that, component AB is transported back into the workspace again. In sequence b, there is no need to unmount any part. Sequence a is obviously inefficient and may cause much uncertainty. In other words, the greater the number of times of unmounting components required by an assembly sequence, the more inefficient the assembly sequence. In this paper, due to the small -workspace feature of microassembly, the number of times necessary for the mounting of parts is selected as the search criteria to find the assembly sequence that has a few a number of times for the mounting of parts as possible. This paper proposes the following approach to search the assembly sequence. The relation graph of the assembly is used to search the optimal assembly sequence. Heuristic approaches are adopted in order to reduce the search times: 1. Check nodes connected with more than two nodes. If the mating directions of its connected nodes are different, mark them as inactive nodes, whereas mark the same mating directions as active mating direction. 2. Select a node that is not an inactive node. Mark the current node as the base node (part). The first base part is fixed on the workspace with the mating direction upside (this is done in the CAD model). Compare the size (e.g., weight or volume) of the base part with its connected parts, which can be done easily by reading the bill of materials (BOM) of the assembly. If the base part is much smaller, then mark it as an inactive node. 3. Select a node connected with the base node as an assembly node (part). Check the mating direction if the base node needs to be unmounted from the workspace. If needed, update a variable, say mount++. Reposition the component (note that there may be not only the base part in the workspace; some other parts may have been assembled with the base part) in the workspace so that the mating direction is kept upside. 4. In the CAD model, move the assembly part to the base part in the possible mating direction, while checking if interference (collision) occurs. If interference occurs, mark the base node as an inactive node and go to step 2, whereas select the Operation type according to parts’ geometric features. In this step, an Obstacle Box is also computed. The box, which is modeled as a cuboid, includes all parts in the workspace. It is used to calculate the collision-free path to move the assembly part, which will be introduced in the following section. The Obstacle Box is described by a position vector and its width, height, and length. 5. Record the assembly sequence with the Operation type, the mating direction, and the grasping position. 6. If all nodes have been searched, then mark the first base node as an inactive node and go to step 2. If not, select a node connected with the assembly node. Mark it as an assembly node, and the assembly node is updated as a base node. Check if there is one of the mating directions of the assembly node that is same as the mating direction of the former assembly node. If there is, use the former mating direction in the following steps. Go to step 3. After searching the entire graph, we may have several assembly sequences. Comparing the values of mount, the more efficient one can be selected. If not even one sequence is returned, then users may have to select one manually. If there are N nodes in the relation graph of Fig. 2b, all of which are not classed as inactive node, and each node may have M mating directions, then it needs MN computations to find all assembly sequences. But because, usually, one part only has one mating direction, and there are some inactive nodes, the computation should be less than MN. It should be noted that, in the above computation, several coordinate systems are involved, such as the coordinates of the assembly sequence, the coordinates of the base part, and the coordinates of the assembly. The relations among the coordinates are represented by a 4×4 transformation matrix, which is calculated based on the assembly CAD model when creating the relations graph. These matrixes are stored with all of the related parts in the database. They are also used in skill decomposition. 3 Skill decomposition and execution 3.1 Definition of skill primitive Skill primitives are the interface between the assembly planning and robot control. There have been some definitions on skill primitives. The basic difference among these definitions is the skill’s complexity and functions that one skill can fulfill. From the point of view of assembly planning, it is obviously better that one skill can fulfill more functions. However, the control of a skill with many functions may become complicated. In the paper, two separate probes, rather than a single probe or parallel jaw gripper, are used to manipulate the part. Even for the grasp operation, the control process is not easy. In addition, for example, moving a part may involve not only the manipulator but also the worktable. Therefore, to simplify the control process, skills defined in the paper do not include many functions. More importantly, the skills should be easily applied to various assembly tasks, that is, the set of skills should have generality to express specific tasks. There should not be overlap among skills. In the paper, a skill primitive for robot control is defined as: Attributes_i Information necessary for Si to be executed. They can be classified as required attributes and option attributes, or sensory attributes and CAD-model-driven attributes. The attributes are represented by global variables used in different layers. Action_i Robots’ actions, which is the basic sensormotion. Many actions are defined in the system, such as Move_Worktable, Move_Probes, Rotation_Worktable, Rotation_Probes, Touch, Insert, Screw, Grasp, etc. For one skill, there is only one Action. Due to the limited space, the details of actions will not be discussed in this paper. Start_i The start state of Action_i, which is measured by sensor values. End_i The end state of Action_i, which is measured by sensor values. Condition_i The condition under which Action_i is executed. From the above definitions, we may find that skill primitives in the paper are robot motions with start state and end state, and that they are executed under specific conditions. Assembly planning in the paper is to generate a sequence of robot actions and to assign values to attributes of these actions. 3.2 Skill decomposition Some approaches have been proposed for skill decomposition. This paper presents a novel approach to guide the skill decomposition. As discussed above, in the present paper, a task is to assemble the Assembly_Part with the Base_Part. We define the process from the state that Assembly_Part is at a free state to the state that it is fixed with the Base_Part as the assembly lifecycle of the Assembly_Part. In its assembly lifecycle, the Assembly_Part may be at different assembly states.Here shows a shaft’s states shown as blocks and associated workflows of an insertion task. A workflow consisting of a group of skills pushes forward the Assembly_Part from one state to another state. A workflow is associated with a specific skill generator that is in charge of generating skills. For different