15: Visual-Based Control
It is common to talk about a robot moving to an object, but the reality is that the robot is only moving to a pose at which it expects the object to be. This is a subtle but deep distinction. A consequence of this is that the robot will fail to grasp the object if it is not at the expected pose. It will also fail if imperfections in the robot mechanism or controller result in the end-effector not actually achieving the end-effector pose that was specified. In order for this approach to work successfully we need to solve two quite difficult problems: determining the pose of the object and ensuring the robot achieves that pose.
Stepping back for a moment and looking at this problem it is clear that the root cause of the problem is that the robot cannot see what it is doing. If the robot could see the object and its end-effector, it could use that information to guide the end-effector toward the object. This is what humans call hand-eye coordination and what we will call vision-based control or visual servo control — the use of information from one or more cameras to guide a robot in order to achieve a task.
A vision-based control system involves continuous measurement of the target and the robot using vision to create a feedback signal and moves the robot arm until the visually observed error between the robot and the target is zero. Vision-based control is quite different to taking an image, determining where the target is and then reaching for it. The advantage of continuous measurement and feedback is that it provides great robustness with respect to any errors in the system.
The task in visual servoing is to control the pose of the robot’s end-effector, relative to the target, using visual features extracted from the image. The camera may be carried by the robot or fixed in the world, known respectively as end-point closed-loop (eye-in-hand) or end-point open-loop.
Sections cover:
- Position-based visual servoing (PBVS)
- Image feature motion due to camera motion
- Controlling feature motion — image-based visual servoing (IBVS)
- estimating depth
- Performance issues and failure modes
- Servoing using line and ellipse features
Publications
- A tutorial on visual servo control, Hutchinson, Hager & Corke (1996)
- Visual Control of Robots: High-Performance Visual Servoing, 1996, Peter Corke. Now out of print but available for free as an ePrint
- PhD thesis of Peter Corke, 1994, dynamics and control of vision-based control systems
- PhD thesis of Francois Chaumette, 1990 (in French). Describes image task-space control and use of features such as points, lines and ellipses
- Visual servo control I: Basic approaches, Chaumette, Hutchinson (2006)
- Potential problems of stability and convergence in image-based and position-based visual, Chaumette (1998). This paper describes the IBVS failure mode where the camera retreats to infinity
Videos
- Visual servoing, IBVS, PBVS, 2.5D, INRIA lagadic
Software
- Epipolar geometry toolbox
- Visual servoing toolbox
- ViSP for Visual Servoing
- Visual servoing simulator (archived), (Java)
These links are kept in one place for every edition and checked monthly. Resources for every chapter.