DLR Logo
Thursday, 2011-01-27
Department of Robotic Systems

Predictive Sensor Based Control of Robots with Positional Interface

Sensor based control with different types of sensors is investigated for robots with a positional interface. The basic approach which is used for example for force control is presented on the page on sensor based control. Characteristics which are fundamental with predictive sensors are explained here. This concerns, for example, visual servoing tasks.

As with the basic approach we assume a position controlled ideal robot which compensates the robot dynamics, provided that the desired positions are consistent. To do so the ideal robot processes not only the current desired position but also the desired positions of some subsequent sampling steps. Thus consistency means that the desired position of a sampling step l is independent of the time instant k in which it has been computed. This can be ensured best with predictive sensors as cameras which are used for visual servoing tasks. (Visual servoing is understood here as the control along a static visible path, not the following of a moving target.)

Sensor data are used to build a representation of the desired path. This can be done by adding the sensed deviations between actual and desired path (given by the image information of a robot mounted camera) to the actual position of the time instant of sensing. The prerequisites of this method are that the sensor has to be calibrated and that the time instant of sensing has to be known. In contrast to other approaches the amount of time-delays is of minor importance. Even asynchronous sensing is possible.

In this image from a robot mounted camera the desired path is represented by a cable. When evaluating the green lines of the image (instead of the whole image) we find five points of the desired path (marked by red blocks). By interpolation, the whole path can be computed and used for feedforward. This effects that the tcp (yellow block) tracks the cable very accurately (see video clips).

The combination of predictive sensors and an ideal robot theoretically allows to control the robot with high speed along a visible line without any control errors. Given a sufficient range of view of the camera, after a single perception of the scene the robot will approach to the desired configuration and execute the desired path showing the bandwidth of the positional control loop.

In practice however the method is sensitive to calibration errors and to errors in the implementation of the ideal robot (see the page on the adaptation of the positional controller). Besides the ideal robot, adaptation is possible with respect to the camera parameters including the mounting on the robot and with respect to the time shift between camera images and the robot's positional measurements. In this way we reached an accuracy of 0.3 mm when following a cable at 0.7 m/s. (see video clips)




Tracking is not restricted to a single degree of freedom. Instead, we either show tracking of a spatial object (video clip) or of position and orientation along a planar path (video clip). A tilted mounting of the camera is useful in combination with tools.

Problems may occur however, if the desired path cannot be executed by the robot, e.g. if the sensed line gives a non continuous path. Then we use impedance-based smoothing. This means that the sensed path is followed, except for the discontinuities. (video clips)

Back to the homepage of Friedrich Lange


Feel free to give comments or suggestions or to order papers in full resolution (sent by physical mail).

F. Lange and G. Hirzinger, "Spatial Vision-Based Control of High-Speed Robot Arms", in: "Industrial Robotics, From Design to Applications", Advanced Robotic Systems, Vienna, Austria, Nov. 2006 

F. Lange, M. Frommberger, and G. Hirzinger, "Is Impedance-Based Control Suitable for Trajectory Smoothing?", 8th Int. IFAC Symposium on Robot Control, Sep 2006, Bologna, Italy 

paper (6 pages, 295k)     related video clips

The paper discusses the difference between explicit sensor-based control and impedance control, using an example of fast vision-based contour tracking. While explicit sensor-based control may cause a rough desired robot path with sharp vertices, impedance control is known to filter the trajectory at the expense of a bigger control error. We present a new implementation and a new approach to impedance-based control where the latter turns out to produce smooth trajectories with small control errors. Both methods are independent of actual control errors but only depend on the desired speed and the sensed shape of the contour.


F. Lange, M. Frommberger, and G. Hirzinger, "Impedance-Based Smoothing for Visual Servoing along Edges", Joint Conference on Robotics ISR2006 / ROBOTIK2006, May 2006, Munich, Germany 

paper (6 pages, 424k)     digest (1 page, 58k)     related video clips

In contrast to explicit sensor-based control where the goal is to minimize the difference between sensed and desired sensor values, impedance-based control also considers the required accelerations and thus smoothes robot motion. Two approaches to impedance control are developed and applied to the sensor-based method in [1] which uses predictive control to track unknown contours at high speed. With the impedance approach the contour is not more required to be smooth.


F. Lange and G. Hirzinger, "Stability Preserving Sensor-Based Control for Robots with Positional Interface", IEEE International Conference on Robots and Automation (ICRA2005), April 2005, Barcelona, Spain 

paper (6 pages, 490k)     digest (1 page, 70k)     related video clips

When industrial robot arms are controlled using sensor data the performance is dependent on the sensor sampling rate, on delays in signal processing, and on the robot dynamics. The paper presents an approach in which control is inherently stable as long as the time instant of sensing is known, independently of delays. In addition to sensor data the method uses the actual robot pose to compute a desired pose which is then controlled by the existing positional control loop. Updated sensor data affect the system as a refined target for positional control. So the positional control and the use of sensor data are decoupled. This is useful for the integration of a priori information on the task. The method is applicable especially for force control tasks as contour following and for visual servoing.


F. Lange and G. Hirzinger, "Calibration and Synchronization of a Robot-Mounted Camera for Fast Sensor-Based Robot Motion", IEEE International Conference on Robots and Automation (ICRA2005), April 2005, Barcelona, Spain 

paper (6 pages, 420k)     digest (1 page, 60k)     related video clips

For precise control of robots along paths which are sensed online it is of fundamental importance to have a calibrated system. In addition to the identification of the sensor parameters - in our case the camera calibration - we focus on the adaptation of parameters that characterize the integration of the sensor into the control system or the application. The most important of such parameters are identified best when evaluating an application task, after a short pre-calibration phase. The method is demonstrated in experiments in which a robot arm follows a curved line at high speed.


F. Lange und G. Hirzinger, "Anwendungsspezifische Adaption für schnelle sensorgestützte Roboterbewegungen (Custom-designed adaptation for fast sensor-based robot motion)", Robotik 2004, Juni 2004, München, Deutschland (in German)

full version (114k)

Usual calibration methods are not always accurate enough to meet the requirements on precision of robots and sensors for fast sensor-based robot motion. Path deviations caused by calibration errors are minimized for a robot mounted camera used for high-speed tracking of a curved line. Adaptation is executed using the application, to reduce the effect of unmodeled features. Inaccuracies of model parameters of the interaction of camera and robot are compensated as well. The experiment quantifies a reduction of the path error by a factor of three, effectuated by the adaptation.


F. Lange and G. Hirzinger, "Spatial Control of High Speed Robot Arms Using a Tilted Camera", Int. Symposium on Robotics, March 2004, Paris, France

full version with online references to video clips (960k)     extended abstract (210k)
video clips of similar experiments

The paper demonstrates that when mounted on a robot end-effector a camera can be used to control the robot arm to move at high speed along a given workpiece. We use a predictive architecture that considers both the robot's dynamics and the workpiece's uncertainties of the pose or shape. To comply with applications such as glue spraying or laser cutting, the camera is mounted laterally with respect to the tool and, for reasons of visibility, tilted. This complicates the control equations. Nevertheless we manage to compute them in real time and to follow a non planar tube at a speed of 0.7 m/s. In spite of a cost effective hardware setup the mean path deviations are less than 1 mm.


F. Lange and G. Hirzinger, "Predictive Visual Tracking of Lines by Industrial Robots ", The International Journal of Robotics Research, Vol. 22, No. 10-11, pp. 889-903, October 2003

draft of article (490k)     related video clips

Many tasks for industrial robots can be described by high precision line following at high speed. This can be executed accurately if the lines are sensed by a camera since then not only the desired pose at the current time step is sensible, but also a segment of the desired path can be predicted. We propose polynomials to represent the progression of the elements of the desired pose. This allows to realize a dynamical sensor control architecture that considers the two main problems: low sampling rate and delays in image processing, and deviations from commanded paths due to the robot dynamics. In contrast to previous publications we now present the complete formulae to control translation and orientation of the robot by tracking (curved) lines that are visible for a single eye-in-hand camera. Experiments using off-the-shelf hardware show that the robot can be precisely controlled at high speed.


F. Lange und G. Hirzinger, "Kameragestützte räumliche Regelung von schnellen Roboterarmen (Camera-based spatial control of fast robot arms)", VDI/VDE Konferenz Applied Machine Vision, Oktober 2003, Stuttgart, Deutschland, VDI-Bericht Nr. 1800, Seiten 77-84 (in German)

full version (290k)     related video clips

The article deals with the integration of non-delayed refinements to programmed robot paths, bearing the robot's dynamics in mind. It is demonstrated that a standard CCD-camera with fast image processing is sufficient to sense such corrections during motion and to execute them with high accuracy without reducing speed. In the experiment a robot follows a bent tube at 0.7 m/s. A camera that is mounted lateral to the tool measures not only the pose of the tube but also its shape in order to keep a tracking error of say 1 mm, e.g. when spraying glue. The whole system is extremely cost effective since additional hardware is limited to a camera and a basic frame grabber.


F. Lange, "Adaptiv vorausplanende Steuerung für schnelle sensorbasierte Roboterbewegungen", Doctoral thesis, University of Karlsruhe, 2003 (only available in German)

The thesis is available on the server of the university of Karlsruhe.
Startseite (Inhaltsverzeichnis und Download)       Übersicht


F. Lange and G. Hirzinger, "Is Vision the Appropriate Sensor for Cost Oriented Automation?", 6th IFAC Symposium on Cost Oriented Automation (Low Cost Automation LCA2001), October 2001, Berlin, Germany,

full version (220k)     video clips of similar experiments

The article points out that a camera is a flexible sensor and that robots can benefit from visual information for different applications. Concerning cost oriented hardware we restrict to standard vision components. We do not need any hardware besides a camera, a low-cost frame grabber, and the robot with its PC-based controller. Software is computationally efficient since only single image rows are evaluated. Our dynamical sensor control architecture distinguishes between robot positional control and refinement of desired positions using vision and / or other sensors, optionally.


F. Lange and G. Hirzinger, "A Universal Sensor Control Architecture Considering Robot Dynamics", International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI2001), August 2001, Baden-Baden, Germany

original version (6 pages, 180k)     extended version (9 pages, 260k)     related video clips

The paper presents a dynamical sensor control architecture that allows robot arms to perform tasks that with conventional feedback of sensor data fail because of delays or deviations due to the robot dynamics. The architecture distinguishes between robot positional control and refinement of desired positions using vision and / or other sensors. Each of these aspects is designed without the knowledge of the other. Communication between sensors and robot control may be slow and asynchronous. Experiments show vision based control along a seen line at a speed of 0.7 m/s. Path deviations are about 0.6 mm.


F. Lange und G. Hirzinger, "Sensorgeführte Roboter für industrielle Aufgaben", VDI/VDE Konferenz ROBOTIK 2000, Juni 2000, Berlin, Deutschland, VDI-Bericht Nr. 1552 (in German)

original version (160k)     extended version (260k)

The article shows that former problems with sensor based path planning and control methods have been solved. An architecture is presented which easily can be integrated into usual robot control devices. The method consists of several modules which allow the user at a reasonable price to apply sensors to improve the accuracy of given motion with respect to measurable invariants, even at high robot speed. The method allows the application of CAD-based planned paths without further teach-in.


F. Lange, J. Langwald, and G. Hirzinger, "Predictive Feedforward Control of High Speed Tracking Tasks", European Control Conference ECC'99, Karlsruhe, Germany, August / September 1999

full version (290k)     summary (50k)

Industrial robots often have to move with an accuracy of less than 1 mm at a speed of more than 1 m/s. Servoing of arbitrary paths therefore requires feedforward control using predictions of future time-steps of the desired motion. These predictions are obtained by camera images which allow the definition of current and future time-steps of the desired path. This is integrated in an architecture which allows learning of the feedforward controller without any knowledge of the dynamical model of the robot. Experiments are presented in which a robot with an end-effector mounted camera is servoed along a curved line.


F. Lange, P. Wunsch, and G. Hirzinger, "Predictive Vision Based Control of High Speed Industrial Robot Paths", IEEE Int. Conference on Robotics and Automation ICRA'98, Leuven, Belgium, May 1998

ps.gz-version (142k)     pdf-version (600k)

A predictive architecture is presented to react on sensor data in the case of high speed motion and low bandwidth sensor data. This concept is used for the vision based control of an industrial robot (KUKA) to track a contour at a speed of 1.6 m/s. The vision task can be performed very fast since only 3 rows of the image are analyzed. In this way an accuracy of 0.3 mm is reached in spite of uncertainties in the robot's kinematic parameters. Vision and control work asynchronously so that even delay times are tolerable during sensing as long as the time-instant of the exposure is known.

Copyright © 2011 German Aerospace Center (DLR). All rights reserved.