Teleoperation
Short answer. Teleoperation is a person driving a robot, and it is the most common way to collect manipulation demonstrations. The data it produces looks like a policy training set: pairs of an observation and the action taken. The pairs are wrong unless the timing is handled. A command issued at one moment arrives at the robot a network delay later, so the observation that belongs with it is the one from when the person decided, not the one from when the command landed. A delay of eighty milliseconds on a fast joint move is several degrees of misalignment, and it is systematic rather than random.
How to do it
What a demonstration actually contains
A teleoperation session produces a stream from the operator side, usually joint targets or end effector poses at the rate of the input device, and a stream from the robot side, usually joint positions and sometimes forces at the rate of the controller. The two streams run at different rates and on different clocks, and the training pair has to be assembled from them.
The pair that a policy learns from is the observation at the moment the operator decided, and the action the operator chose. Recording the observation at the moment the action arrived instead puts the wrong state next to the action, and the error is proportional to how fast the robot was moving. It is zero when the arm is still and largest at the fastest point of the motion.
- operator stream: commands, at the input device rate
- robot stream: states and forces, at the controller rate
- the pair is observation at decision time, action as chosen
Two timing problems that are often confused
Latency is the delay between the operator acting and the robot moving. It shifts the whole command stream relative to the state stream, and it has to be corrected before the pairs are assembled. Sampling rate mismatch is different: the command stream is coarser than the controller rate, so each command has to be held or interpolated to fill the gap. Latency needs a timestamp correction, the rate mismatch needs a resampling rule.
The two are often fixed by the same bridge code and confused with each other. Measuring them separately is worth the effort: latency can be measured by commanding a step and recording when the joint moves, while the resampling error can be computed from the two rates and the peak joint speed.
What the collected data is used for
Demonstrations feed imitation learning, where the policy is trained to reproduce the operator actions given the observation. It also feeds offline evaluation, where a policy is scored on states drawn from the demonstration distribution. Both uses depend on the pairs being correctly aligned, and the second depends on the demonstration covering the states the policy will actually visit.
Coverage is the other half of the problem. A demonstration set collected by one operator on one surface at one speed teaches a policy that distribution and nothing outside it. Recording the conditions alongside the data, and deliberately varying them, is what makes the dataset useful beyond the session that produced it.
Code
Quantify the two timing errors on a joint sweeping at a known speed: the misalignment from link latency, and the resampling error from holding a coarse command stream.
import math
# one joint sweeping sinusoidally at 0.5 Hz, 45 degree amplitude; operator side 100 Hz,
# controller 500 Hz
AMP = 45.0
FREQ = 0.5
OP_HZ = 100
CTRL_HZ = 500
LATENCY = 0.08
def angle(t):
return AMP * math.sin(2 * math.pi * FREQ * t)
def speed(t):
return AMP * 2 * math.pi * FREQ * math.cos(2 * math.pi * FREQ * t)
print("joint sweeps %.0f degrees at %.1f Hz, peak speed %.1f degrees per second"
% (2 * AMP, FREQ, AMP * 2 * math.pi * FREQ))
print()
n_op = int(2.0 * OP_HZ)
worst = 0.0
for i in range(n_op):
t = i / OP_HZ
worst = max(worst, abs(angle(t + LATENCY) - angle(t)))
print("operator commands at %d Hz, the link adds %d ms" % (OP_HZ, LATENCY * 1000))
print(" the joint moves up to %.2f degrees between issuing a command and its arrival"
% worst)
print(" so a recorded pair (observation at t, action at t) is off by that much")
print(" whenever the arm is moving, and the error is zero only at the turning points")
print()
n_ctrl = int(2.0 * CTRL_HZ)
zoh_err = interp_err = 0.0
for k in range(n_ctrl):
t = k / CTRL_HZ
i = int(t * OP_HZ)
t0 = i / OP_HZ
held = angle(t0)
if (i + 1) / OP_HZ <= 2.0:
t1 = (i + 1) / OP_HZ
frac = (t - t0) * OP_HZ
lerp = angle(t0) + frac * (angle(t1) - angle(t0))
else:
lerp = held
true = angle(t)
zoh_err = max(zoh_err, abs(held - true))
interp_err = max(interp_err, abs(lerp - true))
print("upsampling the %d Hz command stream to the %d Hz controller:" % (OP_HZ, CTRL_HZ))
print(" hold the last value max error %.3f degrees" % zoh_err)
print(" linear interpolation max error %.3f degrees" % interp_err)
print()
print("holding is what most bridges do, and it is fine at this speed. the number to")
print("check is whether the peak error stays under the controller's own tracking")
print("error, because past that point the data is describing a trajectory the robot")
print("never followed. the latency term above is the one that has to be corrected")
print("before training: the policy has to see the observation that was current when")
print("the action was chosen, not the one that was current when it arrived.")- Standard library only. The joint is modelled as a pure sinusoid so the timing errors can be computed exactly rather than estimated from noisy data.
- Expected: the joint peaks at 141.4 degrees per second, and with an 80 ms link the joint moves up to 11.28 degrees between issuing a command and its arrival. That is the size of the misalignment if the pairs are assembled without correcting for latency.
- Expected: resampling the 100 Hz command stream up to 500 Hz by holding the last value gives a peak error of 1.306 degrees, while linear interpolation gives 0.005 degrees. The hold error is largest where the joint is fastest, which is the same place the latency error is largest.
- Set OP_HZ to 30 and the hold error rises to several degrees, which is where holding stops being adequate and the bridge needs to interpolate. Set LATENCY to 0.2 and the misalignment exceeds 25 degrees, at which point the operator is steering a robot that has already moved on and the demonstrations are not reproducible.
Where this goes wrong
Assembling pairs without correcting for latency
The observation recorded when a command arrives is the state after the operator already acted on an earlier one. Training on those pairs teaches a policy that reacts to the wrong state, and the error grows with speed, so the policy looks acceptable in slow demonstrations and fails on fast ones. Measure the delay and shift the command stream before pairing.
Treating a held command as a smooth trajectory
Holding the last command value produces a staircase, and the error is largest where the joint moves fastest. If the peak error exceeds the controller tracking error, the recorded trajectory is one the robot never followed. Compare the peak resampling error against the controller error and interpolate when it is larger.
Collecting demonstrations from one operator in one condition
A dataset collected by one person on one surface at one speed teaches a policy that distribution. It will fail on a different surface or a heavier object, and the failure looks like a model problem. Record the conditions with the data and vary them deliberately, including the object mass, the surface and the speed.
Discarding failed demonstrations
Recoveries and near misses are the states where a policy most needs guidance, and they are exactly what a success-only dataset removes. Keep the failures with a flag rather than deleting them, and decide at training time whether to include them. A policy that has never seen a recovery cannot perform one.
When to buy the data instead of building the pipeline
Decide the platform, the sensor set and the timing correction yourself, because the demonstrations have to match the policy you will train. Buy the collection when the hardware or the operator skill is not in house, and specify the robot, the sensors, the command rate, the conditions to vary and the number of demonstrations per condition. Ask for the raw streams alongside the aligned pairs, so the alignment can be checked. We collect teleoperation data with the platform, the sensors and the alignment method documented, and deliver the raw streams with the pairs.
Need data for Teleoperation?
Tell us the language, the hours, and what the data needs to look like. You will get a real number and a real timeline, not a range. If we cannot source it well, we will tell you that instead.
- Pilot batch before the full run, so problems surface early.
- Consent documentation delivered with the data.
- No medical or clinical data. No recorded telephone calls.
- You send the spec Language, hours, recording conditions, and the deadline.
- We reply within two business days A real number and a real timeline, not a range.
- You approve a pilot batch A small run first, so problems surface before the full order.
- Full run, then delivery with documents Audio, transcripts, and the consent record for every speaker.