MURILow-Level
Control
Fabrication
High-LevelControl
How is Compliance used in Locomotion?
Berkeley & Stanford: Measurements of Cockroach Locomotion
What Compliance Strategies in Human-level Tasks?
Harvard & Johns Hopkins: Compliance Learning and Strategies for Unstructured Environments
Fabrication
MURILow-LevelControl High-Level
Control
What strategies are used in insect locomotion and what are their implications?Insect locomotion studies (Berkeley Bio)New measurement capabilities (Stanford)
What motor control adaptation strategies do people use and how can they be applied to robots?
Compliance Learning and Strategies for Unstructured Environments (Harvard & Johns Hopkins)Implications for biomimetic robots (Harvard, Stanford)
Guiding questionsGuiding questions
Yoky Matsuoka and Rob Howe
Harvard University
MURI
High-LevelControl
Impedance Adaptation in Unstructured Environments
MURI
High-LevelControl
• An example of manipulation with impedance
• Why is biology superior to current robots in an unstructured environment?
Manipulation with Impedance
MURI
High-LevelControl
Identify Impedance Learning Strategy in Human
• Two big questions:– What is the initial strategy used to cope with
unknown/unstructured environments?– After learning, what does the biology pick as the
good solution for impedance for a given environment?
How can these solutions help robot’s control strategy?
MURI
High-LevelControl
Comparison Between Analytical and Biological Solutions
• We can mathematically derive optimal impedance for a linear world.– Biological system converges to the analytical solution. ---
great!– Biological system converges to a different solution. --- what
and why: put the biological solution back in the equations and reverse engineer.
• What about a nonlinear varying world where it is difficult to derive the optimal impedance?– What does the biological system do? Can it be modeled as a
solution for robots?
• Goal: Find the “best” impedance.– For this case, find best
Khand.
• Uncertainty in the world– mball, kball, ball(0), and
khand
mhand
khand
mball
kballxball
xhand
x
MURI
High-LevelControl
Example: Linear World --- Catching a Ball
ball
hand
mball
mhand
kball
khand
• Cases:1. Hand stiffnes (khand) is too high
• hand < 0 bounces up
2. Hand stiffness (khand) is too low• xhand > Threshold bottoms out
3. Hand stiffness (khand) is just right• xball xhand until switch is pressed
x
khand 0 infinite
2 3 1
MURI
High-LevelControl
Example: Linear World --- Catching a Ball
mhand
khand
mball
kballxball
xhand
• Solve for xhand(t) and xball(t)– initial condition
• ball(0) > 0
• xball(0) = 0
• hand(0)= 0
• xhand(0) = 0xhand
x
x
m x k x x
m x k x x k xball ball ball hand ball
hand hand ball hand ball hand hand
( )
( )
0
0
MURI
High-LevelControl
Example: Linear World --- Catching a Ball
mhand
khand
mball
kballxball
MURI
High-LevelControl
Analytical Linear World to Biological Motor Control
• The example relates task performance to limb impedance and optimal solution.– Other examples: leg impedance, etc.
• Now measure human strategy….– “System identification” – Need a new technique
MURI
High-LevelControl
Existing System Identification Techniques
• Time invariant systems --- easy– assume constant m, b, and k over time.– apply external impulse perturbation force. – repeat the same condition and average.
)()()()( oxxktxbtxmtF
MURI
High-LevelControl
Existing System Identification Techniques
• Time varying systems• Cannot apply impulses close to each other.
• Need multiple impulses to solve for multiple unknowns.
– PRBS (Lacquaniti, et al. 1993)
)()()()( oxxktxbtxmtF
Setup
Handle
Accelerometer
Data Acquisition
System
Processor
HumanSubject
MURI
High-LevelControl
New System Identification Technique to Observe Learning
Robot
Force Sensor
Monitor
MURI
High-LevelControl
New System Identification Technique to Observe Learning
• Very short duration
• Very clean data from force and acc. sensors
F
m*a
b*v
k*x
m=F/a b= (F-ma)/v k= (F-ma-bv)/x
MURI
High-LevelControl
Testing the New Technique
MURI
High-LevelControl
Video of the task here
• Phantom robot is used as the perturbation/measurement tool.
• Task: balance the moving ball on paddle.
– ball moves at constant speed
– dies when the ball falls off the paddle
– perturbation applied every second
MURI
High-LevelControl
Testing the New Technique
MURI
High-LevelControl
Impedance Change with Learning
k change over time b change over time
m change over time
• Observe the impedance change within one catch
• Observe the impedance change between catches
** under development --- pilot studies underway
MURI
High-LevelControl
Contact Interaction Task -- Impedance Dependent Task
k
b
MURI
High-LevelControl
Current Understanding of the Structure of the Biological Controller
Inverse DynamicsModel
uu
usi
q
q
q q
Long-loopGain
q
uf
ui
q
Forward DynamicsModel 120 msec
30 msec
60 msec
InertialDynamics
From Shadmehr
MURI
High-LevelControl
• Developed a new impedance identification technique– Based on virtual environment --- extremely
versatile– Confirmed ability to measure instantaneous
impedance, characterized changes with learning.
Impedance AdaptationConclusions and Future Work
MURI
High-LevelControl
• Current experiments– Determine human interaction strategies
• initial impedance
• learning characteristics
• final impedance
• Next experiments– Determine human interaction strategies for
nonlinear varying tasks• e.g. plastic deformation (running in sand)
Impedance AdaptationConclusions and Future Work
Control of LocomotionMURI
High-LevelControl
Local controller (single limb):
Control of the limb based on local information:
- position and velocity of the limb
Is the limb far enough to the back so that I can start the return stroke?
- forces acting on the limb
Is the supporting load small enough for me to lift the limb?
Task controller (all limbs):
Coordination with other limbs:
- position of the other legs
Across species, control of limb based on local information appears similar (Cruse, TINS, 90). Coordination with other limbs appears highly species dependent.
MURI
High-LevelControl
Step cycle: generation of power (stance) and return (swing) strokes.
Return phase:
Move the limb from posterior to anterior position along a desired velocity profile.
- Maintain proper impedance to remain stable in case of perturbations
After hitting an obstacle, the limb should converge back to the desired path
- Adapt impedance to allow for generation of desired behavior in the face
of a persistent environment
limb is moving in highly viscous fluid, it must adapt its impedance to the characteristics of the environment.
Impedance control and adaptation in a position control task
Power phase:
Maintain contact, maintain height of load, move limb from front to rear.
Impedance control and adaptation in a contact/force control task
Control of a Limb Based on Local Information
MURI
High-LevelControl
Current Limb Local Control Models for Locomotion in Insects
Cruse et al (Neural Networks 1998): Stick insect model
- limb has little or no inertia
- no muscle like actuators
- controller output is velocity, feedback sensing via position and linear feedback
- no ability to adapt
Essentially a kinematic model of a limb only, with little or no dynamics
This kind of model tells us little about how to design good controllers
MURI
High-LevelControl
General Goals:
1. To understand what impedance strategies a biomechanical controller uses when it moves the limb in a position control task. Apply the results to control of the return phase.
2. To understand impedance strategies of the biomechanical controller in a force control task. Apply the results to the control of the stance phase.
Approach:
Study the human arm’s impedance adaptive control strategies in both position and force control tasks. Test validity of the strategies on a robotic system.
Designing a Single Limb Impedance Controller
MURI
High-LevelControl
Designing a Single Limb Impedance Controller
Task Division:
1. Impedance control at very short time intervals (<10 msec, preflexes)
Yoky Matsuoka and Rob Howe
2. Impedance control at intermediate and long time intervals (<300 msec)
Tie Wang and Reza Shadmehr
3. Test and implementation on a robotic system
Jay Dev Desai and Rob Howe
MURI
High-LevelControl
Challenges:
The biomechanics of the human arm are dominated by multiple feedback loops, with various time delays. Impedance measurements are done through imposition of perturbations and measurement of responses.
• How do time delays affect measures of arm impedance?
• Humans learn internal models when they learn control. How does a change in the internal model affect measures of arm impedance?
• Impedance measures require an estimation of where the system would have been if it had not been perturbed. How well can we do this with a non-stationary system like the human arm?
Quantifying Impedance Control Strategies of a
Biomechanical Controller
MURI
High-LevelControl
Current Understanding of the Structure of the Biological Controller
Inverse DynamicsModel
uu
usi
q
q
q q
Long-loopGain
q
uf
ui
q
Forward DynamicsModel 120 msec
30 msec
60 msec
InertialDynamics
Through modulation of input u to the muscles, impedance of the system is changed. The impedance depends on 3 feedback pathways:1. Near zero-delay mechanical stiffness/viscosity of the muscles (Yoky).2. Short delay sensory feedback through spinal structures.3. Long delay sensory feedback through cortical structures (forward model).
Are muscle “preflexes” enough?
1 cmdt=10ms dt=30msdt=10ms
Intact control systemHigh-level sensory feedback loop disrupted
MURI
High-LevelControl
MURI
High-LevelControl
Impedance of a biological arm: A definition
qd
dMB
dq
dMK
tuqqMqqq
tqtuut
tuqqMqqq
t
uqqM
qqqCqqHqqq
q
q
0
0
0000000
000
:Viscosity
:Stiffness
)(),,(),,(
).(y trajectorparticular a follows arm ,)( and )(when
)(),,(),,(
)( :tenvironmen from torquesExternal
),,( :musclesby produced Torques
),()(),,( : torquesInertial
MURI
High-LevelControl
Estimating Impedance: Theory
uqqqqq
uqquqqMuuqqqqM
uqqMuuqqqqMqqq
uuqqqqMqqqqqq
uuqq
uqqMqqq
du
dM
qd
dM
dq
dM
du
dM
qd
dM
dq
dM
uqq
qqq
00000
000000000
00000000
0000000
00
0000000
)()(
...),,(),,(
),,(),,()()( :)1()2(
(2) ),,(),,(
. and in resulting , :force perturbing a Impose
(1) ),,(),,( :onperturbati a without Dynamics
MURI
High-LevelControl
Estimating Impedance: Requirements
)( :onperturbatiour ofresult a asinput neural musclein change The 3.
)( :onsperturbati no were thereif follow wouldsystem thePath that 2.
),()(),,( :arm theof dynamics Inertial 1.
:estimatemust We
,, :measurecan We
0
tu
tq
qqqCqqHqqq
uqtBqtKqqqdu
dM
u000 )()()()(
MURI
High-LevelControl
1.0 Estimating Inertial Dynamics of the Arm (Theory)
qqqCqqHqqq ),()(),,( : torquesInertial
FJ
qqa
qqqaqqaqqC
aqaa
qaaqaaaqH
T
0)sin(
))(sin()sin(),(
)cos(
)cos()cos(2)(
122
2122222
3223
2232231
),,(
0),,( musclesby produced Torques
qqq
uqqM
. , , parameters estimate :OBJECTIVE
hand on the acting force external :
321 aaa
F
Give a force pulse, use data for up to 14 ms after the pulse to estimate inertial parameters.
MURI
High-LevelControl
0 0.1 0.2 0.3 0.4-2
0
2
4
6
Time (s)
Sho
ulde
r T
orqu
e (N
m)
0 0.1 0.2 0.3 0.4-1
0
1
2
Time (s)
Elb
ow T
orqu
e (N
m)
0 0.1 0.2 0.3 0.41
1.05
1.1
1.15
1.2
Time (s)
Sho
ulde
r A
ngel
(ra
d)
0 0.1 0.2 0.3 0.41.6
1.605
1.61
1.615
Time (s)
Elb
ow A
ngel
(ra
d)
0 0.1 0.2 0.3 0.4-0.5
0
0.5
Time (s)
Sho
ulde
r V
eloc
ity (
rad/
s)
0 0.1 0.2 0.3 0.4-0.2
-0.1
0
0.1
0.2
Time (s)
Elb
ow V
eloc
ity (
rad/
s)
1.1 Estimating Inertial Dynamics of the Arm (Methods)
MURI
High-LevelControl
1.2 Estimating Inertial Dynamics of the Arm (Results)
0 10 20 30 40 50 60 70-4
-2
0
2
4
6S
ho
uld
er T
orq
ue
(Nm
) Experiment Model Error
0 10 20 30 40 50 60 70-4
-3
-2
-1
0
1
2
3
4
Elb
ow
To
rqu
e (N
m)
Index of Array
]0.1293 0.17270.3025 [][321
aaa
MURI
High-LevelControl
2.0 Predicting the Un-perturbed Trajectory (Theory)
][MovPerturbed_
][d_MovUnperturbe
2
)_:_(
1
)1_:1(timerial
2
)_:_(
1
)1_:1(timetrial
timemovtimeprbtrialtimeprbtrial
timemovtimeprbtrialtimeprbtrial
PP
SS
:ionapproximatfor bases estimate to(PCA) Analysis Components Principal Use
pertubed ismovement point when time:_
movementeach ofduration :
movements ofnumber :
matrices data raw ofmean : ,
matrices data raw fromPCA : ,
matrices data raw : ,
timeprb
time
trial
PSPS
PSPS
PS
meanmean
pcapca
Legend
MURI
High-LevelControl
2.1 Trajectory Prediction (Methods)
.predict todata raw theof components principal theuse
wemovements, of data raw theusing of instead that is idea The
))(())()(( :2 Method
movement.unknown epredict th to
function base theas movements of data raw theall use that is idea The
that Asume
predictionfor bases as movements dunperturbe all use :1 Method
methods) (2 predict to , , Use
'2
'1
22'2
11'1
21
222
111
2
21222111111112
2122
211
1
2121
KCK
PPK
PPK
KCK
SSK
SSK
S
SSSSSSSSPPP
KKPSKPSK
PPSS
pca
pca
pca
pca
mean
pcapcameanmeanpcapcamean
MURI
High-LevelControl
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Sho
ulde
r V
eloc
ity(m
/s)
Measured Trajectory Predicted Trajectory
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55-2
-1.5
-1
-0.5
0
0.5
Elb
ow V
eloc
ity(m
/s)
Time (s)
Prediction error (%)
17.47516.89634.36491.9896
17.85197.0604 4.28551.9125
6.05.05.04.04.03.03.02.0 ssssssss
2.2 Trajectory Prediction (velocity)
MURI
High-LevelControl
2.3 Trajectory Prediction (force)
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
0 20 40 60-2
-1
0
1
Conclusion: Position, velocity, and force can be reasonably well predicted for up to 300 msec after the last sampled data point.
Time (10 msec)
For
ce (
N)
MURI
High-LevelControl
3.0 Estimating the Effect of u(t) on Arm Impedance
uqtBqtKqqqdu
dM
u000 )()()()(
Inverse DynamicsModel
uu
usi
q
q
q q
Long-loopGain
q
uf
ui
q
Forward DynamicsModel 120 msec
30 msec
60 msec
InertialDynamics
u is the change in the input to the muscles as a result of our perturbation.
While u cannot be measured directly, we know that it depends on a number of time-delayed, possibly adaptive error feedback systems.
• Time-delayed error feedback from the spinal reflexes
• Time-delayed error feedback from the forward model based cortical pathways
• Input from inverse model based “open-loop” controller
MURI
High-LevelControl
In general, a time delay d in a feedback loop reduces apparent viscosity and adds apparent mass to a system.
Example:
)0( 0)()2/1(
:of impedanceapparent has system resulting The
)(2/1)()()(
:delay on theexpansion seriesTaylor Using
0)(
22
2
dkxxkdbxkdm
ddt
xdd
dtdx
txtx
dtkxxbxm
3.1 Time-delayed effect of u(t) on Arm Impedance
MURI
High-LevelControl
3.2 Estimating u(t) in Terms of Measurable Quantities
1. Effect of Spinal Reflexes
: thatfollowsit
on,perturbati aafter and seriesTaylor a of in termsdelay theRewrite
))(())(( :Assume
muscles theofeedback terror based spinal
muscles theinput to adaptive based, model forward
muscles theinput to adaptive based, model inverse
muscles theinput to
dssdsss
s
f
i
sfi
qdtqbqdtqku
u
u
u
u
uuuu
qdbdkqdkbqku sssssssss )2/1()( 2
MURI
High-LevelControl
3.2 Estimating u(t) in Terms of Measurable Quantities
2. Effect of the Inverse model
Inverse DynamicsModel
uu
usi
q
q
q q
Long-loopGain
q
uf
ui
q
Forward DynamicsModel 120 msec
30 msec
60 msec
InertialDynamics
0on perturbati aafter therefore,0 ii u
dq
du
MURI
High-LevelControl
3.2 Estimating u(t) in Terms of Measurable Quantities
3. Effect of the Forward Model
dfdff
f
dfdff
t
i
qdtqddqqbqdtqddqdqqku
qu
qE
qqbqqku
dtq
ddtqddtqdtqtq
dtdtqtq
qqEqqCuqqMqHq
)(ˆ)(ˆ2/3
: quantities measurable of in terms Rewrite
model. forward theofoutput is ˆ ,0ˆ Assume :1 Case
)ˆ()ˆ(
:assume and expansions seriesTaylor r with thei)( delays Replace
)(ˆ)(2)()(ˆ
:model forward theofon intergrati theeapproximatLinearly
ˆ)(ˆ
),(ˆ),(),,()(ˆ :model forward theofn Computatio
0022
0
2
1
MURI
High-LevelControl
3.2 Estimating u(t) in Terms of Measurable Quantities
4. Effect of Adaptation of the Forward Model
qWHdbWHdkqWdHbWHdku
u
qWHdqWdHqdtqddqqb
qWHdqWHddqdtqddqdqqku
qu
qWE
fffff
f
df
dff
f
1213112
1210
13120
22
:be willadaptation before response respect to
with response adapted in the change the,on perturbati a toresponseIn
)(ˆ
)(ˆ2/3
: quantities measurable of in terms Rewrite
.ˆ and present, ist environmen field force a Assume :2 Case
Predictable changes in impedance should occur as a function of the kind of model that the system learns as it practices movements in an unstructured environment.
If learning is via a forward model, the apparent viscosity must decrease as compared to values obtained before the controller had adapted.
MURI
High-LevelControl
-0.1 -0.05 0 0.05 0.1 0.15-0.05
0
0.05
0.1
0.15
X (m)
Y (
m)
Perturb the movement in different directions
Measuring Impedance of the Moving System
MURI
High-LevelControl
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-8
-6
-4
-2
0
2
4
6
8
10
12
Time into the movement (s)
Interaction force
Inertial dynamics
Impedance Controlled force
acceleration (m/s2 )
velocityx25 (m/sec) positionx50 (m) Fo
rce
(N)
Per
turb
Impedance Early into the Movement
MURI
High-LevelControl
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-8
-6
-4
-2
0
2
4
6
8
10
12
Dynamic force
Interaction force
Impedance controller’s force
Positionx50 (m)
Velocity x 25 (m/s)
Acceleration (m/s2)
Time into the movement (s)
Forc
e (N
)
Impedance in the Middle of the Movement
Per
turb
MURI
High-LevelControl
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5-8
-6
-4
-2
0
2
4
6
8
10
12
Time into the movement (s)
velocityx25 (m/sec)
acceleration (m/s2)
Inertial dynamics
Interaction force
Perturb
Impedance Controlled force
positionx50 (m)
Forc
e (N
)
Impedance Near End of Movement
MURI
High-LevelControl 0
-200
Join
t Sti
ffne
ss (
N.m
/rad
)
100 msec
Stiffness of the System: Results
System Characteristics:
- Initially a very stiff system (likely due to intrinsic muscle stiffness).
- The system yields as the perturbation persists, with stiffness dropping as the short- and long-loop reflex systems take control.
MURI
High-LevelControl
Immediate Plans
Position control task:
- Impose a force field, measure impedance changes as the system adapts
- Do we see evidence for formation of a forward model as indicated by reductions in the system’s viscosity?
Force control task:
- Measure impedance changes in a task that requires maintaining contact (pushing in order to roll a virtual conveyer belt).
- Push too hard, the belt breaks. Push too soft, arm slips on the belt.
- Measure impedance changes as the virtual belt’s dynamics changes, requiring adaptation in the controller.
- Virtual belt is produced by the robot manipulandum.