List of Videos
-
3.1
- Open loop control of the Barrel
- 3.2
- Barrel performing a “lolloping” behavior
- 3.3
- Emerging cooperativity in a chain of TwoWheeleds
- 3.4
- The chain of robots with decentralized control in a regular maze
- 5.1
- Sensitivity and adaptation to environmental changes
- 5.2
- Destabilization and Emergence.
- 6.1
- The role of embodiment and situatedness with wheeled robots.
- 6.2
- Chain of wheeled robots exploring a maze.
- 6.3
- Locomotion of Slider Armband with decentralized control.
- 7.1
- Sweeping mode
- 8.1
- The Semni robot with homeokinetic controller
- 8.2
- Semni getting excited at its resonance frequency
- 8.3
- Oscillatory behavior of Semni
- 8.4
- Different rolling modes of the Spherical
- 8.5
- Spherical in a circular basin
- 8.6
- Spherical adapts to a circular corridor
- 8.7
- Whole body movement with Slinging Snake
- 8.8
- Sweeping mode of Slinging Snake with high frequency catastrophe
- 8.9
- Sensitivity and tolerance to sensor failure
- 8.10
- Walk-like behavior of the Rocking Stamper
- 8.11
- Creativity in unexpected situations
- 8.12
- Emergence of nontrivial modes — Precession of the Barrel
- 8.13
- Decay of nontrivial modes
- 8.14
- Creativity in unexpected situations
- 9.1
- Deprivation of internal forward model and recovery shown with the TwoWheeled
- 10.1
- The challenge for self-organization
- 10.2
- Swinging legs.
- 10.3
- Suspended Humanoid
- 10.4
- Dog “playing” with barrier
- 10.5
- Dog climbing over a barrier
- 10.6
- The HippoDog — The effect of the anatomy
- 10.7
- Initial development of the Humanoid (I)
- 10.8
- Initial development of the Humanoid (II)
- 10.9
- Later developments
- 10.10
- Humanoid exercise at high bar
- 10.11
- Humanoid in Rhönrad
- 10.12
- Fight, Fight, Fight
- 10.13
- More Fighting
- 10.14
- The Snake adapting to its environment
- 10.15
- How the Snake may manage to get out of the pit
- 10.16
- Emergence and decay of collective modes
- 10.17
- Self-rescue scenario with the Humanoid
- 10.18
- The world of playful machines
- 13.1
- Armband learns to locomote — weakly guided
- 13.2
- Armband quickly learns to locomote
- 13.3
- Armband changing the direction of motion
- 16.1
- Illustration of the interaction of different materials.
Chapter 2 Self-Organization in Nature and Machines
Abstract:
Self-organization in the sense used in natural sciences means the spontaneous
creation of patterns in space and/or time in dissipative systems consisting of many individual components.
Central in this context is the notion of emergence meaning the spontaneous creation
of structures or functions that are not directly explainable from the interactions between the constituents of the system.
This chapter presents at first several examples of prominent self-organizing systems in nature with the aim to
identify the underlying mechanisms.
While self-organization in natural systems shares a common scheme,
self-organization in machines is more diversified.
An exception is swarm robotics because of the similarity to a system of many constituents interacting via local laws
as encountered in physics (particles), biology (insects), and technology (robots).
This chapter aims at providing a common basis for a translation
of self-organization effects to single robots considered as complex physical systems
consisting of many constituents that are constraining each other in an intensive manner.
Chapter 3 The Sensorimotor Loop
Abstract:
This chapter aims at providing a basic understanding of the sensorimotor loop as a feedback system.
First we will give some insights into the richness of behavior
resulting from simple closed loop control structures in a robotic system called the Barrel.
This richness is a lesson we can learn from dynamical systems theory:
even very simple systems can produce highly complicated behavior.
Nearly everything is possible in such a feedback system that is provided with enough energy from outside.
Surprisingly, this is accomplished even with extremely simple, fixed controllers, to which we will restrict
ourselves here.
In later chapters we will see how the homeokinetic principle makes theses systems adaptive and
drives them towards specific working regimes of moderate complexity,
loosely speaking somewhere between order and chaos.
3.2 Dominated by Embodiment: The Barrel
3.2.2 Open Loop Control
| Video 3.1: Open loop control of the Barrel The robot is controlled by a periodic control signal driving the internal weights with a phase shift of
π/2. Starting with a very low frequency of the
controller signal, the frequency is doubled at time 2:15, 2:40, 3:05, and
3:30. At 3:45 the Barrel is accelerated by a force (red dot) but is seen to
return rapidly to the original mode of behavior. The higher frequencies very
clearly reveal the difficulties in realizing a fast motion under the open loop control paradigm. |
3.2.3 Closed Loop Control
| Video 3.2: Barrel performing a “lolloping” behavior. A fixed closed loop controller (C11=C12=2, C21=C22=−1,
h1,2 = 0) was used. The robot jumps by rapidly moving the internal weight
while rolling from left ro right. |
3.3 Analyzing the Loop
3.3.5 Effective Bifurcation Point and Explorative Behavior
| Video 3.3: Emerging cooperativity in a chain of TwoWheeleds. The arena has no obstacles. Control is completely decentralized, but the
individual wheels spontaneously cooperate in the working regime close to the
effective bifurcation point. |
| Video 3.4: The chain of robots with decentralized control in a regular maze. Spontaneous cooperativity and sensitive reactions to collisions helps the
chain to navigate in the maze without any proximity sensors. |
Chapter 4 Principles of Self-Regulation — Homeostasis
Abstract:
We have seen
in Chap. 3 that there are specific working regimes in closed sensorimotor loops
where the agents exhibit interesting behaviors. The challenge is now to develop
general principles so that the agent finds these regions by itself. One
essential point at this level of autonomy is the ability to survive in
hostile situations, which, as a first prerequisite, requires a certain stability
against external perturbations. An example of this is homeostasis, one of the
prominent self-regulation scenarios in living beings. This chapter introduces Ashby´s
homeostat as a concrete example from cybernetics and develops a general principle of
self-regulation as a first step towards
a general basis for the self-organization of behavior.
Chapter 5 A General Approach to Self-Organization — Homeokinesis
Abstract:
In this chapter we will introduce the concept of homeokinesis, formulate it
in mathematical terms, and develop a first understanding of its functionality.
The preceding chapter on homeostasis made clear
that the objective of “keeping things under control” cannot lead to a system
which has a drive of its own to explore its behavioral options in a self-determined manner. This is not
surprising since the homeostatic objective drives the controller to minimize
the future effects of unpredictable perturbations.
This chapter uses a different objective, the so called time-loop error, derives learning
rules by gradient descending that error and discusses first consequences of the new approach.
Minimizing the time-loop error is shown to generate
a dynamical entanglement between state and parameter dynamics that has been termed homeokinesis
since it realizes a dynamical regime jointly involving the physical, the neural,
and the synaptic dynamics of the brain-body system.
5.2 Homeokinetic Learning
5.2.3 Self-Actualization, Adaptivity, and Sensitivity — Example
| Video 5.1:
Sensitivity and adaptation to environmental changes.
The robot starts with a unit initialization, such that it moves only slowly.
After a short time the feedback strength has risen sufficiently such that the robot
starts moving steadily forward and backward, bouncing at the walls.
Then a heavy trailer is connected to the robot.
Initially the robot hardly moves anymore, but the feedback strength is adapted
so that stronger and stronger actions are performed.
Settings: єc=1, єA=0.25. |
5.2.4 The Homeokinetic Barrel
| Video 5.2: Destabilization and Emergence.
Behavior of the Barrel when starting in a situation where the center of
gravity of the Barrel is very low so that the situation is physically
stable. The homeokinetic learning is seen to destabilize the system quite
rapidly (a few seconds real time), getting the Barrel into moving. The parameter dynamics
of both the forward model and the controller can be followed in the
panels at the left and right upper corners, respectively. The panels depict
the course of the parameters in a time window of 250 steps corresponding to
about 10 sec.
After some time stable rolling patterns are emerging. Later on (at time
01:05) the Barrel was stopped by applying a physical force to it (note the
red dot that pulls the robot). After releasing the Barrel, the system recovers
again in a very short time and resumes the rolling patterns in a slightly
modified form. |
Chapter 6 From Fixed-Point Flows to Hysteresis
Oscillators
Abstract:
Homeokinesis realizes the self-organization of artificial brain-body systems by gradient descending the time-loop error,
a quantity that is truly internal to the robot since it is defined exclusively in terms of
its sensorimotor dynamics.
Homeokinesis can therefore be considered as a self-supervised learning procedure with the special effect
of making the brain-body system self-referential.
We will study this phenomenon here in an
idealized one-dimensional world in order to identify
key features of our self-referential dynamical systems
independently of any specific embodiment effects.
In particular, we will gain some insight into the entanglement of
state and parameter dynamics and investigate the way how the latter induces behavioral variability.
6.4 Embodiment and Situatedness — Robotic Experiments
6.4.1 Wheeled Robots
6.4.2 Spontaneous Cooperation in a Chain of Wheeled Robots
| Video 6.2: Chain of wheeled robots exploring a maze.
Five TwoWheeled robots connected to a chain by passive joints. Control is decentralized with each
wheel being controlled individually by a homeokinetic controller.
The position
of the red robot is used for the analysis.
Settings: єc=єA=0.01. |
6.4.3 Emergent Locomotion of the Slider Armband
| Video 6.3: Locomotion of Slider Armband with decentralized control. Each of the 18
joints is controlled individually by a one-dimensional controller with the
corresponding joint position as the only input information. Nevertheless, after some time a
spontaneous cooperation of the segments occurs that causes the
robot to locomote. It can even surpass obstacles. At the walls the robot
manages to invert its velocity. |
Chapter 7 Symmetries, Resonances, and Second Order Hysteresis
Abstract:
This chapter is a continuation of the preceding chapter to two-dimensional systems.
We will identify new features of our self-referential dynamical system
again independently of any specific embodiment effects.
The additional dimension opens the possibility for state oscillations,
where the entanglement of state and parameter dynamics
will lead to interesting phenomena and induces behavioral variability.
The most prominent effects
are driving oscillatory systems into a second
order hysteresis (by a self-organized frequency sweeping effect)
and getting into resonance with latent oscillatory modes of the
controlled system.
7.3 Second Order Hysteresis
7.3.3 Frequency Sweeping in Real Systems
Chapter 8 Low Dimensional Robotic Systems
Abstract:
In this chapter we will demonstrate the performance of the homeokinetic control system
when applied to physical robots.
We will recognize many of the effects observed in idealized world conditions, shown in
the previous chapters, but most
dominantly witness new features originating from the
interaction of the learning dynamics with the respective embodiment.
Among them are non-trivial sensorimotor coordination, excitation of resonance modes,
the adaptation to different environments –
all emerging from the unspecific homeokinetic learning rules.
The entanglement effect
is seen to make emerging motion patterns transient so that the behavioral options
are explored and a playful behavior is observed.
In order to keep things simple enough for analysis we consider here only
low-dimensional systems and leave the high-dimensional ones for Chap. 10.
| Video 8.1: The Semni robot with homeokinetic controller.
Until the first cut in the video only joint angle sensors and position control were used.
At the beginning a frequency wandering is observed .
Eventually the legs hits the ground and the movement is hindered. The
system enters a bias dominated dynamics where a slow oscillation occurs.
Both joints get synchronized through the mechanical limits and the robot performs a
series of flips.
After the first cut all sensors and the voltage control were used.
Note the change in the behavior, which becomes more compliant with the movement of the robot.
Different frequencies of oscillations are developing, depending on the current mode of behavior
and the pose of the robot.
Later in the clip the sensitivity of the controller can be seen.
The robot is touched very gently and the behavior
changes rather strongly.
Also note the reaction when the robot is turned over.
Importantly the controller adapts very quickly and engages the body
into a different oscillatory mode. |
| Video 8.2: Semni getting excited at its resonance frequency.
Setting: Robot with loaded head, acceleration sensors, and voltage control.
At the beginning the leg moves in synchrony with the body at a rather high frequency.
The robot slowly slides into one direction.
Then the robot is manually turned over and starts to swing at a much lower frequency.
The robot’s resonance frequency is exited and the amplitude rises such the robot falls over to the previous position. |
8.2 Spherical
| Video 8.5: Spherical in a circular basin. The robot starts to roll around randomly.
Then it enters a circular trajectory as suggested by the environment. |
| Video 8.6: Spherical adapts to a circular corridor. Later in the clip the robot repeatedly
balances along the wall for some time. |
8.3 Slinging Snake
| Video 8.8: Sweeping mode of Slinging Snake with high frequency catastrophe.
The initial phase illustrates very nicely how the robot enters the spinning mode.
The frequency rises quickly and then is lowered again until the robot comes to a rest.
It will not come out of this situation. |
8.4 Rocking Stamper
| Video 8.10: Walk-like behavior of the Rocking Stamper. The robot rocks in a way that slow forward locomotion occurs.
The first three frames (from left to right) show one full swing.
The remaining ones show how the robot travels.
Frames times relative to the first frame: 0.5, 0.9, 3.2, and 18 sec. |
8.2 Barrel
| Video 8.11: Creativity in unexpected situations. The Barrel was put into an upright
position by an external force about 10 seconds ago.
Internal axes are
horizontal now so that the sensor values, the inclination of the internal
axes, are zero apart from some small sensor noise. The forward model does not get
any reliable information in this situation so that rapid forgetting sets in.
This is counteracted by the controller, which increases weights so that
small perturbations are amplified.
Note that the parameters are quite shaken at the beginning of the clip due
to the moving of the barrel.
After some time the motion of the internal
weights become so strong that the Barrel is tossed over.
The dynamics
of the parameters of the model and the controller can be followed in the
panels at the left and right upper corners, respectively.
The panels depict
the course of the parameters in a time window of 250 steps corresponding to
about 10 sec.
Note that the
scales of the panels change, in particular the model parameters change by two
orders of magnitude. The rapidly oscillating parameters are the bias terms of
both model and controller. |
| Video 8.12: Emergence of nontrivial modes — Precession of the Barrel. Out of the upright position there are different
modes that can emerge. In the video you see the emergence of a precession mode
which lasts for more than five minutes. Parameter and model dynamics are
depicted in the panels in the right and left upper corners (swapped w. r. t. Video 8.2). Note that the scales of the panels change. |
| Video 8.13: Decay of nontrivial modes. One special feature of our approach is that modes
do not last forever, instead the simultaneous learning of model and controller
most often leads to a slow change of the parameters so that the system leaves
the mode. In the present case, the precession mode, after lasting for about
five minutes decays spontaneously. The parameter changes are most prominently in
the diagonal elements of both the model (left) and the controller (right
panel) depicted by the red and purple lines (which almost coincide). |
8.2.2 Creativity Using the Real Barrel
Abstract:
This chapter discusses several aspects concerning the simultaneous learning
of controller and internal model. We start with discussing the bootstrapping dilemma
arising in this context and the consequences of insufficient sampling.
It appears that homeokinetic learning solves these problems naturally, which
we illustrate in several examples.
Further, we extend the implementation of the internal model by a sensor-branch.
This is seen to increase the applicability of the homeokinetic controller
because it allows for situations where the sensor values are subject
to an action-independent dynamics.
The extended model is prone to an ambiguity in the learning process,
which can lead to instabilities.
The problem can be resolved if the time-loop error is used as an additional objective
for the model learning.
9.1 Cognitive Deprivation and Informative
Actions
9.1.1 Demonstration by the TwoWheeled
| Video 9.1: Deprivation of internal forward model and recovery shown with the TwoWheeled. The controller was first restricted to only straight driving.
The model degenerates.
After the learning is switched on (robot turns green)
the robot mainly rotates at the spot for a while before driving around normally. |
Chapter 10 High-Dimensional Robotic Systems
Abstract:
This chapter contains many applications of homeokinetic learning to
high-dimensional robotic systems.
The examples chosen for investigation and proposed as experiments to the reader
comprise various robots ranging from dog-like,
to snake-like up to humanoid robots in different environmental situations. The aim of the experiments is to understand
how the controller can learn to “feel” the specific physical properties of the body in
its environment and manages to get in a kind of functional resonance with the physical system. In order to
better bring out the characteristics of homeokinetic learning in these systems, we use a
kind of physical scaffolding, for instance suspending the Humanoid like
a bungee jumper, putting it in the Rhoenrad, or hanging it at the high bar. Interestingly, in all situations the robots
develop whole-body motion patterns that seemingly are related to the specific environmental situation:
the Dog starts playing with a barrier eventually jumping or climbing over it; the Snake develops
coiling and jumping modes; we observe emerging climbing behaviors of a Humanoid like trying to get out of
a pit; and wrestling like scenarios if a Humanoid is encountering a companion.
Eventually, in our robotic zoo all kinds of robots are brought together so that
homeokinesis can prove its robustness against heavy interactions with other robots or
dynamical objects.
Essentially this chapter provides a phenomenological overview
and invites to play around with numerous simulations
to see the “playful machine” in action.
| Video 10.1: The challenge for self-organization. Emergence of embodiment-specific behavior in robots with different
morphology but identical brains. |
10.1 Underactuated and Compliant
| Video 10.2: Swinging legs. The trunk of the Dog is fixed so that the legs can move freely. The motors
are so weak that the controller must learn to excite a resonance mode. There
are no physical cross couplings between the legs. Nevertheless, there are some
frequency and phase correlations observable. |
| Video 10.3: Suspended Humanoid. The robot is pulled upwards by a simulated spring such that the robot can
freely move, but still interacts with the ground. This is a kind of
scaffolding situation in order to give the brain time to accustom to the
body. |
10.2 Dog and HippoDog
| Video 10.4: Dog “playing” with barrier. The Dog video demonstrates the playful manner of exploring the world in
the homeokinetic learning scenario. Note that the robot has nothing but its
joint angle sensors as source of information about its body and the
environment. The Dog is protected by an invisible box on its back
preventing it from falling over. |
| Video 10.5: Dog climbing over a barrier. When encountering a barrier, the Dog can behave in many different ways
since there is no goal prescribed. However, more often than not, the robot
manages to climb over the barrier in a seemingly goal oriented way. In the
video, the Dog has acquired a rather cautious behavior slowly probing
different possibilities of interacting with the barrier. After some time the
left hind leg is swung onto the barrier and after several minutes it climbs
out completely. |
| Video 10.6: The HippoDog — The effect of the anatomy. The HippoDog with its spherical trunk is developing more active and jumpy
motion patterns due to its specific anatomy. |
10.3 Humanoid
| Video 10.8: Initial development of the Humanoid (II). If started in the pit, the robot develops increasingly complex motion
patterns related to that situation. Interestingly,
these patterns keep repeating in the course of time, getting more pronounced
and lasting over a longer time. |
High Bar and Rhönrad - Feeling the Body
Fighting
| Video 10.12: Fight, Fight, Fight. If robots come into close contact they may excite each other to complex
motion patterns. This is amazing since the only way of “feeling” each other
is by the perturbations in the sensorimotor dynamics caused by the physical
forces exerted in the interaction. Adherence is due to normal friction but
essentially also by a failure of the ODE physics engine, which after heavy
collisions produces an unrealistic interpenetration effect, which acts like a
special gripping mechanism. So, the “fighting” is a truly emerging
phenomenon not expected before we observed it. |
10.4 Snakes — Adaptation and Spontaneity
| Video 10.14: The Snake adapting to its environment. Under strong physical constraints, the homeokinetic learning finds a way to
keeping the robot active while taking account of the geometry of the vessel. |
| Video 10.15: How the Snake may manage to get out of the pit. The first picture in the sequence shows the snake in a seemingly relaxed
situation. However, as the video shows there are tensions building up in the
body and after some time it suddenly crunches into a tight bundle of which it
unrolls with very high velocity. By the inertia effects it manages to nearly
jump out of this very deep pit despite the quite weak motor forces. |
10.5 Self-Rescue Scenario
10.6 World of Playful Machines
Chapter 11 Facing the Unknown — Homeokinesis in a New Representation*
Abstract:
Homeokinesis has been introduced and analyzed in the preceding chapters on the basis of
the time-loop error (TLE). This chapter presents an alternative approach to the general
homeokinetic objective by introducing a new representation of the sensorimotor dynamics. This new representation
corrects the state dynamics for the predictable changes in the sensor values so that the
transformed state is constant except for the interactions with the unknown part of the dynamics. The single-step
interaction term will be seen to be identical to the TLE so that
the learning dynamics is not altered. However, besides giving
an additional motivation for the TLE, this chapter
will extend the considerations to the case of several time steps and will eventually consider
infinite time horizons making contact with the global Lyapunov exponents and chaos theory.
Chapter 12 Guided Self-Organization — A First Realization
Abstract:
We introduce here an in the following chapters guided self-organization as the combination
of specific goals with self-organizing control.
As a first realization we propose in this chapter the guidance
with supervised learning signals.
First, we investigate how these signals can be incorporated into
the learning dynamics and present then a simple scenario with direct motor teaching signals.
We find that the homeokinetic controller explores around the given motor patterns
and thus may find a more suitable behavior for the particular body.
Second, we transfer this into a teaching at the level of sensor signals,
which is very natural in our setup.
This mechanism of guidance builds the basis for
higher level guiding mechanisms as discussed in Chap. 13.
Chapter 13 Channeling Self-Organization
Abstract:
Many desired behaviors are distinguished by a certain structure in the motor
or sensor activity. In particular the phase relation between different motors or sensors
capture a lot of this structure.
We will now propose a way to embed these relations as soft constrains to the learning system,
such that we break certain symmetries and let desired behaviors emerge.
Starting from the guidance by teaching we introduce the concept of cross-motor teaching
that allows to specify abstract relations between motor channels.
First we study simple pairwise relations and shape the behavior
of the TwoWheeled robot to drive mostly straight by a relation between both motor neurons.
Then we will consider a high-dimensional robot—the Armband
and demonstrate fast locomotion behaviors from scratch by guided self-organization.
| Video 13.1: Armband learns to locomote — weakly guided. Behavior of the robot with cross-motor teaching and weak guidance (γ=0.001).
A slow locomotive behavior with different velocities is
exhibited. Explorative actions cause the
posture of the robot to vary in the course of time. |
| Video 13.2: Armband quickly learns to locomote. Behavior of the robot with cross-motor teaching and medium guidance (γ=0.003).
Comparable fast locomotive behavior emerges quickly and is persistent.
Nevertheless the velocity varies.
Only small exploratory actions are takes, such that the posture is mainly
constant. |
| Video 13.3: Armband changing the direction of motion.
The behavior of the robot with cross-motor teaching if the connections are changed.
The video starts with a fast locomotive behavior to the left (k=1).
At time 5:00 the couplings are changed (k=0) and the robot slowly stops.
A period of probing actions follows until a reversed locomotion starts
to show up. |
Chapter 14 Reward-Driven Self-Organization
Abstract:
In this chapter we investigate how to guide the self-organization process
by providing an online reward or punishment.
The starting point for the following considerations is
that the homeokinetic controller explores the behavioral space
of the controlled system and that those
behaviors which are well predictable will persist longer than others.
The idea we pursue in this chapter is
to regulate the lifetimes of the transient according to the
reward or punishment.
The mechanism is applied to the Spherical with two goals,
fast motion and curved rolling.
Chapter 15 Algorithmic Implementation
Abstract:
This chapter presents a unified algorithm implementing the
homeokinetic learning rules including
a number of extensions partly discussed already in earlier chapters of this book.
We continue with some guidelines and tips on how to use the homeokinetic “brain.”
We discuss techniques and special methods to make the
self-supervised learning of embodied systems more reliable from the
practical point of view.
This includes the regularization procedures for the singularities in the time-loop
error and different norms of the error for the
gradient descent.
The internal complexity of the controller and the model is extended
by the generalization to multilayer networks.
Apart from that the computational complexity of the learning algorithm
will be reduced essentially by easing non-trivial matrix inversions.
This is important for truly autonomous hardware realizations.
Chapter 16 The LpzRobots Simulator
Abstract:
In this chapter we describe our robot simulator.
We start with the overall structure of the software package
containing the controller framework, the physics simulator
and external tools.
The controller framework makes it very easy to develop and test
our algorithms, be it in simulations or with real robots.
The physics simulator can handle rigid bodies with fixed geometric representation that
are connected by actuated joints.
Particular efforts have been undertaken to develop an elaborated treatment of
physical object interactions including friction, elasticity, and slip.
The chapter also briefly discusses the generation of virtual creatures, the user interface
and the most important features of the LpzRobots simulation environment.
| Video 16.1:
Illustration of the interaction of different materials.
All spheres have the same size and mass.
The substances are foam (yellow),
rubber (dark brown), plastic (white), and metal (silver).
Note the different penetrations (second picture) and bouncing heights. |
Content: © Copyright 2011, Ralf Der and Georg Martius;
Impressum.
Parts of the page have been translated from LATEX by
HEVEA.
