Get in touch

Technology

How we train robot swarms.

Our approach: train teams of robots in simulation with multi-agent reinforcement learning, test every policy against simple baselines, then run it on each robot behind a safety layer. We are building this platform now, in simulation first; the hardware and field steps describe how a policy reaches a partner’s robots. Each step is explained in plain words first, then in technical detail.

  • Multi-agent RL
  • Centralized training, decentralized execution
  • Sim-to-real
  • Onboard safety layer

Overview

One loop: simulate, train, evaluate, deploy.

A single robot can be handled with careful engineering: a planner, a controller, a set of rules. A team in the same field brings new problems. Each robot sees only part of the scene, radio links drop, and the right move for one depends on what the others are about to do.

With multi-agent reinforcement learning (MARL), the robots practice together in simulation across a wide variety of situations, and their coordination comes from that experience rather than from rules someone had to foresee.

Every policy takes the same path around the loop, and the sections below take its steps one at a time.

The Iresine training loop Four steps in a cycle. Simulate: many randomized worlds run in parallel. Train: the robots learn as one team. Evaluate: each candidate policy is compared against baselines, and only a policy that passes moves on. Deploy: the policy runs onboard each robot behind a safety layer. Field data then flows back into simulation and the loop repeats, improving the team policy each time. experience candidate policy if it passes field data Team policy improves each loop Simulate parallel worlds Train as one team Evaluate vs. baselines Deploy on each robot
The loop. Nothing reaches a robot without passing evaluation first. What the robots meet in the field then makes the next round of simulation more realistic, so each pass starts from a better model.

01 Simulate

Swarm-scale practice before any robot moves.

In plain words

Robots learn by trial and error, and trial and error is slow, costly and sometimes dangerous on real hardware. So training happens in simulation: virtual robots with modeled bodies, sensors and radios, in virtual environments, with many copies of the world running at once.

Why it matters for swarms

Coordination only appears when the whole team is in the scene. A swarm simulator has to model every robot and the links between them, and run fast enough to give the team a vast amount of shared experience. Changing the conditions on every run keeps the team from memorizing one world.

  • Calm, open field
  • Gusty wind
  • Dense obstacles
  • Noisy sensors
  • Patchy radio
  • A robot drops out
Many copies of a scenario run side by side. Each one gets different conditions, so the team cannot memorize a single world.

How we do it

  • Parallel environments

    Many independent copies of a scenario run side by side, so the team gathers experience far faster than real time and every update draws on a wide spread of situations.

  • Physics and sensor models

    Motion, contact and payload are modeled, along with what each sensor actually reports: limited range and field of view, noise, dropouts and delay.

  • Communication models

    Radio range, bandwidth, latency and packet loss are part of the simulation, so robots learn with the links they will really have, not perfect ones.

  • Domain randomization

    Each run varies conditions such as friction, mass, wind, sensor noise, starting positions and team size, which pushes policies toward behavior that holds across all of them.

  • Procedural scenarios

    Maps, obstacles and mission goals are generated rather than hand-built, which gives wide variety and lets difficulty be dialed up step by step for curricula.

02 Train

Trained together, so each robot can act on its own.

In plain words

Reinforcement learning is learning from reward. A robot tries an action, the simulator scores the result, and over many attempts the robot’s policy (the network that turns what it senses into what it does) shifts toward actions that earn more reward. In cooperative multi-agent learning the reward is tied to what the team achieves, so robots learn to help each other instead of competing for credit.

Why it matters for swarms

The hard part in a team is credit assignment: when the area gets covered in time, which robot’s choices made the difference? Centralized training with decentralized execution helps with it. During training a shared critic sees the whole team and judges how good the situation is. At run time the critic is gone and each robot’s policy uses only what that robot can sense and hear.

The key idea

Learn with the full picture. Act with a local one.

Simulation can show the critic everything. A robot in the field only has its own sensors and what its neighbors tell it, so that is what its policy is built to use.

How we do it

  • Centralized training, decentralized execution (CTDE)

    Training may draw on the full simulator state, but the policy’s inputs are limited to one robot’s own observations and the messages it receives. Anything that needs the global view lives in the critic, which is left behind when training ends.

  • MAPPO-style actor-critic

    Each robot’s actor (its policy) is trained with proximal policy optimization, a stable and well-studied method that keeps each update small. A shared critic that sees the whole team estimates value, which makes each robot’s learning signal far less noisy.

  • Parameter sharing

    Robots of the same type share one set of policy weights, so every robot’s experience improves all of them. Each still acts on its own inputs, and an optional role input lets shared policies specialize.

  • Curricula

    Training starts small and calm, with fewer robots, simpler maps and gentler conditions, and grows harder as the team improves. Hard coordination problems become learnable one step at a time.

  • Population and self-play training

    Where a task has opponents or many possible partners, policies train against past versions of themselves and a varied population of teammates, so they do not overfit to one partner’s habits.

  • Rewards you can inspect

    Rewards are written as explicit, documented terms and checked for loopholes before long runs, because agents learn to exploit whatever a reward accidentally pays for.

During training in simulation

A shared critic sees the whole team

The critic gets every robot’s observations and actions, judges how well the team is doing, and feeds that back to each robot’s policy.

Training: a shared critic sees every robot Four robots, A to D. Arrows run from every robot up to one shared critic, carrying all observations and actions. A feedback line runs from the critic down the side and along under the robots, with an arrow up into each one: the feedback trains one shared policy that every robot runs a copy of. Shared critic sees every robot all observations + actions feedback A B C D feedback trains one shared policy, copied to every robot

At run time on each robot

Each robot acts on what it senses and hears

The critic is set aside. Each robot’s policy uses only its own sensors and short messages from nearby teammates, so there is no central link to lose.

Execution: each robot uses only local information The same four robots with no critic present. Each robot has its own sensor cone pointing in a different direction, and dashed message links connect neighboring robots. Every robot decides on its own from its sensors and local messages. No critic needed at run time each robot’s own sensors A B C D local messages between neighbors
Centralized training, decentralized execution (CTDE). The same policy runs in both panels. Only the critic, and the whole-team view it depends on, stays behind in simulation.

03 Coordinate

Teamwork that holds up under real-world limits.

In plain words

Coordination is what training produces: robots that split up the work, cover for each other and share what matters, without waiting for orders. The catch is that each robot sees only part of the world, and the radio links between them are limited and unreliable.

Why it matters for swarms

A policy that only works with perfect information, a fixed team size and every robot online will fail on its first day outside. So the policy architecture and the training conditions are built around the limits a real swarm lives with.

One robot’s local neighborhood A robot in the center with a dashed circle showing its radio range. Four teammates inside the circle are linked to it by lines of different thickness, showing how much attention it pays to each. One teammate inside the range has dropped out and has no link. Four robots outside the circle are out of range and ignored. radio range
  • This robot
  • Teammates it can hear. Thicker line, more attention
  • Dropped out: the rest adapt
  • Out of range: not part of its decision
The policy reads a variable set of neighbors rather than a fixed list, so the same network can be trained and tested as the neighborhood grows, shrinks or loses members.

How we handle real limits

  • The limit Each robot sees only part of the scene

    Acting under partial observability

    Policies carry a short memory, recurrent or attention-based, of what the robot has seen and heard, so it can act sensibly on an incomplete picture.

  • The limit Radio links are slow, short-range and lossy

    Learned communication under bandwidth limits

    Robots learn what to say as well as what to do: compact messages to nearby teammates, trained under the bandwidth, range and dropout rates of the real radio.

  • The limit Team size changes from mission to mission

    Policies that work across team sizes

    Attention and permutation-invariant layers let a robot read however many neighbors it has, in any order, so one policy can be trained and tested across a range of team sizes.

  • The limit Robots fail, run low or lose contact

    Robust to dropouts

    Robots are removed, delayed or silenced during training, so the rest of the team learns to re-divide the work instead of stalling.

  • The limit Real fleets mix different machines

    Roles and mixed teams

    Aerial and ground robots, or scouts and workers, get role-aware or per-type policies that are trained together and share one message format.

Evaluate

Measured against baselines, not demos.

In plain words

Evaluation means running a trained policy side by side with simpler options on the same scenarios: a scripted rule, random actions and robots that do nothing. If the policy does not clearly beat them, learning has not added anything yet. We do this at every stage, on scenarios the policy has never seen, with controls.

Why it matters for swarms

Swarm behavior is noisy: the same policy can look brilliant on one random seed and poor on the next. A single good run means little. Careful comparisons across many fresh seeds are the only way to know whether learning added anything, and the only kind of result a partner should accept.

Same scenarios, same seeds, same metrics

  • Learned policy
  • Scripted rules
  • Random actions
  • Idle robots

Before a policy moves on, it has to

  1. Beat the scripted rules, not just random and idle behavior
  2. Hold up on fresh random seeds it never trained on
  3. Survive stress tests: sensor noise, weak radio, lost robots
  4. Show through ablations which of its parts actually matter
  5. Pass a review of replays for odd or unsafe strategies

Passes: hardware tests and field trials

Falls short: back to simulation and training

The evaluation gate every candidate policy goes through.

How we do it

  • Baselines, every time

    Scripted rules (what an engineer would write by hand), random actions and idle robots run the same scenarios, so the learned policy has something real to beat.

  • Fresh random seeds

    Evaluation runs on seeds and scenario variations held out from training, which exposes policies that memorized their practice worlds.

  • Controls and ablations

    We remove one ingredient at a time, such as communication, the shared critic or a reward term, to check which parts actually earn their place.

  • Stress tests

    Teams face more noise, weaker radios, lost robots and team sizes outside the training range, to find where behavior bends and where it breaks.

  • Spread, not best runs

    Results are reported across seeds with their variation. Failures are kept and studied, never filtered out.

  • Watching the behavior

    Rollouts are replayed and reviewed, because a good score can hide a strange or unsafe strategy that games the reward.

04 Deploy

From simulation to real robots, with guardrails.

In plain words

A policy that works in simulation still has to survive real motors, real sensors and real weather. Deployment is the work of closing that gap, shrinking the policy to run on each robot’s own computer, and wrapping it in rules it cannot break.

Why it matters for swarms

Swarms work where networks are weak and no one can watch every robot. Running the policy onboard means no single server or link can stop the team. A safety layer means learned behavior always stays inside limits an engineer set and can check.

  1. Learned team policy

    A small distilled network on the robot’s own computer

  2. Safety layer

    Can override the policy

    Constraints, geofences, speed and separation limits, fallback behaviors

  3. Existing autonomy stack

    For example ROS 2: localization, perception, motion control

  4. Robot hardware

    Sensors, actuators and the radio for local messages

On each robot. Sensor data and teammates’ messages flow up to the policy. Its commands flow down through the safety layer before they reach the controllers.

How we do it

  • Domain randomization

    Policies trained across randomized physics and sensors treat the real world as one more variation rather than a surprise.

  • System identification

    We measure the real robot, its mass, motor response, sensor noise and latency, and tune the simulator to match, so the gap is small to begin with.

  • Distillation into small onboard networks

    A large training-time policy is distilled into a compact, fast network that fits the robot’s embedded computer and power budget.

  • A safety layer around learned behavior

    A layer of hard rules sits between the policy and the motors and can override it: geofences, speed and separation limits, and fallback behaviors such as hold, return or land.

  • Onboard, alongside your autonomy stack

    The policy runs on each robot next to its existing software, such as ROS 2, taking inputs from localization and perception and handing commands to the controllers you already trust.

  • Staged rollout and field feedback

    Hardware tests come first, then small supervised field trials with the partner. Logs from those trials flow back into simulation for the next round of training.

Glossary

The terms, in plain words

The vocabulary of multi-agent learning, without the math. These are the words you will hear when we talk about a project.

MARL Multi-agent reinforcement learning
Reinforcement learning with many learners in the same world at once. Each robot’s behavior is learned while its teammates are learning too, so the team discovers how to work together.
Policy
The learned decision-maker on a robot: a neural network that takes what the robot senses, plus what it hears from teammates, and outputs what it should do next.
Reward
The score the simulator gives for what happened, such as area searched or fire line held, minus penalties for collisions or wasted energy. Training shifts the policy toward earning more of it.
CTDE Centralized training, decentralized execution
Learn with the full picture, act with a local one. During training a critic sees the whole team; in the field each robot uses only its own sensors and messages.
Critic
A helper network used only during training. It estimates how good the team’s situation is, which tells each policy whether its recent actions helped.
Partial observability
No robot sees everything. Range, obstacles and field of view limit what each one knows, so it has to act on an incomplete picture.
Sim-to-real
Moving a policy from simulation onto physical robots without losing the behavior, despite the differences between any model and reality.
Domain randomization
Varying the simulator’s physics, sensors and scenery on every run, so that to the policy the real world looks like one more variation.
Baseline
A simple point of comparison, such as a scripted rule, random actions or doing nothing. A learned policy has to beat these before its results mean anything.
Seed
The number that sets up a simulation’s random choices, such as where a fire starts or where robots begin. A fresh seed means a scenario the policy has never practiced on.
Ablation
A test that removes one ingredient, such as communication or a reward term, to check whether that ingredient actually helps.
Curriculum
A training plan that starts with easy versions of a task, such as fewer robots or calmer conditions, and makes it harder as the team improves.

FAQ

Common questions

Straight answers about how multi-agent learning behaves on real robots.

Why not run the whole fleet from one central controller?

A central controller needs a reliable link to every robot and has to reason about the whole team at once. As the fleet grows it becomes a bottleneck, and if the controller or its network fails, every robot stops.

We use central information only during training. In the field each robot decides for itself from its own sensors and messages from nearby teammates, so the team keeps working when links are weak or a robot drops out. A planner or operator can still set the mission and high-level goals; the robots handle the moment-to-moment coordination.

How many robots can one policy handle?

Policies are designed to generalize across team sizes. Each robot reads a variable set of neighbors rather than a fixed list, and training mixes different team sizes, so one policy is not tied to one head count.

The practical limit depends on the task, the sensors and the radio, so we measure it for each mission with stress tests instead of quoting a single number.

How do you keep learned behavior safe?

In layers. A safety layer outside the learned policy enforces constraints, geofences, speed and separation limits, and switches to simple, predictable fallback behaviors when something is wrong. The policy cannot override it.

Before any field trial, policies are stress-tested on unseen and degraded scenarios and their replays are reviewed. Rollout is staged: simulation first, then hardware tests, then supervised trials.

Do we need our own simulator?

No. If you already have a simulator or digital twin, we start by checking whether it can run many robots, in many parallel copies, fast enough for multi-agent training, and build on it where we can.

If you do not have one, building the simulation is the first technical step of the work, scoped to what matters for your mission: the robots, their sensors and radios, and the environment.

What hardware do the robots need?

Less than you might expect. Policies are distilled into small networks meant to run on the robot’s own embedded computer, next to its existing autonomy stack.

Each robot needs the sensors the mission calls for, a way to estimate where it is, and a radio for local messages. Exact requirements depend on the task, and we work them out at the start.

How do you know a policy actually works?

We do not treat a rising training curve as proof. Every policy is compared against scripted rules, random behavior and idle robots on fresh random seeds it never trained on, with controls and ablations, and it is stress-tested before any field trial.

If it does not clearly beat the simple baselines, it does not move forward.

Can one system coordinate mixed teams, such as drones and ground robots?

That is a design goal. Different robot types can get their own policies, or one policy with a role input, trained together in the same simulation so they learn to complement each other and share a common message format.

Have a question that is not here?

Send us a note about your robots and your mission, and we will answer it directly.

Ask us directly ahsan@iresine.ai

Building a robot fleet that needs to think as one?

Tell us about your robots, your environment and the behavior you need. We will tell you honestly whether multi-agent learning can help, and how.

Start a conversation ahsan@iresine.ai