Sites

Menu
These are the docs for 13.5, an old version of SpatialOS. The docs for this version are frozen: we do not correct, update or republish them. 14.0 is the newest →

Load balancing

If you’re using a legacy version of the Runtime, you’ll need to upgrade in order to use the load balancing strategies on this page.

This page outlines the load balancing strategies available in SpatialOS, and explains how to configure them. Using these strategies you can adjust how many components on entities in the world each server-worker has write access to, so that your server-workers don’t get overloaded. You can also scale the number of server-workers in the world, adding or removing server-workers as needed.

You can set up load balancing to occur dynamically throughout the lifecycle of your deployment or to conform to a static model of allocation.

You set your load balancing configuration for each server-worker type in the launch configuration file, in the load_balancing field:

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "physics",
            // strategy goes here
        },
        {
            "layer": "chat",
            // strategy goes here
        }
    ]
},

How to approach load balancing

You should regularly test your game with more than one server-worker instance of a type.

This way, your code will be able to handle multiple server-worker instances from the start, making it much easier to scale.

Strategies

Rectangular grid (rectangle_grid)

This strategy divides the world up into a rectangular grid of the specified number of rows and columns. There are no overlap regions between the server-worker instance authority regions.

When you’d want to use this strategy:

  • For small deployments with just a few server-worker instances
  • When you want to know where the boundaries are for testing
  • When you don’t want overlap regions

Example configuration

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "physics",
            "rectangle_grid": {
                "cols": 2,
                "rows": 2
            }
        }
    ]
},

Entity ID sharding (entity_id_sharding)

This is a non-spatial load balancing strategy: it does not take into account the positions of the entities it is balancing. Instead, it distributes work evenly across the configured number of server-worker instances using a hash of the entity ID.

You’d want to use this strategy for server-worker types that have no logic that depends on where entities are in the world. For example:

  • An auction house or chat system
  • A player account management system

Example configuration

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "chat",
            "entity_id_sharding": {
                "numWorkers": 5
            }
        }
    ]
},

Hexagonal grid (hex_grid)

This strategy tiles the entirety of the game world as a hexagonal grid, and then places a server-worker instance with a circular authority region at the centre of each hexagon. This minimizes the surface area between server-worker instances.

You specify the number of server-worker instances you want to use, and a grid that covers the world will be generated automatically.

When you’d want to use this strategy:

  • Experimenting with worker authority overlap
  • If you want it to be efficient for minimizing migrations between server-workers

Overlap regions

In this strategy, server-worker instance authority regions can overlap with each other.

In an overlap region, it’s valid for either server-worker instance to have authority over the appropriate components (the components are not guaranteed to all come over together, though in practice they probably will).

Entity components will generally stay on the server-worker instance they’re already on instead of swapping over in the overlap region. This means that authority doesn’t thrash for entities between the two server-worker instances.

In the diagram below, you can see that the overlap region can form quite a lot of the world when you use a hexagonal grid configuration with few server-worker instances. The black line represents the world boundaries; a hexagonal grid with four server-worker instances has been generated to cover that world.

Hexagonal load-balancing showing overlap regions

Example configuration

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "physics",
            "hex_grid": {
                "num_workers": 4
            }
        }
    ]
},

Points of interest (points_of_interest)

In this strategy, you specify key locations in the world that should not be simulated by more than one server-worker instance.

You provide a list of “seeds” or points of interest. From those seeds, the world is divided into regions. Each region consists of the area of the world that is closest to the point of interest (similar to a Voronoi diagram using the Manhattan distance).

Once the world has been split into regions, the strategy distributes the regions amongst a specified number of server-worker instances.

Example configuration

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "physics",
            "points_of_interest": {
                "num_workers": 2,
                "points": [
                    {
                        "x": 100,
                        "z": 100
                    },
                    {
                        "x": 100,
                        "z": -100
                    },
                    {
                        "x": -100,
                        "z": 100
                    },
                    {
                        "x": -100,
                        "z": -100
                    }
                ]
            }
        }
    ]
}

This configuration specifies four points of interest and two server-worker instances. Since there are four points of interest, there are four regions: top left, top right, bottom left and bottom right.

The load balancer distributes the four regions amongst the two server-worker instances that are available. It will try to make sure that it picks regions which are adjacent to each other. This minimizes the total area that each server-worker instance will have to simulate. In this example, the server-worker instances will either split the regions such that both the regions on the left are on one server-worker instance and both regions on the right are on the other server-worker instance; or split the regions such that both regions on the top are on one server-worker instance and both regions on the bottom are on the other server-worker instance. The load balancer will not split the regions such that diagonal points are on the same server-worker instance.

Don’t start workers

If you don’t want SpatialOS to start any worker instances of a type, don’t specify a layer for it: leave it out of the configuration file.

When you’d want to use this:

  • Usually, just for client-worker types

A mixture of strategies

You can also use different strategies for different layers.

Example configuration

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "physics",
            "rectangle_grid": {
                "cols": 2,
                "rows": 2
            }
        },
        {
            "layer": "chat",
            "hex_grid": {
                "num_workers": 4
            }
        },
    ]
},

Manual server-worker connection

Sometimes you might want to connect a server-worker manually (an unmanaged worker), for testing or debugging purposes.

There are two ways of doing this, depending on whether:

  • you want an unmanaged worker to be solely responsible for simulating a layer, or
  • you want an unmanaged worker to participate in simulating a layer, alongside managed (server-side) workers

Unmanaged worker solely responsible for simulating a layer

Tell the Runtime not to start any workers for a given layer. You can still specify a load-balancing strategy, but you’ll start and connect any managed workers yourself. To do this, set the option manual_worker_connection_only for that layer:

"load_balancing": {
    "layer_configurations": [
        {
            "layer": "my_worker_attribute",
            "hex_grid": {
                "num_workers": 4
            },
            "options": {
                "manual_worker_connection_only": true
            }
        }
    ]
},

Then you can start workers:

If you don’t set this option and connect an unmanaged worker, the worker you connect will not be given authority over anything (unless you’ve explicitly specified that worker instance in the ACL). Essentially, the Runtime will treat it like a client-worker.

Unmanaged worker participating in simulating a layer, alongside managed workers

You might want an unmanaged worker to participate in load balancing alongside managed workers (for example, if you’re running a worker from your local machine with a debugger or profiler attached).

In this case, use the new worker replacement command in the spatial CLI:

Local:

spatial local worker replace --existing_worker_id <UnityWorker1> --replacing_worker_id <MyDebugWorker>

Cloud:

spatial project deployment worker replace --existing_worker_id <UnityWorker1> --replacing_worker_id <MyDebugWorker> --project_name <MyProject>

Known issues

  • In the Inspector, the coordinates for a worker that isn’t authoritative over anything are 0,0,0 (the center of the world), until the worker becomes authoritative over something.

Search results

Was this page helpful?

Thanks for letting us know!

Thanks for your feedback

Need more help? Ask on the forums