Sites

Menu
These are the docs for 13.8, an old version of SpatialOS. The docs for this version are frozen: we do not correct, update or republish them. 14.0 is the newest →

Commands

Commands are similar to remote procedure calls (RPCs). There are two main types of commands: component commands and world commands.

Component commands

Component commands are commands that are invoked on a component inside an entity by a sending worker, and are responded to by whichever target authoritative worker has write access authority over the component. These commands can perform any action, since you define their behaviour in your worker code. By default, commands are routed through SpatialOS, which ensures that the target authoritative worker actually had authority over the component at the time of the command execution. However, you can specify that the command should be handled locally when the worker sending the command currently believes to have write access authority over the target component - we refer to locally-handled commands as “short circuiting”. This avoids the round trip to SpatialOS in some cases, but can lead to situations where a worker executes a command despite just having lost authority over its component.

You define commands in your schema, as part of a component. Every command defines a request and response message that are sent back and forth as payload between the sending and the target authoritative worker. To process those requests and responses, your worker code needs to implement two handler functions for each command:

  • The target authoritative worker needs to implement a command request handler that will process incoming requests and return responses.
  • The sending worker needs to implement a command response handler that will process returning responses for command requests previously sent from this worker.

For details on how to implement commands, see the SDK-specific docs:

Setting deadlines

Workers sending component command requests can set an optional deadline after which the command will time out if it wasn’t fully processed by that point. There is a maximum deadline of five seconds, which will also be used if you don’t specify a deadline yourself. Be careful about setting deadlines that are too short: the target authoritative worker might have time to process the command in a simple test scenario, but when it is under more load from simulating many entities it could take longer to respond.

While SpatialOS tries hard to return command responses to the sending worker as fast as possible, it cannot guarantee that this response arrives on that worker before the deadline expires on its side. This means that you cannot use deadlines to decide on the sending worker whether a component command was or wasn’t processed on the target authoritative worker.

Handling failures and retries

When a worker sends a component command, its bridge needs to determine the target authoritative worker and then forward the request to that worker’s bridge. When the target authoritative worker receives the command request, the command request handler function in the worker code will be invoked and return a response back to its bridge. The target authoritative worker’s bridge will then send that response back to the sending worker’s bridge, which will notify the sending worker. The sending worker’s command response handler in the worker code will receive a command response op that contains either the payload of the command response as defined in schema, or a status code and error message explaining the failure.

Because component command execution requires these multiple hops through the deployment, it can fail for a variety of reasons. While a failing response op always contains a human-readable error message that can be manually inspected when debugging, robust error handling requires worker code to implement automated retries based on the status code.

The following subsections explain the currently implemented failure status codes, their possible failure scenarios, and the recommended retry behavior:

Timeout

Meaning: The command request wasn’t processed within the specified deadline.

Failure scenarios:

  • The command timed out locally because the sending worker’s bridge didn’t respond in time. This can happen when this worker’s bridge took to long to process the request or is overloaded.
  • The sending worker’s bridge timed out because the target authoritative worker’s bridge didn’t respond in time. This can happen when that worker’s bridge took to long to process the request or is overloaded.
  • The target authoritative worker didn’t return a command response in time because:
    • It took to long to process the request or is overloaded.
    • It worker doesn’t implement a command handler in its worker code for this component command despite having authority over this component.
    • It processed the command, but the response didn’t make it back to its bridge before the deadline expired.

Recommended retry behavior: Automatically retry with a backoff. Don’t keep retrying immediately because in the case of overload, this will make the problem even worse. If commands keep timing out without any sign of overload, make sure that all your workers register command handlers for all your component commands.

Not found

Meaning: The command’s component or its entity couldn’t be found by some Runtime component.

Failure scenarios:

  • The command request’s specified component or entity wasn’t known to the sending worker’s bridge because:
    • The component or entity didn’t exist.
    • The component or entity got deleted in the meanwhile.
  • The command request’s specified component or entity wasn’t known to the target authoritative worker’s bridge because:
    • The component or entity got deleted in the meanwhile.
    • The target authoritative worker was just about to gain interest over it.
  • The command request’s specified component or entity are no longer known to the target authoritative worker’s bridge after the worker processed the command because:
    • The component or entity got deleted in the meanwhile.
    • The target authoritative worker lost both authority and interest over it in the meanwhile.

Recommended retry behavior: Check if your worker still knows about the component and the entity. If it believes that it still exists, this likely was a transient failure and it should retry the command immediately.

Authority lost

Meaning: The authority of the target authoritative worker over the command’s component couldn’t be confirmed.

Failure scenarios:

  • The sending worker’s bridge couldn’t find the target authoritative worker for this component command because:
    • The target authoritative worker just connected or disconnected.
    • Authority over the command’s component wasn’t delegated to any worker yet.
  • The supposed target authoritative worker’s bridge determined that its worker wasn’t authoritative over the command’s component because:
    • It wasn’t notified about its worker gaining authority over the component yet.
    • Its worker just lost authority over the command’s component.
  • The target authoritative worker’s bridge determined that its worker was no longer authoritative after the worker executed the command because authority was delegated to a different worker in the meanwhile.

Recommended retry behavior: Check if your worker still knows about the component and the entity. If it believes that it still exists, this likely was a transient failure and it should retry the command immediately.

Application error

Meaning: The command failed because of a worker-side error.

Failure scenarios:

  • The worker sent an invalid command request to its bridge. This can happen when using the C SDK.
  • The target authoritative worker’s command handler in the worker code returned a failure.

Recommended retry behavior: This depends on your command and on the exact reason for the failure. To better distinguish your own failure cases, it might be useful to define your own failure status codes in schema and send successful command responses that include them to indicate failures.

Internal error

Meaning: The command failed because of a SpatialOS-internal error.

Failure scenarios:

  • A bug in SpatialOS.

Recommended retry behavior: Don’t retry, since the command is likely to fail again for the same reason. A bug might exist in SpatialOS. Raise a support request or ask on our forums.

World commands

World commands are built-in SpatialOS commands, invoked on the SpatialOS world. They allow you to directly manipulate your simulated world by creating entities, deleting entities and querying the world for entities.

In order to send a world command, a worker must have permission to do so. For more information, see the Worker permissions page.

Creating and deleting entities

To create an entity, you set up a template specifying the components contained within that entity, and the initial values of their properties, then send an entity creation command.

You can optionally specify an entity ID when creating an entity, which must have been previously reserved through an entity ID reservation command call. Entity ID reservations are only stored for a limited team for each worker and will expire if unused or if too many further entity IDs are requested from the same worker. Your workers can reserve multiple entity IDs at once by specifying the number of entity IDs they would like to reserve.

To delete an entity, you need to know its EntityId. One way of finding this out is using an entity query (see below).

For details on how to create and delete entities, see the SDK-specific docs:

Querying the world

You can run entity queries in order to get information about entities in the world. To do this, you build a query, and then use a world command to execute it.

Entity queries are useful if you need to get information about an entity at a particular time.

If you need regular updates about an entity, it’s better to use query-based interest. This causes your worker to get incremental updates about the matching components, rather than all of the information about it.

Entity queries can search for the entities with the following attributes:

  • a specific EntityId
  • a specific component
  • within a specific sphere in the world

and any combination of the above, using AND/OR.

Based on the set of entities that match it, a query can return:

  • snapshots of those entities, including all components
  • snapshots of those entities, including only the components you specify
  • the number of entities

You should keep entity queries as limited as possible. All queries hit the network and cause a Runtime lookup, which is expensive even in the best cases. This means you should:

  • always limit queries to a specific part of the world, such as a sphere or a specific component that isn’t present on every entity
  • only return the information you need from queries: specify which components you need to know about
  • if you’re looking for entities that are within your worker’s checkout radius, search internally on the worker instead of using a query

For details on implementing entity queries, see the SDK-specific docs:

Handling failures and retries

Similarly to component commands, world commands can also fail and your workers might have to retry them. World commands cannot fail with the “authority lost” status code. The following subsections explain the currently implemented failure status codes and the recommended retry behavior:

Timeout

Meaning: The command request wasn’t processed within the specified deadline.

Failure scenarios:

  • The command timed out locally because the sending worker’s bridge didn’t respond in time. This can happen when this worker’s bridge took to long to process the request or is overloaded.
  • The sending worker’s bridge timed out because the Runtime component responsible for processing the command took to long to process the request or is overloaded.

Recommended retry behavior: Automatically retry with a backoff. Don’t keep retrying immediately because in the case of overload, this will make the problem even worse. If commands keep timing out without any sign of overload, make sure that all your workers register command handlers for all your component commands.

Not found

Meaning: The entity your worker tried to delete could not be found.

Failure scenarios:

  • This status code can only be returned by the entity deletion world command, in which case it indicates that the entity your worker tried to delete no longer exists.

Recommended retry behavior: Since the entity you tried to delete no longer exists, it doesn’t make sense to retry this command.

Permission denied

Meaning: Your worker didn’t have permission to issue this world command.

Failure scenarios:

  • The worker permissions for the sending worker’s worker type don’t include the necessary permission to execute the specified world command.
  • The worker tried to create an entity with an invalid entity template that specified components that are reserved for system-managed entities. Such entities can only be created and deleted by the Runtime.
  • The entity your worker tried to delete is system-managed. Such entities can only be created and deleted by the Runtime.

Recommended retry behavior: Don’t retry. Since worker permissions cannot change at runtime, retried command requests of the same type will fail as well. Instead, check your worker permissions to make sure your worker has the appropriate permissions to execute this kind of command.

Application error

Meaning: The command failed because of a worker-side error.

Failure scenarios:

  • The worker tried to create an entity with an invalid entity template:
    • The entity template was invalid according to the specified schema. This can happen when using the C SDK.
    • The entity template didn’t include an EntityAcl component.
    • The entity template’s EntityAcl specified an invalid component ID or an invalid worker attribute.
    • The entity template didn’t include a Position component.
    • The entity template’s Position component specified a position with non-finite coordinates or one that is outside of the specified world dimensions.
  • The worker tried to create an entity with invalid entity ID because:
    • An entity with that ID already exists.
    • The specified entity ID wasn’t previously reserved.
    • The specified entity ID’s reservation expired.
  • The worker tried to reserve an invalid number of entity IDs.

Recommended retry behavior: Don’t retry. Since the world command request was malformed, it will fail the same way again if you retry. Make sure to check the Runtime logs, which will in some cases contain detailed information on the world command failure.

Internal error

Meaning: The command failed because of a SpatialOS-internal error.

Failure scenarios:

  • A bug in SpatialOS.

Recommended retry behavior: Don’t retry, since the command is likely to fail again for the same reason. A bug might exist in SpatialOS. Raise a support request or ask on our forums.



————
2019-07-24 Page updated with limited editorial review: extended component commands description, added section on setting deadlines for component commands, added sections for handling failures and retries for component and world commands

Search results

Was this page helpful?

Thanks for letting us know!

Thanks for your feedback

Need more help? Ask on the forums