Sites

Menu

Commands

Commands are similar to remote procedure calls (RPCs).

There are two main types of command:

  • component commands
  • world commands

Component commands

A sending worker instance invokes component commands on a component inside an entity, and whichever target worker instance has write access authority over the component responds to these commands.

Component commands can perform any action, because you define their behavior in your worker code.

By default, commands are routed through SpatialOS, which ensures that the target worker instance has write access authority over the component when it executes the command.

However, you can specify that the command should be handled locally when the worker instance sending the command believes that the it itself currently has write access authority over the target component. We refer to locally-handled commands as “short circuiting”. This avoids the round trip via SpatialOS in some cases, but it can lead to situations where a worker instance executes a command despite just having lost write access authority over its component.

Implementing component commands

You define component commands in your schema, as part of a component. Every command defines a request and response message that are sent back and forth as payload between the sending and the target worker instance.

To process those requests and responses, your worker code needs to implement two handler functions for each command:

  • The target worker type needs to implement a command request handler to process incoming requests and return responses.
  • The sending worker instance needs to implement a command response handler to process returning responses for command requests previously sent from this worker instance.

For details on how to implement commands, see the docs in C++, C#, and Java.

Setting deadlines

Worker instances that send component command requests can set an optional deadline after which the command times out if it hasn’t been fully processed. The maximum (and default) deadline is five seconds.

Try to avoid setting deadlines that are too short: the target worker instance might have time to process the command in a simple test scenario, but when it is under more load from computing many entities it could take longer to respond.

While SpatialOS tries hard to return command responses to the sending worker instance as fast as possible, it cannot guarantee that the response arrives on that worker instance before the deadline expires. This means that the sending worker instance can’t use deadlines to decide whether a component command was processed on the target worker instance.

Handling failures and retries

When a worker instance sends a component command, its bridge needs to determine the target worker instance and then forward the request to that worker instance’s bridge. When the target worker instance receives the command request, this invokes the command request handler function in the worker code and returns a response back to its bridge.

The target worker instance’s bridge then sends that response back to the sending worker instance’s bridge, which notifies the sending worker instance.

The sending worker instance’s command response handler in the worker code receives a command response op that contains either the payload of the command response as defined in schema, or a status code and error message explaining the failure.

Because component command execution requires these multiple hops through the deployment, it can fail for a variety of reasons. While a failing response op always contains a human-readable error message that you can manually inspect when debugging, robust error handling requires worker code to implement automated retries based on the status code.

The following subsections explain the currently implemented failure status codes, their possible failure scenarios, and the recommended retry behavior.

Timeout

Meaning: The command request wasn’t processed within the specified deadline.

Failure scenarios:

  • The command timed out locally because the sending worker instance’s bridge didn’t respond in time. This can happen when the worker instance’s bridge takes too long to process the request, or is overloaded.
  • The sending worker instance’s bridge timed out because the target worker instance’s bridge didn’t respond in time. This can happen when the worker instance’s bridge takes too long to process the request, or is overloaded.
  • The target worker instance didn’t return a command response in time because:
    • it took too long to process the request or was overloaded.
    • it doesn’t implement a command handler in its worker code for this component command, despite having write access authority over this component.
    • it processed the command, but the response didn’t reach its bridge before the deadline expired.

Recommended retry behavior: Automatically retry with a backoff. Don’t keep retrying immediately because in the case of overload, this makes the problem worse.

If commands keep timing out without any sign of overload, make sure that all your worker types register command handlers for all your component commands.

Not found

Meaning: Some part of the Runtime couldn’t find the command’s component or its entity.

Failure scenarios:

  • the component or entity doesn’t exist.
  • the component or entity was deleted after the command was sent but before it was received.

Recommended retry behavior: Don’t automatically retry. Since the specified component or entity no longer existed when the command request reached the sending worker instance’s bridge, retrying would cause it to fail again.

Authority lost

Meaning: There was a failure relating to finding the responsible target worker instance’s bridge for this component command.

Failure scenarios:

  • The sending worker instance’s bridge couldn’t find the target worker instance for this component command because:
    • the target worker instance recently connected or disconnected.
    • write access authority over the command’s component wasn’t yet delegated to a worker instance.
  • The command request’s specified component or entity wasn’t known to the target worker instance’s bridge because:
    • the component or entity was deleted after the command was sent but before it was received.
    • the target worker instance does not yet have active read access to the component.
  • The supposed target worker instance’s bridge determined that its worker instance did not have write access authority over the command’s component because:
    • it hadn’t yet been notified about its worker instance gaining write access authority over the component.
    • its worker instance recently lost write access authority over the command’s component.
  • The command request’s specified component or entity are no longer known to the target worker instance’s bridge after the worker instance processed the command, because:
    • the component or entity was deleted after the command was sent but before it was received.
    • the target worker instance lost both write access authority and active read access over the component or entity, after the command was sent but before it was received.
  • The target worker instance’s bridge determined that its worker instance no longer had write access authority after the worker instance executed the command, because write access authority was delegated to a different worker instance between the target worker instance receiving the command request and its bridge receiving the command response.

Recommended retry behavior: Check if your worker instance is still receiving updates about the component. If it believes that they still exist, it’s likely this was a temporary failure and you need to implement automated retries (based on status codes) in your worker code.

Application error

Meaning: The command failed because of a worker-side error.

Failure scenarios:

  • The worker instance sent an invalid command request to its bridge. This can happen when using the C SDK.
  • The target worker instance’s command handler in the worker code returned a failure.

Recommended retry behavior: This depends on your command and on the exact reason for the failure. We recommend you define your own failure status codes in schema. Then, indicate failures by returning successful command response messages that include these codes, so that you can set up behavior based on them.

Internal error

Meaning: The command failed because of a SpatialOS error.

Failure scenarios:

  • A bug in SpatialOS.

Recommended retry behavior: Don’t retry; the command is likely to fail again for the same reason. Raise a support request (for customers with a service agreement) or ask on our forums.

World commands

World commands are built-in SpatialOS commands, invoked on the SpatialOS world. You can use them to directly manipulate your SpatialOS world by creating entities, deleting entities and querying the world for entities.

In order to send a world command, a worker instance must have permission to do so. For more information, see the Worker permissions page.

Creating and deleting entities

To create an entity, you set up a template specifying the components contained within that entity and the initial values of their properties, then send an entity creation command.

You can optionally specify an entity ID when creating an entity, which you must have previously reserved through an entity ID reservation command call.

Entity ID reservations are only stored for a limited team for each worker instance, and they expire if unused or if the same worker instance requests too many further entity IDs.

Your worker instances can reserve multiple entity IDs at once by specifying the number of entity IDs to reserve.

To delete an entity, you need to know its EntityId. One way of finding this out is using an entity query (see below).

For details on how to create and delete entities, see the docs in C++, C#, and Java.

Querying the world

You can run entity queries so that worker instances receive information about entities in the world. To do this, you build a query, and then use a world command to execute it.

Entity queries are useful if you need to get information about an entity at a particular time.

If you need regular updates about an entity, it’s better to use query-based interest. This causes your worker instance to receive incremental updates about the components that match queries, rather than all of the information about an entity.

Entity queries can search for entities with the following attributes:

  • a specific EntityId
  • a specific component
  • within a specific sphere in the world

and any combination of the above, using AND/OR.

Based on the set of entities that match it, a query can return:

  • snapshots of those entities, including all components
  • snapshots of those entities, including only the components you specify
  • the number of entities

You should keep entity queries as limited as possible. All queries hit the network and cause a Runtime lookup, which is expensive even in the best cases. This means you should:

  • always limit queries to a specific part of the world, such as a sphere or a specific component that isn’t present on every entity
  • only return the information you need from queries: specify which components you need to know about
  • if you’re looking for entities that are within your worker instance’s interest radius, search internally on the worker instance instead of using a query

For details on implementing entity queries, see the docs in C++, C#, and Java.

Handling failures and retries

Similarly to component commands, world commands can also fail and your worker instances might have to retry them. (However, world commands cannot fail with the “authority lost” status code.)

The following subsections explain the currently implemented failure status codes, their possible failure scenarios, and the recommended retry behavior.

Timeout

Meaning: The command request wasn’t processed within the specified deadline.

Failure scenarios:

  • The command timed out locally because the sending worker instance’s bridge didn’t respond in time. This can happen when this worker instance’s bridge takes too long to process the request or is overloaded.
  • The sending worker instance’s bridge timed out because the part of the Runtime responsible for processing the command takes too long to process the request, or was overloaded.

Recommended retry behavior: Automatically retry with a backoff. Don’t keep retrying immediately because in the case of overload, this makes the problem worse. If commands keep timing out without any sign of overload, make sure that all your worker types register command handlers for all your component commands.

Not found

Meaning: The entity that your worker instance tried to delete could not be found.

Failure scenarios:

  • This status code can only be returned by the entity deletion world command, in which case it indicates that the entity your worker instance tried to delete no longer exists.

Recommended retry behavior: Because the entity you tried to delete no longer exists, you should not retry this command.

Permission denied

Meaning: Your worker instance didn’t have permission to issue this world command.

Failure scenarios:

  • The worker permissions for the sending worker instance’s worker type don’t include the necessary permission to execute the specified world command.
  • The worker instance tried to create an entity with an invalid entity template that specified components that are reserved for system-managed entities. Only the Runtime can create and delete these entities.
  • The entity your worker instance tried to delete is system-managed. Only the Runtime can create and delete these entities.

Recommended retry behavior: Don’t retry. Since worker permissions cannot change at runtime, retried command requests of the same type will fail as well. Instead, check your worker permissions to make sure your worker instance has the appropriate permissions to execute this kind of command.

Application error

Meaning: The command failed because of a worker-side error.

Failure scenarios:

  • The worker instance tried to create an entity with an invalid entity template:
    • The entity template was invalid according to the specified schema. This can happen when using the C SDK.
    • The entity template didn’t include an EntityAcl component.
    • The entity template’s EntityAcl specified an invalid component ID or an invalid worker attribute.
    • The entity template didn’t include a Position component.
    • The entity template’s Position component specified a position with non-finite coordinates or one that is outside of the specified world dimensions.
  • The worker instance tried to create an entity with invalid entity ID:
    • An entity with that ID already exists.
    • The specified entity ID wasn’t previously reserved.
    • The specified entity ID’s reservation expired.
  • The worker instance tried to reserve an invalid number of entity IDs.

Recommended retry behavior: Don’t retry. Since the world command request was malformed, it will fail the same way again if you retry. Check the Runtime logs, which in some cases contain detailed information on the world command failure. (In the Logs viewer, these are any logs with the fabric tag but without the worker tag. You also see them when you run spatial [alpha] local launch.)

Internal error

Meaning: The command failed because of a SpatialOS-internal error.

Failure scenarios:

  • A bug in SpatialOS.

Recommended retry behavior: Don’t retry; the command is likely to fail again for the same reason. Raise a support request (for customers with a service agreement) or ask on our forums.



————
2019-09-19 Page updated with editorial review: Full copy-edit
2019-07-24 Page updated with limited editorial review: Extended “Component commands” description, added “Setting deadlines”, added “Handling failures and retries” sections

Search results

Was this page helpful?

Thanks for letting us know!

Thanks for your feedback

Need more help? Ask on the forums