Serialization reference
Unlike the C++, C# and Java SDKs, which provide generated classes that you can make use of to
interact with schema in your workers, the C API leaves code generation completely up to you. To let you do this, the C API exposes a serialization library (contained within improbable/c_schema.h) with the Schema_ prefix. This page explains the two different methods of providing serialization code, and provides a reference of how to serialize various schema types.
Handle types
For each component known to a particular worker, the C API make use of 4 different data handle types:
ComponentData- Represents a component at a single point in time. Should contain all fields.ComponentUpdate- Represents a component update. Can contain any number of fields from the component, as it represents a diff of some component data between two points of time.CommandRequest- Represents some command request data. Should contain all fields of the command request type.CommandResponse- Represents some command response data. Should contain all fields of the command response type.
In the other SDKs, these handle types are hidden (because they’re dealt with by the schema compiler); in the C API, they’re fully exposed. It’s up to you to serialize the above data (either directly when triggering an operation or by providing vtable functions, as described in the next section).
Specifying serialization code
All 4 handle types have the following structure (using Worker_ComponentUpdate as an example):
typedef struct Worker_ComponentUpdate {
Worker_ComponentId component_id;
Schema_ComponentUpdate* schema_type;
Worker_ComponentUpdateHandle* user_handle;
} Worker_ComponentUpdate;
When using any handle type for any reason (for example, sending a component update), you are given
the choice between either directly creating a serialized object (schema_type), or
passing in your own custom handle (user_handle), which will later get serialized by a
vtable function.
Serialize directly
Using this method, you perform the serialization directly when performing an operation (for example, sending a component update). This is the simplest way to serialize and deserialize data. To enable this, you need to specify a default vtable (so it applies to all components) which is empty (so it uses manual serialization):
Worker_ComponentVtable default_vtable;
memset(&default_vtable, 0, sizeof(Worker_ComponentVtable));
Worker_ConnectionParameters params = Worker_DefaultConnectionParameters();
// ...
params.default_component_vtable = &default_vtable;To serialize a component update directly, you create the Schema_ComponentUpdate object (more on
this below), serialize your data into it, then set the
schema_type field to the Schema_ComponentUpdate object, leaving user_handle as NULL. When
you do this, the C API will use this serialized data (and take ownership of it, so you don’t need to
free anything) without doing any more work. For example:
Worker_ComponentUpdate update;
update.component_id = 54;
update.schema_type = Schema_CreateComponentUpdate(54);
// serialize into update.schema_type.
Worker_Connection_SendComponentUpdate(connection, 1, &update);Deserialization is similar. For example, when processing a component update op, a
Worker_ComponentUpdate will be provided with schema_type assigned to some serialized data
received from the network (and user_handle set to NULL). You can then read this component update
using the schema functions, as described below. For example:
Worker_ComponentUpdate* update;
if (op->op_type == WORKER_OP_TYPE_COMPONENT_UPDATE) {
update = &op->component_update.update;
// deserialize update from "update->schema_type", depending on the component ID.
}Vtables
One drawback to the above approach is that serialization / deserialization needs to be done correctly in all the places that a particular component is used, so it’s easy to miss a case somewhere and end up with a bug which is difficult to track. Another drawback is that serialization is done in the game thread, rather than being done in the internal C API connection threads (assuming the language being used with the C API supports this).
The other option for serializing data is using “vtables” (named after the concept used to implement
polymorphism). A vtable is a struct containing a component ID, and a set of 16 different function
pointers (4 for each handle type). Instead of providing a serialized object in the schema_object
field, you specify a custom handle in the user_handle field, leaving schema_object as NULL.
This custom handle can be any object you like, as it’s a typedef to void*. For each handle type,
you must provide the following functions:
Serialize- converts a custom handle into a serializedSchema_*object.Deserialize- converts a serializedSchema_*object into a custom handle.Copy- creates a copy of a custom handle object (managed then deleted by the C API using yourFreemethod).Free- frees a custom handle allocated usingCopy.
These functions have the following signature (using component update as an example):
typedef void Worker_ComponentUpdateFree(Worker_ComponentId component_id, void* user_data, Worker_ComponentUpdateHandle* handle);
typedef Worker_ComponentUpdateHandle* Worker_ComponentUpdateCopy(Worker_ComponentId component_id, void* user_data, Worker_ComponentUpdateHandle* handle);
typedef uint8_t Worker_ComponentUpdateDeserialize(Worker_ComponentId component_id, void* user_data, Schema_ComponentUpdate* source, Worker_ComponentUpdateHandle** handle_out);
typedef void Worker_ComponentUpdateSerialize(Worker_ComponentId component_id, void* user_data, Worker_ComponentUpdateHandle* handle, Schema_ComponentUpdate** target_out);
Note that each function signature contains a user_data parameter. This is unrelated to the
user_handle specified in the handle type. Instead, this is an arbitrary pointer that is set inside
the Worker_ComponentVtable object and also controlled by you. You can leave this as NULL if
it’s not needed, but one use case for this parameter could be to pass in a dynamic language VM
context so you can invoke some dynamic language code from your C callback.
The simplest implementation of these methods would use malloc and free in C to allocate a struct
you manage yourself, and using the pointer returned by malloc as your
Worker_ComponentUpdateHandle. If the C API is used in a language which doesn’t support pointers, a
more complicated example could use a hash map or dictionary. You’d use an index as a key to a blob
of data, and use that index as your Worker_ComponentUpdateHandle. This is similar to the approach
used in the C# and Java SDKs.
A simplified example in pseudo-C# (assuming that C bindings have been set up) could look like:
// An example of a generated struct for "Position".
struct Position
{
public double x;
public double y;
public double z;
};
// A class which manages a dictionary from an integer to a blob of managed memory.
class Serialization
{
internal static Dictionary<Integer, Object> ObjectHandles;
internal static Integer AddObject(object o) { ... }
internal static void RemoveObject(Integer handle) { ... }
}
// An example of a generated class which implements the vtable functions for "Position".
class PositionSerialization
{
static unsafe void ComponentUpdateFree(
ComponentId componentId,
void* userData,
ComponentUpdateHandle* handle)
{
// Removes the object with index 'handle' from the dictionary. The object will get garbage
// collected later.
Serialization.RemoveObject(handle);
}
static unsafe ComponentUpdateHandle* ComponentUpdateCopy(
ComponentId componentId,
void* userData,
ComponentUpdateHandle* handle)
{
// This function will not copy the data, but create a second handle which points to the same
// data.
object o;
Serialization.ObjectHandles.TryGetValue(handle, out o);
if (o != null)
{
// Allocate a second handle to the same data.
Integer newHandle = Serialization.AddObject(o);
return newHandle;
}
return null;
}
static unsafe byte ComponentUpdateDeserialize(
ComponentId componentId,
void* userData,
SchemaComponentUpdate* source,
ComponentUpdateHandle** targetOut)
{
var position = new Position();
// read `source` using schema APIs and write data to position struct.
// return 0 (false) if serialization fails.
Integer newHandle = Serialization.AddObject(position);
*targetOut = newHandle;
return 1; // (true)
}
static unsafe void ComponentUpdateSerialize(
Worker_ComponentId componentId,
void* userData,
ComponentUpdateHandle* handle,
SchemaComponentUpdate** targetOut)
{
object o;
Serialization.ObjectHandles.TryGetValue(handle, o)
if (o == null)
{
return;
}
var position = (Position) o;
// Read position struct and write data using schema APIs to *targetOut.
}
}
Serialization reference
We provide a C example project which serializes components with both approaches described above. The remainder of this page is a reference which describes in detail how to use the schema C API to serialize different schema types, such as commands, lists, options, etc.
In the code examples, we assume that Schema_Object* fields_object; has been defined already and
is set to the schema object which contains the fields of the Example schema type at the beginning
of each section. More about schema objects can be found below.
Primitive types
enum MyEnum {
FOO = 1;
BAR = 2;
BAZ = 3;
}
type Example {
int32 value = 3;
EntityId entity_id = 4;
MyEnum my_enum = 5;
}
To write a primitive type to a schema object, you can use any of the Schema_Add* functions,
depending on the primitive type. For example, to write an integer 1234 to the field above, you can
write the following code:
Schema_AddInt32(fields_object, 3, 1234);To read a primitive type from a schema object, you can use any of the Schema_Get* functions,
depending on the primitive type. For example, to read the integer in the field above into a
variable, you can write the following code:
int my_value = Schema_GetInt32(fields_object, 3);The mapping from schema type to function family is given below:
| .schema type | function family |
|---|---|
| int32 | Int32 |
| int64 | Int64 |
| uint32 | Uint32 |
| uint64 | Uint64 |
| sint32 | Sint32 |
| sint64 | Sint64 |
| fixed32 | Fixed32 |
| fixed64 | Fixed64 |
| sfixed32 | Sfixed32 |
| sfixed64 | Sfixed64 |
| bool | Bool |
| float | Float |
| double | Double |
| string | Bytes |
| EntityId | EntityId |
| bytes | Bytes |
| user-defined enum | Enum |
| user-defined type | Object |
An EntityId is stored as an int64 under the hood, but for clarity, we expose separate functions
for serializing and deserializing entity IDs. For example:
Worker_EntityId my_value = Schema_GetEntityId(fields_object, 4);An enum is stored as an uint32 under the hood, but similarly to EntityId, we expose separate
functions for serializing and deserializing enums. For example:
typedef enum MyEnum { FOO = 1, BAR = 2, BAZ = 3 } MyEnum;
MyEnum my_value = (MyEnum)Schema_GetEnum(fields_object, 5);Lists
type Example {
list<float> value = 6;
}
A list field is a field whose ID occurs 0 or more times. In generated code, you add values to a list field any number of times to populate your list (whilst for singular primitive types mentioned above, you must add a value exactly once). An example of writing a list in this way can be the following (using the above list field as an example):
float my_list[4] = {1, 2, 3, 4};
Schema_AddFloat(fields_object, 6, my_list[0]);
Schema_AddFloat(fields_object, 6, my_list[1]);
Schema_AddFloat(fields_object, 6, my_list[2]);
Schema_AddFloat(fields_object, 6, my_list[3]);However, this is relatively slow in practice due to having to iterate over the list and copy the
data. The array can be provided to the C API without copying using the Schema_Add*List family of
functions. As a result, the above code can be simplified to:
float my_list[4] = {1, 2, 3, 4};
Schema_AddFloatList(fields_object, 6, my_list, 4);When reading a list field from a schema object, you can:
- obtain the count with the
Schema_Get*Countfamily of functions. - retrieve individual elements with the
Schema_Index*family of functions. - copy the entire list into a buffer managed by yourself using the
Schema_Get*Listfamily of functions.
For example:
/* access the 3rd element (index 2). */
float single_element = Schema_IndexFloat(fields_object, 6, 2);
/* obtaining the complete list. */
uint32_t count = Schema_GetFloatCount(fields_object, 6);
float* array_data = (float*)malloc(sizeof(float) * count);
Schema_GetFloatList(fields_object, 6, array_data);Strings and bytes
type Example {
string value_string = 1;
bytes value_bytes = 2;
}
String and bytes fields are both treated as bytes in the C API, because they are both treated in the same way over the wire. In SpatialOS, the only difference is how code is generated for them in the C++, C#, and Java SDK, and how they’re visualized in the inspector. To write a byte buffer to a schema object (without copying), you can do this with the following code:
/* write string. ensure to exclude the null terminator. */
const char* text = "Hello World.";
Schema_AddBytes(fields_object, 1, (const uint8_t*)text, sizeof(char) * strlen(text));
/* write bytes */
uint32_t bytes_length;
unsigned char* bytes = create_byte_buffer(&bytes_length);
Schema_AddBytes(fields_object, 2, bytes, bytes_length);Sometimes, it’s difficult to ensure that the byte buffers live long enough. In this case, you can
reserve a buffer to a specified length using Schema_AllocateBuffer, which is managed by the C API
and is guaranteed to have the same lifetime as the schema object. You can modify the above example
to use this function in the following way:
/* write string. */
const char* text = "Hello World.";
uint32_t text_length = sizeof(char) * strlen(text); /* ensure to exclude null-terminator. */
uint8_t* text_buffer = Schema_AllocateBuffer(fields_object, text_length);
memcpy(text_buffer, text, text_length);
Schema_AddBytes(fields_object, 1, text_buffer, text_length);
/* write bytes */
uint32_t bytes_length;
unsigned char* bytes = create_byte_buffer(&bytes_length);
uint8_t* byte_buffer = Schema_AllocateBuffer(fields_object, bytes_length);
memcpy(byte_buffer, bytes, bytes_length);
free_byte_buffer(bytes);
Schema_AddBytes(fields_object, 2, byte_buffer, bytes_length);To read bytes from a schema object, you can make use of Schema_GetBytesLength to obtain the length
of the byte buffer, and Schema_GetBytes to get a pointer to the byte buffer itself. For example:
/* read string. */
uint32_t text_length = Schema_GetBytesLength(fields_object, 1);
const uint8_t* text_buffer = Schema_GetBytes(fields_object, 1);
/* ensure to include space for null terminator. */
char* text = (char*)malloc(text_length + 1);
memcpy(text, text_buffer, text_length);
text[text_length] = '\0';
/* read bytes. */
uint32_t bytes_length = Schema_GetBytesLength(fields_object, 2);
const uint8_t* bytes = Schema_GetBytes(fields_object, 2);
/* do something with bytes and bytes_length. */Objects
type MyType {
int32 some_value = 1;
float another_value = 2;
}
type Example {
MyType data = 4;
}
Objects in schema are containers that contain fields, including potentially more object fields. You can manipulate object fields in a similar way to how you manipulate primitive types, but can also manipulate fields within those object fields.
To write an object following the above schema, you could use the following code:
Schema_Object* type = Schema_AddObject(fields_object, 4);
Schema_AddInt32(type, 1, 1234);
Schema_AddFloat(type, 2, 1234.0f);Similarly, we can read the object using the following code:
typedef struct MyType {
int32_t some_value;
float another_value;
} MyType;
Schema_Object* type = Schema_GetObject(fields_object, 4);
MyType data;
data.some_value = Schema_GetInt32(type, 1);
data.another_value = Schema_GetFloat(type, 2);Options
type Example {
option<uint32> value = 2;
}
An option field is a field whose ID occurs either 0 or 1 times. It can be thought of as a list
which can have either 0 or 1 elements. Therefore, options are used in a similar way to lists. To
write an option value, you can call Schema_Add* to set the option to a value, or you can keep the
value empty by omitting the call to Schema_Add*. For example:
/* places 1234 in a uint32 option field. */
Schema_AddUint32(fields_object, 2, 1234);When reading an option field from a schema object, you can use the Schema_Get*Count family of
functions to determine whether the option field is set. For example:
uint32_t* value = NULL;
if (Schema_GetUint32Count(fields_object, 2) == 1) {
/* obtain the value. */
value = (uint32_t*)malloc(sizeof(uint32_t));
*value = Schema_GetUint32(fields_object, 2);
}Maps
type Example {
map<uint32, float> value = 1;
}
A map field is a field which stores key-value pairs (with unique keys). In terms of serialization,
it is functionally equivalent to reading and writing a list of objects, where each object is a
key-value pair. The field IDs of the key and value are 1 and 2 respectively, which are also defined
with the SCHEMA_MAP_KEY_FIELD_ID and SCHEMA_MAP_VALUE_FIELD_ID constants respectively. To write
a map, you can use the following code to add a single key value pair [100 -> 25.0f]:
/* for each key value pair.. */
Schema_Object* kvpair = Schema_AddObject(fields_object, 1);
Schema_AddUint32(kvpair, SCHEMA_MAP_KEY_FIELD_ID, 100);
Schema_AddFloat(kvpair, SCHEMA_MAP_VALUE_FIELD_ID, 25.0f);Similarly, to read a map field, you would make use of Schema_GetObjectCount to obtain the number
of key-value pairs, then use Schema_IndexObject to read them. For example:
uint32_t map_size = Schema_GetObjectCount(fields_object, 1);
for (uint32_t i = 0; i < map_size; ++i) {
Schema_Object* kvpair = Schema_IndexObject(fields_object, 1, i);
uint32_t key = Schema_GetUint32(kvpair, SCHEMA_MAP_KEY_FIELD_ID);
float value = Schema_GetFloat(kvpair, SCHEMA_MAP_VALUE_FIELD_ID);
// use 'key' and 'value'
(void)key;
(void)value;
}Component data
component TestComponent {
id = 10000;
int32 foo = 1;
float bar = 2;
}
Component data is represented as a single object, known as the “fields” object. If you
create a Schema_ComponentData object to be saved to a snapshot, or you receive some component data
when handling an “AddComponentOp”, you can access the “fields” object using the
Schema_GetComponentDataFields function. If the fields object does not exist yet, then this
function will automatically create it for you. This “fields” object will contain the fields of your
component (defined either inline in the component, or using the data statement in .schema). For
example, to create a component data object that corresponds to TestComponent:
Schema_ComponentData* schema_data = Schema_CreateComponentData(10000);
Schema_Object* fields_object = Schema_GetComponentDataFields(schema_data);
Schema_AddInt32(fields_object, 1, 1234);
Schema_AddFloat(fields_object, 2, 55.0f);
/* use schema_data somewhere. */Similarly, you can read a component data object with the following code:
/* assume we have a "Schema_ComponentData* schema_data", possibly contained in a
* Worker_SnapshotData or in a vtable callback. */
Schema_Object* fields_object = Schema_GetComponentDataFields(schema_data);
int32_t foo = Schema_GetInt32(fields_object, 1);
float bar = Schema_GetFloat(fields_object, 2);Component updates
type TestEvent {
uint32 counter = 1;
float size = 2;
}
component TestComponent {
id = 10000;
int32 foo = 1;
float bar = 2;
event TestEvent my_event;
}
A component update can be thought of as a diff between two component snapshots. They contain two objects, a “fields” object and an “events” object.
Fields
In terms of serialization, the “fields” object is equivalent to the “fields” object in a
component data. However, all primitive fields are treated as option<...>’s
instead (options, lists and maps are treated the same). That means that you can choose to not call
Schema_Add* for a particular field ID, and that field will not be included in the component
update. For example:
Schema_ComponentUpdate* schema_update = Schema_CreateComponentUpdate(10000);
Schema_Object* fields_object = Schema_GetComponentUpdateFields(schema_update);
Schema_AddInt32(fields_object, 1, 1234);
/* use schema_update somewhere. */When receiving a component update, there is no guarantee that a field will contain a value. Similar
to options, you should use the Schema_Get*Count family of functions to check whether
a field is contained within an update. To read an update to TestComponent:
/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
* Worker_SnapshotUpdate or in a vtable callback. */
Schema_Object* fields_object = Schema_GetComponentUpdateFields(schema_update);
int32_t foo;
float bar;
if (Schema_GetInt32Count(fields_object, 1) > 0) {
foo = Schema_GetInt32(fields_object, 1);
}
if (Schema_GetFloatCount(fields_object, 2) > 0) {
bar = Schema_GetFloat(fields_object, 2);
}Clearing fields
So far, it’s possible to write an option, list or map which as at least one element in it, otherwise
the update is considered to not have a change to that field ID. However, in some cases, you wish to
assign an option, list or map field to the empty state rather than leave it unchanged. The
Schema_*ComponentUpdateClearedField family of functions can be used for this purpose. For example,
to clear a list field (by setting the field to an empty list), you can write:
/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
* Worker_SnapshotUpdate or in a vtable callback. */
/* specify that this update sets field 3 to the empty option/list/map. */
Schema_AddComponentUpdateClearedField(schema_update, 3);When receiving a component update, you can obtain the list of fields to assign to the empty option/list/map with the following code:
/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
* Worker_SnapshotUpdate or in a vtable callback. */
uint32_t cleared_field_count = Schema_GetComponentUpdateClearedFieldCount(schema_update);
Schema_FieldId cleared_field_id;
for (uint32_t i = 0; i < cleared_field_count; ++i) {
cleared_field_id = Schema_IndexComponentUpdateClearedField(schema_update, i);
/* assigned `cleared_field_id` to the empty option/list/map. */
}Events
Events are stored as lists of objects which contain the event data, as part of the “events” object. Each field ID in the “events” object corresponds to the “event ID”, which is a 1-based index depending on the order of the events in the component.
To trigger a single ‘my_event’ event as part of a component update, you can use the following code:
Schema_ComponentUpdate* schema_update = Schema_CreateComponentUpdate(10000);
Schema_Object* events_object = Schema_GetComponentUpdateEvents(schema_update);
Schema_Object* event_data = Schema_AddObject(events_object, 1); /* event ID 1 */
Schema_AddUint32(event_data, 1, 122); /* counter */
Schema_AddFloat(event_data, 2, 25.0f); /* size */
/* use schema_update somewhere. */When receiving a component update, you need to ensure that all instances of each event type are processed, as a single component update can contain many copies of the same event being triggered. For example:
/* assume we have a "Schema_ComponentUpdate* schema_update", possibly contained in a
* Worker_SnapshotUpdate or in a vtable callback. */
Schema_Object* events_object = Schema_GetComponentUpdateEvents(schema_update);
/* iterate over all 'my_event' events in this update */
Schema_Object* event_data;
for (uint32_t i = 0; i < Schema_GetObjectCount(events_object, 1); i++) {
event_data = Schema_IndexObject(events_object, 1, i); /* event ID 1 */
uint32_t counter = Schema_GetUint32(event_data, 1);
float size = Schema_GetFloat(event_data, 2);
// use 'counter' and 'size'.
(void)counter;
(void)size;
}Command requests and responses
type CmdRequest {
bytes data = 1;
float value = 2;
}
type CmdResponse {
int32 value = 1;
}
component TestComponent2 {
id = 10001;
command CmdResponse my_command(CmdRequest);
}
Command requests and response objects are serialized in almost exactly the same way as
component data. The only difference is that you need to call
Schema_GetCommandRequestObject to access the command request object from an instance of
Schema_CommandRequest*, or Schema_GetCommandResponseObject to access the command response object
from an instance of Schema_CommandResponse*. In addition, you need to specify a “command ID” when
creating a command request or response object. This ID is similar to an “event ID”, because it is
also a 1-based index depending on the order of the commands in the component. To write a command
request or response, you can use the following code (using request as an example):
Schema_CommandRequest* schema_request = Schema_CreateCommandRequest(10001, 1);
Schema_Object* request_object = Schema_GetCommandRequestObject(schema_request);
uint32_t bytes_length;
const uint8_t* bytes_buffer = create_byte_buffer(&bytes_length);
Schema_AddBytes(request_object, 1, bytes_buffer, bytes_length);
Schema_AddFloat(request_object, 2, 55.0f);
/* use schema_request somewhere. */Similarly, you can read a command request or response object with the following code (using request as an example):
/* assume we have a "Schema_CommandRequest* schema_request", possibly contained in a
* Worker_SnapshotData or in a vtable callback. */
Schema_Object* request_object = Schema_GetCommandRequestObject(schema_request);
const uint8_t* bytes_buffer = Schema_GetBytes(request_object, 1);
uint32_t bytes_length = Schema_GetBytesLength(request_object, 1);
float value = Schema_GetFloat(request_object, 2);