This article was written on Pragma Engine version 0.0.94.

The Content Data System: Part 2 #

In the second article of the Content Data series, we’ll delve further into Pragma Engine’s Content Data system by showing you how to create and interact with your own content data. Make sure you check out the previous Content Data article to get a quick overview of how content is handled in Pragma Engine.

Adding Content to Pragma Engine #

Overview of adding content data to Pragma Engine.

Creating content in Pragma Engine can be broken down into the following steps, which we’ll cover in the sections below:

1. Creating the protobuf template data, which will be represented in provided content’s ext fields.
2. Adding and cataloging content in JSON files.
3. Defining and associating data with a Content Handler.
4. Running apply commands to verify and generate a readable JSON package for Pragma Engine’s services.

Creating the protos #

All protobuf data–for provided and custom content–is created and edited in the platform’s 5-ext/ext-protos folder. Remember that protos are used as templates for defining other types of content throughout Pragma Engine. This type of template data will be written as nested protobuf messages for provided content, and regular protobuf messages for custom content.

The location of Proto data in 5-ext/ext-protos.

To see an in-depth tutorial on how to write your own content using protos, check out our Instanced Item series article on Creating Custom Content for Instanced Items.

Remember to only edit files in the 5-ext folder. If you edit files in the 1-protos folder, your protobuf definitions will not register with Pragma Engine and could cause engine errors.

Creating the JSON #

JSON data–which is used to catalog and define provided and custom content–is created and edited in the platform’s 5-ext/content/src folder. This data is written using specific, predefined data type fields associated with the content’s JSON structure.

When defining your JSON catalogs, make sure that JSON objects can represent nested messages with all their fields filled out and completed–not just a filled ext field.

The location of JSON data in 5-ext/content/src.

JSON content data requires two specific files in the /src folder:

• <content-type>.json contains the list of specific content objects (such as StackableSpecs.json from Pragma Engine’s provided content). These files are also often described as JSON catalogs.
• <content-type>.metadata houses metadata information pertaining to the specific type of content data (such as StackableSpecs.metadata from Pragma Engine’s provided content). The metadata information in these files are links to the content-type’s specific Content Handler and version incrementer. We’ll explain what these two metadata types mean in the next section below.

The /src folder is empty by default when installing and running a fresh build of Pragma Engine. If you want to define JSON data using provided content paths, run the following make command in bash using platform as the working directory. This will generate readily usable and out-of-the-box inventory content files for defining your content data.

make init-inventory-content

To define JSON data using custom content, you’ll need to create and populate two required <content-type>.json and <content-type>.metadata files for your desired content. This also involves creating Kotlin logic for the content’s associated Content Handler, which will explain in the section below.

Defining Content Handlers #

Content handlers are the bridges for plugins and other services to use and modify content data; they contain and provide access to specific validated and versioned content data. All Content Handlers are created in Kotlin and assigned within a content-type’s metadata file. They are responsible for validating, containing, and providing access to content data via the ContentData interface, which we’ll explain later on in the article.

How Content Handlers operate with content data.

When a Content Handler is assigned to a <content-type>.metadata file, you can see the appropriate Content Handler defined in the nested applyTriggers’s field. Below is an example of the StackableSpecsHandler in the provided StackableSpecs.metadata file.

{
  "applyTriggers": {
    "contentHandler": "StackableSpecsHandler"
  },
  "versionsToRetain": 1
}

All provided content in the Inventory service comes with default Content Handlers, like the StackableSpecs example above. If you’re defining your own custom content, you’ll need to create your own Content Handler associated with the content.

Creating a custom-defined Content Handler #

All custom-defined Content Handlers should be created and populated in the relevant service directory within 5-ext/ext/src/main/kotlin. You can also define a custom Content Handler for provided content as well. This can help enable certain unique features like cross-content validation across ext fields, which we’ll cover later on in the article.

When creating your own custom-defined Content Handler, you’ll need to import the necessary content and packages associated with what you want your Content Handler to manage. Then, create a class with the following information: the handler’s name, the ContentHandler definition and its associated content-type package, and the custom proto’s location, name, and class.

class [handler name]: ContentHandler<[package].[proto name]>([package].[proto name]::class)

After the class has been defined, override the idFromProto function that returns the Proto ID as a string.

override fun idFromProto(proto: [package].[proto name]): String {
    return proto.[id field]
    }

The finished, overridden function should look something like this:

package demo.content

import pragma.content.ContentHandler

class MyCustomHandler: ContentHandler<CustomContent.MyCustomProto>(CustomContent.MyCustomProto::class) {
    override fun idFromProto(proto: CustomContent.MyCustomProto): String {
        return proto.id
    }
}

Once the Content Handler has been created, go to 5-ext/content/src and assign the Content Handler to a metadata file nested under applyTriggers.

{
  "applyTriggers": {
    "contentHandler": "[handler_name]"
  },
  "versionsToRetain": 1
}

Now that we’ve gone over how to create your own custom content in protos, JSON, and with Content Handlers, let’s discuss how you can apply and verify changes to your content data with the proper validation steps.

Applying and validating content data changes #

It’s easy to make typos or miss certain data-fields when defining and creating content data, especially when working with nested proto messages. Even more difficulties can arise if these incorrect changes in content data make it to live clients or versions of your game. To remedy these scenarios, we can run specific bash commands that validate and check that all new changes to Pragma Engine are written with the correct syntax and required data-fields.

Validated JSON and Protos content in the package folder.

When you validate your content in Pragma Engine, a JSON file is generated in the 5-ext/content/package folder for other Pragma Engine services to use and read. This is to prevent any unprompted data migrations or changes when editing your content data. This also makes sure that Pragma Engine’s services only use the most up to date and working content data in your backend.

Make sure you do not make any files in the /package folder yourself, as this would skip the entire validation step. We also recommend never editing the package files manually, as this can cause major content data errors.

You can validate your content two different ways in Pragma Engine: by using basic validation and cross-content validation.

Basic Validation #

For provided content, you’ll need to use the following bash command and use platform as the working directory to validate and apply any content data changes. Make sure you do this step after you’re finished creating, editing, or removing any content data in your Pragma Engine.

make ext-contentdata-apply

Pragma Engine validates content across nested IDs (outside of any ext data) during the contentdata-apply command step. It checks that all JSON data matches protobuf schema directly, and any invalid content will prevent changes to the engine from being applied.

Invalid and valid content data changes will appear in the bash command log once the contentdata-apply command is finished running. It’s important to note that if one content data change is invalid during the contentdata-apply command step, none of the changes will be applied–even if some of the content data changes were valid during the initial contentdata-apply command.

In addition to the validation of proto schemas and provided content in JSON, you can write custom validation processes for protos and custom ext content.

Custom Cross-content Validation #

For custom content, you can add a validation step across nested ext fields by adding the overridden validateExtWithAllContent function to the Content Handler associated with the ext content.

If you want to validate ext data from provided content, you must write a new Content Handler (even though the provided content already has an associated Content Handler). The new and additional Content Handler needs to inherit information from the provided Content Handler as well, in order to maintain non-ext content validation steps.

When adding the overridden validateExtWithAllContent function to a Content Handler, you’ll need do the following steps:

1. Override the validateExtWithAllContent and inside the functions parameters pull out relevant Content Data from the contentByType map.
   
   The keys will be strings of content types, so when writing code for these keys use the content type’s file name without the JSON extension. You can also add a value inside this function for InventoryCrossContentValidator(), which makes the pulling of relevant content types easier when validating across provided Inventory content.

   override fun validateExtWithAllContent(contentByType: Map<String, ContentData<*>>) {
       val inventoryValidator = InventoryCrossContentValidator(contentByType)
}

2. Create new values in the function that pull out the nested content IDs from the ext templates. These values must be stored in the function by using mutableSetOf<CorrespondingIds>().
   
   CorrespondingIds is a class that holds specific data types of versioned content data via two different strings: idToValidate and parentId. In short, this class ensures error messages are clear for debugging when validating your specific ext content.

3. Once you’ve fleshed out the function by pulling out the ext’s nested content IDs, confirm that all the nested IDs are valid and a part of the relevant content data types by using:

ContentData.validateIds(idsToValidate: Set<CorrespondingIds>, contentType: String)

Cross-content validation is highly dependent on custom proto structure and the way you write your content data, so with that in mind, the way you write and structure the logic for validateExtWithAllContent will vary. Generally speaking, you’ll want to pull out all the nested content IDs into their own variable in the function, and use protobuf compiled methods to validate each individual data type within the ext template.

Now, in the next section of the article, let’s break down how we can start using all our provided and custom content from the Content Data system.

Accessing and Leveraging Content Data with Plugins #

All types of content in Pragma Engine must be accessed using the ContentDataNodeService. The ContentDataNodeService uses getHandler() functions to return deserialized JSON files containing specified proto data. The service works in tandem with the Content Data system to return usable JSON data via RPCs.

How Pragma Engine’s ContentDataNodeService and ContentData interface gives plugins access to your content.

Enabling content access #

When you create content in Pragma Engine, all of the data is stored and made in the Content Data service. However, RPCs and other service calls can’t manipulate that data directly–that’s what the ContentDataNodeService is for. The ContentDataNodeService calls the ContentData interface to return requested content data in an accessible form for RPCs to use. This accessible form, which are usually deserialized JSON files, is then accessible for RPCs through the ContentDataNodeService.

When the ContentDataNodeService fetches specific Content Data, it uses the following call which all content in Pragma Engine uses by default. This call requires the content proto and the .json file resourcePath as a string.

    fun <ContentProto : GeneratedMessageV3> getHandler(resourcePath: String): ContentData<ContentProto> {
        val contentType = nameWithoutExtension(resourcePath)
        return cachedContentHandler.getOrPut(contentType) { contentHandler(resourcePath) } as ContentData<ContentProto>
    }

When this getHandler() runs, it returns the deserialized JSON data from the ContentData interface. Remember that the data the ContentData interface sends to these handlers is read-only, but the getHandlers() returns only the requested data through a map data-type where the map’s keys are the data’s content IDs.

The Content Data interface uses two important functions to provide requested data information: the get(id: String) function, which takes in the requested data’s ID, string, and the data’s content proto, and the contains(id: String) helper function, which provides a boolean to confirm the requested content is available.

Plugins can access and request all content visible to the ContentDataNodeService, and you usually won’t need to implement a custom plugin that needs to directly call this service on its own. However, if you make any type of content that is completely separate from Pragma’s supported content paths with player data, you can implement and call the ContentDataNodeService directly by adding the following line of code to your plugin’s run() function:

content.init(nodeServicesContainer[ContentDataNodeService::class])

And that’s our overview of Pragma Engine’s Content Data system! To see all the features and services we discussed in action, check out our Stackable Item series and Instanced Item series.

For more information, check out the rest of the articles in this series:

Content Data System Part I
Content Data System Part II (this article)