Custom Metrics #

You can use the built-in MetricsManager interface to create metrics for your custom services. Pragma Engine uses the Micrometer metrics library. For more information about Micrometer, see the Micrometer documentation.

Pragma defined metrics and your custom metrics are sent to a composite meter metrics registry. You can use the registry to retrieve from or send data to third party metrics integrations. For more information about configuring the metrics registry, see Metrics Reporting.

Create custom metrics #

Use one of the following procedures to get started with custom metrics. This procedure uses the Pragma Engine matchmaking service as an example.

Metrics object definition

  1. Review the following supported metrics types, implemented in the Pragma MetricsManager.
  2. Navigate to your service definition.
    1. Create a Metrics object.
    2. Define your metrics.
Example: Metrics object definition
object Metrics {
    const val ENTER_QUEUE_KEY = "pragma.matchmaking.enterMatchmaking.enterQueue"
    const val ENTER_QUEUE_WITH_GAME_INSTANCE_KEY = "pragma.matchmaking.enterMatchmaking.enterQueueWithGameInstance"
    const val SKIP_QUEUE_KEY = "pragma.matchmaking.enterMatchmaking.skipQueue"
    const val PLAYER_COUNT_KEY = "pragma.matchmaking.playerCount"
    const val GAME_INSTANCE_COUNT_KEY = "pragma.matchmaking.gameInstancesInQueue"
    const val PAUSED_GAME_INSTANCE_COUNT_KEY = "pragma.matchmaking.pausedGameInstancesInQueue"
    const val PLAYER_LEAVE_KEY = "pragma.matchmaking.playerLeave"
    const val PARTY_TIMER_KEY = "pragma.matchmaking.party"
    const val GAME_INSTANCE_TIMER_KEY = "pragma.matchmaking.gameInstance"
}
  1. Use the metrics in your service.
Example: Metrics usage in function
suspend fun addMatchmakingGameInstance(
        queueKey: MatchmakingQueueKey,
        matchmakingGameInstance: MatchmakingGameInstanceImpl
    ) {
        mutex.withLock {
            val queue = getOrPutQueue(queueKey)

            matchmakingGameInstancesById[matchmakingGameInstance.gameInstanceId] = matchmakingGameInstance
                ...
            queue.queue.add(matchmakingGameInstance)
            metricsManager.metricsCounter(MatchmakingService.Metrics.ENTER_QUEUE_WITH_GAME_INSTANCE_KEY).increment()
        }
    }

Inline definition

You can also create and use metrics within your function definitions without first defining them in your service definition.

Example: In function metric definition
metricsManager.metricsCounter( 
    "pragma.payment.steam.finalizeError", "Error", "UnhandledSteamStatus", "SteamStatus", steamQueryParams.status
).increment()

Histograms #

Histograms show the frequency of a specified event. Pragma Engine supports using histograms to represent metrics, but creates them with 276 buckets that are stored as tags. This number of tags can be expensive in some metrics visualization and collection providers. The following metric types require the use of histograms.

  • metricsSummary
  • metricsGaugeWithDescription
  • metricsTimer

To turn on histograms, set enableHistogramMetrics to true in your configuration.

Supported metrics #

Pragma Engine supports the following types of metrics.

metricsCounter

With a counter you can increment a metric by a fixed positive amount, such as the count of requests to your authentication service. For more information about counters, see Counters in the Micrometer documentation.

Parameters

  • name - A unique identifiable name for the counter.
  • tags - Metadata that can help you manage, identify, organize, search for, and filter metrics.

Example

This example adds a game instance to the queue and increments the enterQueueWithGameInstance metric.

queue.queue.add(matchmakingGameInstance)
metricsManager.metricsCounter(MatchmakingService.Metrics.ENTER_QUEUE_WITH_GAME_INSTANCE_KEY).increment()
metricsTimer

Timers measure short-duration latencies and the frequency of events. Timers report the total time and the count of events as separate time series. To use timers, turn on enableHistogramMetrics. For more information about timers, see the Micrometer timers documentation.

Parameters

  • name - A unique identifiable name for the timer.
  • tags - Metadata that can help you manage, identify, organize, search for, and filter metrics.

Example

The following example creates a function to monitor when a party leaves matchmaking. This function records details such as the reason the party left and their game server version as tags.

private fun recordPartiesLeaveMatchmakingMetrics(
        matchmakingParties: List<MatchmakingPartyImpl>,
        outcome: Metrics.MatchmakingOutcome
    ) {
        pragmaNode.metricsSummary(
            Metrics.PLAYER_LEAVE_KEY,
            "outcome",
            outcome.toString(),
            "queue",
            toJsonString(matchmakingParties.first().matchmakingKey),
            "gameServerVersion",
            matchmakingParties.first().gameServerVersion
        ).record(matchmakingParties.sumOf { it.players.size }.toDouble())
        matchmakingParties.forEach {
            metricTimers.remove(it.matchmakingId)?.stop(
                pragmaNode.metricsTimer(
                    Metrics.PARTY_TIMER_KEY,
                    "outcome",
                    outcome.toString(),
                    "queue",
                    toJsonString(it.matchmakingKey),
                    "gameServerVersion",
                    it.gameServerVersion
                )
            )
        }
    }

metricsTimer #

Creates a Timer.Sample that when stopped reports the sample time to the metricsTimer.

metricsGauge

Gauges are like a speedometer and can go up and down to show a value at a point in time. Gauges are useful monitoring values that have an upper bound, such as a size of a collection or the number of threads in a running state. For more information about metrics gauges, see the Micrometer Gauges documentation.

The MetricsManager interface contains methods for building gauges to observe numeric values, collections, and maps.

Parameters

  • name - A unique identifiable name for the gauge.
  • tags - Metadata that can help you manage, identify, organize, search for, and filter metrics.

Additional parameters for metricsGuage

  • number - The mutable object containing the value to be monitored. The object must implement the Number interface.

Additional parameters for metricsGaugeCollectionSize

  • collection - The mutable object containing the collection size to be monitored. The object must implement the Collection interface.

Additional parameters for metricsGaugeMapSize

  • map - The mutable object containing the map size to be monitored. The object must implement the Map interface.

Example

The following example creates a metrics gauge that records the number of requests given to a specific request type.

class InternalRpcMetricCounters(private val metricName:String, private val jumpData: JumpData) {
    private var counters = ConcurrentHashMap<Int, AtomicLong>()

    fun get(requestType: Int, metricsManager: MetricsManager): AtomicLong {
        return counters.getOrPut(requestType) {
            val t = AtomicLong(0)
            metricsManager.metricsGauge(
                "pragma.internal.${metricName}",
                t,
                "requestType",
                jumpData.getProtoInfoForType(requestType).protoShortName
            )
        }
    }
}
metricsGauge with toDoubleFunction

Gauges are like a speedometer and can go up and down to show a value at a point in time. Gauges are useful monitoring values that have an upper bound, such as a size of a collection or the number of threads in a running state. For more information about metrics gauges, see the Micrometer Gauges documentation.

Parameters

  • name - A unique identifiable name for the gauge.
  • stateObject - The object that maintains the monitored value.
  • toDoubleFunction - A function that accepts a stateObject and returns a double value for the gauge to record.
  • tags - Metadata that can help you manage, identify, organize, search for, and filter metrics.

Example

The following example creates a metrics gauge for matchmaking queues.

private fun registerMetricsGauge(
        key: MatchmakingQueueKey,
        queue: MatchableQueue,
        name: String,
        toDoubleFunction: ToDoubleFunction<Deque<MatchableQueueMember>>
    ) {
        metricsManager.metricsGauge(
            name,
            queue.queue,
            toDoubleFunction,
            "queue",
            toJsonString(key.extMatchmakingKey),
            "gameServerVersion",
            key.gameServerVersion,
            "instanceId",
            serviceInstanceId.toInstanceIdString()
        )
    }
metricsSummary

A metrics summary tracks the distribution of events. For example, you could use a distribution summary to measure the size of requests going to a game server. For more information about metrics summaries, see Distribution Summaries in the Micrometer documentation.

To use metrics summaries, turn on enableHistogramMetrics.

Parameters

  • name - A unique identifiable name for the summary.
  • tags - Metadata that can help you manage, identify, organize, search for, and filter metrics.

Additional parameters for metricsSummaryWithDescription

  • baseUnit - The unit of measurement used by the distribution summary.
  • description - Help text that describes the recorded values.

Example

The following example creates a metrics summary that monitors the queue size based on players leaving the queue or game.

private fun recordPartiesLeaveMatchmakingMetrics(
   matchmakingParties: List<MatchmakingPartyImpl>,
   outcome: Metrics.MatchmakingOutcome
) {
    pragmaNode.metricsSummary(
        Metrics.PLAYER_LEAVE_KEY,
        "outcome",
        outcome.toString(),
        "queue",
        toJsonString(matchmakingParties.first().matchmakingKey),
        "gameServerVersion",
        matchmakingParties.first().gameServerVersion
    ).record(matchmakingParties.sumOf { it.players.size }.toDouble())
    matchmakingParties.forEach {
        metricTimers.remove(it.matchmakingId)?.stop(
            pragmaNode.metricsTimer(
                Metrics.PARTY_TIMER_KEY,
                "outcome",
                outcome.toString(),
                "queue",
                toJsonString(it.matchmakingKey),
                "gameServerVersion",
                it.gameServerVersion
            )
        )
    }
}

Testing custom metrics #

You can verify that you implemented your custom metrics correctly by checking your local Prometheus instance or through the local metrics endpoint.

Local Prometheus #

When you run Pragma Engine a local instance of Prometheus also runs in the background. You can use Prometheus to view the incoming metrics before they are sent to Grafana for visualization on your defined dashboards.

To access your local Prometheus:

  1. Build Pragma Engine from the platform directory make build
  2. Run Pragma Engine from the platform directory make run
  3. Navigate to the Prometheus dashboard in your browser localhost:9090/

If your new metrics don’t appear in Prometheus, try viewing the local metrics endpoint before troubleshooting your custom metrics.

Local metrics endpoint #

Local metrics endpoints store your metrics for visualization or collection services to use. You can access these endpoints at the following addresses:

  • Game PragmaNode metrics - localhost:10300/metrics
  • Social PragmaNode metrics - localhost:11300/metrics

Troubleshooting #

If your metrics aren’t appearing in the local Prometheus or metrics endpoint consider the following to begin troubleshooting your new custom metrics.

  • Ensure your custom plugin is running.
  • Review your configuration files and verify that you added your custom service to the plugin configuration.
  • If the metric is a histogram, verify that you have turned on histograms in the configuration.

Metrics
Metrics Reporting