Engine metrics #

Pragma records metrics for its own runtime alongside your custom metrics. This page covers the built-in metrics most useful for diagnosing node health and service behavior. It isn’t an exhaustive list.

Pragma provides Grafana dashboards that chart many of these metrics, including memory, garbage collection pauses, thread states, PlayerData operations, and cache activity. Use this page to interpret those panels or to query metrics the dashboards don’t chart.

Prometheus metrics are recorded by default and don’t require extra configuration. But other vendors can be configured. See: reporting backend.

Names are shown as exported to Prometheus, which is what you query in Grafana. The suffix tells you the metric type:

  • Counters end in _total and only increase. Query them with rate() or increase().
  • Timers are exported as a family of series, _seconds_bucket, _seconds_count, _seconds_sum, and _seconds_max, shown below as _seconds_*.
  • Gauges report a current value. They have no _total or timer suffix, though some end in a unit such as _bytes or _seconds.

Datadog and StatsD use the registered name instead, with dots in place of underscores and without the unit or _total suffixes. For example, pragma_engine_messages_outbound_total is pragma.engine.messages.outbound.

Common tags #

Every metric carries these tags, so you can filter or group any query by node:

TagDescription
clusterNameThe cluster the node belongs to.
backendTypeThe node’s backend type, such as GAME or SOCIAL.
nodeNameThe node’s name.

The Grafana dashboards use these tags for their Backend Type, Node Prefix, and Node Name filters. Node Prefix narrows the Node Name list to nodes whose names start with the selected value.

JVM and process #

Every node reports the standard Micrometer JVM and system metrics:

AreaMetrics
Memoryjvm_memory_*, plus jvm_memory_usage_after_gc: the fraction of long-lived heap still in use after garbage collection.
Garbage collectionjvm_gc_*, plus jvm_gc_overhead: the fraction of recent CPU time spent in garbage collection.
Threadsjvm_threads_live, jvm_threads_daemon, jvm_threads_peak, and jvm_threads_states (tagged state).
File descriptorsprocess_files_open_files and process_files_max_files. Reported on Linux and macOS only.
Uptimeprocess_uptime_seconds and process_start_time_seconds.
JIT compilationjvm_compilation_time_ms_total.
OtherClass loading (jvm_classes_*), CPU (system_cpu_*, process_cpu_*), and log events (logback_events_total).

A sustained rise in jvm_memory_usage_after_gc or jvm_gc_overhead indicates heap pressure. process_files_open_files approaching process_files_max_files indicates the node is close to running out of sockets and file handles.

Internal message routing #

These metrics cover messages passed between services, both within a node and between nodes.

MetricTagsDescription
pragma_engine_messages_outbound_totalroute, messageType, typeMessages sent to another service. route is local when the target service is on the same node, external when it’s on another node.
pragma_engine_messages_inbound_totalmessageType, typeMessages received from another service.
pragma_engine_serviceRpc_response_seconds_*typeRound-trip time of service-to-service RPCs, from send to response.
pragma_engine_serviceRpc_timedOut_totalrequestType, targetInstanceIdService-to-service RPCs that received no response before timing out.

messageType is request, response, notification, or serviceError. type is the message’s proto name.

A high share of external routes means services that call each other often are on different nodes, and every call crosses the network.

PlayerData operations #

MetricTagsDescription
pragma_playerdata_operation_seconds_*method, sessionType, inBatch, resultTime to run a single PlayerData operation. inBatch is true for operations run as part of a batch.
pragma_playerdata_batch_operations_seconds_*sessionType, operationCount, resultTime to run a batch of operations.

result records how the operation ended:

ValueMeaning
OKThe operation succeeded.
APPLICATION_ERRORThe operation threw an application error, which is returned to the caller.
SERVICE_ERRORThe operation threw a service error.
INTERNAL_ERRORThe operation threw any other exception.

Cancelled operations aren’t recorded. Filter on result="OK" when charting latency so fast failures don’t pull the numbers down.

Caches #

Engine caches, such as parties in PartyService and gameInstances in GameInstanceService, report these metrics, tagged cacheName, service, and instanceId:

MetricDescription
pragma_caches_cacheSizeNumber of entries in the cache.
pragma_caches_cacheAdd_totalEntries added.
pragma_caches_cacheAccess_totalCache reads.
pragma_caches_cacheMiss_totalReads that found no entry, so the cache loaded it or handled the unknown key.
pragma_caches_cacheRemovals_totalEntries deleted by the service, such as when a party disbands.
pragma_caches_cacheEvictions_totalEntries removed by an evicting cache’s sweep because they weren’t accessed within maxTimeBetweenAccessMillis. Sweeps only evict once the cache exceeds its configured size.
pragma_caches_cacheSweepItems_totalAll removal attempts from any source, including removals and evictions, and attempts that found no entry.

Removals and evictions are counted separately. A rising eviction rate with steady removals means entries are expiring rather than being cleaned up by the service that owns them.


Metrics Visualization
Logging