»Nomad Autoscaler Telemetry

The Nomad Autoscaler agent collects various runtime metrics about the performance of different libraries and subsystems. These metrics are aggregated on a ten second interval and are retained for one minute. To configure the telemetry output please see the agent configuration.

This data can be accessed via the /v1/metrics HTTP endpoint, via sending a signal to the Nomad Autoscaler process or via a number of integrations.

To view this data via sending a signal to the Nomad Autoscaler process: on Unix, this is USR1 while on Windows it is BREAK. Once Nomad Autoscaler receives the signal, it will dump the current telemetry information to the agent's stderr.

This telemetry information can be used for debugging or otherwise getting a better view of what Nomad is doing.

Below is sample output of a telemetry dump:

[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 219856.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4316568.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 36243.000
[2020-08-25 10:01:20 +0100 BST][S] 'nomad-autoscaler.runtime.gc_pause_ns': Count: 5 Min: 38083.000 Mean: 69764.400 Max: 122291.000 Stddev: 31487.808 Sum: 348822.000 LastUpdated: 2020-08-25 10:01:26.574809 +0100 BST m=+1.241576679
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4370504.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 220853.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 37240.000

[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 219856.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4316568.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 36243.000
[2020-08-25 10:01:20 +0100 BST][S] 'nomad-autoscaler.runtime.gc_pause_ns': Count: 5 Min: 38083.000 Mean: 69764.400 Max: 122291.000 Stddev: 31487.808 Sum: 348822.000 LastUpdated: 2020-08-25 10:01:26.574809 +0100 BST m=+1.241576679
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4370504.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 220853.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 37240.000

»Runtime Metrics

The runtime metrics help understand the Nomad Autoscaler agent's memory and load pressure performance.

Metric	Description	Type
`nomad-autoscaler.runtime.num_goroutines`	Number of running goroutines	Gauge
`nomad-autoscaler.runtime.alloc_bytes`	The number of allocated heap bytes	Gauge
`nomad-autoscaler.runtime.sys_bytes`	The total bytes of memory obtained from the OS	Gauge
`nomad-autoscaler.runtime.malloc_count`	Cumulative count of heap objects allocated	Gauge
`nomad-autoscaler.runtime.free_count`	Cumulative count of heap objects freed	Gauge
`nomad-autoscaler.runtime.heap_objects`	Number of allocated heap objects	Gauge
`nomad-autoscaler.runtime.total_gc_pause_ns`	Cumulative nanoseconds in GC stop-the-world pauses	Gauge
`nomad-autoscaler.runtime.total_gc_runs`	Number of completed GC cycles	Gauge
`nomad-autoscaler.runtime.gc_pause_ns`	Number of nanoseconds to complete the last GC cycle	Timer

»Policy Metrics

Policy metrics provide insights into the performance of the Nomad Autoscaler's policy handling.

Metric	Description	Type	Labels
`nomad-autoscaler.policy.total_num`	The number of policies currently held within the autoscaler	Gauge
`nomad-autoscaler.policy.source.error_count`	Tracks the number of errors generated by the policy sources	Counter	policy_source

»Scaling Metrics

Scaling metrics provide insight into the performance of scaling actions as well as overall success and failure counters.

Metric	Description	Type	Labels
`nomad-autoscaler.scale.evaluate_ms`	The time taken to evaluate the checks within a single policy	Timer	policy_id, target_name
`nomad-autoscaler.scale.invoke_ms`	The time taken to invoke scaling based on the scaling evaluations	Timer	policy_id, target_name
`nomad-autoscaler.scale.invoke.success_count`	Tracks the number of successful scaling actions triggered	Counter
`nomad-autoscaler.scale.invoke.error_count`	Tracks the number of unsuccessful scaling actions triggered	Counter

»Plugin Metrics

Plugin metrics provide insight into the performance of Nomad Autoscaler plugins and help identify potential bottle necks or latency issues.

Metric	Description	Type	Labels
`nomad-autoscaler.plugin.manager.access_ms`	The time taken to dispense a plugin	Timer
`nomad-autoscaler.target.status.invoke_ms`	The time taken to perform the target plugin status call	Timer	policy_id, plugin_name
`nomad-autoscaler.target.scale.invoke_ms`	The time taken to perform the target plugin scale call	Timer	policy_id, plugin_name
`nomad-autoscaler.apm.query.invoke_ms`	The time taken to perform the APM plugin query call	Timer	policy_id, plugin_name
`nomad-autoscaler.strategy.run.invoke_ms`	The time taken to perform the strategy plugin run call	Timer	policy_id, plugin_name