Search Nomad documentation » The Nomad Autoscaler agent collects various runtime metrics about the performance
of different libraries and subsystems. These metrics are aggregated on a ten
second interval and are retained for one minute. To configure the telemetry output
please see the agent configuration .
This data can be accessed via the /v1/metrics HTTP endpoint, via sending a
signal to the Nomad Autoscaler process or via a number of integrations.
To view this data via sending a signal to the Nomad Autoscaler process: on Unix,
this is USR1 while on Windows it is BREAK. Once Nomad Autoscaler receives
the signal, it will dump the current telemetry information to the agent's stderr.
This telemetry information can be used for debugging or otherwise
getting a better view of what Nomad is doing.
Below is sample output of a telemetry dump:
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 219856.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4316568.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 36243.000
[2020-08-25 10:01:20 +0100 BST][S] 'nomad-autoscaler.runtime.gc_pause_ns': Count: 5 Min: 38083.000 Mean: 69764.400 Max: 122291.000 Stddev: 31487.808 Sum: 348822.000 LastUpdated: 2020-08-25 10:01:26.574809 +0100 BST m=+1.241576679
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4370504.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 220853.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000
[2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 37240.000
[2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 219856.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4316568.000 [2020-08-25 10:01:20 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 36243.000 [2020-08-25 10:01:20 +0100 BST][S] 'nomad-autoscaler.runtime.gc_pause_ns': Count: 5 Min: 38083.000 Mean: 69764.400 Max: 122291.000 Stddev: 31487.808 Sum: 348822.000 LastUpdated: 2020-08-25 10:01:26.574809 +0100 BST m=+1.241576679 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.alloc_bytes': 4370504.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.malloc_count': 220853.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.free_count': 183613.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.policy.total_num': 0.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.num_goroutines': 12.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_pause_ns': 348822.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.total_gc_runs': 5.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.sys_bytes': 74793216.000 [2020-08-25 10:01:30 +0100 BST][G] 'nomad-autoscaler.pathfinder.runtime.heap_objects': 37240.000 » The runtime metrics help understand the Nomad Autoscaler agent's memory and load
pressure performance.
Metric Description Type nomad-autoscaler.runtime.num_goroutinesNumber of running goroutines Gauge nomad-autoscaler.runtime.alloc_bytesThe number of allocated heap bytes Gauge nomad-autoscaler.runtime.sys_bytesThe total bytes of memory obtained from the OS Gauge nomad-autoscaler.runtime.malloc_countCumulative count of heap objects allocated Gauge nomad-autoscaler.runtime.free_countCumulative count of heap objects freed Gauge nomad-autoscaler.runtime.heap_objectsNumber of allocated heap objects Gauge nomad-autoscaler.runtime.total_gc_pause_nsCumulative nanoseconds in GC stop-the-world pauses Gauge nomad-autoscaler.runtime.total_gc_runsNumber of completed GC cycles Gauge nomad-autoscaler.runtime.gc_pause_nsNumber of nanoseconds to complete the last GC cycle Timer
» Policy metrics provide insights into the performance of the Nomad Autoscaler's
policy handling.
Metric Description Type Labels nomad-autoscaler.policy.total_numThe number of policies currently held within the autoscaler Gauge nomad-autoscaler.policy.source.error_countTracks the number of errors generated by the policy sources Counter policy_source
» Scaling metrics provide insight into the performance of scaling actions as well
as overall success and failure counters.
Metric Description Type Labels nomad-autoscaler.scale.evaluate_msThe time taken to evaluate the checks within a single policy Timer policy_id, target_name nomad-autoscaler.scale.invoke_msThe time taken to invoke scaling based on the scaling evaluations Timer policy_id, target_name nomad-autoscaler.scale.invoke.success_countTracks the number of successful scaling actions triggered Counter nomad-autoscaler.scale.invoke.error_countTracks the number of unsuccessful scaling actions triggered Counter
» Plugin metrics provide insight into the performance of Nomad Autoscaler plugins
and help identify potential bottle necks or latency issues.
Metric Description Type Labels nomad-autoscaler.plugin.manager.access_msThe time taken to dispense a plugin Timer nomad-autoscaler.target.status.invoke_msThe time taken to perform the target plugin status call Timer policy_id, plugin_name nomad-autoscaler.target.scale.invoke_msThe time taken to perform the target plugin scale call Timer policy_id, plugin_name nomad-autoscaler.apm.query.invoke_msThe time taken to perform the APM plugin query call Timer policy_id, plugin_name nomad-autoscaler.strategy.run.invoke_msThe time taken to perform the strategy plugin run call Timer policy_id, plugin_name
