June 20-22 Announcing HashiConf Europe full schedule: keynotes, sessions, labs & more Register Now
  • Overview
    • Batch Processing Workloads
    • Edge Workload Management
    • Non-Containerized Application Orchestration
    • Simple Container Orchestration
  • Enterprise
  • Tutorials
  • Docs
  • API
  • Plugins
  • Tools
  • Community
GitHub—Stars on GitHub
Download
    • v1.3.x (latest)
    • v1.2.x
    • v1.1.x
    • v1.0.x
    • v0.12.x
    • v0.11.x
    • Overview
    • Quickstart
      • Overview
      • Requirements
      • Reference Architecture
      • Deployment Guide
    • Windows Service
    • Overview
    • Specific Version Details
    • Overview
    • Consul
    • Consul Service Mesh
    • Vault Integration

    • Overview
    • Architecture
      • Overview
      • Base
      • Task Drivers
      • Devices
      • Storage
      • Overview
      • Internals
      • Preemption
    • Consensus Protocol
    • Filesystem
    • Gossip Protocol
    • Security Model
    • Overview
    • acl
    • audit
    • autopilot
    • client
    • consul
    • plugin
    • sentinel
    • search
    • server
    • server_join
    • telemetry
    • tls
    • ui
    • vault
    • Overview
      • Overview
      • bootstrap
      • policy apply
      • policy delete
      • policy info
      • policy list
      • token create
      • token delete
      • token info
      • token list
      • token self
      • token update
    • agent
    • agent-info
      • Overview
      • exec
      • fs
      • logs
      • restart
      • signal
      • status
      • stop
      • Overview
      • validate
      • Overview
      • fail
      • list
      • pause
      • promote
      • resume
      • status
      • unblock
      • Overview
      • list
      • status
      • Overview
      • allocs
      • deployments
      • dispatch
      • eval
      • history
      • init
      • inspect
      • plan
      • periodic force
      • promote
      • revert
      • run
      • scale
      • scaling-events
      • status
      • stop
      • validate
      • Overview
      • get
    • monitor
      • Overview
      • apply
      • delete
      • inspect
      • list
      • status
      • Overview
      • config
      • drain
      • eligibility
      • status
      • Overview
      • api
      • autopilot get-config
      • autopilot set-config
      • debug
      • keygen
      • keyring
      • metrics
      • raft info
      • raft list-peers
      • raft logs
      • raft remove-peer
      • raft state
      • snapshot agent
      • snapshot inspect
      • snapshot restore
      • snapshot save
      • snapshot state
      • Overview
      • status
      • Overview
      • apply
      • delete
      • init
      • inspect
      • list
      • status
      • Overview
      • apply
      • dismiss
      • info
      • list
      • Overview
      • policy info
      • policy list
      • Overview
      • apply
      • delete
      • list
      • read
      • Overview
      • force-leave
      • join
      • members
      • Overview
      • service delete
      • service info
      • service list
    • status
      • Overview
      • gc
      • reconcile summaries
    • ui
    • version
      • Overview
      • create
      • delete
      • deregister
      • detach
      • init
      • register
      • snapshot create
      • snapshot delete
      • snapshot list
      • status

    • Overview
      • Overview
      • Expressions
        • Overview
          • chunklist
          • coalesce
          • coalescelist
          • compact
          • concat
          • contains
          • distinct
          • element
          • flatten
          • index
          • keys
          • length
          • lookup
          • merge
          • range
          • reverse
          • setintersection
          • setproduct
          • setunion
          • slice
          • sort
          • values
          • zipmap
          • can
          • convert
          • try
          • bcrypt
          • md5
          • rsadecrypt
          • sha1
          • sha256
          • sha512
          • formatdate
          • timeadd
          • base64decode
          • base64encode
          • csvdecode
          • jsondecode
          • jsonencode
          • urlencode
          • yamldecode
          • yamlencode
          • abspath
          • basename
          • dirname
          • file
          • fileexists
          • fileset
          • pathexpand
          • cidrhost
          • cidrnetmask
          • cidrsubnet
          • cidrsubnets
          • abs
          • ceil
          • floor
          • log
          • max
          • min
          • parseint
          • pow
          • signum
          • chomp
          • format
          • formatlist
          • indent
          • join
          • lower
          • regex_replace
          • replace
          • split
          • strrev
          • substr
          • title
          • trim
          • trimprefix
          • trimspace
          • trimsuffix
          • upper
          • uuidv4
          • uuidv5
      • Locals
      • Syntax
      • Variables
    • artifact
    • affinity
    • check_restart
    • connect
    • constraint
    • csi_plugin
    • device
    • dispatch_payload
    • env
    • ephemeral_disk
    • expose
    • gateway
    • group
    • job
    • lifecycle
    • logs
    • meta
    • migrate
    • multiregion
    • network
    • parameterized
    • periodic
    • proxy
    • reschedule
    • resources
    • restart
    • scaling
    • service
    • sidecar_service
    • sidecar_task
    • spread
    • task
    • template
    • update
    • upstreams
    • vault
    • volume
    • volume_mount
    • Overview
    • Docker
    • Isolated Fork/Exec
    • Java
    • Podman
    • QEMU
    • Raw Fork/Exec
      • Overview
      • containerd
      • Firecracker driver
      • Jailtask driver
      • Lightrun
      • LXC
      • Pot
      • Rkt Deprecated
      • Rookout
      • Singularity
      • systemd-nspawn
      • Windows IIS
      • Overview
      • ECS
    • Overview
      • Overview
      • Nvidia
      • USB Beta
  • Schedulers
    • Overview
    • Runtime Environment
    • Variable Interpolation
    • Overview
      • Overview
      • apm
      • dynamic_application_sizing
      • http
      • nomad
      • policy
      • policy_eval
      • source
      • strategy
      • target
      • telemetry
    • API
    • CLI
    • Policy
    • Telemetry
      • Overview
        • Overview
        • Datadog
        • Nomad API
        • Prometheus
        • Overview
        • Dynamic Application Sizing Average
        • Dynamic Application Sizing Max
        • Dynamic Application Sizing Percentile
        • Fixed Value
        • Pass-Through
        • Target Value
        • Threshold
        • Overview
        • Amazon Web Services Autoscaling Group
        • Azure Virtual Machine Scale Set
        • Dynamic Application Sizing
        • Google Cloud Engine Managed Instance Group
        • Nomad Task Group
      • Community
      • Overview
      • Checks
      • Node Selector Strategy
        • Overview
        • Base
        • APM
        • Strategy
        • Target
    • Overview
    • Operating Nomad Agents
    • Monitoring Nomad
    • Metrics Reference
    • Cluster Management
    • Transport Security
    • Access Control

    • Overview
    • Alternative to Kubernetes
    • Supplement to Kubernetes
  • Nomad Ecosystem
  • Nomad Partnerships
  • Who Uses Nomad
    • Overview
      • Overview
      • FAQ
  • FAQ
Type '/' to Search

»Isolated Fork/Exec Driver

Name: exec

The exec driver is used to simply execute a particular command for a task. However, unlike raw_exec it uses the underlying isolation primitives of the operating system to limit the task's access to resources. While simple, since the exec driver can invoke any command, it can be used to call scripts or other wrappers which provide higher level features.

»Task Configuration

task "webservice" {
  driver = "exec"

  config {
    command = "my-binary"
    args    = ["-flag", "1"]
  }
}
task "webservice" {
  driver = "exec"

  config {
    command = "my-binary"
    args    = ["-flag", "1"]
  }
}

The exec driver supports the following configuration in the job spec:

  • command - The command to execute. Must be provided. If executing a binary that exists on the host, the path must be absolute and within the task's chroot. If executing a binary that is downloaded from an artifact, the path can be relative from the allocations's root directory.

  • args - (Optional) A list of arguments to the command. References to environment variables or any interpretable Nomad variables will be interpreted before launching the task.

  • pid_mode - (Optional) Set to "private" to enable PID namespace isolation for this task, or "host" to disable isolation. If left unset, the behavior is determined from the default_pid_mode in plugin configuration.

Warning: If set to "host", other processes running as the same user will be able to access sensitive process information like environment variables.

  • ipc_mode - (Optional) Set to "private" to enable IPC namespace isolation for this task, or "host" to disable isolation. If left unset, the behavior is determined from the default_ipc_mode in plugin configuration.

Warning: If set to "host", other processes running as the same user will be able to make use of IPC features, like sending unexpected POSIX signals.

  • cap_add - (Optional) A list of Linux capabilities to enable for the task. Effective capabilities (computed from cap_add and cap_drop) must be a subset of the allowed capabilities configured with allow_caps.
config {
  cap_add = ["net_raw", "sys_time"]
}
config {
  cap_add = ["net_raw", "sys_time"]
}
  • cap_drop - (Optional) A list of Linux capabilities to disable for the task. Effective capabilities (computed from cap_add and cap_drop) must be a subset of the allowed capabilities configured with allow_caps.
config {
  cap_drop = ["all"]
  cap_add  = ["chown", "sys_chroot", "mknod"]
}
config {
  cap_drop = ["all"]
  cap_add  = ["chown", "sys_chroot", "mknod"]
}

»Examples

To run a binary present on the Node:

task "example" {
  driver = "exec"

  config {
    # When running a binary that exists on the host, the path must be absolute.
    command = "/bin/sleep"
    args    = ["1"]
  }
}
task "example" {
  driver = "exec"

  config {
    # When running a binary that exists on the host, the path must be absolute.
    command = "/bin/sleep"
    args    = ["1"]
  }
}

To execute a binary downloaded from an artifact:

task "example" {
  driver = "exec"

  config {
    command = "name-of-my-binary"
  }

  artifact {
    source = "https://internal.file.server/name-of-my-binary"
    options {
      checksum = "sha256:abd123445ds4555555555"
    }
  }
}
task "example" {
  driver = "exec"

  config {
    command = "name-of-my-binary"
  }

  artifact {
    source = "https://internal.file.server/name-of-my-binary"
    options {
      checksum = "sha256:abd123445ds4555555555"
    }
  }
}

»Capabilities

The exec driver implements the following capabilities.

FeatureImplementation
nomad alloc signaltrue
nomad alloc exectrue
filesystem isolationchroot
network isolationhost, group
volume mountingall

»Client Requirements

The exec driver can only be run when on Linux and running Nomad as root. exec is limited to this configuration because currently isolation of resources is only guaranteed on Linux. Further, the host must have cgroups mounted properly in order for the driver to work.

If you are receiving the error:

* Constraint "missing drivers" filtered <> nodes
* Constraint "missing drivers" filtered <> nodes

and using the exec driver, check to ensure that you are running Nomad as root. This also applies for running Nomad in -dev mode.

»Plugin Options

  • default_pid_mode (string: optional) - Defaults to "private". Set to "private" to enable PID namespace isolation for tasks by default, or "host" to disable isolation.

Warning: If set to "host", other processes running as the same user will be able to access sensitive process information like environment variables.

  • default_ipc_mode (string: optional) - Defaults to "private". Set to "private" to enable IPC namespace isolation for tasks by default, or "host" to disable isolation.

Warning: If set to "host", other processes running as the same user will be able to make use of IPC features, like sending unexpected POSIX signals.

  • no_pivot_root (bool: optional) - Defaults to false. When true, the driver uses chroot for file system isolation without pivot_root. This is useful for systems where the root is on a ramdisk.

  • allow_caps - A list of allowed Linux capabilities. Defaults to

["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod",
 "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"]
["audit_write", "chown", "dac_override", "fowner", "fsetid", "kill", "mknod",
 "net_bind_service", "setfcap", "setgid", "setpcap", "setuid", "sys_chroot"]

which is modeled after the capabilities allowed by docker by default (without NET_RAW). Allows the operator to control which capabilities can be obtained by tasks using cap_add and cap_drop options. Supports the value "all" as a shortcut for allow-listing all capabilities supported by the operating system.

Warning: Allowing more capabilities beyond the default may lead to undesirable consequences, including untrusted tasks being able to compromise the host system.

»Client Attributes

The exec driver will set the following client attributes:

  • driver.exec - This will be set to "1", indicating the driver is available.

»Resource Isolation

The resource isolation provided varies by the operating system of the client and the configuration.

On Linux, Nomad will use cgroups, and a chroot to isolate the resources of a process and as such the Nomad agent must be run as root. Some Linux distributions do not boot with all required cgroups enabled by default. You can see which cgroups are enabled by reading /proc/cgroups, and verifying that all the following cgroups are enabled:

$ awk '{print $1 " " $4}' /proc/cgroups
#subsys_name enabled
cpuset 1
cpu 1
cpuacct 1
blkio 1
memory 1
devices 1
freezer 1
net_cls 1
perf_event 1
net_prio 1
hugetlb 1
pids 1
$ awk '{print $1 " " $4}' /proc/cgroups
#subsys_name enabled
cpuset 1
cpu 1
cpuacct 1
blkio 1
memory 1
devices 1
freezer 1
net_cls 1
perf_event 1
net_prio 1
hugetlb 1
pids 1

»Chroot

The chroot is populated with data in the following directories from the host machine:

[
  "/bin",
  "/etc",
  "/lib",
  "/lib32",
  "/lib64",
  "/run/resolvconf",
  "/sbin",
  "/usr",
]
[
  "/bin",
  "/etc",
  "/lib",
  "/lib32",
  "/lib64",
  "/run/resolvconf",
  "/sbin",
  "/usr",
]

The task's chroot is populated by linking or copying the data from the host into the chroot. Note that this can take considerable disk space. Since Nomad v0.5.3, the client manages garbage collection locally which mitigates any issue this may create.

This list is configurable through the agent client configuration file.

github logoEdit this page
DocsAPIResourcesPrivacySecurityPress KitConsent Manager