»Remote Task Drivers

Nomad 1.1.0 introduces support for Remote Task Drivers. Remote Task Drivers allow custom task driver plugins to execute tasks using nonlocal runtimes such as cloud container runtimes. Without this support, task driver plugins trying to manage remote tasks would run into the following problems:

  1. When draining a node, Nomad stops all allocations on that node before rescheduling them.

  2. When a node is down, Nomad reschedules all allocations onto other nodes.

These 2 behaviors are optimal for traditional task drivers where the task process is colocated with the Nomad agent. If the Nomad node is down or drained, the allocations should be considered down or be drained as well.

However these 2 behaviors do not apply to tasks executing on remote runtimes. If the Nomad node managing them goes down, a new Nomad node should be able to manage them without restarting the task. Likewise if the Nomad node managing the remote task is drained, a new Nomad node should manage the remote task without requiring it be stopped and restarted.

The Remote Task Driver feature in Nomad 1.1.0 improves these behaviors for custom plugins that advertise the RemoteTasks capability.

»Caveats

Due to the exerpimental nature there are a number of standard Nomad features which Remote Task Drivers do not support by default.

»Resources

The resources stanza has not been altered for remote task drivers. Since remote tasks do not consume local resources, remote task drivers should not use the existing resources stanza and instead implement their own resource parameters in their task.config block.

Jobs using remote task drivers should use the minimum allowed resources in their task.resources stanza:

resources {
  cpu    = 1
  memory = 10
}
resources {  cpu    = 1  memory = 10}

»Nomad Client Features

Remote task drivers defer most Nomad client features to the driver plugin.

Since the allocation directory is local to the Nomad node, unless a remote task driver is able to remotely mount or copy its contents, the following features will be unavailable:

  • artifact - artifacts are downloaded to the local Nomad allocation directory.
  • dispatch_payload - dispatch payloads are written to the local Nomad allocation directory.
  • ephemeral_disk - ephemeral disks are local to the Nomad node and therefore not applicable to remote task drivers.
  • template - templates are rendered in the local Nomad allocation directory.
  • vault - a secret token may be retrieved but the task driver may not place the token file in the expected location.
  • volume - volume and volue mounts assumed tasks have access to the Nomad node's local disk are unlikely to work with remote task drivers.

Furthermore since networking is completely handled by the remote runtime the behavior of the following features is completely driver dependent:

  • network - group networks are created on the local Nomad node, and task networks are up to the remote task driver to implement.
  • connect - since group networks and sidecars are local to the Nomad node, Consul Connect sidecars will not work as expected.