Container Network Interface (CNI)

Nomad has built-in support for scheduling compute resources such as CPU, memory, and networking. Nomad's network plugin support extends this to allow scheduling tasks with purpose-created or specialty network configurations. Network plugins are third-party plugins that conform to the Container Network Interface (CNI) specification.

Network plugins need to be installed and configured on each client. The Nomad installation instructions recommend installing the CNI reference plugins because certain Nomad networking features, like bridge network mode and Consul service mesh, leverage them to provide an operating-system agnostic interface to configure workload networking.

Custom networking in Nomad is accomplished with a combination of CNI plugin binaries and CNI configuration files.

CNI reference plugins

The CNCF maintains a set of reference CNI plugins for basic network operations. These plugins are required for Nomad's bridge networking mode and for integrating with Consul service mesh.

See the Linux post-install steps for installing CNI reference plugins.

To use Nomad's transparent_proxy feature, you will also need the consul-cni plugin.

CNI plugins

Spec-compliant plugins should work with Nomad, however, it's possible a plugin vendor has implemented their plugin to make non-standard API calls, or it is otherwise non-compliant with the CNI specification. In those situations the plugin may not function correctly in a Nomad environment. You should verify plugin compatibility with Nomad before deploying in production.

CNI plugins are installed and configured on a per-client basis. Nomad consults the path given in the client's cni_path to find CNI plugin executables.

CNI configuration files

The CNI specification defines a network configuration format for administrators. It contains directives for both the orchestrator and the plugins to consume. At plugin execution time, this configuration format is interpreted by Nomad and transformed into arguments for the plugins.

Nomad reads the following extensions from the cni_config_dir— /opt/cni/config by default:

.conflist files are loaded as network configurations that contain a list of plugin configurations.
.conf and .json files are loaded as individual plugin configurations for a specific network.

Using CNI networks with Nomad jobs

To specify that a job should use a CNI network, set the task group's network mode attribute to the value cni/<your_cni_config_name>. Nomad will then schedule the workload on client nodes that have fingerprinted a CNI configuration with the given name. For example, to use the configuration named mynet, you should set the task group's network mode to cni/mynet. Nodes that have a network configuration defining a network named mynet in their cni_config_dir will be eligible to run the workload.

Nomad's `bridge` configuration

Nomad itself uses CNI plugins and configuration as the underlying implementation for the bridge network mode, using the loopback, bridge, firewall, and portmap CNI plugins configured together to create Nomad's bridge network.

The following is the configuration template Nomad uses when setting up these networks with the template placeholders replaced with the default configuration values for bridge_network_name, bridge_network_subnet, and an internal constant that provides the value for iptablesAdminChainName. This is a convenient jumping off point to discuss a worked example.

{  "cniVersion": "0.4.0",  "name": "nomad",  "plugins": [    {      "type": "loopback"    },    {      "type": "bridge",      "bridge": "nomad",      "ipMasq": true,      "isGateway": true,      "forceAddress": true,      "hairpinMode": false,      "ipam": {        "type": "host-local",        "ranges": [          [            {              "subnet": "172.26.64.0/20"            }          ]        ],        "routes": [          { "dst": "0.0.0.0/0" }        ]      }    },    {      "type": "firewall",      "backend": "iptables",      "iptablesAdminChainName": "NOMAD-ADMIN"    },    {      "type": "portmap",      "capabilities": {"portMappings": true},      "snat": true    }  ]}

For a more thorough understanding of this configuration, consider each CNI plugin's configuration in turn.

loopback

The loopback plugin sets the default local interface, lo0, created inside the bridge network's network namespace to UP. This allows workload running inside the namespace to bind to a namespace-specific loopback interface.

bridge

The bridge plugin creates a bridge (virtual switch) named nomad that resides in the host network namespace. Because this bridge is intended to provide network connectivity to allocations, it's configured to be a gateway by setting isGateway to true. This tells the plugin to assign an IP address to the bridge interface

The bridge plugin connects allocations (on the same host) into a bridge (virtual switch) that resides in the host network namespace. By default Nomad creates a single bridge for each client. Since Nomad's bridge network is designed to provide network connectivity to the allocations, it configures the bridge interface to be a gateway for outgoing traffic by providing it with an address using an ipam configuration. The default configuration creates a host-local address for the host side of the bridge in the 172.26.64.0/20 subnet at 172.26.64.1. When associating allocations to the bridge, it creates addresses for the allocations from that same subnet using the host-local plugin. The configuration also specifies a default route for the allocations of the host-side bridge address.

firewall

The firewall plugin creates firewall rules to allow traffic to/from the allocation's IP address via the host network. Nomad uses the iptables backend for the firewall plugin. This configuration creates two new iptables chains, CNI-FORWARD and NOMAD-ADMIN, in the filter table and add rules that allow the given interface to send/receive traffic.

The firewall creates an admin chain using the name provided in the iptablesAdminChainName attribute. For this case, it's called NOMAD-ADMIN. The admin chain is a user-controlled chain for custom rules that run before rules managed by the firewall plugin. The firewall plugin does not add, delete, or modify rules in the admin chain.

A new chain, CNI-FORWARD is added to the FORWARD chain. CNI-FORWARD is the chain where rules will be added when allocations are created and from where rules will be removed when those allocations stop. The CNI-FORWARD chain first sends all traffic to NOMAD-ADMIN chain.

You can use the following command to list the iptables rules present in each chain.

$ sudo iptables -L

portmap

Nomad needs to be able to map specific ports from the host to tasks running in the allocation namespace. The portmap plugin forwards traffic from one or more ports on the host to the allocation using network address translation (NAT) rules.

The plugin sets up two sequences of chains and rules:

One “primary” DNAT (destination NAT) sequence to rewrite the destination.
One SNAT (source NAT) sequence that will masquerade traffic as needed.

You can use the following command to list the iptables rules in the NAT table.

$ sudo iptables -t nat -L

Create your own

You can use this template as a basis for your own CNI-based bridge network configuration in cases where you need access to an unsupported option in the default configuration, like hairpin mode. When making your own bridge network based on this template, ensure that you change the iptablesAdminChainName to a unique value for your configuration.