newrelic-agent-control

Agent Type Overview

Agent Type Definition is a YAML file that defines an agent’s configuration and behavior. It consists of three main sections: metadata, deployment, and variables.

By defining these three sections, developers can create a customizable and flexible agent type that can be used in various environments.

On top of those sections, every file declares a top-level protocol_version that versions the schema language the file is written against.

Protocol version

protocol_version is a top-level field — separate from the three sections below — that declares the version of the agent-type schema language itself: the set of fields and their meaning that Agent Control knows how to parse, including the shape of the metadata. It is decoupled from both the agent type version (the definition’s semver) and the Agent Control release version.

It is a quoted MAJOR.MINOR string (e.g. "1.0"). The value must be quoted, otherwise YAML parses 0.1 as a float and the field is rejected.

It is parsed and validated on its own, at the registry ingestion boundary, before the rest of the document is interpreted — so it can gate files whose metadata or other sections use a shape this Agent Control would not otherwise understand. Each Agent Control release understands a single maximum protocol version, and the protocol_version is treated as a single ordered MAJOR.MINOR value. The compatibility rules are:

For example, an Agent Control supporting 1.6 accepts everything up to 1.6 (including 0.9 and 1.0..=1.6) and rejects anything newer (1.7, 2.0, …).

Metadata

The metadata section contains information about the agent type: its name, version, namespace, and the target platform.

namespace: newrelic
name: com.newrelic.opentelemetry.collector
version: 0.0.1
platform: host
operating_system: linux
namespace: newrelic
name: com.newrelic.opentelemetry.collector
version: 0.0.1
platform: kubernetes

The platform (and operating_system when applicable) drives how the rest of the document is parsed: the deployment block is dispatched to the on-host or Kubernetes deserializer based on these fields.

Variables

The variables section allows developers to define variables that end users can set. These variables can adjust the agent’s or system’s configuration.

variables:
  config_agent:
    description: "Newrelic infra configuration"
    type: yaml
    required: false
    default: {}
    classification: config
  backoff_delay:
    description: "seconds until next retry if agent fails to start"
    type: string
    required: false
    default: 20s
    deprecated: true
  enable_file_logging:
    description: "enable logging the on host executables' logs to files"
    type: bool
    required: false
    default: false

Nested variable names are supported. For instance:

variables:
  log:
    level:
      description: "Log level"
      type: string
      required: false
      default: info

All variables have a few common attributes:

Both classification and deprecated are accepted and ignored by Agent Control — they carry no runtime behavior and don’t affect resolution, validation, or defaults.

In terms of variable types, we currently support the following types listed in this source file:

Deployment

The deployment section indicates how the agent should be executed and how its health should be checked.

Note you can reference the variables defined in the variables section using ${nr-var:variable_name}. And this is valid for nested variables as well: following the example above, you would be able to use ${nr-var:log.info}.

Template Functions

You can enhance templated variables by applying functions to them, enabling transformations as needed.

Functions are pipelined, meaning the output of each transformation serves as the input for the next one: ${nr-var:variable_name | func1 | func2 | ... | funcN}.

Indent(n)

The indent function indents each new line with n spaces. Essentially, it adds n spaces after each \n. For example, ${nr-var:key|indent 2} will prepend 2 spaces to the beginning of each line in the string produced by the variable.

This is particularly useful when rendering YAML inside a multiline string where the YAML being rendered requires specific indentation, as shown below:

multi_line_string: |
  fixed_key:
    ${nr-var:yaml_variable | indent 2 }

On Host Deployment

For on-host deployment (platform: host, operating_system: linux or windows), use the following format:

deployment:
  enable_file_logging: ${nr-var:enable_file_logging}
  health:
    interval: 5s
    timeout: 5s
    checks:
      - kind: Process
      - kind: Http
        path: "/v1/status"
        port: 8003
  executables:
    - id: newrelic-infra
      path: /usr/bin/newrelic-infra
      args:
        - --config
        - ${nr-var:config_agent}
      env: "NRIA_PLUGIN_DIR="${nr-sub:shared_filesystem_dir}/infra-agent-ohi-configs" NRIA_STATUS_SERVER_ENABLED=true"
      restart_policy:
        backoff_strategy:
          type: fixed
          backoff_delay: ${nr-var:backoff_delay}

In this section:

These diverse options offer extensive customization for your agent’s deployment.

Restart Policy

restart_policy provides a set of instructions on how and when the agent process should be restarted. It’s crucial for maintaining the agent’s availability and reliability, particularly in case of unexpected failures or problems.

In the backoff_strategy we have:

On Host Health

The health section in the deployment configuration is where you can specify how to monitor the health status of the agent. This is critical for maintaining the reliability of your agent and ensuring that it’s functioning correctly. It uses an explicit checks: list where every entry is discriminated by an explicit kind: field:

health:
    interval: 5s
    timeout: 5s
    checks:
      - kind: Process
      - kind: Http
        path: "/v1/status"
        port: 8003
        healthy_status_codes: [200, 203, 204]

In this configuration:

By finely tuning these parameters, developers can closely monitor the agent’s performance and address issues instantly. Adopting a robust health check strategy helps minimize downtime and keeps your system resilient and reliable.

Additionally, alternate protocols and interfaces can be mentioned under health - for instance, a cmd interface to run a command or script, or a file interface to read a specific file for agent status. However, as of current updates, these methods are not implemented yet.

# ...
health:
  interval: 30s
  timeout: 5s
  cmd:
    command: "newrelic-agent-control --status"
    healthy_codes: [0] 
    unhealthy_string: ".*(unhealthy|fatal|error).*"
# ...
health:
  interval: 30s
  timeout: 5s
  file:
    path: "/etc/newrelic-infra/health.lock"
    should_be_present: true
    unhealthy_string: ".*(unhealthy|fatal|error).*"

On Host Version

On-host agents do not define a version-check command. The agent version reported to Fleet Control (the agent.version identifying attribute) is derived from the OCI package the agent is deployed from, using the version configured under deployment.packages.<package_id>.download.oci.version:

deployment:
  packages:
    newrelic-infra:
      download:
        oci:
          repository: newrelic-infra
          version: ${nr-var:package_version}

The resolved value of that version field is published as agent.version.

When an agent type defines more than one package, the package to report is selected explicitly with the top-level deployment.reported_version_package field, which must name one of the declared packages:

deployment:
  reported_version_package: newrelic-infra   # required when more than one package is declared
  packages:
    newrelic-infra:
      download:
        oci:
          repository: newrelic-infra
          version: ${nr-var:package_version}
    nri-flex:
      download:
        oci:
          repository: nri-flex
          version: ${nr-var:flex_version}

Rules:

See On Host Packages for the full package configuration.

Kubernetes Deployment

The Agent Control leverages Flux to act as an operator running Helm commands (install, upgrade, delete) as needed based on the provided configurations.

Then, for a Kubernetes deployment (platform: kubernetes), we use the following format:

deployment:
  # See com.newrelic.infrastructure Agent type for description of fields.
  health:
    interval: 30s
  objects:
    repository:
      apiVersion: source.toolkit.fluxcd.io/v1
      kind: HelmRepository
      metadata:
        name: ${nr-sub:agent_id}
      spec:
        interval: 30m
        provider: generic
        url: https://helm-charts.newrelic.com
    release:
      apiVersion: helm.toolkit.fluxcd.io/v2
      kind: HelmRelease
      metadata:
        name: ${nr-sub:agent_id}
      spec:
        interval: 3m
        chart:
          spec:
            chart: nr-k8s-otel-collector
            version: ${nr-var:chart_version}
            sourceRef:
              kind: HelmRepository
              name: ${nr-sub:agent_id}
            interval: 3m
        install:
          disableWait: true
          disableWaitForJobs: true
          replace: true
        upgrade:
          disableWait: true
          disableWaitForJobs: true
          cleanupOnFail: true
          force: true
        values:
          ${nr-var:chart_values}

Kubernetes Objects

Repository

This is the K8s object whose kind is HelmRepository. It contains all the info to retrieve Helm charts.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: open-telemetry
  namespace: default
spec:
  interval: 1m
  url: https://open-telemetry.github.io/opentelemetry-helm-charts

Visit Flux API reference for v1 for detail spec information.

Release

This is the K8s object whose kind is HelmRelease. It has all the data to deploy a chart (version, values, tests, post-install actions).

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: otel-collector
  namespace: default
spec:
  interval: 1h0m0s
  chart:
    spec:
      chart: opentelemetry-collector
      version: '>=0.60.0 <1.0.0'
      sourceRef:
        kind: HelmRepository
        name: open-telemetry
        namespace: default
  releaseName: otel-collector
  targetNamespace: default
  values:
    mode: deployment

Visit Flux API reference for v2 for detail spec information.

Kubernetes Health

Kubernetes already implements Pod lifecycle, and Liveness, Readiness and Startup Probes mechanism which is a standard for all containers running on the cluster and can be used as a generic interface to understand the health of a sub agent.

Any agent deployed in Kubernetes can be composed of several components and those components deployed under different Pods and Replication Controllers. For instance, nri-kubernetes contains 1 DaemonSet and 2 Deployments.

That’s why the Agent Control leverages the Kubernetes Rust SDK to retrieve the health of standard replication controllers (Deployment, DaemonSet, StatefulSet) of the Agent at a configurable interval.

As a result, the health section for a Kubernetes deployment is as simple as this:

deployment:
  health:
    interval: 30s
  objects:
    ...

Users can currently only configure the interval of those periodic health check, within the Agent Type. However, in the future, we could offer the end users the possibility of selecting what information should be retrieved.

Kubernetes Version

Version is checked periodically by querying the corresponding k8s object in the cluster. The Agent Type allows setting up the version check interval and initial delay:

deployment:
  version:
    interval: 120s # Defaults to 60s..
    initial_delay: 10s # Defaults to 30s.

Development Custom agent types

Agent Control ships a set of embedded agent types (see agent-control/agent-type-registry/newrelic/). You can add your own — or override an embedded one — by placing the definition YAML in the dynamic agent types directory. Custom definitions take precedence over any other definition with the same id (namespace + name + version).

The current layout expects a directory at /etc/newrelic-agent-control/dynamic-agent-types/ containing one file per agent type.

Before dropping a definition into that directory, you can check its schema (required fields, field types, and format constraints) and semantics (every ${nr-var:X} reference in deployment has a matching variables declaration) without running Agent Control at all:

newrelic-agent-control-cli agent-type validate --file my-custom-agent-type.yaml

On-host

This guideline shows how to build a custom agent type and integrate it with the agent control on-host. The telegraf agent is used as a reference.

  1. Create a file with the agent type definition

     # namespace: newrelic, external, other
     namespace: external
     # name: reverse FQDN that uniquely identifies the agent type
     name: com.influxdata.telegraf
     # version: semver scheme
     version: 0.0.1
     # protocol_version: quoted MAJOR.MINOR of the agent-type schema language
     protocol_version: "1.0"
     # platform: host or kubernetes
     platform: host
     # operating_system: required when platform is host. linux or windows
     operating_system: linux
    
     # variables:
     #   my_var_1:
     #     description: "Variable description here"
     #     type: string
     #     required: false
     #     default: "default value"
    
     variables:
       config_file:
         description: "Telegraf config file path"
         type: string
         required: false
         default: "/path/to/telegraf.conf"
       backoff_delay:
         description: "seconds until next retry if agent fails to start"
         type: string
         required: false
         default: 20s
    
     deployment:
       executables:
         - id: telegraf
           path: /usr/bin/telegraf
           args:
             - --config
             - ${nr-var:config_file}
           env: ""
           restart_policy:
             backoff_strategy:
               type: fixed
               backoff_delay: ${nr-var:backoff_delay}
    
  2. Copy the agent type definition to the folder /etc/newrelic-agent-control/dynamic-agent-types
  3. Use the new type in the agents config for the agent control:

     # fleet_control:
     # ...
        
     agents:
       my-telegraf-collector:
         agent_type: "external/com.influxdata.telegraf:0.0.1"
    
  4. If any required variable has been defined in the type or any default value for variables needs to be customized, then define a values.yaml in /etc/newrelic-agent-control/fleet/agents.d/my-telegraf/values.yaml:

     config_file: /custom/path/to/file
     backoff_delay: 30s
    
  5. Restart Agent Control.

Kubernetes

In-cluster, the same /etc/newrelic-agent-control/dynamic-agent-types/ directory is populated by mounting a ConfigMap. Each ConfigMap key becomes a file inside the directory, so a single ConfigMap can carry multiple agent types.

  1. Create the ConfigMap in the namespace where Agent Control runs. The --from-file=<key>=<path> form names each entry; that <key> is the file name that will appear under the mount:

    kubectl create configmap dynamic-agent \
      --from-file=dynamic-agent-type=./agent-control/agent-type-registry/newrelic/kubernetes-com.newrelic.infrastructure-0.1.0.yaml \
      -n newrelic-agent-control
    

    For multiple types, append more --from-file=<key>=<path> — one per file.

  2. Mount that ConfigMap on the Agent Control pod through the deployment chart values, matching the layout used in the dynamic-agent-type k8s e2e (test/k8s-e2e/dynamic/ac-values-dynamic.yml):

    agentControlDeployment:
      chartValues:
        # [...]
        extraVolumeMounts:
          - name: dynamic
            mountPath: /etc/newrelic-agent-control/dynamic-agent-types
            readOnly: true
        extraVolumes:
          - name: dynamic
            configMap:
              name: dynamic-agent
    
  3. Reference the type by its id in the AC config and supply variable values through agentsConfig. Each key under an agent’s agentsConfig entry maps to a variable declared by the type:

    agentControlDeployment:
      chartValues:
        config:
          agents:
            infra:
              agent_type: "newrelic/com.newrelic.infrastructure:0.1.0"
     # [...]
    

    To pick up a fresh ConfigMap, helm upgrade (or restart the AC pod).