Welcome to the Agent Control codebase! This is intended to be an entry point for the high-level documentation of this project.
If you are a new team member for Agent Control or even New Relic as a whole, or from another team wanting to integrate your product or agent with New Relic Control, keep reading! If you are acquainted with Agent Control already and want to push changes to the codebase, please read CONTRIBUTING.md.
Rust API reference: the rendered
rustdocfor the workspace crates is published at https://newrelic.github.io/newrelic-agent-control/. Build it locally withmake doc-open.
Agent Control (AC) is a tool intended to be installed in customer’s hardware, be it used as a traditional server (physical or virtual machines, cloud computing resources such as AWS EC2, what we usually call on-host) or a Kubernetes cluster. From there, it orchestrates the lifecycle of instrumentation agents supported by New Relic for that kind of hardware, like the infrastructure agent, the OpenTelemetry Collector, APM agents or Kubernetes-specific integrations.
Where each agent could be installed using their own means and managed separately before, AC now enables a control plane for the customer, exposing a unified interface for managing all of these observability agents at scale. This control plane can be managed remotely from the New Relic platform via Fleet Control (FC). The set of instrumentation agents managed by AC on a certain server or Kubernetes cluster is called a fleet.
The controlled observability agents will still send their Metrics, Events, Logs and Traces (MELT) on their own terms, as expected of them.
Receiving instructions from either static configuration or from FC, AC will create workloads for the specified observability agents of the fleet covering the customer’s observability needs. On traditional servers, these workloads will usually be simply processes with their configurations for the installed agents, as Operating System threads spawned by AC. On Kubernetes, these will be the appropriate and equivalent Kubernetes resources, usually in the form of a Helm release.
The configuration of these agents, and even the agents themselves that compose the fleet, can be modified by interacting with FC. AC will receive the remote configuration from FC and apply the needed changes on the fleet.
Besides creating, modifying and deleting these workloads, AC is also capable of checking their health and their current configuration (as AC sub-agents, see Applying configurations in the “Integrating with Agent Control” section). It will periodically collect and send this information back to FC, so customers can have a quick glance at the general status of their observability fleet deployment. This health information can also be checked locally by interacting with the AC process via an HTTP endpoint (by default at localhost:51200/status).
The communication between AC and FC is done over HTTPS and is generally compliant with the Open Agent Management Protocol (OpAMP), which is, on their own words:
[…] a network protocol for remote management of large fleets of data collection Agents.
We maintain our own Rust library for OpAMP clients on a separate repository: newrelic-opamp-rs.
From the point of view of OpAMP, AC is also an agent. That’s why, even though each workload will have its own OpAMP communication channel when managed by AC, we commonly refer to the various observability workloads that AC can manage as sub-agents.
[!IMPORTANT] Our reliance on OpAMP is especially important, since its specification informs much of the current and planned capabilities of Agent Control (like package downloading, see below) and its relationship with Fleet Control. After the installation process, OpAMP over HTTPS is the only means of network communication used by the AC process as of now.
Given the nature of Kubernetes resources, Agent Control is capable of downloading arbitrary versions of the observability agents it is managing. For on-host, arbitrary agent downloads and upgrades is not yet supported and AC will only manage the agents already present on the system, be it because they were declared dependencies on the package manager used to install AC (so were installed alongside it), or because they were available before.
Go to how ac remote update works.
The following diagram captures the essentials of what is explained above.
%%{ init: { 'theme': 'neutral' }}%%
flowchart TB
classDef central fill:#00E580;
subgraph New Relic
subgraph FC [Fleet Control]
F@{ label: FC Deployment }
ACRC@{ shape: doc, label: AC remote configuration}
WRC@{ shape: docs, label: Agents remote configurations}
end
M@{ shape: procs, label: MELT endpoints }
end
subgraph C [Customer domain]
SC@{ shape: doc, label: Static Configuration}
W@{ shape: procs, label: "Observability agent workloads (processes, Kubernetes resources)"}
subgraph AC [Agent Control]
A@{ label: Agent Control process }
S@{ shape: procs, label: Supervisors}
H@{ shape: procs, label: Health statuses}
E@{ shape: procs, label: Effective configs}
end
end
class A central
class F central
F -->|makes available| ACRC & WRC
SC & ACRC -->|is read by| A
WRC -->|are read by| S
A -->|creates| S
S -->|manage| W
S -->|check status of| W
S -->|determine| H & E
H & E -->|are reported to| F
W -->|send metrics to| M
The avid reader might be asking at this point “How does AC know the way to create the resources for a particular agent?”, or perhaps the more interesting “How can I teach AC to deploy my agent?” To find answers for these pressing questions, head to the document for Integrating with Agent Control.
For the standard installation of Agent Control, either on-host or Kubernetes, head to the official documentation on New Relic’s website.
For configuration details refer to CONFIG.md.
For details related to developing for AC, like setting up the developer environment, some implementation details, troubleshooting, etc, go to DEVELOPMENT.md.
As mentioned above, if you are interested in making AC capable of managing your own agents, please go to Integrating with Agent Control. Take into account that, as of now, a separate effort must be done for FC. That ensures your agent can be properly represented in New Relic’s web UI and remote configs can be exposed for AC to retrieve.
This section lists all external endpoints that AC and the agents it manages connect to. Use this as a reference when configuring firewall allowlists.
These are the endpoints the AC process itself directly communicates with at runtime.
| Endpoint | Port | Protocol | Purpose |
|---|---|---|---|
opamp.service.newrelic.com |
443 | HTTPS | Fleet Control communication — US region |
opamp.service.eu.newrelic.com |
443 | HTTPS | Fleet Control communication — EU region |
opamp.service.jp.newrelic.com |
443 | HTTPS | Fleet Control communication — JP region |
| Endpoint | Port | Protocol | Purpose |
|---|---|---|---|
system-identity-oauth.service.newrelic.com |
443 | HTTPS | OAuth2 token renewal (used by US, EU, and JP regions) |
AC uses OCI registries as storage for different types of artifacts, including on-host agent packages, and other AC custom artifacts, having the default registry set to docker.io. By default, AC will pull these artifacts from the newrelic namespace on Docker Hub (docker.io/newrelic).
Pulling from Docker Hub involves more than the docker.io hostname itself. The table below lists the endpoints Docker Hub pulls typically go through:
| Endpoint | Port | Protocol | Purpose |
|---|---|---|---|
registry-1.docker.io |
443 | HTTPS | Docker Hub registry API (image manifests and layers) |
auth.docker.io |
443 | HTTPS | Docker Hub authentication (pull tokens) |
production.cloudfront.docker.com |
443 | HTTPS | Docker Hub layer storage (CDN-backed blob storage) |
Note: these endpoints are owned and operated by Docker, not New Relic, and are subject to change without notice on Docker’s end. For the authoritative, up-to-date list, refer to Docker’s own allowlist documentation.
AC fetches public keys (JWKS) from these endpoints to verify the signatures of New Relic-issued artifacts, including remote configurations and OCI agent packages.
| Endpoint | Port | Protocol | Purpose |
|---|---|---|---|
publickeys.newrelic.com |
443 | HTTPS | Public key fetch for signature verification — US region |
publickeys.eu.newrelic.com |
443 | HTTPS | Public key fetch for signature verification — EU region |
publickeys.jp.newrelic.com |
443 | HTTPS | Public key fetch for signature verification — JP region |
If AC’s built-in OpenTelemetry self-instrumentation is enabled (see CONFIG.md), AC sends its own telemetry to:
| Endpoint | Port | Protocol | Purpose |
|---|---|---|---|
otlp.nr-data.net |
4317 | HTTPS/gRPC | AC self-telemetry (OTLP) — US region |
otlp.eu01.nr-data.net |
4317 | HTTPS/gRPC | AC self-telemetry (OTLP) — EU region |
otlp.jp.nr-data.net |
4317 | HTTPS/gRPC | AC self-telemetry (OTLP) — JP region |
AC uses cloud provider instance metadata services to enrich its own telemetry with resource attributes.
| Endpoint | Protocol | Cloud provider |
|---|---|---|
169.254.169.254 |
HTTP | AWS EC2, Azure |
metadata.google.internal |
HTTP | Google Cloud (GCP) |
The following endpoints are only required when running AC on Kubernetes, where it manages agents via Helm releases and OCI image pulls.
| Endpoint | Port | Protocol | Purpose |
|---|---|---|---|
helm-charts.newrelic.com |
443 | HTTPS | Default New Relic Helm chart repository |
For a complete and up-to-date list of endpoints that New Relic agents connect to, refer to the New Relic networks documentation.