> ## Documentation Index
> Fetch the complete documentation index at: https://filetrace.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Filetrace metrics with Prometheus

> Enable the agent's metrics page and scrape it with Prometheus: listen address, scrape job, every metric and label, PromQL examples and how series stay bounded.

The Filetrace agent can serve a page of metrics in the Prometheus text format: bytes that sockets sent and received per process, file bytes sent per transfer method and confidence, transfers started and in progress, open connections, lost events, and the agent's own CPU time and memory. This page shows how to turn the page on and scrape it, lists every metric and label, gives PromQL examples, and explains how the number of series stays bounded.

## Turn on the metrics page

**1. Enable the page in the configuration file**

Add the `[prometheus]` section to `/etc/filetrace/config.toml`:

```toml theme={null}
[prometheus]
enabled = true
listen = "127.0.0.1:9464"
```

**2. Restart the agent**

The agent reads the file only when it starts:

```bash theme={null}
sudo systemctl restart filetrace-agent
```

The agent's log then reports `serving metrics on http://127.0.0.1:9464/metrics`.

**3. Fetch the page**

```bash theme={null}
curl -s http://127.0.0.1:9464/metrics
```

## Listen address

`listen` is an address and a port. The address is numeric: an IPv4 address, or an IPv6 address in brackets such as `[::1]:9464`. `localhost` stands for `127.0.0.1`; no other host name is resolved. The page is served over TCP only, so a `unix:` path is refused.

The default, `127.0.0.1:9464`, accepts connections from the same machine only. For a Prometheus server on another machine, set `listen` to an address of an interface that server can reach.

> [!WARNING]
>
> The page is served without authentication and without TLS. Anyone who can reach the address can read process names and traffic volumes. Keep it on the loopback address or on a network that only your monitoring can reach.

The agent checks the address when it starts. An address that is not in one of these forms is a configuration error (exit code 6), and an address the agent cannot listen on, for example because the port is taken, stops the agent with exit code 8 and the message `Cannot serve metrics on` followed by the address and the reason. See [exit codes](https://filetrace.mintlify.site/reference/exit-codes.md).

## HTTP behaviour

| Request | Response |
| - | - |
| `GET /metrics` | `200 OK` with the page, `Content-Type: text/plain; version=0.0.4; charset=utf-8` |
| `GET` of any other path | `404 Not Found` |
| Any other method | `405 Method Not Allowed` |

* A query string is ignored: `/metrics?name=value` returns the page.
* The page is built from the agent's state at each request; nothing is cached between scrapes.
* Every response closes the connection (`Connection: close`).
* The agent serves at most 8 connections at a time, and a connection that has not received its response within 5 seconds is closed. A request whose headers exceed 2 KB is dropped.

## Scrape configuration

A minimal Prometheus scrape job for an agent on the same host:

```yaml theme={null}
scrape_configs:
  - job_name: filetrace
    static_configs:
      - targets: ["127.0.0.1:9464"]
```

Prometheus requests `/metrics` unless `metrics_path` says otherwise, so the job needs no path. To scrape several hosts, list each host's `listen` address under `targets`. The `host_name` label of `target_info` names the host each target runs on.

## Metrics

| Metric | Type | Labels | Help text |
| - | - | - | - |
| `filetrace_network_io_bytes_total` | counter | `process_executable_name`, `network_transport`, `network_io_direction` | Bytes that local sockets accepted from processes and delivered to them. |
| `filetrace_file_sent_bytes_total` | counter | `process_executable_name`, `filetrace_transfer_method`, `filetrace_transfer_confidence` | File bytes sent over the network. |
| `filetrace_transfers_started_total` | counter | `process_executable_name`, `filetrace_transfer_method`, `filetrace_transfer_confidence` | File transfers observed. |
| `filetrace_transfers_active` | gauge | `filetrace_transfer_confidence` | File transfers in progress. |
| `filetrace_connections_active` | gauge | | Sockets that are open or have recent traffic. |
| `filetrace_processes_active` | gauge | | Processes with network traffic in the last few seconds. |
| `filetrace_events_processed_total` | counter | | Events the agent took from the kernel. |
| `filetrace_events_dropped_total` | counter | `filetrace_loss_kind` | Events lost before the agent could record them. |
| `process_cpu_time_seconds_total` | counter | | Processor time the agent has consumed. |
| `process_memory_usage_bytes` | gauge | | Resident memory of the agent. |
| `target_info` | gauge | `service_name`, `service_version`, `host_name`, `os_type` | Target metadata. Always 1. |

The last two `process_` metrics describe the agent itself, not the processes it observes. The [metrics and attributes reference](https://filetrace.mintlify.site/integrations/telemetry-reference.md) gives the OpenTelemetry name and unit of each metric.

A complete page:

```text theme={null}
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_name="filetrace",service_version="26.11.1.80",host_name="web-01",os_type="linux"} 1
# HELP filetrace_network_io_bytes_total Bytes that local sockets accepted from processes and delivered to them.
# TYPE filetrace_network_io_bytes_total counter
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="tcp",network_io_direction="receive"} 61127424
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="tcp",network_io_direction="transmit"} 62000000
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="udp",network_io_direction="transmit"} 50000
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="udp",network_io_direction="receive"} 50000
# HELP filetrace_file_sent_bytes_total File bytes sent over the network.
# TYPE filetrace_file_sent_bytes_total counter
filetrace_file_sent_bytes_total{process_executable_name="nginx",filetrace_transfer_method="sendfile",filetrace_transfer_confidence="exact"} 50000000
filetrace_file_sent_bytes_total{process_executable_name="python",filetrace_transfer_method="read-write",filetrace_transfer_confidence="inferred"} 12000000
# HELP filetrace_transfers_started_total File transfers observed.
# TYPE filetrace_transfers_started_total counter
filetrace_transfers_started_total{process_executable_name="nginx",filetrace_transfer_method="sendfile",filetrace_transfer_confidence="exact"} 1
filetrace_transfers_started_total{process_executable_name="python",filetrace_transfer_method="read-write",filetrace_transfer_confidence="inferred"} 1
# HELP filetrace_transfers_active File transfers in progress.
# TYPE filetrace_transfers_active gauge
filetrace_transfers_active{filetrace_transfer_confidence="inferred"} 1
filetrace_transfers_active{filetrace_transfer_confidence="strong"} 0
filetrace_transfers_active{filetrace_transfer_confidence="exact"} 1
# HELP filetrace_connections_active Sockets that are open or have recent traffic.
# TYPE filetrace_connections_active gauge
filetrace_connections_active 3
# HELP filetrace_processes_active Processes with network traffic in the last few seconds.
# TYPE filetrace_processes_active gauge
filetrace_processes_active 6
# HELP filetrace_events_processed_total Events the agent took from the kernel.
# TYPE filetrace_events_processed_total counter
filetrace_events_processed_total 49
# HELP filetrace_events_dropped_total Events lost before the agent could record them.
# TYPE filetrace_events_dropped_total counter
filetrace_events_dropped_total{filetrace_loss_kind="event_queue"} 0
filetrace_events_dropped_total{filetrace_loss_kind="counter_table"} 0
filetrace_events_dropped_total{filetrace_loss_kind="socket_identity"} 0
# HELP process_cpu_time_seconds_total Processor time the agent has consumed.
# TYPE process_cpu_time_seconds_total counter
process_cpu_time_seconds_total 0.049848
# HELP process_memory_usage_bytes Resident memory of the agent.
# TYPE process_memory_usage_bytes gauge
process_memory_usage_bytes 12222464
```

## Labels

| Label | Values |
| - | - |
| `process_executable_name` | The kernel's name for the process, at most 15 bytes, as in the `PROCESS` column of the commands |
| `network_transport` | The transport protocol, such as `tcp` or `udp` |
| `network_io_direction` | `transmit` (sent by the process) or `receive` (delivered to it) |
| `filetrace_transfer_method` | `sendfile` or `read-write` |
| `filetrace_transfer_confidence` | `exact` or `inferred`. `filetrace_transfers_active` also has a `strong` series, which stays at 0 because no probe produces that level |
| `filetrace_loss_kind` | `event_queue`, `counter_table` or `socket_identity`; see [loss kinds](https://filetrace.mintlify.site/integrations/telemetry-reference.md#loss-kinds) |
| `otel_metric_overflow` | `true`, only on the overflow series described under [Series limit](#series-limit) |

A label value that is not valid UTF-8 is written with `?` in place of the offending bytes. [Attribution and confidence](https://filetrace.mintlify.site/concepts/attribution.md) explains the methods and confidence levels.

## Query examples

```promql theme={null}
# Outbound bytes per second, by process
sum by (process_executable_name) (
  rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m]))

# Share of outbound bytes that a file accounts for
sum(rate(filetrace_file_sent_bytes_total[5m]))
  / sum(rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m]))
```

More queries that use only the metrics and labels above:

```promql theme={null}
# File bytes sent per second, by process and confidence
sum by (process_executable_name, filetrace_transfer_confidence) (
  rate(filetrace_file_sent_bytes_total[5m]))

# The five processes that sent the most file bytes in the last hour
topk(5, sum by (process_executable_name) (increase(filetrace_file_sent_bytes_total[1h])))

# Transfers started in the last hour, by method
sum by (filetrace_transfer_method) (increase(filetrace_transfers_started_total[1h]))

# Transfers in progress, by confidence
sum by (filetrace_transfer_confidence) (filetrace_transfers_active)

# Lost events in the last 10 minutes, by kind; returns a series only when some were lost
sum by (filetrace_loss_kind) (increase(filetrace_events_dropped_total[10m])) > 0

# Outbound bytes per second, by host
sum by (host_name) (
  rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])
  * on (job, instance) group_left (host_name) target_info)

# Series that went over [limits] metric_series
{otel_metric_overflow="true"}

# Processor time of the agent, in CPU seconds per second
rate(process_cpu_time_seconds_total{job="filetrace"}[5m])
```

Other targets can export `process_cpu_time_seconds_total` and `process_memory_usage_bytes` too, so select the agent's series by `job` as in the last query.

## Series limit

Three counters carry a process name and therefore grow with the number of distinct processes on the host: `filetrace_network_io_bytes_total` has one series per process, transport and direction, and `filetrace_file_sent_bytes_total` and `filetrace_transfers_started_total` have one per process, method and confidence.

`[limits] metric_series` bounds the label sets of these three counters together. The default is 2048, and the setting accepts 64 to 4194304:

```toml theme={null}
[limits]
metric_series = 2048
```

Once the limit is reached, a new label set does not get a series of its own. Its amounts are added to one overflow series per metric, which has the single label `otel_metric_overflow="true"`, as OpenTelemetry SDKs do. Series that already exist keep counting, and nothing is evicted, so a sum over all series of a metric stays correct. The other metrics have fixed label sets and do not count against the limit.

## Counter behaviour

* Counters start at zero when the agent starts. A restart of the agent is a counter reset, which `rate()` and `increase()` handle.
* A counter never goes backwards while the agent runs. Counters are kept apart from the agent's tables, which forget their least recently used entries, and a label set stays on the page for as long as the agent runs.
* Bytes of a read-write transfer are counted once the transfer is reported, from its first byte. Until the association reaches the reporting thresholds it adds nothing to `filetrace_file_sent_bytes_total`; then the counter takes all of its bytes at once. `filetrace_transfers_started_total` counts each transfer once, when it is first reported.
* Byte counts are what the local socket accepted from the process or delivered to it. They are not bytes on the wire and not what the remote end acknowledged.

## No paths or addresses in labels

No metric is labelled with a file path, an IP address or a port: those belong to events. To see which files went where, use the [OTLP log records](https://filetrace.mintlify.site/integrations/opentelemetry.md), the [gRPC API](https://filetrace.mintlify.site/integrations/grpc-api.md), or [`filetrace files`](https://filetrace.mintlify.site/cli/files).

## Exclude the agent's own traffic

The agent's own network traffic, such as its responses to Prometheus scrapes and its OTLP requests to a collector over TCP, loopback included, is counted like that of any other process. To leave it out:

```toml theme={null}
[filters]
exclude_processes = ["filetrace-agent"]
```

> [!NOTE]
>
> `exclude_processes` applies to everything the agent records, not only to metrics: the excluded process no longer appears in the commands, the log records or the API either.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.