Scrape Filetrace metrics with Prometheus

The Filetrace agent can serve a page of metrics in the Prometheus text format: bytes that sockets sent and received per process, file bytes sent per transfer method and confidence, transfers started and in progress, open connections, lost events, and the agent’s own CPU time and memory. This page shows how to turn the page on and scrape it, lists every metric and label, gives PromQL examples, and explains how the number of series stays bounded.

Turn on the metrics page

Enable the page in the configuration file

Add the [prometheus] section to /etc/filetrace/config.toml:

[prometheus]
enabled = true
listen = "127.0.0.1:9464"

Restart the agent

The agent reads the file only when it starts:

sudo systemctl restart filetrace-agent

The agent’s log then reports serving metrics on http://127.0.0.1:9464/metrics.

Fetch the page

curl -s http://127.0.0.1:9464/metrics

Listen address

listen is an address and a port. The address is numeric: an IPv4 address, or an IPv6 address in brackets such as [::1]:9464. localhost stands for 127.0.0.1; no other host name is resolved. The page is served over TCP only, so a unix: path is refused. The default, 127.0.0.1:9464, accepts connections from the same machine only. For a Prometheus server on another machine, set listen to an address of an interface that server can reach.

The page is served without authentication and without TLS. Anyone who can reach the address can read process names and traffic volumes. Keep it on the loopback address or on a network that only your monitoring can reach.

The agent checks the address when it starts. An address that is not in one of these forms is a configuration error (exit code 6), and an address the agent cannot listen on, for example because the port is taken, stops the agent with exit code 8 and the message Cannot serve metrics on followed by the address and the reason. See exit codes.

HTTP behaviour

Request	Response
`GET /metrics`	`200 OK` with the page, `Content-Type: text/plain; version=0.0.4; charset=utf-8`
`GET` of any other path	`404 Not Found`
Any other method	`405 Method Not Allowed`

A query string is ignored: /metrics?name=value returns the page.
The page is built from the agent’s state at each request; nothing is cached between scrapes.
Every response closes the connection (Connection: close).
The agent serves at most 8 connections at a time, and a connection that has not received its response within 5 seconds is closed. A request whose headers exceed 2 KB is dropped.

Scrape configuration

A minimal Prometheus scrape job for an agent on the same host:

scrape_configs:
  - job_name: filetrace
    static_configs:
      - targets: ["127.0.0.1:9464"]

Prometheus requests /metrics unless metrics_path says otherwise, so the job needs no path. To scrape several hosts, list each host’s listen address under targets. The host_name label of target_info names the host each target runs on.

Metrics

Metric	Type	Labels	Help text
`filetrace_network_io_bytes_total`	counter	`process_executable_name`, `network_transport`, `network_io_direction`	Bytes that local sockets accepted from processes and delivered to them.
`filetrace_file_sent_bytes_total`	counter	`process_executable_name`, `filetrace_transfer_method`, `filetrace_transfer_confidence`	File bytes sent over the network.
`filetrace_transfers_started_total`	counter	`process_executable_name`, `filetrace_transfer_method`, `filetrace_transfer_confidence`	File transfers observed.
`filetrace_transfers_active`	gauge	`filetrace_transfer_confidence`	File transfers in progress.
`filetrace_connections_active`	gauge		Sockets that are open or have recent traffic.
`filetrace_processes_active`	gauge		Processes with network traffic in the last few seconds.
`filetrace_events_processed_total`	counter		Events the agent took from the kernel.
`filetrace_events_dropped_total`	counter	`filetrace_loss_kind`	Events lost before the agent could record them.
`process_cpu_time_seconds_total`	counter		Processor time the agent has consumed.
`process_memory_usage_bytes`	gauge		Resident memory of the agent.
`target_info`	gauge	`service_name`, `service_version`, `host_name`, `os_type`	Target metadata. Always 1.

The last two process_ metrics describe the agent itself, not the processes it observes. The metrics and attributes reference gives the OpenTelemetry name and unit of each metric. A complete page:

# HELP target_info Target metadata
# TYPE target_info gauge
target_info{service_name="filetrace",service_version="26.11.1.80",host_name="web-01",os_type="linux"} 1
# HELP filetrace_network_io_bytes_total Bytes that local sockets accepted from processes and delivered to them.
# TYPE filetrace_network_io_bytes_total counter
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="tcp",network_io_direction="receive"} 61127424
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="tcp",network_io_direction="transmit"} 62000000
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="udp",network_io_direction="transmit"} 50000
filetrace_network_io_bytes_total{process_executable_name="nginx",network_transport="udp",network_io_direction="receive"} 50000
# HELP filetrace_file_sent_bytes_total File bytes sent over the network.
# TYPE filetrace_file_sent_bytes_total counter
filetrace_file_sent_bytes_total{process_executable_name="nginx",filetrace_transfer_method="sendfile",filetrace_transfer_confidence="exact"} 50000000
filetrace_file_sent_bytes_total{process_executable_name="python",filetrace_transfer_method="read-write",filetrace_transfer_confidence="inferred"} 12000000
# HELP filetrace_transfers_started_total File transfers observed.
# TYPE filetrace_transfers_started_total counter
filetrace_transfers_started_total{process_executable_name="nginx",filetrace_transfer_method="sendfile",filetrace_transfer_confidence="exact"} 1
filetrace_transfers_started_total{process_executable_name="python",filetrace_transfer_method="read-write",filetrace_transfer_confidence="inferred"} 1
# HELP filetrace_transfers_active File transfers in progress.
# TYPE filetrace_transfers_active gauge
filetrace_transfers_active{filetrace_transfer_confidence="inferred"} 1
filetrace_transfers_active{filetrace_transfer_confidence="strong"} 0
filetrace_transfers_active{filetrace_transfer_confidence="exact"} 1
# HELP filetrace_connections_active Sockets that are open or have recent traffic.
# TYPE filetrace_connections_active gauge
filetrace_connections_active 3
# HELP filetrace_processes_active Processes with network traffic in the last few seconds.
# TYPE filetrace_processes_active gauge
filetrace_processes_active 6
# HELP filetrace_events_processed_total Events the agent took from the kernel.
# TYPE filetrace_events_processed_total counter
filetrace_events_processed_total 49
# HELP filetrace_events_dropped_total Events lost before the agent could record them.
# TYPE filetrace_events_dropped_total counter
filetrace_events_dropped_total{filetrace_loss_kind="event_queue"} 0
filetrace_events_dropped_total{filetrace_loss_kind="counter_table"} 0
filetrace_events_dropped_total{filetrace_loss_kind="socket_identity"} 0
# HELP process_cpu_time_seconds_total Processor time the agent has consumed.
# TYPE process_cpu_time_seconds_total counter
process_cpu_time_seconds_total 0.049848
# HELP process_memory_usage_bytes Resident memory of the agent.
# TYPE process_memory_usage_bytes gauge
process_memory_usage_bytes 12222464

Labels

Label	Values
`process_executable_name`	The kernel’s name for the process, at most 15 bytes, as in the `PROCESS` column of the commands
`network_transport`	The transport protocol, such as `tcp` or `udp`
`network_io_direction`	`transmit` (sent by the process) or `receive` (delivered to it)
`filetrace_transfer_method`	`sendfile` or `read-write`
`filetrace_transfer_confidence`	`exact` or `inferred`. `filetrace_transfers_active` also has a `strong` series, which stays at 0 because no probe produces that level
`filetrace_loss_kind`	`event_queue`, `counter_table` or `socket_identity`; see loss kinds
`otel_metric_overflow`	`true`, only on the overflow series described under Series limit

A label value that is not valid UTF-8 is written with ? in place of the offending bytes. Attribution and confidence explains the methods and confidence levels.

Query examples

# Outbound bytes per second, by process
sum by (process_executable_name) (
  rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m]))

# Share of outbound bytes that a file accounts for
sum(rate(filetrace_file_sent_bytes_total[5m]))
  / sum(rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m]))

More queries that use only the metrics and labels above:

# File bytes sent per second, by process and confidence
sum by (process_executable_name, filetrace_transfer_confidence) (
  rate(filetrace_file_sent_bytes_total[5m]))

# The five processes that sent the most file bytes in the last hour
topk(5, sum by (process_executable_name) (increase(filetrace_file_sent_bytes_total[1h])))

# Transfers started in the last hour, by method
sum by (filetrace_transfer_method) (increase(filetrace_transfers_started_total[1h]))

# Transfers in progress, by confidence
sum by (filetrace_transfer_confidence) (filetrace_transfers_active)

# Lost events in the last 10 minutes, by kind; returns a series only when some were lost
sum by (filetrace_loss_kind) (increase(filetrace_events_dropped_total[10m])) > 0

# Outbound bytes per second, by host
sum by (host_name) (
  rate(filetrace_network_io_bytes_total{network_io_direction="transmit"}[5m])
  * on (job, instance) group_left (host_name) target_info)

# Series that went over [limits] metric_series
{otel_metric_overflow="true"}

# Processor time of the agent, in CPU seconds per second
rate(process_cpu_time_seconds_total{job="filetrace"}[5m])

Other targets can export process_cpu_time_seconds_total and process_memory_usage_bytes too, so select the agent’s series by job as in the last query.

Series limit

Three counters carry a process name and therefore grow with the number of distinct processes on the host: filetrace_network_io_bytes_total has one series per process, transport and direction, and filetrace_file_sent_bytes_total and filetrace_transfers_started_total have one per process, method and confidence. [limits] metric_series bounds the label sets of these three counters together. The default is 2048, and the setting accepts 64 to 4194304:

[limits]
metric_series = 2048

Once the limit is reached, a new label set does not get a series of its own. Its amounts are added to one overflow series per metric, which has the single label otel_metric_overflow="true", as OpenTelemetry SDKs do. Series that already exist keep counting, and nothing is evicted, so a sum over all series of a metric stays correct. The other metrics have fixed label sets and do not count against the limit.

Counter behaviour

Counters start at zero when the agent starts. A restart of the agent is a counter reset, which rate() and increase() handle.
A counter never goes backwards while the agent runs. Counters are kept apart from the agent’s tables, which forget their least recently used entries, and a label set stays on the page for as long as the agent runs.
Bytes of a read-write transfer are counted once the transfer is reported, from its first byte. Until the association reaches the reporting thresholds it adds nothing to filetrace_file_sent_bytes_total; then the counter takes all of its bytes at once. filetrace_transfers_started_total counts each transfer once, when it is first reported.
Byte counts are what the local socket accepted from the process or delivered to it. They are not bytes on the wire and not what the remote end acknowledged.

No paths or addresses in labels

No metric is labelled with a file path, an IP address or a port: those belong to events. To see which files went where, use the OTLP log records, the gRPC API, or filetrace files.

Exclude the agent’s own traffic

The agent’s own network traffic, such as its responses to Prometheus scrapes and its OTLP requests to a collector over TCP, loopback included, is counted like that of any other process. To leave it out:

[filters]
exclude_processes = ["filetrace-agent"]

exclude_processes applies to everything the agent records, not only to metrics: the excluded process no longer appears in the commands, the log records or the API either.

Exporters and API

API reference

Scrape Filetrace metrics with Prometheus

Turn on the metrics page

Listen address

HTTP behaviour

Scrape configuration

Metrics

Labels

Query examples

Series limit

Counter behaviour

No paths or addresses in labels

Exclude the agent’s own traffic

​Turn on the metrics page

​Listen address

​HTTP behaviour

​Scrape configuration

​Metrics

​Labels

​Query examples

​Series limit

​Counter behaviour

​No paths or addresses in labels

​Exclude the agent’s own traffic

Turn on the metrics page

Listen address

HTTP behaviour

Scrape configuration

Metrics

Labels

Query examples

Series limit

Counter behaviour

No paths or addresses in labels

Exclude the agent’s own traffic