Export events and metrics over OTLP

The Filetrace agent includes an OTLP exporter that sends events as OpenTelemetry log records, and the same metrics the Prometheus page shows, to a collector over OTLP/gRPC. This page covers the [otlp] settings, the endpoint forms, a collector configuration that receives the data, the format of the log records, and how the exporter batches, retries and buffers when the collector cannot be reached.

Turn on the exporter

Run a collector on the same host

The exporter speaks OTLP over gRPC without TLS, so it sends to an OpenTelemetry Collector or Grafana Alloy on the same machine, which forwards to Loki, Elasticsearch, Splunk, Datadog or a SIEM with credentials and encryption of its own. See Collector configuration.

Enable the exporter

Add the [otlp] section to /etc/filetrace/config.toml:

[otlp]
enabled = true
endpoint = "127.0.0.1:4317"

Restart the agent

sudo systemctl restart filetrace-agent

The agent’s log reports exporting events and metrics to 127.0.0.1:4317 over OTLP/gRPC.

Settings

Every key of the [otlp] section, with its default:

[otlp]
enabled = false
endpoint = "127.0.0.1:4317"
logs = true
metrics = true
events = ["file_transfer", "event_loss"]
file_paths = true
batch_size = 1024
flush_interval = "5s"
metrics_interval = "30s"
timeout = "10s"

boolean

default:"false"

Turns the exporter on. With logs and metrics both false, the exporter does not start.

string

default:"127.0.0.1:4317"

Where the collector receives OTLP over gRPC: a numeric address and port, or unix: followed by the path of a Unix socket. See Endpoint forms.

boolean

default:"true"

Sends the events of the types in events as log records.

boolean

default:"true"

Sends the metrics every metrics_interval.

list of strings

default:"[\"file_transfer\", \"event_loss\"]"

The event types that leave as log records. Any type that filetrace events knows: process_exec, process_fork, process_exit, socket_connect, socket_listen, socket_established, socket_close, file_seen, network_send, network_receive, file_transfer and event_loss. A name that is not an event type is a configuration error.

boolean

default:"true"

With false, log records leave out file paths. A file is then described by its device and inode, which remain.

integer

default:"1024"

The most log records in one export request, from 16 to 4096.

duration

default:"5s"

How often the log records collected so far are sent, from "100ms" to "1h".

duration

default:"30s"

How often the metrics are sent, from "1s" to "1h".

duration

default:"10s"

How long one export request may take, from "1s" to "5m". A request that takes longer counts as failed and is retried.

A duration is a quoted number with the unit ms, s, m, h or d. The configuration reference describes the rest of the file.

Endpoint forms

Form	Example
IPv4 address and port	`"127.0.0.1:4317"`
IPv6 address in brackets and port	`"[::1]:4317"`
Unix socket	`"unix:/path/to/otlp.sock"`

localhost stands for 127.0.0.1; no other host name is resolved. The agent checks the endpoint when it starts, and an endpoint in none of these forms is a configuration error (exit code 6). A collector that is not running yet is not an error: the exporter keeps trying, as described under Delivery and retries.

Collector configuration

A minimal OpenTelemetry Collector configuration that receives both signals and prints them:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:    {receivers: [otlp], exporters: [debug]}
    metrics: {receivers: [otlp], exporters: [debug]}

To forward to a backend, replace the debug exporter with the exporter your backend documents. A sketch that forwards both signals over OTLP/HTTP with TLS and a bearer token taken from the collector’s environment:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
processors:
  batch: {}
exporters:
  otlphttp:
    endpoint: https://otlp.example.com
    headers:
      Authorization: "Bearer ${env:OTLP_TOKEN}"
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Grafana Alloy receives OTLP as well and can take the collector’s place; point endpoint at the address its OTLP gRPC receiver listens on. If a collector turns the metrics into Prometheus metrics, their names and labels are those of the Prometheus page, so the same PromQL works either way.

Log records

Events of the types listed in events leave as log records, one per event except where batching sums the bytes of several:

Field	Value
Event name	`filetrace.<type>`, such as `filetrace.file_transfer`
Body	The event type as a string, such as `file_transfer`
Severity	`INFO`; `WARN` for `event_loss`
Timestamp	When the event happened
Observed timestamp	When the batch that holds the record was prepared
Attributes	The event’s fields, named as listed below

Every request carries the resource attributes service.name (filetrace), service.version, host.name and os.type, and the instrumentation scope filetrace with the agent’s version. A transfer as a collector’s debug exporter prints it:

EventName: filetrace.file_transfer
Body: Str(file_transfer)
Attributes:
     -> filetrace.event.type: Str(file_transfer)
     -> process.pid: Int(1245)
     -> process.parent_pid: Int(1)
     -> process.executable.name: Str(nginx)
     -> process.user.id: Int(33)
     -> file.path: Str(/var/www/video.mp4)
     -> file.inode: Str(366815)
     -> filetrace.file.device: Int(64768)
     -> file.size: Int(50000000)
     -> filetrace.socket.id: Int(421)
     -> network.transport: Str(tcp)
     -> network.type: Str(ipv4)
     -> network.local.address: Str(192.0.2.7)
     -> network.local.port: Int(443)
     -> network.peer.address: Str(198.51.100.10)
     -> network.peer.port: Int(48175)
     -> filetrace.transfer.method: Str(sendfile)
     -> filetrace.transfer.bytes: Int(50000000)
     -> filetrace.transfer.confidence: Str(exact)

A read-write transfer adds filetrace.transfer.bytes_read, and one still below the reporting thresholds has no filetrace.transfer.confidence. A deleted file adds filetrace.file.deleted.

Attributes by event type

Event types	Attributes
All	`filetrace.event.type`
`process_exec`, `process_fork`, `process_exit`	`process.pid`, `process.parent_pid`, `process.executable.name`, `process.user.id`, `process.executable.path`; `process_exit` adds `process.exit.code`, or `filetrace.process.exit_signal` when a signal ended the process
`socket_connect`, `socket_listen`, `socket_established`, `socket_close`	The process attributes when the socket has a known owner, `filetrace.socket.id`, `network.transport`, `network.type`, `network.local.address`, `network.local.port`, `network.peer.address`, `network.peer.port`
`network_send`, `network_receive`	The process and socket attributes, `network.io.direction`, `filetrace.network.bytes`
`file_seen`	`file.path`, `file.inode`, `filetrace.file.device`, `file.size`, `filetrace.file.deleted`
`file_transfer`	The process, file and socket attributes, `filetrace.transfer.method`, `filetrace.transfer.bytes`, `filetrace.transfer.bytes_read`, `filetrace.transfer.confidence`
`event_loss`	`filetrace.loss.events`, `filetrace.loss.kind`

An attribute whose value is not known is left out. The metrics and attributes reference gives the type and meaning of each attribute, and event types describes the events.

Batching and aggregation

Log records are sent every flush_interval. Bytes of one transfer, or of one socket and peer, that arrive within an interval are summed into a single record, which keeps the time and confidence of the latest part. A long transfer therefore produces one record per interval, and the bytes of its records add up to the transfer. A batch also leaves before the interval ends when it holds batch_size records, or when its request approaches 3 MB, which keeps each request under the 4 MB that a collector accepts by default. Byte counts are what the local socket accepted, not bytes on the wire. A network or transfer event carries the bytes of one collection interval of the agent, not of one system call; see records and identifiers.

Metrics

Metrics are sent every metrics_interval, cumulative since the agent started: every counter point carries the agent’s start time as its start timestamp. Counters are monotonic sums with cumulative temporality; the other metrics are gauges. The metrics, their units and attributes are those listed in the metrics and attributes reference.

Delivery and retries

The exporter has one request of each signal in flight at a time.

Log records. When the collector cannot be reached, or answers with a status worth retrying (UNAVAILABLE, DEADLINE_EXCEEDED, RESOURCE_EXHAUSTED, ABORTED, CANCELLED, OUT_OF_RANGE or DATA_LOSS), the agent keeps the request and tries again after a pause that grows from one second to thirty. Meanwhile up to 16 batches, at most 8 MB together, wait behind it. Beyond that the oldest waiting batch is given up, and the agent’s log reports gave up exporting with the number of records, at most once a minute.
Rejected batches. A batch that the collector refuses with any other status is given up, and the agent logs the status and the collector’s message. When the collector accepts a request but rejects some records, the agent logs how many and why.
Metrics. A metrics request that fails is not sent again. The next collection, metrics_interval later, carries the cumulative values and makes up for it.

The agent logs cannot export to when the collector becomes unreachable and exporting to ... again when it answers again. When the agent stops, it sends the batch it is filling and a last collection of metrics, and gives them up to one second to leave, unless the collector is already known to be unreachable. Batches waiting for a collector live in memory and do not survive a restart.

File paths and text encoding

file_paths = false leaves file paths out of the log records. Device and inode remain, so records about the same file can still be told apart. A path or process name that is not valid UTF-8 is sent with ? in place of the offending bytes.

Not included

The exporter sends log records and metrics; it does not send OpenTelemetry traces, and it speaks OTLP over gRPC only, not OTLP over HTTP.

Exporters and API

API reference

Export events and metrics over OTLP

Turn on the exporter

Settings

Endpoint forms

Collector configuration

Log records

Attributes by event type

Batching and aggregation

Metrics

Delivery and retries

File paths and text encoding

Not included

​Turn on the exporter

​Settings

​Endpoint forms

​Collector configuration

​Log records

​Attributes by event type

​Batching and aggregation

​Metrics

​Delivery and retries

​File paths and text encoding

​Not included

Turn on the exporter

Settings

Endpoint forms

Collector configuration

Log records

Attributes by event type

Batching and aggregation

Metrics

Delivery and retries

File paths and text encoding

Not included