List and stream transfers with the gRPC API

The Filetrace agent can serve a gRPC API, filetrace.v1.FiletraceService, that reports the agent’s status, lists the file transfers it holds and streams transfer activity as it is recorded. This page shows how to turn the API on, who may connect, how to call it with grpcurl and buf curl, how filters and streams behave, and how to generate a client. The gRPC API reference describes every message and field.

Turn on the API

Enable the API in the configuration file

Add the [grpc] section to /etc/filetrace/config.toml:

[grpc]
enabled = true
listen = "unix:/run/filetrace/api.sock"

Restart the agent

sudo systemctl restart filetrace-agent

The agent’s log reports serving the gRPC API on unix:/run/filetrace/api.sock.

Listen address and access

listen is either unix: followed by a socket path, or a numeric address and port. Who may connect depends on the form:

Form	Example	Who may connect
Unix socket	`"unix:/run/filetrace/api.sock"`	Root and the user the agent runs as, like the command-line socket
IPv4 address and port	`"127.0.0.1:9500"`	Whoever can connect to the address
IPv6 address in brackets and port	`"[::1]:9500"`	Whoever can connect to the address

The agent creates the Unix socket with mode 0600, rejects any other user that connects (and logs rejected an API client that is neither root nor the agent's user), and removes the socket file when it stops. With an address and port, the agent logs a warning when it starts that the API is open to whoever can connect. localhost stands for 127.0.0.1; no other host name is resolved.

The API has no TLS and no authentication of its own. On an address and port, anyone who can reach it can list the files the host sends and where they go. Prefer the Unix socket, which admits only root and the user the agent runs as.

Under the packaged systemd service the file system is read-only to the agent except for its runtime directory /run/filetrace, and /tmp is private to the service, so keep the socket in /run/filetrace.

Methods

Method	Returns
`GetStatus`	Version, host, backend, mode, start time, totals, whether events were lost lately
`ListTransfers`	Transfers over their whole observed lifetime, as `filetrace files` lists them; at most 1000
`WatchTransfers`	A stream of the file bytes attributed in each collection interval, until the caller cancels

The schema

The server speaks gRPC over HTTP/2 without TLS and without server reflection, so a client names the schema itself. The service is defined in proto/filetrace/v1/filetrace.proto in the Filetrace repository. The examples below run from a clone of the repository, where proto is the import path:

git clone https://github.com/OrlovEvgeny/filetrace.git
cd filetrace

Call the API

The examples use sudo because the default socket admits only root and the agent’s user. Status of the agent:

sudo grpcurl -plaintext -unix -import-path proto -proto filetrace/v1/filetrace.proto \
    /run/filetrace/api.sock filetrace.v1.FiletraceService/GetStatus

Transfers to a network:

sudo grpcurl -plaintext -unix -import-path proto -proto filetrace/v1/filetrace.proto \
    -d '{"filter": {"remote": "203.0.113.0/24"}}' \
    /run/filetrace/api.sock filetrace.v1.FiletraceService/ListTransfers

Transfer activity on files under /data, as it happens, with buf curl:

sudo buf curl --schema proto --protocol grpc --http2-prior-knowledge \
    --unix-socket /run/filetrace/api.sock -d '{"filter": {"path": "/data/*"}}' \
    http://localhost/filetrace.v1.FiletraceService/WatchTransfers

Both tools take the request and print the response in protobuf’s JSON form. Enum values are written by name, such as "METHOD_SENDFILE", 64-bit numbers appear as strings, and fields at their zero value are left out of the output.

Filters

ListTransfers and WatchTransfers take a filter. Every field that is set narrows the result; a field left at zero or empty does not filter.

Field	Selects	Command-line equivalent
`pid`	One process ID	`--pid`
`process`	Process name; `*` matches any run of characters	`--process`
`path`	File path; `*` matches any run of characters, `/` included	`--path`
`remote`	Remote address or CIDR network, such as `"203.0.113.0/24"`	`--remote`
`port`	Local or remote port	`--port`
`transport`	`TRANSPORT_TCP` or `TRANSPORT_UDP`	`--protocol`
`method`	`METHOD_SENDFILE` or `METHOD_READ_WRITE`	`--method`
`confidence`	Exactly this level: `CONFIDENCE_UNKNOWN`, `CONFIDENCE_INFERRED`, `CONFIDENCE_STRONG` or `CONFIDENCE_EXACT`	`--confidence`

A filter that cannot be applied, such as a remote that is neither an address nor a network, or a port above 65535, ends the call with INVALID_ARGUMENT and a message that names the field. Filters describes the matching rules the command-line options share with the API.

ListTransfers

ListTransfers returns the transfers the agent holds, each covering its whole observed lifetime, most recently active first. The request takes, besides the filter:

Field	Effect
`active_only`	Only transfers still in progress
`include_unattributed`	Also lists sent bytes that no file accounts for, as `CONFIDENCE_UNKNOWN` without a `file`, like `filetrace transfers`. They are left out when the filter sets `path`, `method`, or a `confidence` other than `CONFIDENCE_UNKNOWN`
`min_bytes`	Smallest number of bytes sent
`limit`	The most transfers to return. At most 1000, which is also what 0 asks for

sudo grpcurl -plaintext -unix -import-path proto -proto filetrace/v1/filetrace.proto \
    -d '{"filter": {"method": "METHOD_SENDFILE"}, "active_only": true, "limit": 10}' \
    /run/filetrace/api.sock filetrace.v1.FiletraceService/ListTransfers

The response also carries observed_since_unix_nano, the time from which the agent has observed (nothing earlier is known to it), and lost_events, the events the agent has lost so far. Above zero, the list may be incomplete, and each transfer that was in progress while events were lost has incomplete set. A response that would exceed 4 MB fails with RESOURCE_EXHAUSTED and the message The result is too large. Narrow it with a filter or a limit.

WatchTransfers

WatchTransfers streams transfer activity from the moment the call starts; it sends nothing from before. Each message carries one TransferEvent: the file bytes attributed to one socket during one collection interval of the agent, the same records that filetrace events --type file_transfer prints.

The stream runs until the caller cancels it or the agent stops.
confidence is that of the whole association so far, and CONFIDENCE_UNSPECIFIED while the association is still below the reporting thresholds. A confidence filter selects only events that have that level.
A watcher that reads too slowly is not sent what it cannot take: once more than 1 MB is waiting unread, further events are left out. The next message the watcher receives carries missed, the number of events left out before it.
The agent serves at most 16 watchers at a time. Another WatchTransfers call fails with RESOURCE_EXHAUSTED and Too many watchers.

Errors and limits

Status	Message	Cause
`INVALID_ARGUMENT`	Names the field, such as `remote is neither an IP address nor a CIDR network.`	A filter value that cannot be applied
`INVALID_ARGUMENT`	`The request is not what the method takes.`	The request does not decode as the method’s request message
`RESOURCE_EXHAUSTED`	`The result is too large. Narrow it with a filter or a limit.`	A `ListTransfers` response above 4 MB
`RESOURCE_EXHAUSTED`	`Too many watchers.`	16 `WatchTransfers` calls are already open
`RESOURCE_EXHAUSTED`	`the request is too large`	A request above 64 KB
`UNIMPLEMENTED`	`No such method.`	A method the service does not have
`UNIMPLEMENTED`	`compressed messages are not accepted`	A compressed request message

The server serves up to 16 connections at a time, with up to 32 concurrent calls on each. A request whose content type is not application/grpc is answered with HTTP status 415.

Generate a client

The schema is a plain proto3 file, so any protobuf and gRPC toolchain can generate a client from it. With Buf, copy filetrace/v1/filetrace.proto into a proto directory of your project, keeping its path, and add a buf.gen.yaml that names the plugins of your language; for Python, for example:

version: v2
inputs:
  - directory: proto
plugins:
  - remote: buf.build/protocolbuffers/python
    out: gen
  - remote: buf.build/grpc/python
    out: gen

buf generate

Connect the generated client to the Unix socket, or to the address and port in listen, without TLS.

​Turn on the API

​Listen address and access

​Methods

​The schema

​Call the API

​Filters

​ListTransfers

​WatchTransfers

​Errors and limits

​Generate a client

Turn on the API

Listen address and access

Methods

The schema

Call the API

Filters

ListTransfers

WatchTransfers

Errors and limits

Generate a client