Syscall behavior tracing¶

The other domains report what the host contains: its processes, its ports, its files, its packages. --trace reports what the software does. It samples the system calls that each process makes.

A compromised binary keeps its name, its path and often its checksum. Its behavior changes. A process that is named [kworker/3:2] looks like a kernel thread in the process table. If it calls init_module and connect, it is not one.

delta --processes --trace
delta --trace --trace-window 3000 --json
delta --cfg policy.json --processes --trace --interval 60

Delta samples one observation window for each collection cycle. The default window is 1 second. For each process it reports the number of syscalls, which syscalls they were, and which behavior classes those syscalls belong to. Add --trace-args to also decode the arguments of selected syscalls.

This is a sample, not a recording. The agent keeps no syscall log.

Backends¶

Each platform and each kernel gives a different way to observe syscalls. Delta selects the best backend that is available. To select a backend manually, use --trace-backend <name>. The default value is auto.

Backend	Platform	Requirements	Operation
`ebpf`	Linux	`CAP_BPF` or root, and an unrestricted `bpf(2)`	A BPF program on the `raw_syscalls:sys_enter` tracepoint counts `(pid, syscall)` pairs in a kernel hash map. It observes all processes and stops no process. The cost is some tens of nanoseconds for each syscall.
`ptrace`	Linux	`CAP_SYS_PTRACE` or root, or Yama `ptrace_scope=0`	`PTRACE_SEIZE` and `PTRACE_SYSCALL` on a limited set of target processes. This backend also operates on a kernel that has no BPF. Each traced syscall causes two stops, thus Delta traces some processes and not the full host.
`seccomp`	Linux 5.6 or later	none: the traced process installs the filter itself	seccomp user notification on a command that Delta starts (`--trace-run`). This backend gives the most exact data of the three. It cannot attach to a process that already runs.
`es`	macOS	the endpoint-security entitlement, root, and `DELTA_TRACE_ES=ON`	Endpoint Security notify events: exec, task-port acquisition, kext load, mount, setuid. These are events and not syscalls. macOS supports no closer equivalent.
`procinfo`	macOS	none	`proc_pidinfo(PROC_PIDTASKINFO)` syscall counters. Delta reads them two times and calculates the difference. The counters give no syscall identity, but they show which process became loud.

A backend that is not available is not an error. The report gives the name of the backend that operated, or none, and a note with the reason. The policy checks stay inert.

Delta does not trace under --root. An offline image has no kernel that operates.

Windows has no backend. The domain is part of the build and Delta accepts --trace, but the report contains "backend": "none" and a note. The syscalls policy then applies to nothing. All other checks operate as usual.

Note

Delta observes syscalls. It does not block them. The seccomp backend answers each notification with SECCOMP_USER_NOTIF_FLAG_CONTINUE, thus a traced command operates as it operates without a supervisor. The ptrace backend writes no register of a traced process and removes none of its signals. This agrees with the report-only design of the agent (see Report-only builds).

Behavior classes¶

A syscall name has a meaning only for a reader who knows the kernel ABI. Delta thus puts each observation into a behavior class, which gives the function of the syscall. Policies and drift keys use these classes.

Class	Syscalls (examples)	Reason to monitor it
`exec`	`execve`, `execveat`	A process started a program
`ptrace`	`ptrace`, `process_vm_readv`, `process_vm_writev`	Access into a different process: injection, credential theft
`module`	`init_module`, `finit_module`, `delete_module`	A process loaded kernel code (`T1547.006`)
`bpf`	`bpf`	Recent rootkits use BPF (`T1014`)
`kexec`	`kexec_load`, `kexec_file_load`	A process replaced the kernel (`T1542`)
`mount`	`mount`, `umount2`, `pivot_root`, `chroot`	Changes to the filesystem topology, container escape (`T1611`)
`namespace`	`unshare`, `setns`	A process entered or left an isolation boundary
`privilege`	the `setuid` family, `capset`	Privilege changes (`T1548`)
`network`	`socket`, `connect`, `bind`, `sendto`	A process that did not use the network started to use it
`memexec`	`mprotect`, `memfd_create`, `userfaultfd`	Executable memory, and payloads that use no file (`T1055`)
`keyring`	`add_key`, `request_key`, `keyctl`	Access to stored credentials (`T1555`)
`fsperm`	`chmod`, `chown`, `setxattr`, `link`	Changes to permissions and ownership (`T1222`)

Syscall numbers are different on each architecture. Delta builds its name table from the SYS_* macros of the toolchain, thus the numbers agree with the target that you compile for. Delta reports a syscall that is not in the table as nr:<n>. This form is also a valid policy key.

Argument decoding¶

--trace-args gives the data that a count cannot give: the file, the address, the permissions. An openat is a usual operation, but an openat of /etc/shadow is not. A connect is a usual operation, but a connect to an unexpected address is not.

delta --processes --trace-args
delta --trace-run "/tmp/suspicious" --trace-args

Text output then contains a decoded line for each such syscall:

Syscall trace (ebpf, 1000 ms): 65058 call(s), 65058/s
  - nginx (pid 812): 9120 call(s) [network]
      epoll_wait=3011 accept4=812 openat=54 connect=12
       openat path=/var/log/nginx/access.log flags=wronly|append
       connect addr=10.0.0.9:5432
       mprotect prot=rwx

Delta decodes only the syscalls in the table below and counts all the others. There are three reasons for this limit:

Delta must read the memory of a different process.
A full argument log is too large for the device and for the link.
A path or a peer address is more sensitive than a syscall name.

Decoding is off by default.

Syscalls	Decoded data
`execve`, `execveat`	the program to start
`open`, `openat`, `openat2`	the path, the access mode and the creation flags
`unlink`, `unlinkat`, `rename`, `renameat2`, `truncate`	the paths
`chmod`, `fchmodat`, `chown`, `lchown`, `fchownat`	the path, and the mode for the `chmod` family
`link`, `linkat`, `symlink`, `symlinkat`	the two ends of the link
`setxattr`, `lsetxattr`	the path and the name of the attribute
`mount`, `umount2`, `chroot`, `pivot_root`	the source and the target
`connect`, `bind`, `sendto`	the peer as `ip:port`, or as `unix:/path`
`socket`	the domain and the type. `AF_PACKET` with `SOCK_RAW` is a sniffer.
`mprotect`, `pkey_mprotect`	the protection bits in the form `rwx`. Memory that is writable and also executable is a sign of an injected payload.
`memfd_create`	the name of the anonymous file. A payload that uses no file needs it.
`finit_module`, `kexec_file_load`	the descriptor. Delta also resolves the file behind it when it can.
`delete_module`, `add_key`, `request_key`	the name of the module, or the type of the key
`ptrace`, `process_vm_readv`, `process_vm_writev`	the request and the target pid
`kill`, `tgkill`	the target pid and the name of the signal
`unshare`, `setns`	the namespaces

Delta removes duplicates. One syscall with the same arguments is one line with a count. A loop on one file thus gives one entry and not one thousand entries. Delta keeps a maximum of 24 different decoded lines for each process.

Decoding by backend¶

To decode arguments, a backend must have the arguments of the syscall. It must also read the memory of the caller before the syscall completes. The backends are different:

ptrace decodes all arguments. The traced process stops at the entry of the syscall, thus Delta reads its memory with process_vm_readv while nothing changes. Delta also resolves a descriptor through /proc/<pid>/fd. Delta decodes each occurrence.
seccomp decodes all arguments. The supervised process is blocked, but a blocked process can still stop to exist. Delta thus asks the kernel to confirm the notification after each read, and discards the data that the kernel does not confirm.
ebpf decodes in the kernel. Two limits are the result. Delta keeps one example for each pair of process and syscall in a window. It also reads the strings of the first two arguments only, thus a path in a later argument stays undecoded. This backend needs a mounted tracefs.
es on macOS gives the subject of each event directly. Examples are the executable of an exec and the file of an open.
procinfo on macOS has counters only and decodes nothing.

The cost is a function of the decoded syscalls and not of all syscalls. The eBPF program makes one more hash lookup for each syscall, and reads memory for the selected syscalls only. The ptrace backend reads memory only for a syscall that already stopped in front of it.

Sensitive data¶

A string from a traced process is untrusted data. Delta replaces each byte that is not printable ASCII, and limits the length. A crafted path thus cannot put terminal escape sequences on the screen of an operator, or newlines into the log of a collector.
Delta reads only. It writes no register, it changes no syscall, and it replaces no argument.
A decoded report contains file paths and remote addresses. Decoding is optional for this reason.

Policy¶

The domain is inert until the norm contains a syscalls block. This agrees with the other inventory checks:

{ "norm": {
    "syscalls": {
      "forbidden": ["init_module", "finit_module", "kexec_load", "bpf"],
      "forbidden_args": ["path=/etc/shadow", "prot=rwx", "domain=AF_PACKET"],
      "deny_classes": ["module", "kexec", "ptrace"],
      "max_rate": 200000,
      "max_process_rate": 50000
    }
} }

forbidden gives the syscalls that no process on the device can use. Delta reports one violation for each process that used one. The severity is high.
forbidden_args gives text that no process can produce in a decoded argument line. Delta matches it as a substring, for example path=/etc/shadow, prot=rwx or addr=203.0.113.. One rule can thus select a path prefix, an address range, or a set of flags. This rule needs --trace-args. Without decoded arguments the rule matches nothing. The severity is high.
deny_classes gives the behavior classes that no process can show. A class is more general than a list of names. It also stays correct when the kernel adds a syscall, because a new syscall that loads a module is also in the module class. Delta ignores a class name that it does not know, thus an older agent can read a newer policy.
max_rate and max_process_rate give a maximum number of syscalls for each second, for the host and for one process. The severity is medium. These limits show a loop or a denial-of-service pattern that no single syscall shows. Delta calculates a rate only for a window of 200 ms or more.

Delta reports the findings in the syscalls category. Each finding gives the name of the process and an ATT&CK technique. A SIEM can thus use them as it uses the other findings of Delta.

In monitoring mode¶

With --interval the domain gives data to the two parts of Continuous monitoring:

The behavior classes of each process become drift keys in the form sc:<process>:<class>. If nginx starts to load kernel modules, the cycle reports + sc:nginx:module. Syscall volume produces no key, because it changes between cycles.
The syscall rate of the host goes to the behavioral model as the syscall_rate metric. The monitor reports a large change as a ~ anomaly.

Tracing one command¶

--trace-run <command> starts a command under the seccomp supervisor, reports each syscall that the command made, and exits. Use it to examine a suspicious file from a device, or to make the forbidden list for a known workload.

delta --trace-run "/usr/bin/curl -s http://example.com/"
delta --trace-run "/tmp/suspicious" --cfg policy.json --json

Delta divides the command line at each space and starts the program directly. It uses no shell, and it does no quoting and no globbing. An agent that can start a shell line is a remote-execution tool. This agent starts only what the local operator gives to it.

--cfg also applies in this mode. Delta checks the traced command against the same syscall policy that it applies to a host, and a violation gives a non-zero exit code. --trace-window gives the maximum time for the command. The default value is 10 seconds. Delta stops a command that uses more time.

Cost and limits¶

Window. Delta samples one window for each collection cycle. The default window is 1 second and the maximum is 60 seconds (--trace-window). The collection waits for this time, thus keep the window much shorter than --interval.
eBPF costs the host one map lookup and one atomic increment for each syscall. Delta wakes up one time at the end of the window. Use this backend on a busy device.
ptrace stops the traced process two times for each syscall. It attaches to a maximum of 16 processes and 64 threads, and it stops after 50000 syscall stops. The traced processes operate more slowly during the window. Delta does not trace PID 1 and does not trace itself.
Limits. Delta keeps a maximum of 128 processes, 48 different syscalls for each process, and 24 decoded argument lines for each process. Above these limits the totals and the class flags stay correct, the lists stop to increase, and the report contains truncated. The system-wide backend reads its counters in order of volume, thus the limit keeps the processes that do the most kernel work.
Containers. A system-wide eBPF sample sees all PID namespaces. Delta resolves a process name from the /proc that the agent sees, thus it reports a pid from a different namespace as ?.

Build¶

DELTA_FEATURE_TRACE controls the domain. The default value is ON. DELTA_TRACE_ES controls the macOS Endpoint Security backend. The default value is OFF, because this backend operates only in a binary that has the endpoint-security entitlement of Apple.

cmake -S . -B build -G Ninja -DDELTA_FEATURE_TRACE=OFF     # no tracing
cmake -S . -B build -G Ninja -DDELTA_TRACE_ES=ON           # macOS, entitled build

See Build- & run-time features.