ptrace (“process trace”) is a system call in Unix and Unix-like operating systems that intercepts system calls. It’s a powerful tool that enables tools like debuggers (e.g.,
gdb), reverse engineering tools, tracing, code injection, and even simple sandboxing. (see proot for an example of a
ptrace sandbox). The most interesting part of
ptrace is that you can do all of these things completely in user space (even sandboxing!).
Sandboxing. Roughly how a
ptrace sandbox works:
- Fork a child process to run the untrusted code
- Set the
ptracesyscall to trace the child process and intercept and monitor the child’s syscalls
- Inspect the child process syscalls and arguments. Maybe enact some sort of security process, logging, modification, or something else.
Google’s gVisor can use ptrace as a backend for its sandboxing. It’s a lot more complicated than a vanilla
ptrace sandbox and works much more like User-mode Linux (UML).
strace (Linux) and
ktrace (macOS), which are userspace monitoring utilities that expose
ptrace functions as a binary. You can monitor or modify system calls for a running program. Solaris had
DTrace (by Bryan Cantrill).
Some other cool ideas you could do with
- Time-travel debugging (record the whole state with
- Process-aware firewalling
- Transparent network traffic encryption (intercept
- Resource limits and throttling (intercept
- Live process migration or checkpointing (capture the entire state and copy somewhere else)
- Custom process scheduling (intercept
- Testing (fault injection, fuzzing, etc.)