ptrace (“process trace”) is a system call in Unix and Unix-like operating systems that intercepts system calls. It’s a powerful tool that enables tools like debuggers (e.g., gdb), reverse engineering tools, tracing, code injection, and even simple sandboxing. (see proot for an example of a ptrace sandbox). The most interesting part of ptrace is that you can do all of these things completely in user space (even sandboxing!).
Sandboxing. Roughly how a ptrace sandbox works:
- Fork a child process to run the untrusted code
- Set the
ptracesyscall to trace the child process and intercept and monitor the child’s syscalls - Inspect the child process syscalls and arguments. Maybe enact some sort of security process, logging, modification, or something else.
Google’s gVisor can use ptrace as a backend for its sandboxing. It’s a lot more complicated than a vanilla ptrace sandbox and works much more like User-mode Linux (UML).
Monitoring: ptrace powers strace (Linux) and ktrace (macOS), which are userspace monitoring utilities that expose ptrace functions as a binary. You can monitor or modify system calls for a running program. Solaris had DTrace (by Bryan Cantrill).
Some other cool ideas you could do with ptrace
- Time-travel debugging (record the whole state with
ptrace) - Process-aware firewalling
- Transparent network traffic encryption (intercept
send/recv sendto/recvfrom) - Resource limits and throttling (intercept
mmap,read,write, etc.) - Live process migration or checkpointing (capture the entire state and copy somewhere else)
- Custom process scheduling (intercept
fork,clone,exit,wait,waitpid, etc.) - Testing (fault injection, fuzzing, etc.)