The ptrace syscall

Apr 23, 2023

ptrace (“process trace”) is a system call in Unix and Unix-like operating systems that intercepts system calls. It’s a powerful tool that enables tools like debuggers (e.g., gdb), reverse engineering tools, tracing, code injection, and even simple sandboxing. (see proot for an example of a ptrace sandbox). The most interesting part of ptrace is that you can do all of these things completely in user space (even sandboxing!).

Sandboxing. Roughly how a ptrace sandbox works:

  • Fork a child process to run the untrusted code
  • Set the ptrace syscall to trace the child process and intercept and monitor the child’s syscalls
  • Inspect the child process syscalls and arguments. Maybe enact some sort of security process, logging, modification, or something else.

Google’s gVisor can use ptrace as a backend for its sandboxing. It’s a lot more complicated than a vanilla ptrace sandbox and works much more like User-mode Linux (UML).

Monitoring: ptrace powers strace (Linux) and ktrace (macOS), which are userspace monitoring utilities that expose ptrace functions as a binary. You can monitor or modify system calls for a running program. Solaris had DTrace (by Bryan Cantrill).

Some other cool ideas you could do with ptrace

  • Time-travel debugging (record the whole state with ptrace)
  • Process-aware firewalling
  • Transparent network traffic encryption (intercept send/recv sendto/recvfrom)
  • Resource limits and throttling (intercept mmap, read, write, etc.)
  • Live process migration or checkpointing (capture the entire state and copy somewhere else)
  • Custom process scheduling (intercept fork, clone, exit, wait, waitpid, etc.)
  • Testing (fault injection, fuzzing, etc.)