# tracexec

A small utility for tracing execve{,at} and pre-exec behavior.

tracexec helps you to figure out what and how programs get executed when you execute a command.

It's useful for debugging build systems, understanding what shell scripts actually do, figuring out what programs
does a proprietary software run, etc.

- [Installation Guide](INSTALL.md)

## Showcases

### Perfetto Trace Export

tracexec supports exporting exec traces to perfetto trace format,
which could be viewed in the [Perfetto UI](https://ui.perfetto.dev/).
The trace follows a tree format in the UI, where processes resulting from successful execs are represented as slices and exec failures are represented as instant events.

The following video shows analyzing the build process of tracexec with itself:

https://github.com/user-attachments/assets/fc825d55-eb2f-43cd-b454-a11b5f9b44bc

The shape of the traces in the Perfetto UI could give you a rough idea of how
parallel the build is at process-level. The trace tree and details of slices
enable identification of bottlenecks, troubleshooting, and a deep understanding
of how the build works.

Start collecting a perfetto trace with the following command:

```bash
tracexec collect --format=perfetto -o out.pftrace -- cmd
```

### TUI mode with pseudo terminal

In TUI mode with a pseudo terminal, you can view the details of exec events and interact with the processes
within the pseudo terminal at ease.

![TUI demo](https://github.com/kxxt/tracexec/blob/v0.15.1/screenshots/tui-demo.gif?raw=true)

### Tracing setuid binaries

With root privileges, you can also trace setuid binaries and see how they work.
But do note that this is not compatible with seccomp-bpf optimization so it is much less performant.
You can use eBPF mode which is more performant in such scenarios.

```
sudo tracexec --user $(whoami) tui -t -- sudo ls
```

![Tracing sudo ls](https://github.com/kxxt/tracexec/blob/v0.15.1/screenshots/tracing-sudo.png?raw=true)

Nested setuid binary tracing is also possible: A real world use case is to trace `extra-x86_64-build`(Arch Linux's build tool that requires sudo):

![Tracing extra-x86_64-build](https://github.com/kxxt/tracexec/blob/v0.15.1/screenshots/tracing-nested-setuid.gif?raw=true)

In this real world example, we can easily see that `_FORTIFY_SOURCE` is redefined from `2` to `3`, which lead to a compiler error.

### Use tracexec as a debugger launcher

tracexec can also be used as a debugger launcher to make debugging programs easier. For example, it's not trivial or convenient
to debug a program executed by a shell/python script(which can use pipes as stdio for the program). The following video shows how to
use tracexec to launch gdb to detach two simple programs piped together by a shell script.

https://github.com/kxxt/tracexec/assets/18085551/72c755a5-0f2f-4bf9-beb9-98c8d6b5e5fd

Please [read the gdb-launcher example](demonstration/gdb-launcher/README.md) for more details.

### eBPF mode

The eBPF mode is currently experimental.
The minimum supported kernel version is 5.17.

The following examples shows how to use eBPF in TUI mode.
The `eBPF` command also supports regular `log` and `collect` subcommands.

#### System-wide Exec Tracing

```bash
sudo -E tracexec ebpf tui
```
[ebpf-system-wide-tui.webm](https://github.com/user-attachments/assets/12cec4ef-8884-4580-a93a-c9144ec7102b)

#### Follow Fork mode with eBPF

```bash
sudo -E tracexec --user $(whoami) ebpf tui -t -- bash
```

[ebpf-follow-forks.webm](https://github.com/user-attachments/assets/997e1992-df85-4d45-ae68-faf693c6b99b)

### Log mode

In log mode, by default, `tracexec` will print filename, argv and the diff of the environment variables and file descriptors.

example: `tracexec log -- bash` (In an interactive bash shell)

[![asciicast](https://asciinema.org/a/sNptWG6De3V5xwUvXJAxWlO3i.svg)](https://asciinema.org/a/sNptWG6De3V5xwUvXJAxWlO3i)

### Reconstruct the command line with `--show-cmdline`

```bash
$ tracexec log --show-cmdline -- <command>
# example:
$ tracexec log --show-cmdline -- firefox
```

[![asciicast](https://asciinema.org/a/AWTG4iHaFPMcEGCVtqAl44YFW.svg)](https://asciinema.org/a/AWTG4iHaFPMcEGCVtqAl44YFW)

### Try to reproduce stdio in the reconstructed command line

`--stdio-in-cmdline` and `--fd-in-cmdline` can be used to reproduce(hopefully) the stdio used by a process.

But do note that the result might be inaccurate when pipes, sockets, etc are involved.

```bash
tracexec log --show-cmdline --stdio-in-cmdline -- bash
```

[![asciicast](https://asciinema.org/a/NkBTaoNHS7P7bolO0hNuRwGlQ.svg)](https://asciinema.org/a/NkBTaoNHS7P7bolO0hNuRwGlQ)

### Show the interpreter indicated by shebang with `--show-interpreter`

And show the cwd with `--show-cwd`.

```bash
$ tracexec log --show-interpreter --show-cwd -- <command>
# example: Running Arch Linux makepkg
$ tracexec log --show-interpreter --show-cwd -- makepkg -f
```

[![asciicast](https://asciinema.org/a/7jDtrlNRx5XUnDXeDBsMRj09p.svg)](https://asciinema.org/a/7jDtrlNRx5XUnDXeDBsMRj09p)

## Usage

General CLI help:

```bash
Core crate of tracexec [Internal implementation! DO NOT DEPEND ON!]

Usage: tracexec [OPTIONS] <COMMAND>

Commands:
  log                   Run tracexec in logging mode
  tui                   Run tracexec in TUI mode, stdin/out/err are redirected to /dev/null by default
  generate-completions  Generate shell completions for tracexec
  collect               Collect exec events and export them
  ebpf                  Experimental ebpf mode
  help                  Print this message or the help of the given subcommand(s)

Options:
      --color <COLOR>      Control whether colored output is enabled. This flag has no effect on TUI mode. [default: auto] [possible values: auto, always, never]
  -C, --cwd <CWD>          Change current directory to this path before doing anything
  -P, --profile <PROFILE>  Load profile from this path
      --no-profile         Do not load profiles
  -u, --user <USER>        Run as user. This option is only available when running tracexec as root
  -h, --help               Print help
  -V, --version            Print version

```

TUI Mode:

```bash
Run tracexec in TUI mode, stdin/out/err are redirected to /dev/null by default

Usage: tracexec tui [OPTIONS] -- <CMD>...

Arguments:
  <CMD>...  command to be executed

Options:
      --successful-only
          Only show successful calls
      --fd-in-cmdline
          [Experimental] Try to reproduce file descriptors in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
      --stdio-in-cmdline
          [Experimental] Try to reproduce stdio in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
      --resolve-proc-self-exe
          Resolve /proc/self/exe symlink
      --no-resolve-proc-self-exe
          Do not resolve /proc/self/exe symlink
      --hide-cloexec-fds
          Hide CLOEXEC fds
      --no-hide-cloexec-fds
          Do not hide CLOEXEC fds
      --timestamp
          Show timestamp information
      --no-timestamp
          Do not show timestamp information
      --inline-timestamp-format <INLINE_TIMESTAMP_FORMAT>
          Set the format of inline timestamp. See https://docs.rs/chrono/latest/chrono/format/strftime/index.html for available options.
      --seccomp-bpf <SECCOMP_BPF>
          Controls whether to enable seccomp-bpf optimization, which greatly improves performance [default: auto] [possible values: auto, on, off]
      --polling-interval <POLLING_INTERVAL>
          Polling interval, in microseconds. -1(default) disables polling.
      --show-all-events
          Set the default filter to show all events. This option can be used in combination with --filter-exclude to exclude some unwanted events.
      --filter <FILTER>
          Set the default filter for events. [default: warning,error,exec,tracee-exit]
      --filter-include <FILTER_INCLUDE>
          Aside from the default filter, also include the events specified here. [default: <empty>]
      --filter-exclude <FILTER_EXCLUDE>
          Exclude the events specified here from the default filter. [default: <empty>]
  -t, --tty
          Allocate a pseudo terminal and show it alongside the TUI
  -f, --follow
          Keep the event list scrolled to the bottom
      --terminate-on-exit
          Instead of waiting for the root child to exit, terminate when the TUI exits
      --kill-on-exit
          Instead of waiting for the root child to exit, kill when the TUI exits
  -A, --active-pane <ACTIVE_PANE>
          Set the default active pane to use when TUI launches [possible values: terminal, events]
  -L, --layout <LAYOUT>
          Set the layout of the TUI when it launches [possible values: horizontal, vertical]
  -F, --frame-rate <FRAME_RATE>
          Set the frame rate of the TUI (60 by default)
  -m, --max-events <MAX_EVENTS>
          Max number of events to keep in TUI (0=unlimited)
  -D, --default-external-command <DEFAULT_EXTERNAL_COMMAND>
          Set the default external command to run when using "Detach, Stop and Run Command" feature in Hit Manager
  -b, --add-breakpoint <BREAKPOINTS>
          Add a new breakpoint to the tracer. This option can be used multiple times. The format is <syscall-stop>:<pattern-type>:<pattern>, where syscall-stop can be sysenter or sysexit, pattern-type can be argv-regex, in-filename or exact-filename. For example, sysexit:in-filename:/bash
  -h, --help
          Print help

```

Log Mode:

```bash
Run tracexec in logging mode

Usage: tracexec log [OPTIONS] -- <CMD>...

Arguments:
  <CMD>...  command to be executed

Options:
      --more-colors
          More colors
      --less-colors
          Less colors
      --show-cmdline
          Print commandline that (hopefully) reproduces what was executed. Note: file descriptors are not handled for now.
      --no-show-cmdline
          Don't print commandline that (hopefully) reproduces what was executed.
      --show-interpreter
          Try to show script interpreter indicated by shebang
      --no-show-interpreter
          Do not show script interpreter indicated by shebang
      --foreground
          Set the terminal foreground process group to tracee. This option is useful when tracexec is used interactively. [default]
      --no-foreground
          Do not set the terminal foreground process group to tracee
      --diff-fd
          Diff file descriptors with the original std{in/out/err}
      --no-diff-fd
          Do not diff file descriptors
      --show-fd
          Show file descriptors
      --no-show-fd
          Do not show file descriptors
      --diff-env
          Diff environment variables with the original environment
      --no-diff-env
          Do not diff environment variables
      --show-env
          Show environment variables
      --no-show-env
          Do not show environment variables
      --show-comm
          Show comm
      --no-show-comm
          Do not show comm
      --show-argv
          Show argv
      --no-show-argv
          Do not show argv
      --show-filename
          Show filename
      --no-show-filename
          Do not show filename
      --show-cwd
          Show cwd
      --no-show-cwd
          Do not show cwd
      --decode-errno
          Decode errno values
      --no-decode-errno
          Do not decode errno values
      --successful-only
          Only show successful calls
      --fd-in-cmdline
          [Experimental] Try to reproduce file descriptors in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
      --stdio-in-cmdline
          [Experimental] Try to reproduce stdio in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
      --resolve-proc-self-exe
          Resolve /proc/self/exe symlink
      --no-resolve-proc-self-exe
          Do not resolve /proc/self/exe symlink
      --hide-cloexec-fds
          Hide CLOEXEC fds
      --no-hide-cloexec-fds
          Do not hide CLOEXEC fds
      --timestamp
          Show timestamp information
      --no-timestamp
          Do not show timestamp information
      --inline-timestamp-format <INLINE_TIMESTAMP_FORMAT>
          Set the format of inline timestamp. See https://docs.rs/chrono/latest/chrono/format/strftime/index.html for available options.
      --seccomp-bpf <SECCOMP_BPF>
          Controls whether to enable seccomp-bpf optimization, which greatly improves performance [default: auto] [possible values: auto, on, off]
      --polling-interval <POLLING_INTERVAL>
          Polling interval, in microseconds. -1(default) disables polling.
      --show-all-events
          Set the default filter to show all events. This option can be used in combination with --filter-exclude to exclude some unwanted events.
      --filter <FILTER>
          Set the default filter for events. [default: warning,error,exec,tracee-exit]
      --filter-include <FILTER_INCLUDE>
          Aside from the default filter, also include the events specified here. [default: <empty>]
      --filter-exclude <FILTER_EXCLUDE>
          Exclude the events specified here from the default filter. [default: <empty>]
  -o, --output <OUTPUT>
          Output, stderr by default. A single hyphen '-' represents stdout.
  -h, --help
          Print help

```

Collect and export data:

```
Collect exec events and export them

Usage: tracexec collect [OPTIONS] --format <FORMAT> -- <CMD>...

Arguments:
  <CMD>...  command to be executed

Options:
      --successful-only
          Only show successful calls
      --fd-in-cmdline
          [Experimental] Try to reproduce file descriptors in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
      --stdio-in-cmdline
          [Experimental] Try to reproduce stdio in commandline. This might result in an unexecutable cmdline if pipes, sockets, etc. are involved.
      --resolve-proc-self-exe
          Resolve /proc/self/exe symlink
      --no-resolve-proc-self-exe
          Do not resolve /proc/self/exe symlink
      --hide-cloexec-fds
          Hide CLOEXEC fds
      --no-hide-cloexec-fds
          Do not hide CLOEXEC fds
      --timestamp
          Show timestamp information
      --no-timestamp
          Do not show timestamp information
      --inline-timestamp-format <INLINE_TIMESTAMP_FORMAT>
          Set the format of inline timestamp. See https://docs.rs/chrono/latest/chrono/format/strftime/index.html for available options.
      --seccomp-bpf <SECCOMP_BPF>
          Controls whether to enable seccomp-bpf optimization, which greatly improves performance [default: auto] [possible values: auto, on, off]
      --polling-interval <POLLING_INTERVAL>
          Polling interval, in microseconds. -1(default) disables polling.
  -p, --pretty
          prettify the output if supported
  -F, --format <FORMAT>
          the format for exported exec events [possible values: json-stream, json, perfetto]
  -o, --output <OUTPUT>
          Output, stderr by default. A single hyphen '-' represents stdout.
      --foreground
          Set the terminal foreground process group to tracee. This option is useful when tracexec is used interactively. [default]
      --no-foreground
          Do not set the terminal foreground process group to tracee
  -h, --help
          Print help

```

eBPF backend supports similar commands:

```
Experimental ebpf mode

Usage: tracexec ebpf <COMMAND>

Commands:
  log      Run tracexec in logging mode
  tui      Run tracexec in TUI mode, stdin/out/err are redirected to /dev/null by default
  collect  Collect exec events and export them
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

```

## Profile

`tracexec` can be configured with a profile file. The profile file is a toml file that can be used to set fallback options.

The profile file should be placed at `$XDG_CONFIG_HOME/tracexec/` or `$HOME/.config/tracexec/` and named `config.toml`.

A template profile file can be found at https://github.com/kxxt/tracexec/blob/main/config.toml

As a warning, the profile format is not stable yet and may change in the future. You may need to update your profile file when upgrading tracexec.

## Known issues

- Non UTF-8 strings are converted to UTF-8 in a lossy way, which means that the output may be inaccurate.
- For eBPF backend, it might be impossible to show some details of the tracee, See https://mozillazg.com/2024/03/ebpf-tracepoint-syscalls-sys-enter-execve-can-not-get-filename-argv-values-case-en.html
- The output is not stable yet, which means that the output may change in the future.
- Test coverage is not good enough.
- The pseudo terminal can't pass through certain key combinations and terminal features.

## Origin

This project was born out of the need to trace the execution of programs.

Initially I simply use `strace -Y -f -qqq -s99999 -e trace=execve,execveat <command>`.

But the output is still too verbose so that's why I created this project.

## Credits

This project takes inspiration from [strace](https://strace.io/) and [lurk](https://github.com/JakWai01/lurk).
