strace in production: when your process won't start and logs say nothing
You know the situation. A service starts, sits there, and produces exactly zero log lines before eventually timing out. You've checked the config, the permissions, the env vars. Nothing. This is where strace earns its place. The scenario A daemon process refuses to start cleanly under systemd but works fine when you run it by hand. No errors, no logs, just silence. Classic case of something…
In the scenario where a service starts but fails to produce any log lines, strace becomes an invaluable tool. A common issue arises when a daemon process refuses to start cleanly under systemd, yet functions well when run manually. The silence is deceptive, masking a problem occurring before the application can log anything.
To diagnose, locate the process ID using `systemctl status your-daemon`, then attach strace: `strace -p -f -e trace=write`. The `-f` flag ensures child processes are followed, crucial for daemons spawning workers. `-e trace=write` filters to display write syscalls, where the application's output attempts are usually made.
Upon stracing, one case showed the process repeatedly trying to open a file in `/var/run/` that didn't exist and wasn't being created by the init script. The application called `getcwd()` and died silently when the expected path couldn't be resolved. The write(2) call was destined for `stderr`, which systemd swallowed unless `StandardOutput=journal` was set. The solution was a simple addition to the init script, discovered only through strace.
Key strace flags: `-p` for process ID, `-f` for following forks, `-e trace=...` to filter syscalls, `-t` for timestamped lines, `-r` to save to a file, and `-c` for a summary of syscall counts after detachment. Be cautious, as strace adds overhead; avoid running it on high-throughput production processes continuously. Use it during maintenance windows or on test instances.
Running strace incurs performance costs, so avoid permanent use on production processes. After use, use `strace -p -c -f` to get a syscall count summary, helping with profiling without noise. Remember, strace provides direct access to kernel messages, revealing what the process cannot through its own logging.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
