strace in production: when your process won't start and logs say nothing

In the scenario where a service starts but fails to produce any log lines, strace becomes an invaluable tool. A common issue arises when a daemon process refuses to start cleanly under systemd, yet functions well when run manually. The silence is deceptive, masking a problem occurring before the application can log anything.

To diagnose, locate the process ID using `systemctl status your-daemon`, then attach strace: `strace -p -f -e trace=write`. The `-f` flag ensures child processes are followed, crucial for daemons spawning workers. `-e trace=write` filters to display write syscalls, where the application's output attempts are usually made.

Upon stracing, one case showed the process repeatedly trying to open a file in `/var/run/` that didn't exist and wasn't being created by the init script. The application called `getcwd()` and died silently when the expected path couldn't be resolved. The write(2) call was destined for `stderr`, which systemd swallowed unless `StandardOutput=journal` was set. The solution was a simple addition to the init script, discovered only through strace.

Key strace flags: `-p` for process ID, `-f` for following forks, `-e trace=...` to filter syscalls, `-t` for timestamped lines, `-r` to save to a file, and `-c` for a summary of syscall counts after detachment. Be cautious, as strace adds overhead; avoid running it on high-throughput production processes continuously. Use it during maintenance windows or on test instances.

Running strace incurs performance costs, so avoid permanent use on production processes. After use, use `strace -p -c -f` to get a syscall count summary, helping with profiling without noise. Remember, strace provides direct access to kernel messages, revealing what the process cannot through its own logging.