Five Linux Field Guides: Production Traps, SSH, Small VPSes, Debugging, and Maintenance

1. Common Linux production issues Production mishaps can arise without any alarming crashes. They might occur when a deployment executes under your account but not for the service user, a disk fills up due to still-open deleted logs, or a configuration alteration takes effect following an unexpected restart. Testing as the service user A command that operates smoothly in your shell might not function within a service.

Your login account might possess distinct group membership, environment variables, working directory, or access to a secret file compared to the account operating the application. Verify the configured identity and paths instead of presuming they align with your shell: systemctl show myapp -p User -p Group -p WorkingDirectory If feasible, attempt a read or write operation using that account with sudo -u appuser ... .

For file access, examine every parent directory, not merely the file itself: a process necessitates search (x) permission on each directory along its path. Assuming a service inherited your shell environment Variables exported in .bashrc or an interactive shell typically aren't accessible to a system service. Position service-specific settings in an explicit location, such as a systemd unit's EnvironmentFile=, and restrict that file's access if it contains confidential information.

After modifying a unit, reload systemd's unit definitions and restart or reload the service as appropriate. Mistaking free space for free inodes A filesystem may exhibit available capacity but lack available inodes for new files. Examine both: df -h / df -i / A workload generating numerous small cache files may exhaust inodes before consuming substantial disk space.

Identify the directory generating files and rectify its retention behavior rather than indiscriminately deleting data. Deleting a log still held open by a process Deleting a sizable log file does not invariably free up its space when a running process continues to hold it open. If df indicates a full filesystem but directory totals do not account for it, scrutinize open deleted files: sudo lsof +L1 Utilize the application's supported log-reopen mechanism or a scheduled service restart.

Refrain from truncating random open files without comprehending the application's logging behavior. Modifying configurations without a rollback path Prior to altering a production configuration, retain a verified copy, verify its syntax using the application's own checker, and implement the change in a reversible manner. For SSH or firewall modifications, maintain an active session open and validate a second connection before severing it.

For systemd unit edits, systemctl cat myapp displays the effective unit and its drop-ins. Prevention revolves around making assumptions transparent: which user operates the service, which filesystem stores its data, and which exact configuration file the process reads. To interpret service status before making changes, refer to the systemctl status guide.