5 Linux Security Checks I Never Skip on Production Servers

In the realm of Linux server management, even a fully patched and operational system can harbor hidden security vulnerabilities. Over two decades of experience in IT infrastructure and information security has taught one valuable lesson: a robust server does not guarantee optimum security. Here are five critical areas I meticulously examine, particularly on production systems that have been in service for extended periods.

1. SSH Configuration: SSH often marks the entry point for many security assessments. I meticulously scrutinize the SSH settings, including password authentication, root login, and any configuration overrides. Tools like `sshd -T | grep -Ei` help unveil the effective SSH settings. However, it's crucial to bear in mind that match rules could yield different outcomes based on the user or source address.

Before implementing any SSH changes, running `sudo sshd -t` ensures the configuration's validity. Above all, never sever an active SSH session until a successful reconnection has been established, for a simple SSH hardening tweak could inadvertently disrupt production access.

2. User Accounts and Privileged Access: Over time, temporary accounts created for troubleshooting or legacy applications might evolve into permanent fixtures, unnoticed by system administrators. Commands like `getent passwd`, `getent group`, and `sudo awk` allow me to identify these lingering accounts. For RHEL-based systems, checking the wheel group using `getent group wheel` adds another layer of scrutiny.

It's pivotal to question the necessity of each account and their associated privileges, ensuring they are still relevant and required.

3. Listening Services: A seemingly healthy server may inadvertently expose services without explicit intent. By running `sudo ss -tulpen`, I assess the listening ports, paying special attention to those on 0.0.0.0 or ::, which could potentially be accessible from the internet. However, the mere presence of such services doesn't automatically imply internet exposure.

Factors like firewall rules, routing configurations, containerization, and cloud security groups all play a role in actual exposure. Yet, every unexpected listening service warrants a closer look.

4. Firewall Configuration: Over time, firewall rules can accumulate, making it challenging to discern which connections are truly necessary. On Ubuntu systems utilizing UFW, `sudo ufw status verbose` provides a detailed view of the active firewall rules. For RHEL systems with firewalld, `sudo firewall-cmd --list-all` serves the same purpose.

Additionally, reviewing the active nftables ruleset with `sudo nft list ruleset` offers further insight. It's not enough to merely confirm that the firewall is enabled; understanding the permitted connections, their justification, and whether these exceptions are still relevant is of utmost importance.

5. Security Updates and Configuration Drift: While patching the operating system is crucial, it does not inherently transform a system into a hardened fortress. On Ubuntu, `apt list --upgradable` and on RHEL-based systems, `sudo dnf check-update` help identify available updates. However, it's essential to review the purpose of these updates, their potential operational impact, and any requisite restarts before deployment.

Beyond mere patching, configuration drift poses another challenge. Over time, as servers pass through multiple hands - administrators, troubleshooters, during migrations, and during application upgrades - security exceptions can become entrenched within the environment without anyone recalling the rationale behind their introduction. Periodic configuration reviews become indispensable to address this issue.

While manual review of these settings can be insightful, it can become laborious when managing multiple Linux systems. This is where tools like NextGenAlert Linux Security Audit come into play. A free Community Edition tool designed to review common Linux security settings, NextGenAlert generates reports in HTML, PDF, and JSON formats.

The checks are run locally, providing a comprehensive overview that could otherwise be overlooked. For those managing Ubuntu, RHEL, AlmaLinux, or other supported Linux systems, exploring the free tool is advised. However, it's important to remember that automated audits are but a starting point, not a substitute for a deep understanding of the environment.