Urgent.News

What's breaking now, across thousands of outlets.

Tech

Debugging Linux Performance with /proc, PSI, and Cgroups

Debugging Linux Performance with /proc , PSI, and Cgroups When a Linux application becomes slow, the first instinct is often to inspect application code, check CPU usage, or restart the affected process. But Linux performance problems are not always caused by application-level code. A container may be experiencing CPU throttling even when the host has available CPU capacity. A process may have…

When a Linux application becomes slow, the instinct is often to examine the application code, check CPU usage, or restart the process. However, performance issues may not stem from application-level code. A container might be experiencing CPU throttling even when the host has available capacity. A process may have low CPU usage but spend time waiting for I/O.

Memory pressure may appear even if the system shows sufficient memory. Understanding which resource is constrained, which process or container is affected, and whether supported measurements validate the suspected cause is crucial for effective performance debugging.

Begin by identifying the problem's scope. Examine host-level metrics (CPU, memory, disk, and network activity), container-level resource limits, CPU throttling, and memory constraints, and process-level metrics (CPU consumption, memory usage, I/O activity, and scheduling behavior). These levels are interrelated, but they are not interchangeable.

High memory usage on the host does not necessarily mean one process is responsible. Low host CPU utilization does not confirm a container has sufficient capacity, as its cgroup may impose a lower limit.

Collect CPU measurements over time from /proc/stat. These counters are cumulative, so isolated readings are insufficient to understand current CPU activity. Collect measurements over a consistent interval and compare changes. Look at user CPU time, system CPU time, idle CPU time, I/O wait, steal time, and context switches. A process may appear idle even if it has accumulated significant CPU time over hours.

For processes, inspect CPU time using /proc/PID/stat or tools like ps. Rate calculations should be based on observation windows, not cumulative counters.

Memory pressure, not just memory usage, needs investigation. Linux caching can make free memory appear abundant, but this does not independently establish that the system is under pressure. Start with memory totals and available memory from /proc/meminfo. Examine memory-related counters like pgfault, pgmajfault, and oom_kill in /proc/vmstat.

Linux Pressure Stall Information (PSI) provides visibility into resource stalls. Review /proc/pressure/memory to see if tasks are experiencing delays due to memory constraints. Correlate memory availability, page faults, memory pressure, and OOM-related events for a comprehensive understanding.

When dealing with containers, inspect cgroups. Even if a host has available memory and CPU capacity, a container may be constrained by its own limits. For cgroup v2, examine the relevant workload's cgroup directory, such as /sys/fs/cgroup/cpu.max, /sys/fs/cgroup/cpu.stat, /sys/fs/cgroup/memory.current, /sys/fs/cgroup/memory.max, and /sys/fs/cgroup/memory.events.

Do not assume the root directory contains the correct values, as the workload's cgroup may be nested. Check CPU quota, available capacity, throttled usage, memory usage relative to the configured limit, and OOM events in cgroups. Cgroup v1 uses different file layouts and counter names, so collectors must account for the version in use.

Once the affected process is identified, inspect its individual resource usage. In /proc/PID/status, find the Name, State, VmRSS, VmSwap, voluntary_ctxt_switches, and nonvoluntary_ctxt_switches fields. For example, use ps -p 1234 -o pid,comm,%cpu,%mem,nlwp,state to summarize process-level information. Context switches and I/O counters are more useful as changes or rates than isolated totals. If permissions allow, examine /proc/PID/io and /proc/PID/fd for additional I/O activity and open file descriptor information.

Block-device statistics and network connections can reveal disk and network bottlenecks. Linux exposes disk statistics in /proc/diskstats, and network activities can be inspected using ss -tan. Pay attention to disk latency, accumulated I/O activity, socket states, queue lengths, and TCP retransmission trends. A non-zero queue or retransmission counter alone does not confirm a bottleneck. Correlate these observations with application latency and changes over time.

Active profiling should be used sparingly. Passive telemetry should be the first step. When more detailed investigation is needed, tools like strace and perf can provide insights into system calls, scheduling, and process-level activity. However, active profiling introduces overhead and may require additional permissions or capabilities. Keep profiling targeted, use bounded durations, and understand the impact before enabling it.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Make the agent use the existing Link component, not a raw <a>, for internal routes

The agent adds a “Back to settings” control as <a href="/settings"> . The app already has a Link (or RouterLink , or Next.js Link ) that prefetches, keeps the client transition, and applies the active…

  • Use existing Link component for internal routes
  • Avoid raw a tags for internal navigation
  • Link component offers prefetching, transitions, active classes

CVE-2026-111181: When Uncaught Stream Errors Escape Routing Controls

CVE-2026-111181: When Uncaught Stream Errors Escape Routing Controls CVE-2026-111181 exposes a boundary that many application teams do not map explicitly: request parsing happens before the route…

  • High-severity denial-of-service vulnerability in Express and body-parser
  • Uncaught stream errors escape routing controls due to request parsing
  • Upgrade to body-parser 1.20.9 or 2.4.0 to remediate

More from Thursday 19 November →