How i did the Multi VM monitoring

I once managed a situation with five servers transmitting metrics to a monitoring dashboard, but I couldn't differentiate between which server each metric came from. As I added more servers running the same service on four to five machines, all metrics from all machines appeared in the same dashboard, lacking any indication of the originating server. This made it impossible to identify issues, as I couldn't tell if CPU spikes or error increases originated from server 1 or server 4.

The root cause was that each server was supposed to tag its metrics with its unique name, but my configuration setting failed to pass this information to the metric-sending code. The config file had the necessary value, but it wasn't reaching the metric-sending code as expected. Consequently, all servers sent metrics without any name or used a default name, without any errors or warnings.

To resolve this issue, I eliminated the reliance on a configuration file setting. Instead, I set the server's name explicitly when it started up, ensuring it was passed directly and explicitly to the metric-sending code. This direct handover eliminated any assumption that the setting would magically appear. After implementing this change, I verified its effectiveness by checking the actual metrics in the dashboard and confirming that each server's name appeared correctly.

This experience taught me a valuable lesson: when your application depends on a configuration setting to function correctly, don't assume it will automatically reach the intended code path. Always verify that the setting is present and correctly applied. This precaution becomes crucial when scaling from a single server to multiple servers, especially during critical times like a 2am incident when identifying the problematic machine is vital.