Federated Learning: Why Averaging Model Updates Works - and When It Fails
Most machine learning still assumes a convenient fiction: all the data sits in one place, ready to be shuffled, split, and trained on. Real data rarely cooperates. It lives on phones, in hospitals, in banks, and in factories, where moving it is expensive, slow, or simply not allowed. Federated learning is the attempt to train anyway, without centralising the data. The core idea is disarmingly…
Federated learning enables machine learning models to be trained on data that remains distributed across devices. Instead of moving data to a central location, the model itself is sent to each device. Each device trains locally on its own data, then shares only the update or new model with a central server. The server then averages all these local updates to create an improved global model.
This process, known as Federated Averaging or FedAvg, appears too simple to be effective. However, it generally works and only fails in certain circumstances. FedAvg addresses the challenge of heterogeneous data - data that differs significantly between devices. For example, one hospital might have different patient conditions than another, or users on different phones may type differently.
This non-IID (not independently and identically distributed) data is a major hurdle in federated learning. In a single round of FedAvg, the server sends the current global model to a subset of clients. Each client then runs several steps of stochastic gradient descent (SGD) locally, using only its own data, to obtain a new local model.
The client then sends its update back to the server. The server then computes the next global model by taking a weighted average of all the local updates. Two key design choices affect the performance of FedAvg. First, multiple local SGD steps are performed rather than just one. This reduces communication requirements, which can be a bottleneck when dealing with slow networks or intermittently available devices.
Second, updates are weighted by the size of each client's dataset. A client with more data has a greater influence on the final global model. While these defaults make sense, they can also lead to problems. It may seem counterintuitive, but averaging model parameters does not lead to averaging predictions. For linear models, averaging parameters and predictions yield similar results, but this is not the case for deep neural networks.
Averaging two models trained from different starting points usually results in a poor model, as the same function is being learned in both cases.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
