{
  "id": 17500250,
  "title": "Too big, it explodes. Too small, it crawls.",
  "url": "https://urgent.news/2026/11/01/too-big-it-explodes-too-small-it-crawls",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-01T00:20:00.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/logitsolutionsai/too-big-it-explodes-too-small-it-crawls-pio"
  },
  "original_language": "en",
  "account": "The learning rate, denoted as η (eta), is a crucial factor in training neural networks as it determines the step size taken by the gradient descent algorithm. Every weight update in gradient descent follows the formula w ← w − η ∇L, where ∇L represents the gradient of the loss function. The learning rate is a setting that must be chosen by the practitioner, and it is the first hyperparameter to be tuned.\n\nTo illustrate the importance of the learning rate, consider a single weight 'w' in a simple loss function L(w) = w². The gradient of this loss function is 2w, and the update rule becomes w ← w − η · 2w = (1 − 2η) · w. Starting from w = 2, with different learning rates, the weight behaves differently:\n\n1. With η = 0.01, each step multiplies w by 0.98. After ten steps, w has reached 1.634, and the loss is still 2.67. The weight is crawling towards the minimum.\n2. With η = 0.4, each step multiplies w by 0.2. In ten steps, w reaches 0.008, and the loss is significantly reduced.\n3. With η = 0.9, each step multiplies w by -0.8. The weight oscillates across the minimum, but eventually lands on the other side.\n4. With η = 1.1, each step multiplies w by -1.2. The weight grows exponentially, causing the loss to increase dramatically.\n\nThe optimal learning rate, where the weight reaches the minimum in the fewest steps, is 0.5 for this example. In general, stable training requires 0 ≤ η ≤ 2/λ, where λ is the curvature of the loss function (the second derivative). For the given loss function L = w², the curvature is λ = 2, so the optimal learning rate is 1.\n\nWhen training a real neural network with many weights, the loss function has multiple directions with varying curvatures. The steepest direction dictates the speed limit, while flatter directions progress more slowly. Techniques like learning rate decay schedules can be employed to adapt the learning rate during training. For instance, starting with a higher learning rate like 1e-1 and gradually decreasing it to a smaller value like 1e-4 can help the network converge efficiently.",
  "summary": "The learning rate, step by step: why one number decides whether training works at all. Backpropagation tells every weight which way to move: its gradient. It doesn't say how far. That's the job of the learning rate, written η (eta). Every step of gradient descent is w ← w − η ∇L The gradient comes from the model. The learning rate is a setting you choose, and it is the first one people tune. This…",
  "key_points": [
    "Learning rate determines step size in gradient descent",
    "Practitioner sets learning rate as first hyperparameter",
    "Optimal learning rate 0.5 for given example loss function"
  ],
  "editors_take": "Precise hyperparameter tuning transforms the learning rate from a source of instability or stagnation into a tool for efficient convergence within the complex landscapes of neural network training.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}