Python: Call Multiple APIs in Parallel with asyncio.gather and Semaphore in 10 Minutes
When you need to call dozens of APIs — syncing data, crawling prices, firing webhooks — a sequential for loop adds up the waiting time of every request: 20 requests × 1 second = 20 seconds. asyncio.gather runs many coroutines concurrently inside a single event loop, while asyncio.Semaphore limits how many requests fly at the same time so you don't overwhelm the partner API. This technique applies…
When you need to call multiple APIs simultaneously, sequencing them with loops adds up all the waiting time. Using asyncio.gather and Semaphore, you can run coroutines concurrently inside a single event loop, speeding up I/O-bound tasks like HTTP calls. To avoid overwhelming partner APIs, Semaphore limits the number of concurrent requests.
For each of 20 users, fetching info from a partner API that takes about 1 second each sequentially would take ≈ 20 seconds. By using asyncio.gather with Semaphore(5), you can run up to 5 requests concurrently, reducing the total time to ≈ 4 seconds. To handle errors, use return_exceptions=True, allowing the function to return a list of results including any exceptions that occurred.
Start with Semaphore(5) and adjust based on the API's response limits and observed performance. Remember, this technique is useful for any I/O-bound tasks but not for CPU-bound work.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
