810 blocking calls at once: what the .NET thread pool did to the first burst

A cloud trade copier for MetaTrader was built using a .NET DLL with a synchronous API that blocks the thread until the broker answers. This caused delays when 27 positions closed simultaneously in MT4, with 30 followers each. The measured time from the master's callback to the follower's order call was between 1-3 ms in quiet periods, but during bursts, it increased to 125-190 ms, with a maximum of 244 ms.

Upon investigation, it was discovered that every in-flight order call holds a pool thread for the broker's answer, and during bursts, all calls are blocked simultaneously. The next piece of work needs a new thread, causing the pool to create threads gradually. Creating a thread costs about 0.2-0.5 ms, and with hundreds of blocked threads, these fractions add up to the staircase-like delay.

The thread pool's minimum threads were set too low, causing the pool to reject calls and silently reject them if either value was above the maximum. This did not help as the minimum threads stayed at the core count (4 on the server). The fix was to create threads that never retire, running on their own threads created once, kept forever, and reused.

This change eliminated the staircase-like delay, as each worker is a plain loop over a BlockingCollection. By creating threads before the burst, the worst-case scenario of 480 threads was reached, resulting in an own time of about 1 ms regardless of burst size. The issue also affected an unrelated MT5 node sharing the machine, which slowed down due to the creation of 300-450 threads by the MT4 node.