In 2018 I hand-wrote a C++ deep learning framework so I'd never pad a batch. In 2023 LLM serving landed on the same structure.
There's a sentence in the README of a library I wrote that I've been thinking about lately: "To summarize, we believe that Padding-free Dynamic Batching is the feature that NLPers will dive into but is surprisingly not supported by today's deep learning libraries ." I wrote that around 2021, about InsNet , a C++14 deep learning library I'd been building since 2018. Then transformers ate the…
In 2018, a master's student working on NLP research wrote a C++ deep learning library called InsNet to avoid padding when processing inputs of varying lengths. The library, which grew to over 21,000 lines of C++ code, eliminated the need for padded tensors by representing each value as a flat buffer with a two-integer shape: total element count and width. This design allowed InsNet to keep each value's width exactly as wide as the sentence it represented, without any padding or mask tensors.
InsNet's batching key was designed to deliberately forget sizes, grouping nodes by type signature rather than input shape. This meant that linear layers could batch together even if they processed different numbers of columns, as the column count was not included in the signature. By leaving out the column count, a single batch in InsNet could contain matrices of different shapes, making the most of the computation available.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
