Two days of kernel debugging ended at one default value

A recent debugging session focused on two days of kernel adjustments, ultimately settling on a singular default value. The browser training appeared to have reached a stagnation point, nearing a loss of 2.45. The curve was so smooth that it resembled a model or kernel limit. It became clear that WebGPU kernels were not the root cause of the issue. The browser utilized a learning rate of 3e-3, while the Python reference employed a more conservative rate of 3e-4. Over time, the default drifted by a factor of ten.

Kernel parity tests addressed the numerical operations; however, they did not delve into the configuration defaults. This oversight led to a situation where a correctly implemented model behaved like a broken one. The reference path served as the ultimate authority for hyperparameters and mathematical calculations. The investigation uncovered two additional boundaries.

The default corpus was disappointingly brief, containing only 863 bytes. This limited dataset obscured the model's true capacity behind its small size. The Memory64 benchmark tested an allocation path, but it overlooked the browser bridge essential to the product's functionality. All three failures shared a common trait: a narrow test passed while the actual workflow remained incorrect.

To rectify the situation, end-to-end checks were implemented at three critical boundaries: configuration, data, and integration. The complete debugging record can be found at https://posttrainllm.com/devlog.