What an Over-Engineered Parity Classifier Taught Me About Representation

In an over-engineered parity classifier, the representation proved to be the crucial factor in determining accuracy. Initially, the classifier achieved 69.67% accuracy by processing binary representations through a wavelet transform, summarizing coefficients, clustering them with k-means, and attempting to recover parity from the resulting feature space. However, upon revisiting the project, the author discovered two key issues: label leakage and a non-fully unsupervised classifier.

To address these concerns, the author rebuilt the experiment with a stricter evaluation protocol, using all integers from 0 to 10,000 and splitting the data into training, validation, and held-out test sets. The revised study achieved an 84.26% held-out test accuracy with a 95% Wilson confidence interval of 82.60%–85.79%. This improvement was observed across 20 different stratified train/test splits.

The author then conducted further experiments to understand the representation's role in achieving high accuracy. By masking the natural least significant bit and keeping the rest of the wavelet pipeline unchanged, validation accuracy dropped to 48.15%, essentially random guessing. This finding ruled out the idea that the model was discovering an abstract arithmetic rule of parity. Instead, it suggested that the information was already present in the binary representation.

The author also explored the impact of representation changes on the classifier's performance. When using right padding instead of left-zero padding, validation accuracy dropped to 65.20%. This demonstrated that the position of bits within the signal affected the classifier's ability to recover the parity information. Additionally, the approximation band A3 in a level-3 wavelet decomposition contained almost all of the predictive signal, contrary to the initial expectation that fine-scale detail coefficients would be more important.

The most revealing experiment involved moving the parity-carrying bit to different positions in the signal while keeping the rest of the bits unchanged. The validation accuracy ranged from 48.15% to 98.60%, depending on the parity bit's position. This experiment clearly demonstrated that the model was exploiting a representation-dependent structure created by the wavelet transform, rather than learning an abstract rule of parity.

Boundary handling also played a significant role, with validation accuracy ranging from 54.45% to 83.20% depending on the boundary-extension mode used. Overall, the experiments revealed that the representation was the true story behind the classifier's performance, ruling out the possibility that the model had discovered an abstract arithmetic rule of parity.