Show HN: DeepSeek V3 in ~200 lines of pure Jax

DeepSeek V3, a deep learning model, has been implemented in just 190 lines of pure JAX code. The architecture was developed from the technical report "DeepSeek-V3" and trained on public-domain detective fiction using a MacBook Air. The model utilizes a unique approach, with all parts of the model presented as functions that can be easily read top to bottom in the "deepseek_v3.py" file.

The training process involved processing 207 books from public domain sources, cleaning and tokenizing the text, and generating data files for training and validation.

The model was trained using configurations that included balancing and bias evens, which resulted in a validation loss of 3.921 and 3.923, respectively, at step 2000. This demonstrates the effectiveness of the bias evens in even out the experts without incurring any additional costs. The authors claim that their approach supports the paper's assertion.

It is important to note that DeepSeek V3 is an architecture demonstration rather than a strong language model. The write-up includes an analysis of what worked, what didn't, and what was missing from the implementation. The code comes with a disclaimer regarding the use of checkpoints, stating that only those loaded from trusted sources should be utilized.

The required environment for running the code includes an Apple Silicon Mac, Python 3.12, and uv. Mac users may need to refer to the "Running on CPU" section for alternative instructions.

The code is designed to be compatible with JAX's ROCm backend, allowing for cross-platform compatibility. Users can find the full training logs and checkpoints in the "runs" directory, with different configurations available for comparison. The model's performance on public-domain texts is an important aspect of the demonstration, as it showcases the potential of the approach while also highlighting the limitations due to the nature of the training data.