OpenArch – PyTorch implementations of modern LLM architectures

OpenArch is a PyTorch repository dedicated to implementing modern open-source large language model (LLM) architectures. The project aims to provide clear, readable implementations for learning and comparison purposes, based on Sebastian Raschka's LLM Architecture Gallery. Each model is developed from scratch, one at a time, with the goal of highlighting architectural choices such as attention type, normalization, layer mix, MoE routing, and positional encoding.

The implementations are marked either ✅ as usable for forward passes or 🚧 as still under construction. The full list of 72 architectures mirrors the Architecture Gallery, and contributions are welcome. The repository is licensed under the Apache License 2.0, and individual model implementations follow the licenses of the original models when applicable.

The code is written with readability in mind, prioritizing educational value over performance. This project is a collaborative effort, and the creators welcome new contributors. They encourage users to open an issue before starting a large piece of work to avoid duplicating effort. It's important to note that any errors in the implementations are the responsibility of the authors, as they are based on publicly available information and are not affiliated with or endorsed by the original model authors.

For production use, users are advised to use the official implementations or the transformers library.