Show HN: Mini-AGI – Dynamic continual learning model trained on 8GB VRAM

Mini-AGI is a dynamic continual learning model that was trained on a 8GB VRAM. The project, named for its ambitious features, aims to fulfill all the desired characteristics of an Artificial General Intelligence (AGI) model. A detailed explanation of the model can be found in the Readme, along with a demonstration of the training process.

The scaling graph of the model shows promising results. The model was created with dissatisfaction in mind, allowing for full control over the model's training data. This is achieved through a method called "MoE with a lot of experts that gets added and pruned from the model while it trains", only using a small subset of experts at any given time.

This approach is combined with a batch 1 training on a single continuous stream of data. This allows the model to be bounded only by the disk space in terms of the number of parameters, and makes it possible to work with small VRAM capacities. The big randomized batches and their respective gradients are not needed, making the process more efficient.

The model is being trained on a 7.8B characters corpus, and the weights are expected to be ready in a couple of weeks. The setup is simple, allowing anyone to clone the project, run it, and observe the training process. The creator acknowledges the use of AI in the development of this project, and hopes it is justified given the complexity of the task. Comments and further discussion can be found at the provided URL.