Pathway’s brain-inspired architecture development on Amazon SageMaker HyperPod

Pathway has developed a brain-inspired architecture named BDH (Dragon Hatchling) for use on Amazon SageMaker HyperPod. This architecture performs reasoning in latent space, avoiding the need for intermediate text traces generated through chain-of-thought methods. Unlike traditional transformer models, BDH uses sparse, local interactions between neurons to communicate and maintain state in synapse-like connections, eliminating the need for fixed-size context windows or heavy chain-of-thought tokens.

Traditional large language models (LLMs), including transformers, have had challenges with general intelligence and long-term coherent reasoning. They forget information during long interactions, don't retain knowledge between sessions, and require retraining to acquire new knowledge. BDH-CQ, however, updates the model's internal memory during inference, enabling iterative computation without generating lengthy reasoning traces or necessitating fine-tuning or retraining.

Integrating with PyTorch, BDH can scale training on Amazon SageMaker HyperPod, which efficiently shares compute resources in a resilient, scalable, and cost-effective manner. By reformulating sequence modeling as local graph dynamics on a network of interacting neuron particles, BDH aligns better with computational efficiency and natural intelligence principles.

The architecture is designed to scale efficiently in production environments, with only 5 percent of neurons active at any given time, reducing computation per inference step.