Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

Jul 16, 2026

Agora is a system that enables efficient pretraining of large language models using heterogeneous, individually owned GPUs connected via the internet. By combining bandwidth-efficient pipeline parallelism with fault-tolerant collective operations, Agora successfully trained an 8.6B-parameter model on 500B tokens using 330 contributor nodes over 40 days. The system achieved 63% of the efficiency of a centralized H100 GPU cluster, demonstrating the feasibility of large-scale, decentralized model training.

Why it matters: This work shows that large-scale AI model training can be decentralized and permissionless, potentially broadening access to frontier AI development beyond traditional data centers.

Full story at: arXiv Machine Learning ↗