Together AI's kernel research team develops high-performance GPU kernels such as FlashAttention and ThunderKittens to bridge the gap between hardware and production AI. The team focuses on optimizing low-level operations to improve efficiency and speed for AI workloads.
Why it matters: This work directly impacts the performance and cost of running large-scale AI models by making GPU utilization more efficient.
Together AI has introduced Aurora, an open-source reinforcement learning framework that transforms speculative decoding into a self-improving system. Aurora learns from every request it serves and achieves a 1.25x speedup over a well-trained static speculator.
Why it matters: Aurora enables LLM inference to continuously improve without manual retuning, potentially reducing latency and cost in production.
Together AI introduces a 'Divide & Conquer' framework that breaks long documents into parallel chunks processed by a planner, workers, and manager. This method enables smaller models like Llama-3-70B and Qwen-72B to outperform GPT-4o single-shot on long context tasks.
Why it matters: This approach shows that smaller models can surpass larger ones on long context tasks through orchestration, potentially reducing reliance on massive models.
GitHub has announced Squad, a feature that enables coordinated AI agents to operate directly within repositories using GitHub Copilot. The design emphasizes inspectable, predictable, and collaborative multi-agent workflows, representing a move toward repository-native orchestration for AI agents.
Why it matters: Squad brings multi-agent AI workflows directly into the development environment, making them more transparent and collaborative, which could change how teams automate and manage complex coding tasks.
Together AI has announced a new platform for building real-time voice agents, featuring co-located speech-to-text, large language model, and text-to-speech infrastructure. The system achieves end-to-end latency under 500ms and natively supports Deepgram and Cartesia.
Why it matters: This enables developers to build responsive voice agents with low latency, improving user experience in conversational AI applications.
Together AI has expanded its fine-tuning service to include support for tool calling, reasoning, and vision-language models. The update also enables training of models with over 100 billion parameters, offers up to 6× higher throughput, and provides job cost and ETA estimates.
Why it matters: This update broadens the capabilities of Together AI's fine-tuning platform, enabling developers to customize advanced models for complex tasks involving function calling, reasoning, and multimodal inputs.
Together AI has announced Mamba-3, a state space model (SSM) designed for efficient inference. According to the company, Mamba-3 is faster than Transformers at decode, stronger than its predecessor Mamba-2, and is available as open-source.
Why it matters: Mamba-3 highlights ongoing advancements in state space models as potential alternatives to Transformers for language model inference.
Together AI announced new launches in inference, agents, voice AI, and open models at NVIDIA GTC 2026. The company also hosted technical sessions led by its research and engineering leaders.
Why it matters: These launches expand Together AI's platform with new capabilities, highlighting ongoing innovation in the AI infrastructure sector.
Together AI has rolled out new features for its GPU Clusters, including autoscaling, role-based access control (RBAC), full-stack observability, and self-healing node repair. These enhancements are designed to deliver production-ready GPU infrastructure that scales efficiently and remains resilient for enterprise workloads.
Why it matters: The update addresses enterprise needs for scalable, reliable, and manageable GPU infrastructure for AI workloads.
Together AI has announced FlashAttention-4, a new kernel design that addresses asymmetric hardware scaling by introducing pipelining for maximum overlap, 2-CTA MMA modes to reduce shared memory traffic, and a hardware-software hybrid approach to softmax exponentials. The technique aims to keep pace with GPU throughput outpacing memory bandwidth.
Why it matters: FlashAttention-4 could significantly improve the efficiency of attention mechanisms in large language models, enabling faster training and inference as hardware continues to evolve.
At the AI Native Conf, Together AI announced FlashAttention-4, ThunderAgent, and together.compile, highlighting advancements in kernels, reinforcement learning, and inference optimization. The company stated that these research developments are being deployed directly to production on its AI Native Cloud.
Why it matters: These announcements demonstrate ongoing innovation in AI infrastructure, with Together AI moving new research into production environments.
Together AI introduced Cache-aware Prefill–Decode Disaggregation (CPD), a new inference architecture that separates warm and cold workloads. The approach delivers up to 40% higher throughput and significantly reduces time-to-first-token for long-context LLM serving.
Why it matters: This technique addresses a key bottleneck in serving long prompts, enabling faster responses for applications like document analysis and code generation.
Together AI has announced CoderForge-Preview, an open dataset intended for training efficient coding agents. The dataset is designed to support open-source AI development in code generation and understanding.
Why it matters: This release offers an open resource that could support research and development of coding AI agents and foster collaboration in the field.
Together AI's research finds that leading speech models such as Whisper and Deepgram, despite near-human benchmark scores, fail to correctly transcribe street names 39% of the time. The company outlines a proposed solution to address this significant shortcoming.
Why it matters: This research exposes a major limitation in speech AI that could affect critical real-world uses like navigation and emergency response.
Together AI has introduced Consistency Diffusion Language Models (CDLM), a post-training method that enables exact block-wise KV caching and reduces the number of refinement steps required. This approach achieves up to 14.5x latency improvements over standard diffusion language models without sacrificing output quality.
Why it matters: This development makes diffusion language models more practical for real-time applications by significantly reducing inference time while maintaining quality.
GitHub has published a blog post addressing the 'Eternal September' phenomenon in open source, where reduced contribution friction is creating new challenges for maintainers. The company describes plans to help maintainers adapt, including the introduction of new trust signals, improved triage methods, and community-led solutions.
Why it matters: This highlights GitHub's acknowledgment of sustainability challenges in open source and its intent to provide practical support for maintainers.
Together AI has introduced Dedicated Container Inference, a production-grade orchestration service for custom AI models. The service delivers 1.4x to 2.6x faster inference compared to standard approaches.
Why it matters: This enables enterprises to deploy custom models with significantly improved performance, reducing latency and cost for AI inference at scale.
New research from Together AI finds that different large language model (LLM) families exhibit distinct default behaviors when given no specific prompt. According to the study, GPT models tend to generate code and math, Llama models favor narratives, DeepSeek often produces religious content, and Qwen outputs exam questions.
Why it matters: Understanding these inherent biases is crucial for deploying LLMs in applications where neutrality is important.
Together AI has announced that Rime Arcana V3 Turbo and Rime Arcana V3 are now available on its platform. Users can now access these models through Together AI.
Why it matters: This expands Together AI's model offerings with new versions of the Rime Arcana series.
Together AI has appointed Alon Gavrielov as Vice President of Infrastructure Strategy. The company says this hire deepens its commitment to building reliable, efficient, and scalable infrastructure for AI-native teams.
Why it matters: This hire highlights Together AI's focus on strengthening its infrastructure to support AI-native teams.