Tokens-as-a-Service (TaaS)

This week, Groq announced a $640 million funding round with Fabrica Ventures among the participating investors.

AI chips address two critical phases of the AI lifecycle: the training of new models based on existing data and the inference of those models on live data in production. It is expected that inference demand should become much larger than training (10 to 1?). Groq has developed a revolutionary ultra-low-latency AI inference chip known as Language Processing Unit (LPU).

Groq has become viral because of the impressive demos showcasing open-source LLMs running on inference APIs, where it achieved 10x the throughput of other inference services.

This instant speed has drawn a wave of developers to Groq, leading to the creation of a wide array of new and innovative AI applications and models. Indeed, Groq has quickly expanded its community to 360K+ developers who are building on GroqCloud, its cloud-based platform that offers on-demand access to Groq’s LPUs. Instead of selling hardware, Groq will offer a tokens-as-a-service (TaaS) model through GroqCloud, making access simple and straightforward.

Groq’s chip has a fully deterministic architecture, with no buffers. It has no external memory, and it keeps weights, activations, etc. all on-chip during processing. Because each chip has little memory, no useful models can actually fit on a single chip. Instead, they must utilize many chips to fit the model and network them together.

Jonathan Ross, CEO and founder of Groq, is widely regarded as a genius, besides being a visionary leader and a great salesman. Before creating the Groq’s LPU, he developed Google’s Tensor Processing Unit (TPU), an AI accelerator chip for neural network ML.

Conclusion

With this new funding, Groq is expected to deploy 108K LPUs by the end of Q1 2025, which will be the largest AI inference compute deployment of any non-hyperscaler.

Groq has also established partnerships with Earth, Wind & Power AI computer center in Norway and with Aramco Digital in Saudi Arabia, where it plans to deploy 100K+ LPUs next year.

So, if everything proceeds as planned, Groq is on track to become the fastest-growing startup ever, with revenue expected to soar from a few million dollars to a billion in 2025.