Google has a new TPU v5p chip that it used to train its biggest public model yet, dubbed Gemini. This is an update to the Google Brings TPU v5e that we have seen previously. The new TPU v5p is combined into pods and Google has already demonstrated deploying and training on them at scale.
Google TPU v5p AI Chip Launches Alongside Gemini
The Google TPU v5p is said to offer ~2.3x better performance per dollar than the TPU v4, and training LLMs at 2.8x the speed of its predecessor. The Google TPU v5p is clustered into pods of 8960 chips with 4,800 Gbps/chip of bandwidth and 3x the HBM at 95GB total. Embedding dense models train at around 1.9x the speed of their predecessors.
These new TPUs support features like INT8 for more efficient computing in terms of power and bandwidth. They also offer considerably more performance per chip than the v5e as well as allowing larger topologies.
Final Words
This is a cool disclosure by Google. Historically, Google has only disclosed chip designs after at least one new generation is in production. Still, the company is facing an enormous amount of pressure to stay ahead in the generative AI game. The big part of that is launching huge LLMs like Gemini. The other part is pushing its hardware advancements to ensure it is attracting hardware talent over other companies like NVIDIA.
While this is great to see, at the end of the day, this is a Google Cloud-specific device. That is great if one is using the Google Cloud and has access to that many accelerators to their account. For those who are not on the Google Cloud, this is probably less relevant since it is not an on-prem or cross-cloud chip. For many companies, if they have not been reading STH’s AI system coverage for years and investing in liquid cooling and higher power racks, they may have no choice but to use a cloud solution like this.
One thing is for sure, it is great to see Google showing more frequent hardware disclosures. Hopefully one day we can go see these in person.
… I just want a next gen coral device lololol
In all seriousness, I think their restriction to using their own cloud is hurting their potential, as cloud lock-in is becoming more and more serious a concern for many :-/