Meta unveils second-gen AI training and inference chip

Meta has unveiled its second-generation “training and inference accelerator” chip, or “MTIA”, nearly a year after the first version, and the company says its new part brings substantial performance improvements. 

Meta — as with other tech giants, such as Microsoft, Google, and Tesla — is investing in custom artificial intelligence (AI) hardware to hedge against the monopoly power of the chief GPU supplier, Nvidia. The investment is also a way to ensure the supply of computing given that Nvidia has not been able to produce enough chips to meet demand during the sudden surge in generative AI interest.

Also: 5 ways to prepare for the impact of generative AI on the IT profession

Like the first part, the MTIA version 2 chip consists of a mesh of blocks of circuits that operate in parallel, an “8×8 grid of processing elements (PEs)”. The chip performs 3.5 times as fast as MTIA v1, said Meta. It is seven times faster on AI tasks that involve “sparse” computation, those where variables have a zero value. 

Meta said benefits come from changes to the chip’s architecture and enhanced memory and storage. “We have tripled the size of the local PE storage, doubled the on-chip SRAM and increased its bandwidth by 3.5X, and doubled the capacity of LPDDR5,” said the tech giant.

The chip is built in a 5-nanometer process technology developed by contract chip manufacturing giant Taiwan Semiconductor Manufacturing. 

Also: Want to work in AI? How to pivot your career in 5 steps

The larger chip, measuring 421 square millimeters versus 373 for the v1, holds 2.4 billion gates, said Meta, and performs 103 million floating-point math operations per second. That performance compares to 1.1 billion gates and 65 million operations for the earlier model. 

As with MTIA v1, the new chip runs software that optimizes programs using Meta’s PyTorch open-source developer framework. Two software compilers collaborate — one on the front end compiles the compute graph of a program, and one on the back end is written in the open-source Triton compiler language to generate optimal machine code for the chip. 

Meta said the software development work for MTIA v1 allowed the company to rapidly bring the new chip to fruition from, “going from first silicon to production models running in 16 regions in less than nine months.” The tech company said the chip is being deployed to support ranking and recommendation advertising models.

Also: Nvidia CEO Jensen Huang unveils next-gen ‘Blackwell’ chip family at GTC

Meta said it has designed a rack-mount computer system running 72 MTIA v2s in parallel. “Our design ensures we provide denser capabilities with higher compute, memory bandwidth, and memory capacity,” said Meta. “This density allows us to more easily accommodate a broad range of model complexities and sizes.”

The tech company plans to continue to invest in custom hardware design. “We currently have several programs underway aimed at expanding the scope of MTIA, including support for GenAI workloads,” said Meta. “We’re designing our custom silicon to work in cooperation with our existing infrastructure as well as with new, more advanced hardware (including next-generation GPUs) that we may leverage in the future.”