Nvidia’s rivals have built their own AI chips and now want to make the software switch easier

Nvidia’s rivals want artificial intelligence (AI) developers to switch chips without rewriting their code, as Amazon and Google try to close a software gap that has helped keep the chipmaker ahead.

Amazon released TorchNeuron on October 20 as a public beta. The open-source tool allows developers to run existing PyTorch code on its Trainium2 and Trainium3 AI chips with minimal changes, according to the company.

PyTorch is a widely used framework for developing and running AI models.


Meanwhile, Nvidia’s position in AI computing rests partly on CUDA, its software platform that helps developers run models efficiently on its graphics processing units (GPUs). Its libraries, which give ready-made software functions, and tools for finding errors and improving performance build an ecosystem that rivals must also compete with.

Nvidia’s software ecosystem makes switching chips more complicated because a company considering different hardware must also account for the work involved in adapting its software, checking that models still produce correct results and check performance. Amazon says this engineering work accounts for a significant part of the cost of moving to another chip.

Also Read: Amazon says its AI chips will work with popular coding language, challenging Nvidia

Cutting the work of switching

TorchNeuron lets developers use their usual PyTorch tools to test code, fix errors and run models across multiple Trainium chips. For supported code, Amazon says developers can change the device setting rather than rewrite the model.

Voice AI startup Cartesia used TorchNeuron to run its inference pipeline on Amazon’s Trainium chips with minimal changes to its PyTorch code. “With PyTorch native support on AWS Trainium, we ran our standard inference pipeline with minimal changes to our PyTorch code,” said Arjun Desai, co-founder of Cartesia, in a PyTorch blog post published on October 19.

Speech AI company Deepgram also ported its models to Trainium. “In the benchmarks we evaluated, Trainium processes about 60% more input audio seconds per dollar than our existing GPU baselines,” said Adam Sypniewski, CTO of Deepgram.

The company is also contributing changes to PyTorch to remove assumptions that tie parts of the framework to particular chips. TorchNeuron will use the Apache 2.0 open-source licence, allowing developers to access and modify the software.

Google is pursuing a similar approach with TorchTPU, which connects PyTorch directly to its Tensor Processing Units (TPUs). These are custom chips that Google uses to train and run models, including its own Gemini models, and offers to cloud customers.

Google says TorchTPU is designed to let developers move existing PyTorch workloads with minimal code changes, preserving the main training code and the tools they already use. In an April technical post, Google described its goal as letting developers change the hardware setting in an existing PyTorch script while retaining the core logic of their training code.

Microsoft is also building its own AI chips for Azure, including Maia 200, which it introduced in January to run trained AI models. The company also announced a preview of software tools that include integration with PyTorch.

Compatibility is the first hurdle

These moves show that these companies are trying to make their chips easier to use as they compete with Nvidia. That means supporting the software and tools developers already use, so they do not have to change how they build and run AI models.

However, TorchNeuron’s first release has limits. Amazon’s technical announcement lists unsupported features, including dynamic and symbolic shapes, where the dimensions of data processed by a model can change during execution.

Google also acknowledges that getting code to run on another chip is only part of the task. Its technical post says developers may need to adjust model design or write specialised code to make the best use of TPU hardware.