Fractile is currently developing AI inference accelerators based on in-memory compute and is aiming to be able to run frontier AI models – large language, vision and audio models – two orders of magnitude faster than existing hardware, at a tenfold reduction in cost.
From ChatGPT to the open-source Llama model series, LLMs and other foundation models are finding widespread application and model inference – the process of serving these trained models – is coming the dominant portion of compute costs, exceeding the cost of model training.
Fractile has licensed Andes’ AX45MPV RISC-V vector processor, combined with ACE (Andes Automated Custom Extension) and Andes Domain Library, and plans to incorporate the vector processing unit into the company’s first-generation data centre AI inference accelerator.
Fractile’s uses novel circuits to execute 99.99% of the operations needed to run model inference in on-chip memory, removing the need to shuttle model parameters to and from processor chips and instead baking computational operations into memory directly.
This architecture drives both much higher energy efficiency (TOPS/W) as well as dramatically improved latency on inference tasks (tokens per second per user in an LLM context, for instance).
The company has been betting on inference scaling – leveraging more inference time-compute to improve AI performance – as the next frontier of AI scaling. OpenAI recently released their latest LLM, o1, which requires orders of magnitude more inference compute than previous LLMs. Fractile’s hardware and software stack is built to take models that can still take many seconds to produce an answer on current hardware and make this instantaneous.
As part of the collaboration, Fractile will integrate Andes Technology’s high-performance RISC-V vector processor with its own in-memory computing architecture via ACE.
Commenting Dr. Walter Goodwin, CEO and founder of Fractile, said, “The limitations of existing hardware present the biggest barrier to AI performance and adoption. Andes Technology has unmatched technical and commercial leadership on RISC-V vector processors and is a natural partner for us as we build Fractile’s accelerator systems. Building hardware for AI acceleration is intrinsically hard – the world’s leading models can change overnight, while chips take time to bring to market. Software-programmable vector processors like Andes’ are a key part of staying robust to these changes.“
