All research

ASIC-Inspired Small Language Model (<1B Parameters)

Topics

LLM
Model Architecture
Quantisation
Python

Most language model research focuses on scaling — more parameters, more compute, better performance. This project took the opposite direction: what if you designed a language model the way chip designers design ASICs — fixed-function, inference-optimised, minimum viable capability?

Motivation

ASICs (Application-Specific Integrated Circuits) are efficient because they do one thing extremely well, with no generalisation overhead. The hypothesis: a language model designed with a specific narrow task in mind — and aggressively optimised for inference at that task — could be practical on-device without a GPU.

Architecture Exploration

The research explored: - Reduced-head attention configurations trading quality for speed - Aggressive quantisation (INT8, INT4) from the training stage rather than post-training - Structured pruning to remove redundant attention heads and MLP neurons - Fixed vocabulary and tokenisation tuned for the target domain rather than general text

Findings

Aggressive quantisation and pruning combined can reduce model size by 60–80% with manageable quality loss for narrow tasks. The key insight: most of the quality loss in small models comes from trying to generalise — constraining the task definition early allows much deeper optimisation.

Status

Prototype and research notes. Not published. Ongoing exploration.

Tools

Python, PyTorch, HuggingFace, custom quantisation scripts.