Aaria-LM

Edge-Native Progressive Language Model Architecture
Designed from scratch. Built on consumer hardware. Engineered for the edge.

---

Overview

Aaria-LM is an end-to-end initiative to build, train, and deploy custom Small Language Models (SLMs) on accessible consumer hardware. Rather than treating model creation as an exercise in massive compute, this project treats language model development as a full-stack MLOps engineering problem — from random weight initialization to local edge inference.

The Production Anchor: Aaria-Micro 125M

Metric	Value
Architecture	Decoder-only Transformer (Llama-compatible)
Parameters	162,417,408 (162M)
Components	RoPE + SwiGLU + RMSNorm
Context Window	1,024 tokens
Vocabulary	32,000 tokens (TinyLlama BPE)
Training Data	3.137 Billion tokens across 8 datasets
Token/Param Ratio	19.31:1 (near Chinchilla-optimal)
Training Hardware	NVIDIA Tesla T4 × 2 (Kaggle Free Tier)
Inference Format	GGUF FP16 (311 MB)

Note: As an SFT-aligned model at 162M parameters, Aaria-Micro is a demonstration of training methodology, not a production-ready assistant. It hallucinates frequently and should not be used for factual queries.

---

Training Pipeline

┌──────────────┐     ┌───────────────┐     ┌──────────────┐
│  PyTorch     │ ──▶ │ GGUF          │ ──▶ │ Hugging Face │
│  (Scratch)   │     │ (llama.cpp)   │     │ Model Vault  │
└──────────────┘     └───────────────┘     └──────┬───────┘
                                                   │
┌──────────────┐     ┌───────────────┐             │
│ Local Edge   │ ◀── │ Ollama        │ ◀───────────┘
│ Inference    │     │ Modelfile     │
└──────────────┘     └───────────────┘

Training Curriculum

Stage	Dataset	Tokens	LR	Purpose
1	TinyStories	839.6M	6e-4	Syntactic fluency from scratch
3A	FineWeb-Edu	655.2M	3e-4	World knowledge injection
3B	StarCoderData	327.6M	1.5e-4	Python code structure
3C	Cosmopedia	327.6M	1.5e-4	Mathematical reasoning
5	OpenWebMath	327.6M	2e-4	Formal math (LaTeX, proofs)
6A	Wikipedia	327.6M	1e-4	Encyclopedic polish
6B	FineWeb-Edu	327.6M	1e-4	Domain synthesis
7	no_robots (SFT)	3.65M	3e-5	Chat alignment

---

Quickstart

Option 1 : Directly Pull using Ollama

ollama run Panda_Shrey/aaria-lm-v1

Option 2: PyPI

# Create a virtual environment (required)
python -m venv env
# Windows:
env\Scripts\activate
# Linux/macOS:
source env/bin/activate

# Install
pip install aaria-lm

# Run
aaria chat

The model weights (~326 MB) are automatically downloaded from Hugging Face on first launch.

Option 3: Ollama (Local Edge)

# With the .gguf file and Modelfile in the same directory:
ollama create aaria-structured -f Modelfile
ollama run aaria-structured

Option 4: From Source

git clone https://github.com/Shreyas-S-809/Aaria-LM.git
cd Aaria-LM
python -m venv venv
venv\Scripts\activate        # Windows
pip install -e .
aaria chat

---

Ollama Modelfile

# AARIA-LM-V1 — EDGE INFERENCE CONFIGURATION
FROM ./aaria_micro_125M_day10.1.gguf

# Persona Prior
SYSTEM """You are Aaria, a helpful, highly concise AI assistant."""

# Inference Safety Parameters
PARAMETER temperature 0.6
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.2
PARAMETER num_ctx 2048

# Structural Token Braking (Anti-Looping)
PARAMETER stop "<|end_of_text|>"
PARAMETER stop "<|user|>"
PARAMETER stop "<|assistant|>"
PARAMETER stop "<|end|>"
PARAMETER stop "User:"

# ChatML Structural Wrapper
TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}"""

---

Documentation

Document	Description
Model Card	Industry-standard spec sheet
Training Report	Complete training narrative
Architecture	125M architecture deep dive
Dataset Manifest	Data provenance & licensing
Hyperparameter Registry	Every LR/optimizer decision with rationale
Changelog	Version history
Evaluation Suite	Qualitative capability matrix
Repo Audit	Architecture & infrastructure ledger
Phase Runbooks	Per-phase execution guides
ADRs	Architecture Decision Records
Day-wise Analysis	Per-day training diagnostics
Loss Curves	Training visualization gallery
Training Telemetry	LR, throughput, and loss heatmaps

---

Repository Roadmap

• Phase 1 — Infrastructure Lock: From-scratch PyTorch training loop
• Phase 2 — Quantization Bridge: HuggingFace → GGUF conversion pipeline
• Phase 3 — Distribution Gateway: PyPI CLI engine + Hugging Face Hub
• Phase 4 — Local Containerization: Ollama edge deployment
• Phase 5 — Multi-domain CPT: 8-dataset curriculum (3.1B tokens)
• Phase 6 — SFT Alignment: no_robots chat fine-tuning
• Phase 7 — Stage 8 Global Mixture: 6-stream 100:1 over-training run
• Phase 8 — Scale Up: Port pipeline to 1.5B+ on A100/H100 clusters

---

The Architectural Pivot: 1.3B → 125M

The development of Aaria-LM follows a deliberate two-stage engineering retrospective:

1. The 1.3B Exploratory Run (Scrapped): Tested progressive blockwise depth growth ("Train-Freeze-Stitch") up to ~1.3B parameters. Revealed severe attention degradation and local probability minima when scaling depth without a pre-stabilized base layer.

2. The 125M Production Anchor (Current): Strategic reset to a highly stable 125M model under a tight 20:1 Chinchilla ratio. The objective is infrastructure-driven — lock down the full training → quantization → deployment pipeline before scaling.

See docs/adr/Aaria-LM-1.3B/ for the complete 1.3B post-mortem.

---

Contributing

See CONTRIBUTING.md for development setup, code style guidelines, and contribution process.

License

This project is licensed under the Apache License 2.0.

⚠️ SFT Data Notice: The Stage 7 SFT checkpoint uses training data from HuggingFaceH4/no_robots (CC-BY-NC 4.0), which restricts commercial use of the SFT-aligned model. The pre-SFT checkpoint is fully permissive. See docs/DATASET_MANIFEST.md for details.