ANE Trainer
https://github.com/fspecii/ane-trainer- Category
- AI Tools
- Rank
- No. 1858Tools index
- Pricing
- Open Source
- Type
- TOOL
- Builder
- fspecii
- Date
About
Train LLMs on Apple Neural Engine using dynamic weight IOSurfaces — compile once, no exec restart, 110M params at 77ms/step.
What it does
ANE Trainer is a research implementation of transformer learning on Apple’s dedicated neural accelerator. It builds forward and backward computation graphs from MIL text, sends tensors through shared IOSurface memory, and keeps selected optimizer work on the CPU. The included trainer targets a Stories110M configuration, supports fresh runs, pretrained weights, checkpoints, and terminal monitoring.
Why it's ranked here
The project turns undocumented hardware into a working training backend, with unusually clear optimization history and measured bottlenecks. Its strongest result is architectural, not general availability: fused accelerator kernels, shared-memory activation flow, sparse embedding updates, and concurrent layer updates form a credible end-to-end experiment. Private APIs and narrow hardware support keep it firmly in research territory.
What's good
The design attacks overhead across the whole training loop. Forward activations remain in accelerator-accessible surfaces for backward work. Weight gradients and the large classifier backward pass run on the neural engine. Sparse embedding updates avoid touching most vocabulary rows, while layer optimizer updates run concurrently. The repository also records failed fusion attempts and measured regressions, which makes the performance claims easier to interpret.
Tradeoffs
It requires Apple Silicon and macOS 15 or newer, then relies on undocumented Apple frameworks resolved at runtime. Accelerator arithmetic is fp16, while weights and optimizer moments remain fp32 on the CPU. Causal attention needs a decomposed workaround because the hardware ignores its mask input. Rotary position embeddings are absent from accelerator training. The supplied model shape and data workflow are specialized rather than configurable infrastructure.
How to use it well
Use it for low-level Apple Neural Engine research, MIL experiments, kernel fusion studies, and reproducing the supplied transformer training setup. Start with random initialization or the expected pretrained binary, accumulate steps, checkpoint progress, and watch timing through the terminal dashboard. It does not replace a supported training framework, portable backend, hosted API, or general model-development stack.
Technical notes+
README.md describes runtime MIL generation, in-memory compilation through _ANEInMemoryModelDescriptor, fp16 IOSurface tensors in [1,C,1,S] layout, and fp32 CPU-side Adam state. training/PLAN.md documents the progression to 87 startup-compiled kernels, including fused forward work, fused attention backward work, ANE weight-gradient kernels, sparse embedding Adam, and ANE classifier backward. training/Makefile builds train_large with Objective-C ARC plus Foundation, CoreML, IOSurface, Accelerate, and dl. training/dashboard.py provides a Blessed-based TUI, parses trainer logs, reads checkpoints with NumPy, and optionally gathers process metrics through psutil. training/tokenize.py extracts flat uint16 token IDs from the expected TinyStories archive. training/model.h and training/train.m retain an earlier baked-weight path that recompiles kernels after updates and can fall back to CPU.
Observed
- License
- MIT License
- Primary language
- Objective-C for the trainer and accelerator probes, with Python utilities
- Install surface
- Make target or direct xcrun clang build; core trainer uses Apple system frameworks
- Interfaces
- Command-line trainer and Python terminal dashboard
- Platform support
- Apple Silicon with macOS 15 or newer
- Runtime dependencies
- Private Apple Neural Engine APIs loaded dynamically; dashboard uses blessed and NumPy, with optional psutil
- Training precision
- ANE compute uses fp16; CPU maintains fp32 weights and Adam moments
Read from README.md, training/tokenize.py, training/dashboard.py, LICENSE, inmem_peak.m, sram_bench.m, sram_probe.m, inmem_basic.m, inmem_bench.m, api_exploration.m, training/PLAN.md, training/model.h, training/train.m, training/Makefile, training/README.md.
What it can do
Train large language models on Apple Neural Engine
LLM training data and model configuration → Trained LLM optimized for Apple Neural Engine
Compile model for Apple Neural Engine execution
LLM model architecture and parameters → Compiled model ready for Apple Neural Engine
Execute training steps without application restart
Training batches and dynamic weight updates → Updated model weights via IOSurfaces
Process models with up to 110 million parameters
Large language model with 110M parameters → Training step completion in 77ms
Update model weights dynamically using IOSurfaces
New weight values during training → Real-time weight updates without memory reallocation
Tags
Tech Stack
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.