Overview
SpectraAdamW is an experimental optimizer for fine-tuning large language models on memory-constrained hardware. It targets the optimizer states used by AdamW, combining a factorized second moment with frequency-domain compression of momentum.
The project is currently described as a closed beta, with a Python proof of concept for validating the factorized-variance method and memory reduction. An optimized C++ CUDA backend remains in closed development.
Key Features
- Factorizes the second-moment estimate into row and column means for two-dimensional weight matrices, replacing a full matrix with much smaller tensors.
- Reconstructs the factorized variance through scalar multiplication before each gradient update, keeping the denominator positive and bounded.
- Uses an FFT-based native backend design to isolate high-energy momentum signals and mask lower-impact frequencies.
- Aims to preserve full-precision convergence behavior while reducing optimizer-state overhead by 50%.
Usage
The Python proof of concept is presented as a drop-in replacement for the PyTorch AdamW optimizer. It receives model parameters along with familiar settings such as learning rate and weight decay, then participates in the usual backward-pass and optimizer-step flow.
The implementation focuses on verifying the variance factorization and VRAM savings. Frequency-domain momentum compression belongs to the proprietary native backend rather than the Python proof of concept.
Reported Benchmark
In a simulated update for an 8192-dimension Transformer layer with batch size 128, the project reports optimizer-state memory falling from 256.00 MB with AdamW to 128.16 MB with SpectraAdamW. It also reports loss curves aligning with AdamW by step 40 in that simulation.
These figures describe the project’s stated benchmark configuration and should be interpreted in that specific experimental context.