Parameter Golf

Overview

This is Blake Paulson's non-record contribution to OpenAI's Parameter Golf challenge, where a model must fit inside a 16MB artifact and train within a strict wall-clock budget. The local submission was intentionally evaluated on one 12GB RTX 3080 rather than the official 8×H100 leaderboard hardware.

The resulting run combined a 10-layer 512-dimensional blueprint model with progressive sequence length, multi-scale rotary embeddings, byte-level token side channels, mixed-bit quantization, and compression. The recorded single-seed artifact was 15,900,102 bytes with a quantized val_bpb of 1.5568; it is presented as an exploration, not a leaderboard claim.

Methodology

The work began with a systematic survey and ablation of 12 techniques across three phases on the local GPU. Each candidate was judged against the wall-clock tradeoff, because a technique that improves a step can still lose when its overhead reduces the number of steps completed before the ten-minute cap.

The final stack uses grouped-query attention, tied embeddings, U-Net skips, a 3x MLP, Muon for matrix parameters, Adam for embeddings and scalars, and SWA. Progressive sequence lengths move from 512 to 1024 to 2048 tokens, KV groups use four RoPE bases, byte embeddings add a 64-dimensional UTF-8 side channel, and mixed int5/int6 export is compressed with zstd.

What it does

• 10-layer, 512-dimensional compact language model with GQA
• Tied embeddings, U-Net skips, 3x MLP, Muon, Adam, and SWA
• Progressive sequence-length schedule from 512 to 2048 tokens
• Four-scale RoPE bases assigned across KV groups
• 64-dimensional byte embeddings derived from token UTF-8 bytes
• Mixed-bit quantization and zstd artifact compression
• Ablation log documenting why several 8xH100 techniques failed locally