LLM fine-tuning laboratory for measuring specialization gain, training dynamics, generalization and failure behavior under constrained hardware, with a frozen benchmark and base-vs-tuned evidence.
View repository01 / Overview
CauseTune is a QLoRA fine-tuning laboratory that freezes a held-out benchmark, measures the untouched base model's capability gap, runs one controlled specialization, then re-measures base versus tuned on the same frozen evaluation with quality, efficiency and failure analysis kept as separate evidence.
A single accuracy number does not show whether fine-tuning added a real capability, whether it generalized, or what it cost. CauseTune keeps the benchmark frozen before training, selects checkpoints on validation only, and reports the gain as specific failure modes disappearing rather than one headline score — so a specialization claim is inspectable, not asserted.
02 / Engineering highlights
Problem: a benchmark the model was tuned against measures memorization, not capability. Solution: the challenge benchmark and evaluation contract are established and hashed before training, excluded from the train and validation data, and never used for checkpoint selection.
Problem: an aggregate delta hides whether the model got better or just shifted errors. Solution: base-vs-tuned is decomposed into mechanically observed transitions — 49 cases went base-wrong to tuned-correct against 1 regression — and per-family behavior, so configuration_regression (0% → 100%) and disk_io_saturation (25% → 100%) are visible individually.
Problem: fine-tuning claims often ignore what the run cost. Solution: peak allocated VRAM (5.312 GiB on an 8 GB laptop GPU), trainable-parameter share (0.814%), optimizer steps actually consumed (100 of 600) and wall time are reported as first-class results alongside quality.
Experiment 01 isolated one training-affecting variable: an unshuffled class-contiguous order collapsed held-out accuracy to 26.8% because the final optimizer windows were single-class, while a deterministic seeded shuffle recovered it to 99.2%. Train loss alone was insufficient evidence.
No benchmark-informed oversampling, no LLM judge, no manual output repair, no alternate-checkpoint fishing. Malformed outputs were scored as produced, and metrics were recomputed offline from the persisted 144 predictions without regenerating model outputs.
03 / Engineering concepts
04 / Proof & evidence
Specialization gain
65.28% -> 99.31% diagnosis exact match
Experiment 02 · frozen 144-case synthetic benchmark
Untouched Qwen3-4B versus the tuned adapter on a benchmark frozen before training and excluded from checkpoint selection. +34.03 pp; resolution exact match rose +64.58 pp. Synthetic benchmark accuracy, not production accuracy.
Generalization by slice
STANDARD 100% · HARD 97.92% · TRANSFER 100%
Experiment 02 tuned · held-out slices
The single remaining diagnosis error was one HARD downstream-timeout case predicted as DNS resolution failure. TRANSFER is transfer-style evaluation, not true OOD.
Training efficiency
Stopped at step 100 of 600 · 5.312 GiB peak VRAM
8 GB RTX 5070 Laptop · 0.814% trainable
Early stopping on validation_no_improvement avoided 500 of 600 optimizer updates. 33,030,144 trainable parameters over a 4,055,498,240-parameter logical model.
Causal training-order fix
26.8% -> 99.2% validation accuracy
Experiment 01 · M5 unshuffled vs M6 seeded shuffle
Changing only the training order from class-contiguous to a deterministic seed-42 shuffle recovered held-out performance; all 250 optimizer windows became mixed-class instead of terminal single-class.
05 / Technology stack