LLM fine-tuning laboratory built around the full craft, not one training run: capability-gap measurement, QLoRA recipe selection under a hardware and data budget, held-out generalization testing, failure-mode and calibration analysis, and cross-model replication.
View repositoryCauseTune is a QLoRA fine-tuning laboratory that measures a model's capability gap, selects a training recipe under a frozen validation-only study, then treats the resulting gain as a claim to stress-test rather than a result to publish: does it survive data the adapter never saw, a second model architecture, and evidence deliberately made ambiguous. Quality, efficiency and failure behavior are kept as separate, honestly scoped evidence.
A single accuracy number does not show whether fine-tuning added a real capability, whether it generalizes past the benchmark it was measured on, or what it cost. CauseTune keeps the benchmark frozen before training, selects checkpoints and recipes on validation only, and then goes further: the confirmed gain is re-tested on data the adapter never saw, on a second model family, and under evidence deliberately made ambiguous — the same discipline a fine-tuning recipe needs before anyone trusts it past the benchmark it was tuned on.
Problem: LoRA rank, learning rate and training-data volume are usually picked by intuition or copied from a blog post. Solution: a frozen, validation-only study swept data volume, adapter capacity and learning rate under a fixed hardware budget before any number was reported, so the selected recipe is a defensible choice, not a lucky run.
Problem: a benchmark the model was tuned against measures memorization, not capability. Solution: the challenge benchmark and evaluation contract are established and hashed before training, excluded from the train and validation data, and never used for checkpoint selection.
Problem: a frozen benchmark can still be a proxy the model quietly overfit to. Solution: the tuned adapter was retested on a second, completely independent blind challenge it never saw during training or checkpoint selection — diagnosis held at 98.33% against an untouched-base 66.67%, only 0.98 pp below the original frozen-benchmark result.
Problem: a recipe hitting 100% on validation looks finished. Solution: a cheaper, cost-optimized recipe reached 100% diagnosis, resolution and strict-JSON on validation, but the same independent blind challenge exposed a 5.83 pp diagnosis regression against the original recipe — preserved as evidence instead of retroactively explained away.
Problem: a specialized model can sound certain even when the evidence does not support a diagnosis, and accuracy alone never surfaces that. Solution: sufficient, insufficient, contradictory, ambiguous and out-of-taxonomy evidence were tested directly — 100% accuracy when evidence was sufficient, but an 18.75% (9/48) false-confident diagnosis rate at the boundary, measured directly instead of hidden behind accuracy.
Problem: a result on one model family is easy to overstate as general. Solution: the same controlled methodology was rerun on a second, architecturally different model (Phi-4-mini) — untouched baseline 0% diagnosis, tuned 63.33% (38/60), so the method carries across model families.
One training-affecting variable was isolated by holding everything else fixed: an unshuffled class-contiguous order collapsed held-out accuracy to 26.8% because the final optimizer windows were single-class, while a deterministic seeded shuffle recovered it to 99.2%. Train loss alone was insufficient evidence.
Specialization gain
65.28% -> 99.31% diagnosis exact match
Experiment 02 · frozen 144-case synthetic benchmark
Untouched Qwen3-4B versus the tuned adapter on a benchmark frozen before training and excluded from checkpoint selection. +34.03 pp; resolution exact match rose +64.58 pp.
Generalization by slice
STANDARD 100% · HARD 97.92% · TRANSFER 100%
Experiment 02 tuned · held-out slices
The single remaining diagnosis error was one HARD downstream-timeout case predicted as DNS resolution failure. TRANSFER is transfer-style evaluation, not true OOD.
Training efficiency
Stopped at step 100 of 600 · 5.312 GiB peak VRAM
8 GB RTX 5070 Laptop · 0.814% trainable
Early stopping on validation_no_improvement avoided 500 of 600 optimizer updates. 33,030,144 trainable parameters over a 4,055,498,240-parameter logical model.
Causal training-order fix
26.8% -> 99.2% validation accuracy
Experiment 01 · M5 unshuffled vs M6 seeded shuffle
Changing only the training order from class-contiguous to a deterministic seed-42 shuffle recovered held-out performance; all 250 optimizer windows became mixed-class instead of terminal single-class.
Fresh blind confirmation vs a regression
E02 98.33% held · E04 92.50% regressed -5.83 pp
E05 · independent 120-case blind challenge
Neither adapter had seen this challenge. The original E02 adapter's gain held close to its frozen-benchmark score; E04's cost-optimized recipe, which had saturated validation at 100%, regressed against E02 once tested on unseen data.
Cross-model replication
0% -> 63.33% (38/60) diagnosis exact
E06 · microsoft/Phi-4-mini-instruct
Same controlled methodology, separate frozen held-out benchmark, untouched-baseline 0% diagnosis and 0% strict-JSON compliance.
Failure-boundary false confidence
9/48 = 18.75% false-confident diagnosis
E07 · ambiguous / insufficient / contradictory evidence
The E04 adapter reached 100% accuracy on sufficient-evidence cases but stated a confident diagnosis 18.75% of the time when the evidence at the boundary did not support one.
Cost / quality trade-off
Classified: lower-cost, not quality-preserving
E08 · efficiency-frontier study
E04 reduced training corpus size and wall-clock cost versus E02, but the classification accounts for the E05 blind-challenge regression: cheaper was not quality-preserving once fresh generalization was included.