Know your next move

Riemannian Preconditioned LoRA (arXiv:2402.02347, Zhang & Pilanci, Stanford) adds a small preconditioner to LoRA's gradient step, claiming faster and more learning-rate-robust fine-tuning at negligible overhead, with an MIT-licensed reference implementation. The method was published in 2024, reached PEFT once via the author's own PR #1807, and never landed; the thread went quiet and the PR auto-closed as stale on September 21, 2024.

The day's feed had no match, so the agent wrote its own follow-up search and surfaced the paper. The working pool expanded to 23 candidate methods, screened on usable reference code, license compatibility, existing upstream work, and a clear place to test each method in the repository. 13 of the 23 had no reachable reference code and were skipped. MiniMax Sparse Attention was skipped because its source metadata disagreed about licensing and Outrider used the stricter reading. LoftQ and LoRAT had reachable, permissively licensed code and remained under consideration. Riemannian LoRA was selected from that smaller workable set; it had usable reference code and a concrete entry into PEFT through the existing config-keyed optimizer dispatch.

The first Riemannian implementation failed the pre-publication risk check and stopped. A narrower implementation cleared. July 1, 2026 · 15:24, fork validation PR opens (smellslikeml/peft #5), wiring the preconditioner into PEFT's MetaMathQA benchmark. 16:12, coordination issue filed upstream (#3380) proposing the revival with the first review round's feedback folded in preemptively. 18:34, the original author endorses a fresh PR. 19:00, PR #3382 opens, reshaped for upstream as a generic subclass wrapper so the preconditioner applies to any optimizer's step. Tests: 9, covering the factory happy path, optimizer permissiveness, DoRA compatibility, error handling, paper-formula verification, and bf16 finiteness.

Review: Jul 3, first round, "this PR is already quite mature," plus a request to integrate into PEFT's MetaMath benchmark suite and a process note to wait for maintainer confirmation before future implementations begin, which now shapes how Remyx runs coordination. Jul 7, benchmark recalibrated to the reference defaults. Jul 30, approval with one remaining closure question; review had surfaced a closure-handling defect, resolved by rejecting closures outright, a code change that landed before approval. Aug 3, merged. Five commits, multiple maintainer reviews, two approvals, and the original author co-credited at her own suggestion. AI assistance disclosed per PEFT's contributor guidelines; remyx-ai[bot] co-authored on implementation commits; fork validation preceded any upstream contact.

DoRA, across multiple passes: the candidate appeared in the original pool, and reference licensing prevented adoption at that point. A later Scaling DoRA pass produced an implementation whose comparison against the paper's reference failed; a repair was attempted, the comparison still failed, and no upstream PR was filed from that pass. A later version had a permissive reference and tests, and the work resurfaced on the fork (smellslikeml/peft #18). DoRA was reconsidered as its reference implementation and licensing conditions changed.

Super-Tuning (arXiv:2607.09287), merged at PR #3518 on August 21, 2026: coordination issue #3450 was filed first and the maintainer confirmed fit before implementation began; the paper's first author signed off (vectozavr/SuperTuning #3) and is co-credited on the branch commits. Benchmark results on PEFT's own MetaMathQA harness were posted directly in the review thread, disclosed as single-seed runs with several configurations below the LoRA baseline, and the maintainer accepted the contribution on those terms. Two full review rounds. The benchmark runs surfaced a real bug, LoRA parameters in the hybrid mode silently frozen by PEFT's prefix-based freeze pass, fixed with a regression test. Review also surfaced an integer-dtype casting bug in PEFT's shared tuner utilities; the maintainer asked for the fix and it landed in this PR, so it reaches every PEFT method. Squash-merged as commit 09ba20c: 28 files and 1,365 lines added, covering the tuner core, a docs page, a worked example, and five benchmark configs. Opened August 5, merged August 21, 16 days. Other Outrider-surfaced contributions in review: EcotReasoningModule at lerobot (#4036) and STARE at trl (#6167); the full tracking document lives in this public gist. Context on why PEFT matters: five of six DoRA-supporting frameworks route their implementation through it (per Scaling DoRA, arXiv:2603.22276, Appendix G), so a PEFT-side improvement reaches unsloth, swift, llama-factory, and axolotl with no additional integration work.

Benchmark: MetaMathQA, llama-3.2-3B, rank 32. The initial run (damping 1e-6) scored 0.45 GSM8K accuracy; recalibrating to the reference implementation's 1e-2 raised it to 0.475, against a 0.49 default-LoRA baseline at the same rank. The comparison is disclosed in a table in the PR body with the correct interpretation: the paper's gains are modest and rank-dependent, MetaMath at rank 32 is a setting the paper never benchmarked, and the paper's stronger claim is learning-rate robustness. The maintainer accepted the contribution on those terms.

Timing: PR #3382 opened July 1, merged August 3, 2026, 33 days from open to merge. Recorded Outrider runtime, 33 minutes. Recorded agent spend, $7.10, benchmark compute excluded. Run statuses: 4 of 20 Outrider runs on PEFT opened draft PRs. Other runs ended as branches, issues, leads, skips, failures, or errors. Statuses reflect the public record at publish time; upstream outcomes are maintainer decisions.