Ideas ready to test & merge
A useful technique can sit outside a codebase for years. Outrider works from the context in your repo, plus any brief you hand it, and assembles the change the way maintainers need it: wired to a live call site, tested, benchmarked, and disclosed. The record below is public, including a contribution merged into Hugging Face PEFT.
The idea and the codebase, two years apart
Riemannian-preconditioned LoRA existed as a standalone research artifact. Until it landed in PEFT and the half-dozen libraries downstream that depend on it, most fine-tuning users had no practical way to discover whether the method helped their application.
outside the libraries most fine-tuning runs through
One gem from a goldmine that keeps growing. Results like this rarely make it to testing or integration. Ideas that don't drive improvement cost even more, absorbing engineering time, expensive evals, and online experiments before anything filters them out.
From promising idea to upstream contribution
one Outrider run · a human in the loop at every reviewThe day's feed had no match, so the agent wrote its own search and surfaced the 2024 paper.
Filed the coordination issue first. The original author came back and endorsed the revival.
Draft PR with 9 tests, a guardrail-scoped diff, AI assistance disclosed per repo policy.
Two maintainer rounds answered within days, benchmark tuned in the open.
August 3, 2026. Now shipping in a library with 21,000 stars.
Add Riemannian-preconditioned
LoRA optimizer
this PR is already quite maturehuggingface/peft maintainer · first review
Outrider connected the dots between the paper, the dormant PR, and a live call site. Without a run pointed at PEFT, this integration stays where it sat for two years.
Land once, lift the ecosystem
PEFT is the hub of the fine-tuning ecosystem. Five of six DoRA-supporting frameworks route their implementation through it.
lifted by one PEFT-side improvement
A PEFT-side improvement lifts five of six frameworks with no additional integration work. The same ranking looks for leverage points in your codebase, the changes that lift the most surface in your system.
# per Scaling DoRA, arXiv:2603.22276, Appendix G
One merged PR became a method portfolio
Three parameter-efficient fine-tuning methods, surfaced by Outrider, drafted on the fork, and shepherded upstream. Sized against the year's merged tuner PRs at PEFT.
Neither is an outlier. Both are shaped like contributions PEFT reviewers routinely accept, and the merged one carried 41% test coverage in its diff against a cohort median of 13%.
# statuses reflect the public record at publish time · upstream outcomes are maintainer decisions · most Outrider runs end in a skip, by design
Each change arrives with its evidence
Two builds, your declared criteria, a prediction posted before you merge.
Predicted to meet your declared criteria
- two builds, baseline vs this PR
- 3 seeds · 95% confidence interval
- guardrail metrics enforced alongside the target
- posted as a Check before the merge
A quarter on Validate looks like this
merged changes, the criteria they declared, and what production did · illustrative
changes measured
last 90 days
confirmed improvements
in production
no measurable effect
filtered out
regressions caught
guardrails held
The highest-potential ideas graduate. They earn human attention and expensive offline and online evals. Every result refines the search for the next best idea.
What was automated and what was human
- candidate selection across indexed methods, repos, and models
- the fork implementation, wired to a live call site
- the test suite and benchmark configuration
- the commits and pushes on the draft PR, authored by remyx-ai[bot]
- review-feedback drafts and the evidence on every exchange
- coordination with the original author and maintainer
- the decision to reshape the design for upstream
- the merge itself; the maintainer controlled every decision
Every claim, on the record
The complete detail behind every number on this page, verifiable link by link.
The two-year gap, in detail
Riemannian Preconditioned LoRA (arXiv:2402.02347, Zhang & Pilanci, Stanford) adds a small preconditioner to LoRA's gradient step, claiming faster and more learning-rate-robust fine-tuning at negligible overhead, with an MIT-licensed reference implementation. The author brought it to PEFT herself in PR #1807 (May 2024), reached the maintainer's second review round that July, then coursework intervened, the thread went quiet, and the PR auto-closed as stale on September 21, 2024.
The run, hour by hour
July 1, 2026 · 15:24, fork validation PR opens (smellslikeml/peft #5), wiring the preconditioner into PEFT's MetaMathQA benchmark through its existing config-keyed optimizer dispatch. 16:12, coordination issue filed upstream (#3380) proposing the revival with the first review round's feedback folded in preemptively. 18:34, the original author endorses a fresh PR. 19:00, PR #3382 opens, reshaped for upstream as a generic subclass wrapper so the preconditioner applies to any optimizer's step.
The review, commit by commit
Jul 3, first review: "this PR is already quite mature," plus a request to integrate into PEFT's MetaMath benchmark suite and a process note to wait for maintainer confirmation before future implementations begin, which now shapes how Remyx runs coordination. Jul 7, benchmark recalibrated to the reference defaults. Jul 30, approval with one remaining closure question, resolved by rejecting closures outright. Aug 3, merged. Five commits total, two review rounds, two approvals, and the original author co-credited at her own suggestion.
The honest benchmark
The initial run (damping 1e-6) scored 0.45 GSM8K accuracy on the MetaMath setting; recalibrating to the reference implementation's 1e-2 raised it to 0.475, against a 0.49 default-LoRA baseline at the same rank. That comparison is disclosed in a table in the PR body with the correct interpretation: the paper's gains are modest and rank-dependent, MetaMath at rank 32 is a setting the paper never benchmarked, and the paper's stronger claim is learning-rate robustness. The maintainer accepted the contribution on those terms. A result below baseline, disclosed directly, merged anyway, because the pipeline's job is producing enough structured evidence for a qualified human to make a defensible decision.
The PEFT method portfolio, with links
Super-Tuning & Supra: coordination issue #3450 confirmed by the maintainer before implementation, upstream PR #3518 filed August 5 (+1309 / −3, 24 files), and the paper's first author is a co-author on the branch commits per vectozavr/SuperTuning #3. Scaling DoRA: validated on the fork (smellslikeml/peft #18) with the fused kernel published on the HF Hub; the upstream reference repo has no license file, so we opened the coordination issue and are holding the upstream filing until it resolves, on the two-step path the maintainer recommended. The full tracking document lives in this public gist.
The full record
Repository: huggingface/peft, upstream, preceded by fork validation (smellslikeml/peft #5). Tests: 9, covering the factory happy path, optimizer permissiveness, DoRA compatibility, error handling, paper-formula verification, and bf16 finiteness. Benchmark: MetaMathQA, llama-3.2-3B rank 32. Timeline: opened July 1, merged August 3, 2026, 33 days. Disclosure: AI assistance disclosed per PEFT's contributor guidelines for AI-assisted work, remyx-ai[bot] co-authored on implementation commits, fork validation preceded any upstream contact, and the author endorsed the revival before submission.
Run this loop on your repo
Bring one repository and one metric. We agree on success before the pilot begins.