Know before you merge
Bring a pull request and the goal it needs to meet. Validate proposes the test, runs the approved comparison and returns the evidence and cost to the PR.
Connect the Remyx GitHub App in Studio, then comment @remyx-ai validate on a pull request.
Runs on Remyx compute. First completed validation free per GitHub App installation. Further completed runs use prepaid credits.
Try it on a change you're already making
No separate Validate app. Use the Remyx GitHub App you connect through Studio.
@remyx-ai validateReview the proposed test. The result and cost return to the PR.
Connect GitHub ↗| Measure | Baseline | Change |
|---|---|---|
| Explained variance | 0.9666 | 0.9677 |
| Negative activations | 0% | 49.2% |
Allowed explained-variance loss: 0.02. Required negative activations: at least 1%. Both criteria passed in this run.
15 min from test confirmation to results in this run. One CPU run, one seed.
What you get back on the pull request
What improved, what regressed, and what it cost.
The result
| METRIC | CURRENT | PROPOSED | CHANGE |
|---|---|---|---|
| Answer resolution | 71.4% | 74.0% | +2.6 points |
| p95 latency | 812 ms | 876 ms | +64 ms |
| Cost per 1,000 queries | $0.41 | $0.48 | +17% |
Use the evidence to decide what to pursue, revise, or skip.
Measured on evaluation set v3 across three seeds. The comparison does not show the effect on live users. If a guardrail in the plan fails, the run cannot pass.
The evaluation plan
Does adding a reranker improve answer resolution within the latency limit?
Keep the index, generator and evaluation set fixed.
Validate proposes the plan. You review it on the pull request.
View run details
Results post as a PR comment and a GitHub Check. The executed notebook is committed to the repository, and the run is listed in Studio.
From a proposed change to a measured result
Bring the tests and evaluation assets already in your repo. Validate proposes the comparison, prepares the environment, runs the approved test and returns the evidence to the PR.
Start from any pull request on a connected repository, however the change was authored.
If the run cannot finish within the budget, Validate reports where it stopped. The criteria in the plan stay unchanged.
Start on one change. Expand when it works.
One free validation per GitHub App installation. Credits after that.
The free run is used only when a measurement finishes with a verdict or valid abstention. Further runs draw from a prepaid Remyx balance at the rates shown in Studio. There is no separate validation pack.
Bring Validate to your team
Connect more repositories, tailor the workflow and plan a deployment around your team's needs.
Before you start
Do I need Outrider to use Validate?
No. Validate works on an existing pull request, however the change was authored. Outrider is a separate product that finds and drafts changes worth testing.
What does Validate need from my repository?
An installation of the Remyx GitHub App covering the repository, claimed by your Remyx account, and an open pull request. Validate can use tests and evaluation assets already in the repo and proposes a plan for your review.
What does a passing evaluation establish?
It shows that the change met the agreed criteria on the chosen data and setup. Measure the effect on real users through online experiments and production monitoring.
Where does the evaluation run?
Validate builds the environment and runs the comparison on Remyx compute. Results return to the pull request as a comment, a GitHub Check and an executed notebook, and the run is listed in Studio. Teams that need evaluations to run in their own environment can talk to us.
What happens if a run cannot finish?
Validate reports the stop and reason on the pull request. A validation refused during preflight or stopped before a measurement verdict does not consume the free run or debit the prepaid balance. A confirmed test cannot be silently changed by the repair loop.
What does it cost?
The first completed validation is free per eligible Remyx GitHub App installation. Subsequent completed runs debit actual compute cost and any Remyx-funded model calls from your prepaid balance. Rates, credits and transactions are visible under Studio → Account → Billing. See pricing →
Do I need to connect a model provider?
Not for the first supported validation while the Remyx-funded model allowance is available. You can connect your own provider in Studio and set Validate's model under Account → Validate. Calls made with your own key are billed by your provider, not by Remyx.
Where do I add credits?
Studio → Account → Billing shows your balance, rates and recent transactions. Add $20, $50, $100 or a custom amount from $10. Add credits →
Who decides what merges?
Your team. The Validate Check reports the verdict on the pull request. It blocks a merge only when your team makes it required in branch protection.
Try it on your next pull request
Connect the Remyx App, comment on your PR and review the plan. The first completed validation on that installation is on us.