Quiz settings

Each repository has its own quiz configuration, found under Repositories → (your repo) → Settings. These settings control how quizzes are generated and what counts as passing.

Question format

What kind of questions Reviewsaur generates from the diff.

• Multiple choice (MCQ) — one correct answer per question. Fastest to take.
• Mixed — a blend of multiple-choice and short-answer.
• Open-ended — short-answer questions, graded by Reviewsaur. Best for probing genuine understanding.

All formats are available on every plan.

Difficulty

How deep the questions dig into the change. Every quiz still has the same number of questions — only the depth changes.

• Easy — each question can be answered from a single hunk of the diff. Good for confirming the reviewer opened the PR at all.
• Medium — questions cover the change as a whole: why it was made and how it works end to end. This is the default.
• Hard — questions require combining several parts of the diff, or working out behaviour the change implies but does not spell out. Wrong choices are close to the right one.

Hard also grades open-ended answers more strictly. On Easy and Medium a short answer passes if it shows the reviewer understood the main idea. On Hard the answer has to cover the expected key points; vague or generic answers are marked wrong.

The level is fixed when the quiz is generated, so changing this setting never changes how a quiz already in progress is graded.

Medium by default. Available on every plan.

Question categories

A category is the lens a question takes on the change. Difficulty controls how deep a question digs; the category controls what it asks about. The two are independent — any category can be asked at any difficulty.

Automatic (the default) lets the quiz pick a mix that fits the change. Most questions check understanding of the core change, and the others are added when the PR calls for them: Risk when it touches auth, money, data, or concurrency; Architecture or Trade-off when it touches boundaries, data flow, or a design choice; Verification when behaviour changes without obvious tests.

You can also pick the categories yourself. The quiz then mixes only the ones you picked.

Trade-off questions are graded differently from the rest: several answers can be correct, so any answer that names a real trade-off of the change together with a reason is accepted — including one the rubric did not list. This holds on Hard too.

Architecture and Risk questions, and choosing categories yourself, need the Pro plan. On the free plan quizzes automatically mix Understanding, Trade-off and Verification.

I don't know

When this is on, every question gets an I don't know button.

Clicking it does three things:

• The question counts as wrong. There is no partial credit.
• The correct answer and its explanation appear right away.
• The question locks, so it cannot be answered afterwards.

The result page marks these questions separately from wrong guesses, so you can tell an honest gap from a bad answer.

Off by default. Available on every plan.

Pass threshold

The minimum score — as a percentage — a reviewer needs to pass. For example, a threshold of 70 means they must answer at least 70% of questions correctly.

Lower it to keep the gate light; raise it for high-stakes repositories.

Required approvals per PR

How many different people must each pass a quiz before the Reviewsaur Quiz check turns green. Set it to 1 for solo review, or higher to require multiple independent reviewers to demonstrate understanding.

Multi-reviewer approval is a Pro feature; Free requires one.

Per-developer settings

Team admins can give one person a different difficulty or a different I don't know setting than the repository default. Set either value, or leave it on Inherit, from two places:

• Members → the person's menu → Quiz settings…
• A repository's Settings tab → Per-developer settings

Both edit the same thing. The settings follow the person, so a change in either place applies to every repository of the team. The quiz link in the pull request stays the same for everyone: each person gets their own questions when they open it, picked to match their settings. One person cannot see another person's questions or results.

Generation still runs once per push. The questions are generated for each difficulty level that someone on the team uses, so a team with no overrides pays exactly what it did before. An override added after a quiz was generated applies from the next push; until then that person gets the closest level available.

A Pro feature. Overrides saved on Pro are kept but ignored after a downgrade.

Excluded paths

Glob patterns for files Reviewsaur should ignore when generating a quiz — one pattern per line. Use this for generated files, lockfiles, or docs where a comprehension quiz adds no value.

Examples:

*.md
docs/**
*.lock
package-lock.json

Patterns without a slash match file names at any depth — *.lock matches packages/a/yarn.lock. Patterns with a slash match the full path from the repo root — docs/** matches everything under docs/ but not src/docs.ts.

Excluded files don't count toward the PR size limits, and they are stripped from the diff before quiz generation, so questions are never asked about them. If every changed file in a PR is excluded, no quiz is generated and the check passes automatically.

Trivial changes

Not every pull request needs a quiz. Reviewsaur rates each change while it writes the walkthrough — trivial, normal or complex — and the verdict is shown on the pull request.

A change is trivial when it has no runtime behaviour change, or is small enough that a reviewer can check it by reading the diff once: a typo, a comment, documentation, formatting, a rename with no logic change, a dependency version bump, a config value. The verdict comes from the diff alone — a title that says "trivial fix" does not make it one.

Two rules can pull a verdict back to normal, never the other way:

• Size — a pull request changing more than 100 lines is never trivial.

• Never trivial paths — glob patterns you set per repository. A pull request that touches any of them always gets a full quiz, even when the rest of the change is trivial. Same syntax as excluded paths:

  **/billing/**
  **/migrations/**
  **/auth/**
  

When Skip the quiz for trivial changes is on, a trivial pull request gets a green Reviewsaur Quiz check with no quiz at all, plus one comment giving the reason and the usual walkthrough. Nothing is hidden: the reason is on the pull request.

When it is off — and on the Free plan — a trivial pull request still gets a quiz, but a shorter one: 3 questions instead of 5. That shorter quiz is on every plan.

If the walkthrough fails or times out there is no verdict, and the pull request gets a full quiz. Skipping never happens on a retry after a failed attempt, so a check can never turn green right after someone failed it.

Skipping is a Pro feature; the shorter quiz and never-trivial paths are on every plan.

Need to enforce all this on merge? See GitHub status check. For what each plan unlocks, see Plans & limits.