How we review questions

What the label means

After you answer, some questions show an Independently reviewed note with the sources the review checked. That note means the exact version you answered passed every step below. If a question is edited later, the note comes off until the new version has been reviewed again.

The steps

1. Written to the public outline. Each question in this process targets one task in the public BACB 6th edition Test Content Outline and is written with an explanation for the keyed answer and for every other option.
2. Automated checks. Before any review, software checks the item for wording that could give the answer away (for example, a keyed option that stands out as the longest or most detailed), near-duplicates of existing questions, and missing explanations.
3. The key is sealed. The keyed answer is locked with a cryptographic seal before review starts, so any later change to it, for example to match a reviewer's answer, would be detected.
4. Blind solve. A separate review process answers the question without seeing the key or the explanations. If its answer differs from the key, or it finds a second defensible answer, the question is held.
5. Guessability check. The same reviewer then asks whether someone with no behavior-analytic training could pick the answer from wording alone. Questions that can be guessed this way are rewritten or retired.
6. Source review. A different review process sees the key, checks it and every explanation against published sources (research articles, the BACB Ethics Code, the BCBA Handbook), and records which sources it inspected.
7. Release. A release switches on only questions with a complete review record, and each release is checked against the exact reviewed version before it runs, so an edited question cannot slip through under an old review. A question that went live by another route can be put through the same steps afterwards; it shows the note only if it passes.

Who does the reviewing

The blind solve and the source review are carried out by separate AI review processes. Each starts fresh, without the author's notes, and the solver never sees the key until its answer is recorded. They are not credentialed human reviewers. Decisions about what is released are made by Contingency Lab's founder.

How much of the bank this covers

85 questions currently carry a complete independent review record. Questions without the note have passed our automated checks and earlier quality audits but have not been through this full review.

Spotted a problem?

Every answered question has a report control under its explanation. Reports go to our review queue.