## Project 66: Test of Humanity

## Detection accuracy

- How many of your intentional problems did the AI reviewer catch?
  - It caught all the problems.
- How many did it miss? Do the misses share a pattern (e.g., all subtle, all in a specific file type)?
  - We didn't see any missed cases.
- Were there any false positives — comments about things that weren't actually problems?
  - There weren't any false positives.
- Did the severity levels match how serious the problems actually were?
  - The severity levels matched the problems, but some of them looked underrated. It said `low` instead of `medium`.
- How actionable were the comments? Could you fix the issue based on the comment alone?
  - The comments were described fully enough to understand the issue and fix it.


## Verification accuracy

- Did the AI correctly identify which issues you fixed?
  - Yes, but we lost one comment after fixing the issue. The AI reviewer didn't mark it as resolved and just removed the comment.
- Did it incorrectly resolve any issues you left broken?
  - All comments were successfully resolved, but one comment was removed.
- Did it miss any fixes you made?
  - No.
- How confident are you in the verification after this test?
  - 90%


## Analytics page

- Did the numbers on the analytics page match your experience?
  - We fixed 3 issues, but it detected only 1.
- What would you add or change about the page?
  - It looks good after the first review.


## Overall experience

- Was it clear whether the AI review was in progress or complete?
  - We missed the AI review on our first PR and merged it, because there weren't any UI notifications that the reviewer would start in the near future.
- How long did the AI review take?
  - 3–5 min
- Did the summary comment posted by the AI reviewer make sense?
  - Yes
- If you added custom AI rules, were they accounted for in the review?
  - Yes, but it didn't count the fixed issue on the analytics page.
- Would you trust this AI reviewer on real PRs after this test?
  - 90% yes.
- What made you smile?
  - It looks similar to the Cursor bot we have in GitHub.
- What felt frustrating or broken?
  - You should check how the AI reviewer resolves issues described in rules, because it feels like there are some troubles with this part.
- On a scale of 1 to 10, how would you rate the overall quality of the AI reviewer?
  - 8
- Are there any additional notes you would like to share?
  - It's not possible to see outdated comments.