- How many of your intentional problems did the AI reviewer catch?
- It caught all the problems.
- How many did it miss? Do the misses share a pattern (e.g., all subtle, all in a specific file type)?
- We didn't see any missed cases.
- Were there any false positives — comments about things that weren't actually problems?
- There weren't any false positives.
- Did the severity levels match how serious the problems actually were?
- The severity levels matched the problems, but some of them looked underrated. It said
lowinstead ofmedium.
- The severity levels matched the problems, but some of them looked underrated. It said
- How actionable were the comments? Could you fix the issue based on the comment alone?
- The comments were described fully enough to understand the issue and fix it.
- Did the AI correctly identify which issues you fixed?
- Yes, but we lost one comment after fixing the issue. The AI reviewer didn't mark it as resolved and just removed the comment.
- Did it incorrectly resolve any issues you left broken?
- All comments were successfully resolved, but one comment was removed.
- Did it miss any fixes you made?
- No.
- How confident are you in the verification after this test?
- 90%
- Did the numbers on the analytics page match your experience?
- We fixed 3 issues, but it detected only 1.
- What would you add or change about the page?
- It looks good after the first review.
- Was it clear whether the AI review was in progress or complete?
- We missed the AI review on our first PR and merged it, because there weren't any UI notifications that the reviewer would start in the near future.
- How long did the AI review take?
- 3–5 min
- Did the summary comment posted by the AI reviewer make sense?
- Yes
- If you added custom AI rules, were they accounted for in the review?
- Yes, but it didn't count the fixed issue on the analytics page.
- Would you trust this AI reviewer on real PRs after this test?
- 90% yes.
- What made you smile?
- It looks similar to the Cursor bot we have in GitHub.
- What felt frustrating or broken?
- You should check how the AI reviewer resolves issues described in rules, because it feels like there are some troubles with this part.
- On a scale of 1 to 10, how would you rate the overall quality of the AI reviewer?
- 8
- Are there any additional notes you would like to share?
- It's not possible to see outdated comments.
