Get the latest tech news

How we made our AI code review bot stop leaving nitpicky comments

Everything that went wrong trying to make our LLMs say less.

This post is adapted from a talk I gave at the Sourcegraph Dev Tools meetup at the Cloudflare office in San Francisco on December 16th, 2024. Since LLMs are “few-shot learners”, we also tried to give Greptile several examples of good and bad comments in the prompt - hoping it would be able to generalize those patterns. If anything, this made the bot even worse because it didn’t actually find a useful pattern across the available samples (some might argue LLMs are architecturally incapable of that), and instead inferred superficial characteristics.

Get the Android app