Get the latest tech news

LLM Shibboleths determine AI effectiveness

Originally posted 2025-05-28 Tagged: llms, software engineering Obligatory disclaimer: all opinions are mine and not of my employer Coding assistants promise to revolutionize software development, but why do some developers sing praises while others find them useless? The answer lies between the keyboard and the chair, but it’s more than just simple user error. Your level of expertise silently shapes the way you interact with the AI, allowing two people to have completely different experiences despite interacting with the same AI on the same subject.

LLM training datasets includes novices asking questions that don’t make sense, as well as experts that know exactly the piece of information they are lacking and are now requesting. “Add a delete endpoint that soft-deletes X.” The phrases “deleted_at” (as opposed to “is_deleted”), “migration”, “CRUD”, “not null”, “soft delete” are all shibboleths that conjure precisely the parts of the AI subconscious that correspond to “experienced backend engineer”, and the text of my prompts match precisely the Git commit summary that an experienced backend engineer would write for that pull request - since that’s presumably what these LLMs were trained on. I could see where the AI was strongest and weakest, and after some false starts where I tried vibecoding and auto-accepting a bit too much code at once, I found a nice rhythm where I could smash out features as fast as I could think through how they should be built.

Get the Android app