Get the latest tech news

Tao on “blue team” vs. “red team” LLMs

In the field of cybersecurity, a distinction is made between the "blue team" task of building a secure system, and the "red team" task of locating vulnerabilities in such systems. The blue team is more obviously necessary to create the desired product; but the red team is just as essential, given the damage that can result from deploying insecure systems. The nature of these teams mirror each other; mathematicians would call them "dual". The output of a blue team is only as strong as its weakest link: a security system that consists of a strong component and a weak component (e.g., a house with a securely locked door, but an open window) will be insecure (and in fact worse, because the strong component may convey a false sense of security). Dually, the contributions to a red team can often be additive: a red team report that contains both a serious vulnerability and a more trivial one is more useful than a report that only contains the serious issue, as it is valuable to have the blue team address both vulnerabilities. (But excessive low-quality reports can dilute attention from critical issues.) Because of this, unreliable contributors may be more useful in the "red team" side of a project than the "blue team" side, though the blue team can still accommodate such contributors provided that the red team is competent enough to catch almost all of the errors that the contributor to the blue team might make. Also, unreliable red team contributions only add value if they _augment_ the output of more reliable members of that team, rather than _replace_ that output, and if their output can be effectively filtered or triaged by more experienced red team members. (1/3)

Get the Android app