Get the latest tech news

Reflections on Qualitative Research

This note offers some opinionated thoughts on why interpretability research may have qualitative aspects be more central than we're used to in other fields. It also aims to describe some heuristics for research taste in qualitative work.

A recent example of this in our own work was our study of tanh-regularization in dictionary learning, where summary statistics initially indicated a lot of promise and it was only later qualitative inspection of features that revealed that we had been led astray. They're so striking that they hold up as notable even if they're cherry picked: Hooke seeing cells through a microscope is just obviously showing something real, even if he looked at hundreds of other things that were uninteresting and didn't report on them. The Structure of Scientific Revolutions – Especially the discussion of particular competing "schools" in research into electricity and how they focused on different phenomena and abstractions; and also early atomic physics and visualizing particles in cloud chambers.

Get the Android app