Several vision models tested on the classic game "Find the odd picture out"? and found that they don't always reason like humans because models, still see the world differently than humans.
This is an important drawback because if a model does't structure its internal representation of world like a human, its decisions can be illogical and unreliable.
🟢 What DeepMind did?
Humans group objects by meaning, while models more often group by visual/textural features. For example, in the task <starfish, cat, fox>, humans choose the starfish because it lives in water, while models choose the cat because the picture stands out in the color palette.
– Took a large visual model, froze it, and artificially attached a small adapter trained only on a dataset with human answers for odd-one-out.
– Generated a huge dataset with millions of odd-one-out decisions using this model.
– Based on this large dataset, fine-tuned other models so that their internal representations became closer to human logic of grouping.
So, it seems like the models were just trained on some children's game, nothing surprising. But it turned out that this qualitatively changed their hidden space. See the third screenshot: on the left, the model before alignment, on the right, after. You can see clear clusters appearing that correspond to human logic (for example, animals, food, etc.). Beautiful, right?
Also, such a model turned out to be more robust in terms of distribution shifts. For example, if the background, lighting, or other conditions in the pictures change, its answers still follow logic.
Overall, it can be expected that such a model will generalize faster than usual.
••••••••••••••••••••••••••••••••••••••••••••••••••••
🤖 Data Science, ML & Big Data with @DataXplore