Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?

I saw this thread on X and get excited.
I didn't go through the paper, but I liked the way of thinking and modeling the problem with information theory. It was inspirational. In addition, imo the main problem is very related to the Godel Incompleteness theorem and/or the halting problem. In a nutshell, the system could interpret a sentence both as an instruction and data, which is the underlying cause of many self-referential paradoxes.

#paper
@SingularThinker