By the mid to late-1800s, the word "nut" was slang to mean a person's head. Not long after, it looks like it also acquired the meaning of someone who was not acting right in the head, e.g., a person acting strange might be described as "nuts" or being "off their nut."