The connection between talking about secret society conspiracies and being crazy is a deliberate misrepresentation by the media to discredit people that point out the obvious.
I don't know if there are secret societies controlling the world or not, but if there are, the NSA has got their phones bugged!
edit: I listened to all 20 minutes of the JFK/secret societies speech. He did not say that they were infiltrating the U.S. Govt. He was talking about international communism and the cold war, referencing Karl Marx by name.