My question is why these powers were so commonly seen in the past, and what has changed in the past that presently we see none of them?
Well what most likely changed is the fact that people are now far less gulable then they used to be. Now we are way more suspicios of anyone displaying any kind of spectacular ability and we tend to first investigate and then believe.
Now i'm not saying iddhis/siddhis are fake since I myself am very interested in the subject. It seems to me that iddhis should be achievable through meditation by exercising the awareness in the same way that we can control a dream only when we are AWARE that it is a dream. So I believe that the same would aply to this reality. Once we develop the awareness to such a degree something like the siddhis should be possible.
So I will not give up on the idea that iddhis/siddhis are possible until I myself will be able to enter and master all the jhanas, and only then see for myself if they are or are not possible.
That being said I do think it is very conveniant and suspicios that many people say "oh super powers or miracles don't matter because they are not the true goal of the path but rather an obstacle since it would increase your ego and bla bla bla". The point is that miracles iddhis super powers or whatever you want to call them are the best way to prove to people that we are living an illusory existence and that there is a way out of it.
SImply put if a monk has iddhis/siddhis I see no valid reason why he should not display them other then the fact that he does not posses such abilites. Displaying such powers could only show once and for all that this reality is not what most of the people think it is. Also it could be proof that the certain master is for real. I mean out of two masters teaching Buddha's way I would certainly trust the one that could also prove to me that what he is teaching really works.
I know the goal of buddhism is the end of suffering not siddhis. But once someone reaches this goal they are not human anymore. They are something else, something beyond, something better. Man is the bridge between beast and superman. So if one passes this bridge and becomes a being that knows no suffering they should also have some new and spectacular abilities.
Of course this is only speculation and until we ourselvs master the jhanas and maybe even become enlightened we cannot say for sure.
You also said that "super-powers and insanity are twins". That could be very accurate. Let's say that through meditation we could learn to use our awareness in such a way that we could alter our perception of reality. So one could control the mind in such a way that he could literraly see himself moving an object with his mind or flying through the air. But that would only be his subjective perception of reality. Like a dream. Others could not see what he would see. So it would appear like a form of insanity to other people when someone says look i can move that object with my mind and the others don't see anything happen.