Yes, true. Unless you can claim that your ancestors have been in the area since at least the end of the Jurassic then you have no claim to exclusive right to live there.
On the other hand, I do think the issue is more complex than this. Without falling into racism and nationalism we do need to find skillful ways to take responsibility for the stability of our societies and the successful integration of communities and individuals into those societies. And if that involves putting brakes on immigration or more education for immigrants then so be it.
Many people are afraid to talk about this especially politicians because of the fear of it being (wrongly) perceived as being racist. But the alternative may be divided communities, resentment, ghettoisation, interracial conflict etc.
We can idealise a society where everyone loves everyone else unconditionally but that's not actual reality. We need to work with actual reality.