Perhaps it is time for that famous metaphor about the lake and the six classes of beings.
This metaphor is in Vasubandhu's 20 Verses
, a Yogācāra text.
You are arguing that there is no basis for the senses of the various beings to apprehend the lake according to their dispositions. There is no form there at all, which is more or less a Yogacaran position. The "lake" is entirely a construction of the mind. Exponents of this position do not deny that they are espousing a straight up idealism.
Who are these exponents? Yogācāra doctrines in no manner espouse any form of idealism. They temporarily introduce a system of consciousness to show the illusory nature of the so-called external material realm. This doesn't sit well with people, much like yourself, so this temporary introduction becomes quite a focal point.
But ultimately Yogācāra doctrines do in fact get to the point of relinquishing this consciousness for its falseness as well. This leaves no room for idealism, since no "mind" is ultimately asserted.
Basically subject (consciousness) is introduced to show the illusory nature of object (material realm). Then once object is seen to be non-existent, then it must follow that there can in reality be no subject.
Of course your next step will be to jump on "nihilism!". It is only your nihilism if you assert something to be denied.
Yogācāra, as with all Mahāyāna schools, ultimately assert nothing.
I am arguing that there is an interdependence between the form of the lake and the senses of the various beings. There is both rupa (the basis) and the rupaskandha (the senses of the various beings) coming into simultaneous, interdependent contact, such that neither the subject nor the object can be established. The "lake" is empty, the senses of the various beings are empty, the significations are empty. But this does not mean that everything is a construction of the mind.
What is your reason to believe in some necessary external stimulant that sparks the apprehension of some appearance? Did not the Śūraṅgama Sūtra state:
"Whatever manifests does so in compliance with karma. Ignorant of that fact, people of the world are so deluded as to assign its origin to causes and conditions or to spontaneity. These mistakes, which arise from the discriminations and reasoning processes of the mind, are nothing but the play of empty and meaningless words."
Whatever manifests (whatever kind of lake) does so in compliance with karma (volition, mind).
Ignorant of that fact, people of the world are so deluded as to assign its origin to causes and conditions (example: "both rupa [the basis] and the rupaskandha [the senses of the various beings] coming into simultaneous, interdependent contact.").
These mistakes arise from the discriminations (example: "form of the lake, senses of various beings") and reasoning processes of the mind (example: "coming into simultaneous, interdependent contact, such that neither the subject nor the object can be established.").