On the Buddhist Tantra talk section there's a question about Samaya....rather than a PM to the original poster, I thought it important to share these words of Thrangu Rinpoche for the benefit of everyone....they are not strictly about Samaya, but pertain to devotion and sacred outlook, which are most certainly related to specific Samaya of Highest Yoga Tantra:
"However, faith in one’s guru is not supposed to be blind faith. It is not the attitude, “My guru is perfect,” even though your guru is not perfect. It is not pretending that your guru’s defects are qualities. It is not rationalizing every foible of the guru into being some kind of superhuman virtue, because after all most gurus are going to have defects.You do not have to pretend that your guru’s defects are qualities. That is not devotion. The object of your devotion is not the foibles, quirks,or defects of your guru, but it is the dharma which your guru teaches you. As long as the dharma is authentic and pure,then that guru is a fit object for your devotion. You are not practicing the guru’s foibles. You are practicing the guru’s teachings, the dharma. The result which you will get, you will get from the dharma that you practice. Therefore, your devotion is principally to the dharma, which is the teaching of the
guru. You need to recognize the defects of your guru as defects. You do not need to pretend that they are anything other than that. But you also need to recognize that the guru’s defects will not hurt you because it is not the guru’s defects that you are creating or cultivating. You are following the teachings fo the guru, and so trust means trust in the teachings, trust in the validity, purity and authenticity of the dharma that you receive from your guru. You do not need to pretend that defects are qualities. You just need to recognize the validity of the teachings themselves. So that is the importance of devotion and sacred outlook."http://www.rinpoche.com/teachings/creation.pdf
From Shenphen Osel, Vol 5, No. I, and I believe this was later published as a book on Kongtrul's Creation and Completion text.
With regard to the original poster's question, if one cannot abide the foibles, quirks, and defects, it is appropriate to seek out another teacher, but it does not mean that one's Dharma Connection with prior guru is somehow invalidated. This is a tricky point, but honesty, with oneself and with others, is key.