Rather than punishing yourself for it, use it as an opportunity to investigate with mindfulness. Examine what was going on, what is going on when you remember it. It sounds like a study in craving. Instead of it just being a mistake, it can become an opportunity for learning. It's all grist for the mill.
Oh yeah, and how about throwing in a little compassion for oneself while you're at it? Couldn't hurt. Everyone knows what it's like to make a mistake when in the grips of craving, and how that leads to suffering. Compassion is always the appropriate response to suffering, even (or especially) when it's toward yourself.
Check out this excerpt from Gil Fronsdal's Issue At Hand:
Sila is usually translated as “virtue” or “ethics,” but we need to be careful not to confuse it with Western ideas of virtue and ethics. A traditional foundation of Western ethics is commandments and values often handed down from a god. These values include ideas about right and wrong, good and evil, and absolute rules that we have to live by. This approach to ethics leads easily to guilt, an emotion that is pervasive in the West, but which is considered unnecessary and counterproductive in Buddhism.
Buddhism understands virtue and ethics pragmatically, based not on ideas of good and bad, but rather on the observation that some actions lead to suffering and some actions lead to happiness and freedom. A Buddhist asks, “Does this action lead to increased suffering or increased happiness, for myself and others?” This pragmatic approach is more conducive to investigation than to guilt.