Bane Of One's Existence

The source of one's constant irritation or misery; a persistent cause of annoyance.

Examples:

• The neighbor's cat is the bane of my existence.

Origin: Late 1500s