Throw Out The Baby With The Bath Water

Discard something valuable along with something not wanted.

Examples:

• We shouldn't throw out the baby with the bath water by voting the bill down.

Origin: First recorded in English 1853