How do you respond to people who use your own language against you in this way:
"Marriage already HAS been redefined. Polygamy was once the norm... acceptable to God, now it's taboo. Gay marriage is no more a violation of natural law than that was."
See the problem is people do not understand how the natural law works. Nor even what the issue here is.
There are three ends of marriage (speaking of natural marriage, the ends are raised up in sacramental marriage)
2. Unity (that is, a common life, aid to each other)- This end is both an end in itself and further serves the end of procreation which is why procreation is the higher end (a common life serves for the bringing up of children)
3. After the fall- relief of concupiscence
Now homosexual relations cannot fit under marriage since, at the very least, the primary end is ontologically impossible. They cannot procreate. So even before calling them a sin, we see that they cannot fit into marriage
Now polyandry, a woman taking multiple husbands, is against the natural law both in its secondary principles (since it weakens the unitive end) but also in the first principles, since additional husbands do not aid procreation, and indirectly hardm it.
However, polygyny (taking multiple wives) does not go against the first principles of the natural law. Both Rachel and Leah could have kids after all! It does, however, go against the secondary principles of natural law, namely the unitive. However, it does not necessarily destroy or prevent that end, it tends against it and weakens it surely. This means that polygyny is not intrinsically evil, but it can only be permitted by God.
And God did that. He allowed polygamy. So no, polygamy was not adultery. It was not a sin, it was allowed by God. But it was allowed by way of a dispensation.
An analogous thing is that natural law goes against incest. Why? Because it demands that we extend the bonds of human society, and also because of mixed affections. The affection I have to a relative should not be mixed with the affection of a wife. However, where is the line drawn? Natural law only intrinsically, in all times and places, forbids relations between direct descendants. Even siblings are not against the natural law in such a way that such relations could never ever happen. They did and they had to. God has since forbidden them absolutely by divine law.
But take an uncle and niece, or two cousins. That also tends against the natural law, but here it can be permitted, and the Church has and does dispense this impediment. Human law serves as a further determination of natural law here, in part because contrary to popular conception natural law is not always a black and white, all or nothing standard. It is more nuanced than that