This isn't quite zero-sum. There is also the aspect that any trade deal that removes existing tariffs represents a tax reduction for any businesses that do international trade, and a corresponding reduction in government revenue for participating countries. On both sides, it represents a wealth transfer away from government and towards business. You may interpret that as good or bad depending on your point of view, but either way it's a policy change.
Another side of this is that free trade allows businesses to move production to whatever country has the weakest labour laws and environmental protections. In effect, it rewards countries economically for having lax regulations, and encourages a "race to the bottom". In the short term, it may be good for workers who now have jobs, but in the long term we don't want employment to only be economically competitive if it's indistinguishable from slavery.
Another side of this is that free trade allows businesses to move production to whatever country has the weakest labour laws and environmental protections. In effect, it rewards countries economically for having lax regulations, and encourages a "race to the bottom". In the short term, it may be good for workers who now have jobs, but in the long term we don't want employment to only be economically competitive if it's indistinguishable from slavery.