The word heresy is thrown around so much as the worst that can be said about another, even Pastor Steve Cioccolanti from Australia said Christians say it over all manner of doctrinal differences. So, he provided the best standard of something that is heresy; that which keeps one from salvation.
Otherwise, every denomination is heretical to the doctrines of ones own denomination, right?
Yes, that's right, ...and, of course, it stands to reason. If one is truly convicted of their exegesis, which we hope that they are, then they must conclude that a contrarian view is either heretical, or consequential. I've said the same thing to anyone who has a 'to each his own' attitude, that it can only mean that you lack zeal or understanding of your position.
But, to your point, since invariably there will always be controversy, we have to limit our accusations of heresy to those doctrines that are more mortal, rather than venial (if I may use Catholic terminology). But, of course, we also have the concepts of orthodoxy and heterodoxy, which may be preferred as they allude lesser to one's salvation, than to their exegesis.