I see what you mean. I think you're right.It's about our bodies so of course it's a women's rights thing. Ultimately, whether men agree or disagree with abortion is irrelevant because the responsibility is always placed upon women - if a man wants an abortion but the woman wants the child, the man just leaves the relationship. If a man refuses to allow a woman to get an abortion, guess who still has to carry the baby for nine months?
There are contraceptive measures for men that are safer and more easily reversed than the ones women use every day, but the vast majority of men refuse to go through with it. The onus is always on us to not get unintentionally pregnant and if we do it's our fault and anti-abortion governments are very clear in that they oppose abortions but will provide fuck all support once that child is born.
This is absolutely a women's right issue first and foremost because it isn't about the child but controlling what a woman can do with her own body.