Considering how the Legion treats women, I don't think there's enough nuance in the world to ever justify siding with them.
"The Legion" as shown in the game isn't the Legion as a whole though. It's just the front line of their war, Caesar's Legion controls an absolutely massive amount of land. Yes, in the Legion camp women were purely slaves and most of the Legion soldiers encountered in the game straight up hate women, but meanwhile Cass mentions how she's able to travel safely through Legion lands, she even mentions how she doesn't even need bodyguards because no one in their right mind would ever try to attack someone, man or woman, in Legion-controlled lands because of the severe punishments for raider actions. Also, the Wiki and J.E. Sawyer himself have this to say about female roles in Caesar's Legion:
"Women are expressly forbidden from fighting and act as support corps. They are caretakers, healers, midwives and breeders, fulfilling roles essential to maintaining the Legion's continuous campaign of expansion. Of particular note are the priestesses, who raise children taken from their parents, and raise them in keeping with Caesar's doctrine.
Even though breeding is incredibly important in the Legion, there isn't any concept of family outside of the Legion's structure. All of the places where the player encounters the Legion are forward camps where direct military service is given the most weight and is of the most immediate importance. Because only males are involved in that service, they look down upon females even though it's incredibly short-sighted.
So basically, lore-wise in Caesar's Legion women are viewed as inferior to men, but they are not merely slaves, but fill roles that Caesar deems fitting for their sex. However, they are generally seen as lesser than men and treated as such (which fits in with the Roman motif, considering women were treated pretty much the same in Roman times). Because all you see in the game are war camps and women aren't allowed in the military, women are nowhere to be found in those camps except as conquered slaves. The original plan was to include some of this stuff in the game, but most of it didn't make it in, which makes the Legion just seem almost comically misogynistic. It also makes playing through the Legion storyline with a female character a bit... odd.
EDIT: Not that I'm defending Caesar's Legion, they are very much the evil choice, no matter how you twist and turn it, but there's a method to their madness. They basically live on the following ideal "Hey, the Roman Empire managed to create a good society for hundreds of years, no reason why we couldn't do the same". They're not evil for the sake of being evil. I also personally love how the most evil faction in the game was created out of the most morally good faction in the game; Joshua Graham and Caesar used to be Followers before finding a pre-war book about the Roman Empire and deciding that the way to save the world was by creating the same kind of order that the Romans did.
