Honestly? This is the first arc in the series that is making me consider dropping it or just waiting for it to be over and reading the TL;DR. I feel like I have no idea what is going on anymore and it feels like a chore to read. What happened Oda? :(
Yeah, there's a lot of characters, but the way the acts have been structured we're mainly focusing on a certain number of characters right now while others are doing stuff in the background.
Besides the Straw Hats, the Heart Pirates, and the minks, there are about 10 really important characters you would have to remember.