As a whole, yeah. X5-X7 really bring down the series as a whole. Zero's weakest entry is 4 and even that's still a perfectly decent game at the end of the day.
Zero 1-4 Vs. X1-X4 is a much more interesting debate.
Aye, it's one of the reasons I wish I'd been able to play X4 - I've played the first three, but never had been able to get ahold of the fourth. Looking forward to a legacy collection.
Personally, there's aspects about each I prefer. I like the art style of Zero and ZX more than the (to me) clunky looking X, but I prefer the wider types of weaponry seen in X. I know Zero tried to use EX skills to emulate it, but it wasn't quite the same. Then ZX came along, and I definitely like what it did with the different forms, but then the sequel had to go and ruin it by taking it way too far...
You also have odd things - X1's relative ease, the extreme difficulty of the earlier Zero games, and then ZX's arguable identity crisis...
It's an interesting conversation for sure. Honestly, right now the only thing making me lean more Zero 1-4 than X 1-4 is something as tiny as art style preference, they're both a solid selection of games.