I do wonder what could make Xenoblade games sell better. The quality is extremely high. Content is there. Reviews favorable.
IMO the biggest issue is IP recognition with more mainstream audiences. Things like shulk in smash are good but then he doesn't appear in next two games. Two titans battling world? Gone. Awesome mechs? Gone. The series has no clear identity and the naming is confusing as hell. X? Is 2 a direct story sequel?
It's just not good marketing (I don't mean advertising).
It's a series with no clear through line.
Three releases in, this is what we can currently expect from Xenoblade games: a main story that will take most people at least 70-80 hours to complete, a complex battle system that will keep layering on mechanics throughout the game, and many other systems with varying degrees of complexity which will be necessary for most players to learn in some capacity.
As it stands, Xenoblade is a daunting experience for multiple reasons and therefore has an inherently limited audience. Cool mechanics like mechs, crossovers, and different art styles can help the series to some degree, but I don't think any of these things are going to take sales to the next level. There would need to be a fundamental shift for the series to hit another gear, and yeah, the battle system seems like the most obvious thing Monolith Soft could drastically change to make the games more "appealing." I selfishly hope they don't because I love the series the way it is, and the combat is a big part of that.
It will definitely be interesting to see what direction their next game goes in after Xenoblade Chronicles 2.