I'd say go for it, even if it is originally just for one game.
Saying that, I'm also the person that bought a £300 Hotas just to play Elite Dangerous, which then cascaded into buying a Rift. This ultimately expanded me into other flight sims, and VR experiences which I have thoroughly enjoyed. Likewise, If you get an X you could branch off into other games using Gamepass.
You've got a great 65" TV, and should do it justice by pairing it with the best hardware. If the reports are to be believed RDR2 on the X will be pushing out twice the amount of pixels: 8.2m compared to 4.1m, and going by previous AAA games, this certainly follows a pattern.
It all depends whether you can trick your cognitive dissonance into saying that's a "slight upgrade" if you don't go for it.
Only you know how much that decision will weigh on your shoulders