Yeah... I can see why the gameplay was such a divisive sticking point for a lot of people when it came out, the slow lumbering movements, the weird design decisions, the unpredictability of your horse at times, and the shooting that took a while to figure out... But I feel like for people who stuck with the game then end up coming around on the gameplay... That the gameplay, controls, etc, feel internally consistent with the world.
I expressed controlling Arthur is different from controlling characters in most other games. Like, I kinda started to feel that Arthur is a character that i'm "Sending" controls to, but he's still his own thing in the world... Kinda like how you said, how he sways, is slow, if he's tired he moves a certain way, and I think of the control of this game being almost like a puppeteer with puppets on a string:
Like you, the human being player, are controlling the wooden block (the controller) which in turn influences the movement of the puppet, but the puppet can have a life of its own because it is it's own creature in a world. I totally get why it's divisive, especially early when you start playing the game, but I think it ends up making the world feel more internally consistent.
Likewise, with horses. When you're riding a horse, you the player are sending controls to arthur, who is in turn riding a horse which is itself a creature in the world. It's a weird thing about the game that I didn't like at first, but came to think of it that way.
