I see people theorizing that Baba was pushed out like Tabata, and it's like... Really? Tabata had something to show for his time at Square. Not even close for Baba.
He was not a great producer for the Tales series. Let's dispel one myth: he was NOT producer on Vesperia! He was brand manager at the time.
His tenure as producer is marked by Graces Wii being a glitchy mess on the Wii. Then Xillia being unfinished. Followed by the half-assed Xillia 2. And ending with Zestiria, which was a mess on its own. Even if you want to dismiss the fan backlash in Japan as waifu wars, there was some weird things going on with the voice actress Komatsu and the story around her character (not blaming her, and it was ridiculous that she was told to apologize to fans). Oh, and Hearts on DS was fine except for the inexplicable decision to have two versions with CG and anime style cutscenes, the former of which went straight to the bargain bin in Japan.
Based on all this, I don't think Baba was a good producer. Two of my examples could be considered flukes, but overall there's a pattern of bad decisions that weren't there for earlier in the series.
Baba is a hack, and I'm pretty sure he was behind the new scenario writer for Xillia 2 and on who had this creepy obsession with grim dark plots and fetuses and harming children as major plot points. It's really unfortunate that he was producer for not one, but two major series anniversaries.
Baba isn't some poor developer down on his luck. His fuck ups seem to be his own.
He doesn't deserve our pity or people acting like he's some great loss for the industry.
I'll give him credit for the Tales series being pushed in the West, though.