So yes, it was racist and he shouldn't have done it, but culturally back then it was pretty acceptable and the question is always whether we should judge people for their past actions by our present standards. And that's a more nuanced question than a lot of people give credit for, I think.
That's the most important thing.
Times change and nobody is perfect. Society changes as well.
Just think about what you are doing today, can you already expect something to be offensive in 20 years?
It's always important to see what this person has been done since then.
Consistently racist and policies against minorities? Hell, time to not vote for this guy (if it was ever acceptable).
Everyone does mistakes. And if people apologize and actually don't do this shit anymore, hey, that's great.
If you do a mistake a second time after being called out, then it's a completely different story.