You sound as level-headed as a decent person could be about it.
A garbage-person would conclude that your interactions is indicative of the makeup of millions of strangers, and would let this inform future interactions with people who don't look like you.
People are very complex, and if cornered - or feeling indignant - often don't shy away from being personally hurtful. If you were a young woman, or an easy-to-spot-non-straight-person, you would be ill-treated along those lines (and it can get incredibly ugly).
For white, blonde people - frankly, people who look like me - the most effective angle "to work" in an ugly, stranger-interaction, is to accuse you of being racist, because it's so difficult to really dispute, and because white people - again, who look like me - tend to bend over backwards when accusted of being racist (partly because they are well keyed-into privilege, and don't want to add to the misery of this world).
The world is very complex, and so are humans. I can't imagine that any random, clever brown person isn't also aware that - if they wanted to act amorally - could probably push well-meaning white people (through shame) to do all kinds of messed up interactions.
For example, (very) young, non-white pickpocketers around the world seem to understand contemporary racial interactions in a very clear and perceptive manner, so why wouldn't a non-white adult also be aware of these mechanics?
Ultimately, this situation sounds like a stressful environment to work within, and I think you're being tripped up a bit by accusations of racism when you try to evaluate whether this is a job you can do (or want to do).
It's understandable that you feel a little tripped up about it, but since you seem concerned about possibly being racist - and posting about it here, on an open forum - it also seems fair to assume that you are trying your best to be egalitarian and fair.
It could be that you're simply in a bad work-situation, and your ability to assess if it is bad for you is distracted by a worry that you might, indeed, be secretly racist.
//Just to be clear - and because I am talking about racial tension as expressed by individual interactions, I want to stress that I am coming at this in good faith. Chances are that what I've said could be misconstrued in terms of ill-will, so I just feel it is important to stress this.
Cheers, love.