Okay so what's the plan, if they won't vote for policies that benefit them or people who reach out to them beyond 'fuck brown people' what do you want to do? Bearing in mind that if you shift over to the right, they'll just go even more extreme and beat you on that front.
Here's one problem I see straight away: you used the word "them".
And why do you assume they all want to say "fuck you" to brown people? Sure, some do, but we can ignore those ones and focus on the others.
Political manifestos shouldn't be written for the working class, they should be written BY the working class.
If Corbynism and New Labour had only one thing in common, it was this: both the 1997 and 2019 manifestos gave off an image that they had been written by inner city cosmopolitan elites.... because that's exactly who wrote them.
The second referendum was a classic example - that policy wasn't cooked up by a voter in Sunderland, it was cooked up by some dickhead in London who didn't want to accept the result. What happened? The working class voters simply just said "why didn't you fuckers respect our brexit vote in the first place?", and voted Tory.
A good start might be to simply ask these people "where do you want to see your town economically in ten years?", get rid of any racist crap, and go from there. Bring some of them into the economic discussion about what they want.