I think that Brexit might be worse because it's a public referendum that seems, at least from an outsider, very difficult to reverse. But, I say this as someone who doesn't fully understand the implications and execution of Brexit. Aside from the Republican Tax Bill and Trump's SCOTUS nominees (neither of whom have really been outsider nominees, more or less run of the mill GOP nominations which you could see under any GOP president... At least, with Gorsuch), most of Trump's impact has been on executive actions and executive policy decisions, which are things easily reversed with the next president.
Another concern is if Trump actively succeeds in furthering his trade war, which typically takes years to reverse (at least, from our last experience with Smoot Hawley). The thing with executive decisions, what Trump has relied most on to pursue his agenda, is that they're quickly overturned and changed as soon as there's a new president, this is what Trump did to Obama's executive decisions, and the next president will likely do the same.
My opinion might change after January 20, 2021. Assuming Trump loses the 2020 election, which isn't even a given at this point, but assuming optimistically he does, I think I'll be most concerned for the future of American democracy from ~Nov 6 2020 (or whatever day the election is) to January 21, 2021, whenever the next president's inauguration is.
Trump attacking political opponents on Twitter, siding with literally the head of the KGB over his own Justice Department, and the dozen or so other institutions he's looking to disrupt are things that make me very concerned... But how Trump will act if he loses, in those ~2 months, has me pretty terrified.