This is even more of a reason to give a polite rejection. It seems like you have difficulty saying no.It's usually more just acquaintances trying to be chummy that do this. With close friends we tend to be more aware of what/how much everyone prefers to drink.
But there's definitely an element of pressure to it. I can just not drink it, of course. But people do get called out for it which is a kind of bullshit element of drinking culture. So let's not pretend it's as simple as "just don't drink it."