I used to feel so bad when I worked nights. Every saturday morning, I'd usually stop by the local convenience store on my way home and buy some beers at like 7:30 am. And I'd drink them that morning before taking a nap. Just wanted to relax after a tough week of work since it was my equivalent of a friday night but it always made me feel super weird and guilty lol.
When I went back to normal hours, I would still day drink occasionally. But yeah it can be a slippery slope depending on your personality. Some people are good at setting reasonable boundaries for themselves, others are not. I was having too many internal debates about how much was too much, where was the exact limit, would one more drink be okay, etc. So I just decided to stop altogether before it turned ugly.