When HDR is turned on in the Windows display settings, doesn't Windows tonemap the desktop so SDR content is supposed to look correct? I know I've seen the option to adjust SDR white levels in the options. Also, games that don't support HDR just change back to SDR as long as they support Exclusive Fullscreen. So, what I'm asking is is it not possible to just keep HDR on all the time? That would get rid of all the switching back and forth when playing games.
I think if you turn HDR on in the display settings this is what triggers the HDR flag to the display. The screen becomes dim because the format of the desktop is entirely different, so it gives you the option to boost the overall brightness to make things useable again.
Meanwhile anything that is HDR compatible in borderless window i.e. Windows Store games can sit quite happily on the desktop, the colours are correct as they detect HDR is enabled in windows and are not affected by the increase in brightness you have applied.
Full Screen Exclusive Dx11 games take responsibility for triggering the HDR and booting up with the correct settings.
You'd like to think that if HDR is already running, then this will also be happy and you can just leave the Windows settings on all the time.
Best thing to do is find a game that has HDR support, runs full screen exclusive and boot it and look in the video settings to see if your HDR calibration options are still there. If so, then you should be able to leave it on all the time.