Someone chime in here! This is why I didn't reply in the first place - my understanding here is limited. This is my understanding of what is going on - it might be wrong but I'm sure someone will say so.
6500K refers to the screen gamma/white point. Adjusting the brightness changes the number of lumens outputted by the LCD - which is why your must recalibrate - in adjusting your display brightness you will shift the gamma point - and therefore must reset this by recalibrating the screen.
I suggest that you leave your gamma at 6.5. If you are on a Mac, you adjust the display brightness in the Displays tab of System Preferences. On a PC, I think it will be in Control Panel. On a laptop, you may have F keys additionally used for screen brightness. Otherwise, you may have a hardware switch on the screen itself for turning it down. I would assume that using hardware control is more accurate than software but could be wrong. Regardless, every time you change the brightness, you must recalibrate.