The difference is the extra 4 pounds of weight of the 500mm f/4 lens. With the 500mm PF I shoot hand held except when using a 1.4x teleconverter and even then all I need is a monopod to keep the lens steady.
For me carrying the 500mm PF and the 80-400mm zoom is roughly the same load as a 500mm f/4 and I do not need to carry a tripod with a gimbal head which saves me 8 lbs of weight and also allows me to shoot faster and at different heights without delay.