Certainly for walking around I prefer the 300mm f/4. I was having some more serious health problems when I went to Bali last year, and I had to rent a 5D Mrkii and a few lenses, because I needed a compromise on weight and functionality which I couldn't get with any of my stuff. D3s, far to heavy, Phase out of the question, Leica, yeah no because I wanted to do some wildlife, and wanted AF camera. So it was a logical choice. I took the 35mm f/1.4, 50mm f/1.2 and 300mm f/4 and couldn't have been happier with it.
I found it was very good optically, and had no issues with the images that I was making handheld.....it is light and it works. I would be willing to bet a fixed 300mm would have better quality then a 100-400 on either Nikon and Canon, and it is prob lighter and better AF no?
No experience with the 400mm f/5.6 but I would be willing to bet it might be worse since its basically the 300mm f/4 with a TC built in...