High ISO capability and image stabilization has changed the equation from the past, when "fast" lenses made possible faster shutter speeds needed to stop action in low light situations. These days you can crank up the ISO and use "slower" lenses and still get faster shutter speeds.
That said, "fast" and "slow" lenses have advantages and disadvantages. Fast lenses let in more light, giving you a brighter image in the viewfinder. They also allow for shallower depth of field giving more selective focus. On the downside, "fast" lenses tend to be bigger, heavier and more expensive than their "slow" equivalents.
I have Canon's 70-200 f/2.8 L IS and it has been a workhorse for me (amateur landscape, architectural, some sports and event photography). I have played with the 70-200 f/4 and it is smaller, lighter, easier to carry and arguably has a slight edge in sharpness. The 70-200 f/2.8 lens is NOT a walk-around lens: too big, heavy and obvious. IMO your 24-105 fits that role much better.
My recommendation to anyone thinking about buying a lens is to borrow or rent one, if possible, for a couple of days. Forums (fora?) like this one can give you lots of good information and opinons, but in the end you'll learn more by actually trying out the lens yourself.
Paul