I have a 5D like you, and my battery of medium primes are the 35mm f/2, the 50mm f/1.4, and the 85mm f/1.8 (I also have a 300mm f/4L, but I guess that is not relevant to the discussion).
The 50mm probably just edges the 85mm in terms of optical quality, but it is a close run thing and both are seriously excellent. The only optical problem with the 50mm is that it shows a bit of barrel distortion, which may or may not be an issue for you. The 85mm is a bit of an oddball focal length, and is really only good for portraits (and boy oh boy is it good for those).
The 50mm has a rather flaky AF mechanism - it claims to be USM, but it isn't the same as the USM drives on Canon's classier / more recent lenses. My lens' AF occasionally sticks, especially when transitioning from long to short distance focus. Not a big problem because you just focus manually until you are close enough and then let AF take over again... however it is an irritant and I have lost several shots as a result, and I don't fully trust the lens. The 85mm doesn't have this issue.
The 35mm f/2 spends much more time on my camera than either of the other two lenses - it is tiny, unobtrusive and light as a feather, and optically it is very good - not quite as good as the others, but pretty damn good nonetheless. The only negative with this lens is the noisy non-USM AF, but personally it doesn't bother me. This, for me, is the ideal lens for street and candids.
What you might consider doing is borrowing a wide to tele zoom, eg the 24-105mm f/4L IS, and playing with it for a while. You will quickly discover your preferred focal length, and that could determine your choice of prime. Or, alternatively, you could just go ahead and buy all 3 primes, as they are all exceptionally cheap considering their optical quality.
On the question of L versus non-L, obviously the cost will be much greater, and your return in terms of IQ and lens speed will be rather marginal. Build quality will be better, but that said the build quality on the three non-L versions is pretty good too. Personally, even if I had all the money in the world, I would probably stick with the non-L primes. The reason is that all the L versions are, relatively speaking, huge and heavy. For me, the whole point of primes is that they should be small, light and unobtrusive - the L class primes are the opposite.
Ed