At one time or another I have owned the 85/1.8, 85/1.2 (v1), 100/2, 100/2.8 macro and 135/2. Keep in mind I use them on full fame cameras.
First off, I can tell you all of them are excellent. That said, both of the L's have the L look -- quite sharp, a bit crisper contrast, and a bit better color saturation than the non L's. The others will render very similar to your 50/1.4.
Your decision is going to be based on your intended use and preferred focal. At the end of the day, I hardly ever grabbed the 135 -- it was arguably the sharpest Canon lens I ever owned, even wide open, but for my style of shooting it was simply too long (on full frame) most of the time. (However, It would have made a nice fast long lens on a 1.6 crop sensor!)
The 100 was great too, but I sold it when I got the macro, feeling the macro woudl serve double duty. It didn't. It is more than sharp enough at close focus distances, but has a different look when used at normal distances. In the end, I use my 90 TSE with tubes or extenders when I want to go close and replaced the 100 macro with an 85/1.2 for the added speed and slighter shorter focal length -- which I now knew I would prefer over the longer 100 and 135. This is a great lens, and I loved using it, but it was so big it rarely went into the bag unless I knew for sure I was going to use it. As a result it was never in the bag when I wanted it...
So I now have the simple little 85/1.8, and carry it ALL THE TIME. For me, it is the ideal second focal to compliment my 50/1.4 and then I also round a trio out with the 28/1.8. I just returned froma trip to Europe where I carried only those three lenses and two 5D bodies -- and never felt like I needed anything else.
A word on the 28/1.8. Previous versions of that lens I tried -- a few years back -- were nothing short of horrible. However, it seems later versions may have been silently improved. This new copy (purchased last fall) is in the same class as the 50/1.4 except for perhaps the extreme corners wide open -- I really like it. Plus it's small and relatively inexpensive.
A shot from the 85 at f2, hand-held through a store window:

Here is another from the 28. Not the best shot, but shows the capabilities of the lens, including its handling of flare -- hand held at 1/15th, f 2.8:

Cheers,