I have an X-pro1 (same sensor as X-e1) I'm getting good results using LR4.2 on 90% of my raw files. The RAW conversion could and should be improved but for my criteria I (and others) find the results to be just fine. Everyone has their own criteria but many Fuji users are pleased with the image quality.
I'm in the same boat. It's interesting how something that is sometimes an issue becomes an all out fact. There are plenty of people (including me) who have seen some of the issues with RAW in ACR/LR, but they are a tiny portion of shots, and only with some subject matter.
Overall, I find the output outstanding.
Another oft-quoted mark against the Fujis is the 'limited' lens selection. For some people it may be limited. There is currently only one zoom. What I find though is that this is a limitation for only a small number of people. It's one of those largely theoretical limitations. It's certainly nice to buy in to a system with a vast lens array, but how many serious photographers buy a vast array, and how many of those actually use them.
If the current or planned Fuji lineup suits your style and subject matter, then compare on that basis. if it doesn't, then don't consider Fuji at all. But if you are in the former camp, the Fuji is, for me, a superior camera. I assume the Fuji lens lineup must be nearly appropriate, because otherwise why would you raise the XE-1 as an option?
I love the OMD on paper. What a versatile camera! But I don't need most of the lenses on offer. More importantly, I don't like the camera 'in person'. The Fujis (X-Pro 1 or XE-1) are well made and have superb control and ergonomics. Output is superb, limited only by the photographer. I don't need the tilting screen of the OMD. The EVF is good, but in my experience no better than the XE-1, which I consider (for my needs) to be superior. The OMD has faster autofocus, but that is of no advantage to my landscape and documentary work.
I suppose I could wander through the advantages and disadvantages of each camera. But I think it's more important to start with what you shoot, and how you shoot. if you want the best possible files in a camera that's easy to control, and you don't need any of the features the OMD boasts, then the Fuji is worth a serious try.
The personal bond you have with a camera is important. I'll repeat: I admire the OMD on paper, but in person it's cramped, has an overly complex UI, boasts features I don't need, and just isn't fun. I find the Fujis' addictive.
As others have noted, it's great to have the choice. I would simply caution against buying on feature list alone. Love your camera and you'll use it a lot more.