I've been searching for a compact system camera to augment (not replace) my big DSLR kit for several years. Started with a Panasonic GF1 with a couple of primes back in 2010 -- a revelation! Excellent image quality in a camera the size of a medium p+s, even at ISO 400 and 800! That 20mm Panny lens is most excellent, too. I am sitting here at my desk looking at an outdoor portrait of my niece shot with that combo, then cropped vertically (so it's about half the original 12 megapixels) and printed to 16x20 inches on my Epson 3800. It's a lovely print and plenty sharp at her eyes. Did I mention that the Panny 20mm is terrific?
The GF1 high ISO quality wasn't good, sadly, so I added a Panasonic GH2. That was my primary small kit for a couple of years -- the two Panasonic cameras, and the 14, 20, and 45mm lenses (the Olympus 45/1.8 is also a very good lens.) The GH2 added a beautiful EVF - much better than the clip-on GF1 finder (ugh) - and excellent images to 1600, with very usable images at 3200.
Then I had a chance to handle the new Fuji X series cameras last month. One of my colleagues let me borrow his X Pro 1 for an afternoon, and another let me handle his for most of a week at a conference. I came home and purchased a pair of XE1 bodies and four lenses.
The Fujis are maddeningly quirky little cameras. Totally annoying and frustrating much of the time. But there is something about the images that is just wonderful, and makes me overlook whatever current quirk is driving me mad.
They are designed for slow, careful, deliberate work. At least, I hope they were designed that way, because that's the only way to use them. The autofocus is inclined to take its own sweet time, and the user interface -- which tries to emulate a mid-20th Century rangefinder, with manual aperture and shutter speed dials -- has enough little bugs and odd design choices to last a lifetime.
Now that I've talked you out of looking at them, let me add that the prime lenses are excellent, the zooms are very, very good, and the final image quality can be spectacular. By the end of the year they will have all the lenses you require in their lineup. My total kit weighs about the same as a single pro Canon body with an f/2.8 zoom, and man my shoulders and neck feel the difference. No, the Fujis won't ever replace my big Canon kit at work, but there I'm shooting things that the Fujis aren't designed to do (sports and other fast moving stuff.) They will almost certainly replace my own personal big Canon kit, and I won't miss it at all. For all the things that a traditional rangefinder camera kit can do, like travel and candids and landscape and portraiture, I am finding that the Fujis are a good choice. The more I use them, the faster and more responsive they become.
Good luck in your search. The other advice you have received about the Olympus OM-D is also good, that's a very nice little camera and the prime lenses are fantastic.