I have used a Fuji X-E1 system for portraits, some fashion/professional work as well as a bit of travel photography. It pretty much replaced my Nikon system (for the moment at least). If you do not need fast AF, tethering or certain lenses unavailable for the Fuji, the X-E1 is almost perfect. It is small, does not get in the way, and seems to generate less of a barrier between the subject and the photographer than a large camera, which is particulary nice for environmental portraits.
In terms of lenses, the 35mm Fuji lens draws very nicely (I know this is subjective), there is a plethora of old lenses to get a particular look, and manual focus works fine as long as the subject doesn't move despite the lack of focus peaking. Personally, I like the Voigtlander 75mm 1.8 for portraits, with a tripod for shallow depth of field images.
I cannot really comment on landscape photography or video, but I have a feeling there might be better options for that.
I hope this helps,
Martin
Update (8/13): The Fuji is great for portraits and stationary subjects. Unfortunately, the AF is not fast enough, even after the latest firmware update, for anything that moves. I am pretty much back to my Nikon setup for most of my current work.