Once you accept the EOS-M's sensor size
It's an APS-C sensor, so there really isn't a whole lot to 'accept', is there?
and complete lack of EVF (not even available as an accessory)
The lack of a viewfinder, optical or electronic is a killer, even optional in the shoe (as with the RX1). Problem with a shoe option is no way to link it to the angle of view of the lens, unless it were electronic and signals were sent through the shoe contacts in some fashion.
If I were going to go for an exchangeable lens option in a smaller, 'streetable', carry around camera, I'd probably look at the X Pro 1, X-E1 or the Nikon 1 (despite its smaller sensor), the Nex line, or maybe M4/3 although some of those are a bit larger.
The idea of being limited by a single focal length is, as others have said, a misleading viewpoint. It can be seen both ways. Limiting and liberating. Also challenging from the standpoint of testing one's ability to create compelling images with a single lens. Who isn't up for a challenge?
I just picked up a 'Texas Leica' for not a bad price to mess around a bit with MF film again. Same consideration wrt a fixed lens. There were plenty of those sold 'in the day', in a variety of formats.