...
If size is a concern to you I would go straight for the micro four-thirds because the lenses on the X-T1 are the same as any APS-C lens.
...
If you go into the telephoto range, yes. Otherwise they are not really as big as my APS-C Nikon.
The 18-55 F2.8-4 is smallish, the 10-24 F4 is a magnitude smaller than the shorter range non-OIS Sigma 8-16 that I have.
The primes are smaller (not that much lighter) than the Nikon APS-C with few exceptions (the Nikon 50 F1.8 D is as small as it gets, the 35 F1.8DX is bigger but as light as the Fuji 35 F1.4 albeit at a smaller aperture).
While I don't have the m43 when I researched the system their lenses were smaller although the difference was not that big if you needed bright lenses.
PS. come to think of when I chose the X-E1 instead of the EM-5 when I added all the lenses that I wanted for each system the m43 was just a little lighter, enough not to care that much; now with brighter bigger lenses from Fuji the temptation might be to go with those which would increase the weight differential