The meaning of sensor size is a bit over-rated I think. We're not talking 8x10" here, an MF sensor has only 1.3-1.6 times larger diagonal than a full-frame 135 camera. The largest medium format sensors just matches the smallest medium format film formats.
With current technology there's a limit to how much photons we can capture per area unit silicon so there must be a certain size to be able to capture a certain quality. Mobile phone sensors are so small that they have issues with noise, ie you cannot reach state of the art image quality with that sensor size. But as you step up in sensor size the photon capture advantage is reduced, and today the difference between a 36x24mm sensor and 54x41mm is negligible in that particular aspect. The difference between 36x24mm and 44x33mm is even smaller. So no, we cannot drop down to mobile phone sensor size (not yet at least), but 36x24mm is up there with as good as it gets.
But you get whole different lens lines with different design targets, different raw conversion etc so you get a different system with a different outcome, which you may or may not prefer. The whole system makes a difference with small contributions here and there, difficult to point out one single aspect as the most important.
As a tech cam user my arguments for using MF is only partly about image quality, it's about having more fun shooting, a camera and workflow that suits my style and personality. An evening out shooting 2-3 hours I get home with 2-4 pictures, but I rarely throw away any of them, I only make the effort to set up the camera if I think it's going to be a good shot. I like that, more time out enjoying photography in the nature and less time in front of the computer. I find the all-mechanical camera to be charming to use, I love having the tools with tilt and shift on all lenses, and use it all the time in my shooting style, which gives that strict upright perspective as we previously saw in large format landscape photography, a style I like. Close to the ground shots, slanting horizons and short depth of fields or other creative effects commonly seen in 135 landscape photography is just not my style.