Far and away the greatest contributor to image quality is sensor size -- capture more photons and a better image results.
Emil, I agree with most of what you say, in particular the part about gathering photons faster from the subject being the main ingredient in lower noise at a given shutter speed. But for comparisons of noise levels when the need for adequate shutter speed forces the use of higher than minimum ISO speed, what matters most for photons captured and thus noise levels is not sensor size but
effective aperture size. You are making the usual assumption of equal ISO speed and thus equal aperture ratio needed for equal shutter speed, which means that the larger sensor and the corresponding longer focal length used give a larger effective aperture size, and thus faster light gathering from same subject. For example, increasing linear format size by 1.4x ("one stop") and thus doubling sensor area also involves increasing focal length and effective aperture diameter by 1.4x and so doubling effective aperture area, which doubles the rate at which photons are gathered from the same subject.
But if one can reduce aperture ratio in the original format by one stop, one again increases effective aperture diameter by 1.4x, doubling light gathering speed, and one can get the same shutter speed with half the ISO speed, giving a similar improvement in noise levels. (We are talking about high ISO situations, so halving ISO speed is possible.)
Of course larger sesors do indirectly allow greater light gathering speed by allowing larger effective aperture sizes, since a given aperture ratio correspond to a larger aperture size with the longer focal length used to get the same FOV with a larger format. Even slow SLR lenses like f/3.5-5.6 zooms are faster (at gathering light) than the fastest current compact digicam lenses, due to larger effective aperture diameters.
But give credit where credit is due for increased speed: to the "big glass", not the "big silicon" alone.