A digital image consists of pixels and really has no physical size in millimeters. If the pixels are of equal quality, it makes no difference whether they were derived from a small or large sensor. However, in the real world, the effects of diffraction and noise often favor large pixels ....
A digital image is a record of an optical image, which most certainly does have physical size, and the optical image has characteristics like diffraction blur which do have physical size. When this information is tranferred into pixels, it is not lost. I guess you can forget about the physical size of the pixels and the sensor, provided you keep track of something equivalent, like the size of the diffraction blur measured in pixels. With this understanding, we proabably have no real disagreement.
(I still maintain though that there is no fundamental difference between recording an image on digital pixels or analog film, at least in the limit where the recording medium has finer resolution than the optical image. So why maintain that film size is significant, but digital sensor size is not? Is it just because film physically reminds you how large it is, but a digital file makes it easy to forget?)
Let me give a simple example where I would claim sensor size matters and I imagine you would claim it does not. I think we will agree on the final conclusions, if not on the semantics :)
A) 50mm lens at f/11 on 1/2" sensor with 4um pixels
B) 50mm lens at f/22 on 1/2" sensor with 4um pixels
C) 100mm lens at f/22 on 1" sensor with 8um pixels
All three images have the same field of view and perspective. All three images have the same pixel count. However, the digital images are not equivalent.
Image A will produce a sharper print than image B (but with less depth of focus), because the optical resolution is higher (less diffraction).
Image C will also produce a sharper print than image B, even though the optical resolution is the same. From my viewpoint, this is because less magnification is required to make the final print. In your view, I suppose it is because the diffraction blur spreads across fewer pixels.
Images A and C will produce similar prints, with comparable sharpness and depth of focus. I concede that in this comparison sensor size does not matter, because optical resolution was scaled to match the sensor size.