I understand diffraction in theory, but it seems that few lenses are truly 'diffraction-limited'.
Actually, all lenses are diffraction-limited (well, with the possible exception of toy lenses such as The Lensbaby). The question is,
at which aperture does the limitation set in? Cheap lenses are diffraction-limited only at small apertures such as f/11 or f/16 and beyond. Good lenses are diffraction-limited from, say, f/5.6 on. And only the finest lenses are diffraction-limited from, say, f/2.8 on. For industrial purposes, there are lenses which are diffraction-limited at f/0.7 ... but they cannot be used for photography. You wouldn't be able to afford them anyway
What I'm wondering is if there are some principles of lens design or set of principles that make one kind of lens able to get closer to the theoretical limits than another.
Yes, sure: quality. In other words: lack of lens aberrations. Being diffraction-limited means: having residual spheric and chromatic aberrations so small that diffraction affects resolution more than anything else.
Another way to word this is were you shopping for a very good macro lens how could you evaluate its potential by looking at its design?
Not at all. Unless you're a seasoned lens designer, there's no way to evaluate potential lens quality from just looking at the design. Furthermore, there are more factors affecting lens quality besides design---precision of manufacture and strictness of quality control, in particular. For end users, it's better to look at the price tag. Excellent lenses are never cheap.
For example does physical aperture design affect this? Number of blades, thickness and shape?
No.
If the light is diffracted by the blades, then won't virtual distance from aperture to film plane make a difference?
Yes---but that distance is strictly determined by focal length and focus distance ... no degree of freedom here.
How about the number of glass elements rear of the aperture?
No.
I'm guessing magnification factor or format size makes a difference?
Yes, indeed. The absolute amount of diffraction is the same for all formats, but the (relative) effect on resolution is the inverse proportion of the linear image size. Larger format, less blur through diffraction. That's why you can stop down a view camera's lens more than the lens on an APS-C-format camera and still get away with no visible diffraction blur.
Magnification also affects the effect of diffraction. Higher magnification, more blur through diffraction. That's why it is so difficult to find a good compromise for depth-of-field and sharpness at really high magnifications way beyond 1:1.
... but still from experience it does seem like a lot of lenses just hold up better as they are stopped down but I can't put my finger on why.
One thing is for sure: there is no way to overcome the effects of diffraction. Diffraction is the same for all lenses. There is no way for a good lens to resolve more line pairs per millimeter at, say, f/16 than another good lens at the same aperture and the same magnification on the same camera. Still there's room to be different---after all, there's more to image quality than just resolution: there may be different micro contrast, different field curvature, different distortion, different colour rendition, different bokeh. In particular, less contrast at the same resolution will yield less acutance, i. e. less visual sharpness.
-- Olaf
EDIT: Err ... wait a minute. I just said, "There is no way for a good lens to resolve more line pairs per millimeter at, say, f/16 than another good lens at the same aperture and the same magnification on the same camera." That's not perfectly true. Pupil magnification does make a difference ... not near infinity but at close focus distance, i. e. at large reproduction ratio. Larger pupil magnification means less diffraction. However it also means less depth-of-field, and when stopping down for the same depth-of-field then diffraction will be the same again.
