Diffraction. in this case , is the bending of light rays at the aperture (iris) edge, It happens at all apertures, but it affects the image more at small apertures as a greater % of the rays are affected.In other words with little holes relatively more light rays don't focus where the lens says they should, as they hit the edge of the iris and bend away from their intended path.
Lens aberrations are usually worse at large lens openings, so best results are usually a compromise and a couple of stops down. Some lenses though, especially some Leitz glass for Leicas, are designed to have minimum aberrations when use wide open ie they are designed for low light photography of moving subjects.
Diffraction effects seem to be more visually acceptable when the iris is composed of a large number of blades, which results in a hole that is more spherical.This is more common in large format cameras where small apertures are required for depth of field purposes. See work by f64 . Note that even if the image is a little soft, thin bladed circular apertures give a pleasant softness and bokeh.
Shutter speed is not an issue.