In reference to image quality, you should never use a filter if you don't require the effects of that filter.
A filter may or may not detract from the image quality. It all depends on the lighting conditions. You're most likely to have IQ issues in bright conditions, and when photographing light sources...such as neon lights. In these cases, reflections start to become an issue. You'll get flare, mirrored points of light, and even ghost images of bright subjects. UV and clear protective filters are most susceptible to these kinds of IQ problems because they're they let through nearly all the light of the scene. A polarizer or one-stop ND filter only lets through half the light of the scene, and so will suffer less from such artifacts.
With modern digital cameras, you don't need UV filters as there's already a UV filter in front of the sensor. To protect the lens, use a lens hood. The lens hood will provide additional protection against flare as well. Use a clear protective filter when there are environmental issues such as salt from sea spray or sulfur from Yellowstone geysers.