Yes, that's basically it: the 17-40 f/4L. The 16-35 f/2.8L buys you one stop wider for about twice the price, and the 14mm prime is too expensive for most users of reduced-frame dSLRs - about three times the price of the 17-40. The other non-L primes seem to be no better than the 17-40 L zoom (20/2.8, 24/2.8, 28/1.8, 28/2.8, 35/2) as long as you don't need a wider aperture than f/4. Hopefully soon Canon will improve their wide angle lenses...
So the only EF-S lens you really need is the 10-22, and you only need it if you have a real need for "super-wide" angles of view, as it's equivalent to the 16-35 on a full-frame camera. But the 17-85 IS lens is a very handy all-around lens, so I own only those two EF-S lenses. The rest are all EF.