I agree Rob. I normally travel with my wife and the expressed purpose of the travel is vacation time for both of us. Photography is a distant second, at best, on the priority list.
So I do two things. First, catch as catch can which usually results in few if any keepers. Second, I get up early and go out alone for an hour or so. That gives me a little time to concentrate on photography without turning it into a 'photo trip' which my wife doesn't care for much.
Rarely, we will make an excursion purely for photography purposes and she will grudgingly be my assistant and gear mule. Very, very rarely.