The 24-105 is not a true zoom Originally the definition of zoom lens required that as focal length changed, focus did not. The 24-105, like a lot of modern "zooms," has an interactive relationship between focus an focal length. If you reframe a shot with the zoom collar, you have to refocus as well, and vice versa. I'm just guessing, but that interactivity between the controls might be what you're seeing.
Incidentally, I used to have a couple of true zooms for 35mm in the film days. I actually liked the "push-pull" zoom controls that were common on those lenses in the 80s, mostly because you could do everything without moving your fingers from one part of the lens to another. But from what I gather, most people did not like them, because there was always a chance you'd change focus as you zoomed. With something lie the 24-105, a push-pull layout would be more useful, if you're trying to do manual focus.