Jeremy, the short answer to your question is that, sometimes, making progress requires taking steps that breaks compatibility with older versions, or at least makes integrations into older versions very difficult and/or very expensive. I confess this is difficult to appreciate unless you are familiar with the process of building large and complex software systems.
I maintain that Adobe's current approach is the preferred choice as it keeps the developers focused on advancing the craft of photography (i.e., technical image quality and workflow), instead of focused on performing tasks that have zero potential to advance the craft.