For 24MP raws, my M1 MacBook Pro* (64gigs of RAM) is very fast in Adobe Camera Raw, Lightroom Classic, and Photoshop. Do I have want bumps for an M4? Sure. But I see no reason at this point, if I upgrade, it will be late next year for an M5. Not that I need that much more speed, but as Macs get older and older, support for newer OSs becomes more necessary with newer hardware to support it. My old 2019 MacBook Pro, which uses an Intel chip, is being used by my wife, and it will not support the next major OS as far as I've seen. So by next year, a new MBP for me, she gets the M1, which should hold us for a while. You can probably get great deals on older than M4 products, I suspect unless you're really pushing much larger images, that will work quite well for you.
*MBP Pro 16" 2022 M1 Max/32-Core GPU | 16-Core Neural Engine/64GB