I use a Nikon D810 which averages about 45Mb for the Raw files and I use a workflow very similar to your own, i.e. I use Lightroom (5.7) as the hub of all my photo-processing, employing Photoshop (CS6) and the suites of Nik and Topaz plug-ins from within Lightroom as required. I always bring back the processed files into LR for finishing, exporting or printing as appropriate.
I have never encountered the problem you identify. Might it perhaps be a result of hardware limitations? I use a 4-year old PC with i7 processor, 16Gb of Ram and the onboard graphics of the motherboard and have neither speed nor file size problems.