I've found some papers apparently come with more dust built into the rolls than others. I don't know if it's brand specific or just random luck. For instance I had a roll of Ilford GFS and after using the entire roll had only one or two prints with a speck that fell off along with the ink. Then I loaded a roll of Hahnemuhle Photo Silk Baryta, which is apparently identical to IGFS, and made 6 prints to get one without multiple white specks. On the last (good) one I fed out enough paper for the print, thoroughly cleaned it with a Swiffer Duster, rolled it back, and then printed it. I also thououghly cleaned the printer multiple times during this ordeal, but I'm convinced the dust came with the paper roll. FWIW, the Swiffer dusters hold onto the dust instead of moving it around and seem to work quite well.
edit: Also note that with these baryta papers after the printer cuts the print off of the roll the paper edge is absolutely loaded with little flakes of what I assume is the paper coating. It you press a finger tip on each side of the cut edge and run them down the cut you'll find them full of these white flakes. After doing that about 3 times they are mostly gone. The point is, when the printer cuts off the print and then winds paper back onto the roll, that cut paper edge effectively distributes dust all through the paper path. How convenient!
edit 2: FWIW, I have not seen this problem with Hahnemuhle Photo Silk Baryta sheets, but then sheets are easy to blow off before using them and rolls are not.