I have done a fair number of borderless test prints with FAP and have seen this problem. I think I had the platten to "Wide" though pretty early on for this type paper. You may want to try that to get rid of the "smudgy" stuff. since the 3800 doesn't have a vacuum many of us go 1 step wider on the platen then usual.
I've been using it on wide also. I was getting head strikes when it was set to auto.