I'm on Windows 10 64 running LR 6.10 and an Epson P800, so I decided to try making a custom paper size. Here's what I tried:
1. In LR I picked an image to pretend to print, and went into the Print module.
2. I picked "Page Setup" (bottom left), and a "Print Setup" menu appeared, with a Paper Size defaulting to "Letter (8 1/2 x 11 in)", with a pull-down menu.
Pulling down that menu revealed a huge list of paper sizes, so I invented one that wasn't already there: 6x9". Since it wasn't in the list, I chose "User Defined" at the bottom of the list. It was a bit unsettling not to find any place to specfy the size I wanted, but I didn't give up. (This may be the point at which you began scratching your head.) I clicked "OK" to exit back into LightRoom.
3. Back in LR I set the margins to 0.5" all around and adjusted the cell size to 5.00 x 8.00". (Still no specific paper size!)
4. I next clicked "Printer" (bottom right), and in that Print menu I clicked "Properties." Part way down the page was "Paper Settings," including "Size," which defaulted to "User Defined." BUT This was different from the previous, similar menu in step 2 (above) , because now there was a new box to the right also labeled "User Defined..." And clicking this finally opened a menu letting me specify width and height of my new paper size, and to give it a "Paper Size Name." I called my size 6x9.
5. At this point I clicked "Cancel," since I don't have any 6x9" paper. But when I next looked at the list of paper sizes from Step 2 (above,) my new size of 6x9 was listed there with all the others!
Please try this and let us know if it works for you. It certainly seems a round-about way to define a paper size, and I suppose there is a good chance that I could have gotten there faster if I had tried the "Printer" button instead of the "Page Setup" button when I first started experimenting.
I hope this solves it for you.
-Eric