I believe that what you're looking for is:
a) a reduced picture element cell size, where a cell is traditionally made up of an integer number of dots in each direction
b) a nicely orthoganal cell matrix, so that you can get the lenticular lens lined up properly
Am I on the right track so far?
If so, you've got the usual tradeoff between the number of possible colours in each cell, and the spatial resolution - just as with any printer that uses discrete dots of ink (but not dye-sub printers, which are more continuous).
So, the Epson uses a stepper size of up to 2880 units/inch in one direction, and something like 720 in the other direction as the actual dot placement resolution, and you choose some (hopefully integer) multiple of that as your screening frequency. Nothing new so far.
But now I wonder about diffusion dot screening, which is what the current Epson and Canon printer drivers use by default. If I understand the way diffusion screening works, the cell size isn't constant - which is great for general inkjet prints, but potentially screws up the orthogonal and regular cell matrix you'd want for the lenticular lens. So I wonder if a RIP that uses the older fixed cell size method might be better suited to having a lenticular lens glued to the print? Maybe I'm wrong about that, since you seem to be getting decent results so far!
Going back to the spatial resolution, I wonder what your success would be if you opted for a larger lens size, but made up for it by using a larger print, and an increased viewing distance? That way the final dot resolution of the printer would be less critical, it might be easier to align the lenticular lens, you'd get better colour depth, and still get the perception of spatial resolution you're after? Of course, the downside is that going bigger makes everything in the process more expensive ...