The problem you are having is that the custom pape size you are setting up is in one of the directions above the maximum paper size for the printer (OSX will alow you to do this without any warnings). Posibly you are putting the width as 50 inches?
When printing with the borderless printers there are normally two options, borderless retain size and borderless auto-expand, with borderless auto-expand the image will be expanded by 1/4" this is to ensure that the printer will print over the edge of the paper and not leave a white border even if the paper skews slightly as its feed through the printer. With retain size you need to ensure that your image is slightly oversize for the above reason.
The banner mode is there to enable long prints to be made from applications that support tileing such as Illustrator where the image is tiled into pages and with the save roll paper checked in the roll paper options the pages will be printed without any visible breaks.