I still can't fix the leaning building and skewed horizontals and verticals, though. Any help would be very welcome!
You can define vertical and horizontal constraining control points with PTgui.
Just go to the control point tab, select the same image in the left and right views then select vertical of horizontal as control point type.
For vertical control points, create a control point by clicking on a vertical line of the building in the left window, and on a lower point of the same vertical in the right window.
Do that both on an image at the extreme left of the pano, and do the same to another one at the extreme right of the pano.
You can do the same for horizontal lines.
Then optimize once more (approve the proposal to change something)... and you should be done.