The sensor size that can be used with existing 35mm lenses is limited by the diameter of the image circle, so for shorter focal lengths at least, you cannot exceed the 43mm of 35mm format, and indeed a little less seems advisable, to deal with slight weakness in the corners that is sometimes revealed by high quality digital sensors.
So you would be limited to 30x30mm. That means that the break even point for getting a larger image compared to a 24x36 sensor would be 24x30, or 5:4 shape. The square camera image would be bigger for shapes form square to 5:4, but smaller for anything from 5:4 through 3:2 and wider. Since the great majority of even fine art prints are in the shape range from 5:4 up, most photographers would more often lose usable image are rather than gaining.
This is before allowing for the fact that the greater frame depth of 30mm (or 36mm) would considerably increase the depth of the mirror and the size of the prism if one stays with the standard SLR approach. Thus it would require a major redesign and up-sizing, and might not even be compatable with current 35mm lenses, since they might need to be mounted further from the focal plane in order to make room for the bigger mirror.
By the time all the necessary changes are made, the camera ends up looking a lot like a 645 medium format digital system.
