If you don't have legacy lenses to contend with, any reasonable sensor size will do... 35mm full-frame is nothing especially optimized - it is an early 20th Century format running 35mm movie film horizontally to get a decent frame size and an easily held camera. It actually predates Oskar Barnack by a couple of decades, although Leica did popularize it.
You don't want to make the frame too small, because you can't shrink the camera beyond a certain point while still having it easy to handle. Micro 4/3 begins to run into this problem - yes, you can make smaller cameras than you can for APS-C or full frame, but they're hard to handle if you have bigger hands (and the easier m43 cameras to handle could fit an APS-C sensor, with the biggest having room for full frame). There are also parts of the camera that either can't shrink or shouldn't - you want a ~3" screen, an EVF with a reasonable eye window, and a decent-sized battery. You don't want to make the frame too big, because the cameras and lenses start to get unwieldy. Hasselblad's H1D fits a 33x44 mm sensor in a small body, but the lenses are relatively large and you give up stabilization. Anything bigger would require very large bodies and lenses (most small film-era lens designs don't work well on digital because of the requirement that the light strikes the sensor closer to head-on) - Mamiya 6 and 7 lenses would be as much of a pain as older, tiny Leica M designs have been.
The original reason for the APS-C sensor size had nothing to do with APS film - it was just about the largest sensor that could be imaged in a single pass on the silicon wafer, and a lot of them fit on the wafer. FF sensors were nearly impossible to make in the early days of affordable digital SLRs, and were very expensive for quite a while thereafter.
I don't know the story of why the manufacturers stuck with a 3:2 aspect ratio - I suspect simply for familiarity. At first, the R&D effort went into getting the cameras out - the manufacturers stuck with lens mounts photographers already had lenses for, and didn't bother making "digital-only" lenses for quite a while. Several popular primes worked out well - a 35mm is a very nice normal lens on APS-C, while 24mm is a moderate wide angle, 20mm is almost wide enough for a true wide angle, and 50mm is a slightly short portrait lens. Long telephotos gained reach, which nobody complained about, especially on resolution-starved early sensors. Zoom lenses were more of an issue - 70-200mm was fine (it's a useful 100-300 equivalent), while most of the zooms that started wide became markedly less useful because they lost too much coverage on the wide end.
By the time Canon, Nikon, Pentax and Minolta started making digital-specific lenses in optimized focal lengths, the vast majority of body sales were on the low end - they needed $300 lenses to go with their $1000 bodies. For the most part, that's what the digital-specific lenses were... There's no reason they couldn't have built a beautiful 18-55mm that sold for $1500 - they just (for the most part) didn't. There were a few nice APS-C lenses in their lines (and there still are) , ranging from Nikon's original 18-70 f3.3-4.5 that was the D70's kit lens to much more expensive 14mm primes, 12-whatever zooms and and f2.8 zooms.
Just as they were broadening their lines of better quality APS-C lenses, full-frame sensors became affordable enough to use in most higher-end cameras. Instead of building lenses to go with APS-C sensors, they had sensors that went with their existing lenses - Canon had 18 years of EF lenses by the time the EOS 5D was a smash hit (they also had the earlier, very expensive full-frame EOS 1Ds, but that was a specialty camera). Nikon had nearly 50 years of F-mount lenses when they started selling a lot of D700s.
The market for APS-C bodies from Canon and Nikon in the wake of the 5D and D700 was strongly skewed down, so the majority of lens designs were built for low-end bodies. Both Canon and Nikon maintained a few higher-end APS-C bodies that were crying out for great APS-C lenses (the D300/D500 line and the D7x00 line for Nikon and the 60D/70D/80D and later the 7D line for Canon), but the big sellers were Rebels and D3x00 bodies, so the innumerable 18-whatever zooms that prized price, compactness and stuffing in as much reach as possible were the order of the day.
Fuji didn't have an existing lens system that made sense for the X-Pro 1 - it was always going to be a new mount. Sales projections were low enough that a custom sensor was out of the question (it used a custom color filter on top of a standard sensor). The 16 MP Sony sensor from the Nikon D7000 (and many other cameras) was readily available and affordable. It happened to be APS-C, so Fuji started building nice lenses to go with an APS-C sensor. They ended up with something no other manufacturer has ever had - a full line of high-quality APS-C specific lenses. Canon or Nikon could have done the same thing with EF-S or DX lenses, and probably would have if FF sensors hadn't come along as film compatibility became less important.
Sony's initial foray into mirrorless with the NEX system prized extreme compactness. The first few bodies had essentially no controls on them, and were so tiny it was remarkable an APS-C sensor fit. They made the E-mount small to make sure it fit on the NEX-3 and NEX-5. Their initial lenses followed the same pattern - compactness over quality, and they have never really released the right lenses for the NEX-7 or the later a6000 series. I've never been able to figure out why they kept the mount for their full-frame line - the sensor barely fit, and it has made lens design a pain,by their own admission at one point. They didn't have any lenses that mattered - no APS-C E-mount lens covered the larger sensor, so they could have easily gone for a larger-diameter version. Their initial FE lenses were compromised, although they've released some beauties since then (not always the most compact, though).
Canon and Nikon have both gone for brand-new mounts for high-end mirrorless, and they've both picked wide, but shallow mounts for easy lens design. Why stick with a sensor size that was originally two frames of movie film? Why not something like 27x32mm, which would be about the same size with a different aspect ratio? Why not 28x34 or even 28x36mm, which would fit easily in those big mounts? Going the other way, why not a 16:9 sensor of about the same area for native 8K video (with sufficient processor power) - Panasonic, are you thinking that way? Probably because they can share sensors with their own FF DSLRs, at least in part? They both rely on lens adapters to broaden compatibility to all of their older lens designs, and a different sensor size would reduce that compatibility - even if the image circle fit, baffles in the lenses might be a problem.
Interestingly, we are reaching a convergence where sensor size and resolution are tightly correlated. Most APS-C sensors today are 24MP, with a pixel pitch very close to 4um, and those sensors are excellent performers. The high resolution FF sensors have pixel pitches a bit larger (about 4.1 - 4.5 um), for resolutions ranging from 42+ to 50MP. You gain a little bit of dynamic range and noise performance (at least on the Sony-sourced versions of those sensors), probably in part from lower base ISOs. Going to a 4um pitch on full-frame would put resolution in the 55-60 MP range - that's a small difference from what we have today, so the R&D may or may not be worth it. Both the Sony 100MP 33x44mm sensor and the 150MP "full-frame 645 (it's not quite)" sensor are right around 4um pitch sensors.
To get a meaningful increase in resolution (it would be around 36MP on APS-C and almost exactly 80MP on full-frame), would require going to a 3.3 um pixel pitch. That sensor exists, and its performance is nothing to write home about - it's the 20MP Micro 4/3 sensor. I'd far rather have a 24 MP X-H1 or a 46 MP D850 with their dynamic range and rendering than a 36 or 80 MP camera with the dynamic range and rendering of modern Micro 4/3. It's not bad, but it doesn't have the richness of other modern sensors.
