Stunning Developments in image processing by Google

[narikin]

Google's new Pixel 3 camera has some stunning 'computational photography' development tricks explained in the tech blog here.

Really clever stuff, using the natural movement of the photographer during multi frame bursts, to mimic pixel shifting, align those in software and get a high res image out of it. (without de-mosaicing, in most cases).

At the moment they are using it as a zoom on the pixel 3's single cameras, but obviously it's a short software step to high resolution full frame pixel shift images.

Apple/ iPhone Xs uses some tricks of its own, but nothing like this level, afaik.

(and... Pixel 3 also gives 'computational RAW' feature - raw DNG images with multi frame combined, and... free unlimited storage on Google Drive/Cloud for all your Pixel images, including those RAW's. It seems that this is the phone for photographers)

[eronald]

I checked out the Pixel side by side with my own phone.
The Pixel is supposed to be able to do simultaneous audio translation to earphones, and what happens when you point it at an object is uncanny, it read my handwriting off my sketchpad. It can identify commercial objects, and I assume faces. Whether you like it or not, I assume it phones home data on what it recognises to both Google and the NSA. And I assume the translator's real purpose is  to provide Google and the NSA searchable text transcriptions of anything within microphone range. 

I think ads tuned to what the Pixel sees around you or hears when "switched off" are not far away. In my opinion this device has as first and foremost goal commercial behavior detection and as a side effect law enforcement surveillance.

In spite of the huge computational abilities, white adaptation of the demo I saw to ambient light was bad,the screen is at least as good as the iPhone's but  tone mapping of images was deficient, and while the functionality seems intrinsically superior to my iPhone Xs Max, the "feature polish" is as usual on Android yucky, leading it to be more of an engineering demo than a quality product. I really would have liked to like it but I didn't.

In summary, I think the design goal of this device is to feed Google by means of distributed object and voice recognition, and path descriptions, more than to serve the user.  As an advertising platform designed to detect behavior patterns and sales opportunities, it is extraordinary, as a surveillance device its capabilities are superb. As a phone, it is stellar or less good, depending whether you come from Android or the other side. As a camera, I think it needs a good third part app to put some flesh on fine bones.

Google's new Pixel 3 camera has some stunning 'computational photography' development tricks explained in the tech blog here.

Really clever stuff, using the natural movement of the photographer during multi frame bursts, to mimic pixel shifting, align those in software and get a high res image out of it. (without de-mosaicing, in most cases).

At the moment they are using it as a zoom on the pixel 3's single cameras, but obviously it's a short software step to high resolution full frame pixel shift images.

Apple/ iPhone Xs uses some tricks of its own, but nothing like this level, afaik.

(and... Pixel 3 also gives 'computational RAW' feature - raw DNG images with multi frame combined, and... free unlimited storage on Google Drive/Cloud for all your Pixel images, including those RAW's. It seems that this is the phone for photographers)

[narikin]

Umm, the Pixel was 2 generations ago.
The Pixel 3 is what we're talking about here...

[NancyP]

I suspect that he meant the Pixel 3.

I too am disturbed by the continual invasion into our privacy. There are no guarantees. Products will be priced differently for different consumers, with low prices going to the debt-less well off and higher prices going to those with significant credit card debt and modest jobs. The advertising has been mentioned above. China uses its combined social media / payment app to monitor purchases, determine if you have patriotic friends or if you hang around specific "trouble makers", if you drink too much, if you search for unusual topics, and can assign a score that determines if you are allowed to rent a desirable apartment, work at a good job, travel, live in a large city, etc. The infrastructure is being rolled out currently in a few large cities and no doubt will be made mandatory nationwide, or at least in small towns to large cities. You know that credit scores are (illegally, I think) used in the USA in screening out job applicants.

Technology is a dictator's wet dream.

[narikin]

Be great to keep this on topic. It's not about cellphone privacy, or Google vs Apple, or spyware in China...  it's about recent 'computational photography' developments, which, as the linked article reveal, have been pushed to a whole new level recently.  It is truly mind boggling what they are able to do with the small cellphone sensors now.

In fact it's because of these small sensors, and their ability to scan/sweep 15+ times in the time it takes an MF sensor to do one capture, that these algorithms can be deployed. There is no way current MF tech can manage to refresh so fast.  Someone here posted about Phase developing 'Frame Averaging' in the coming back. Which is basically where cellphones were 2 or 3 generations ago!

Next step will be to see this tech move across to APS-C then Full Frame, but that's at least 1 or 2 generations away, as it needs much faster sensor sweeps than currently possible.

[eronald]

I don't like staying on topic, so I would remark that a lot of the "false lighting" features or multiple image compositing techniques rely on image segmentation and recognition, ie basically one is inverse-rendering the image from 3D into planes and objects and re-rendering it as it would be in a certain light with a certain lens and F stop. The problem is that this is not photography, it is artificial.

A Selfista could ask the phone to insert her into any scene and it could probably do it.

It's not computational photography, it's VR.

Edmund

[Bart_van_der_Wolf]

I don't like staying on topic, so I would remark that a lot of the "false lighting" features or multiple image compositing techniques rely on image segmentation and recognition, ie basically one is inverse-rendering the image from 3D into planes and objects and re-rendering it as it would be in a certain light with a certain lens and F stop. The problem is that this is not photography, it is artificial.

Hi Edmund,

Not really, it's more like how human vision manages to cope with high dynamic range scenes (adjusting brightness based on a 1-degree average as we scan the scene through pupil contraction/dilation). Not artificial at all, unless poorly executed (by the camera's postprocessing). Google is getting pretty close to how it should be done. The images are often positively amazing.

Cheers,
Bart

[BJL]

The problem is that this is not photography, it is artificial.

Someone has to say it: all photography is artificial—it is just a matter of how much and what sort of artifice one is comfortable with. My favorite example is the much-beloved "crystal ball Christmas decorations" OOF effect ("bokeh!") sometimes produced when using an aperture far larger than the opening of the human iris: we only ever see those effects in photographs, not when actually viewing a scene. (I sometimes call it "upscale lomography".) Of course it is fine to accept such imagery as artistically legitimate, but anyone who does is in no place to declare something a problem simply because it is artificial.

(And of course, "art" and "artificial" have the same core meaning.)

[BJL]

... it's more like how human vision manages to cope with high dynamic range scenes (adjusting brightness based on a 1-degree average as we scan the scene through pupil contraction/dilation). Not artificial at all, unless poorly executed (by the camera's postprocessing). Google is getting pretty close to how it should be done. The images are often positively amazing.

Good point: though it is sometimes a valid criticism that EVFs reduce the contrast of a scene, as does the automatic in-camera HDR processing in the 2018 model iPhones, that is a lot like how our visual system works when we view a scene of high subject brightness range.

("By Google" removed from the subject line, since I am adding in some of the stuff that Apple has done recently.)

[eronald]

Someone has to say it: all photography is artificial—it is just a matter of how much and what sort of artifice one is comfortable with. My favorite example is the much-beloved "crystal ball Christmas decorations" OOF effect ("bokeh!") sometimes produced when using an aperture far larger than the opening of the human iris: we only ever see those effects in photographs, not when actually viewing a scene. (I sometimes call it "upscale lomography".) Of course it is fine to accept such imagery as artistically legitimate, but anyone who does is in no place to declare something a problem simply because it is artificial.

(And of course, "art" and "artificial" have the same core meaning.)

yes well, i just use an iphone these days anyway. but i still think pixel 3 feels unfinished

[Wayne Fox]

Some nice stuff going on with cell phones.

Just wondering why this is in this forum however.

[narikin]

Some nice stuff going on with cell phones.

Just wondering why this is in this forum however.

Because... Phase just announced 'Frame Averaging' for MF backs, so I thought it worth knowing where the cutting edge is in that regard, for MF users.

Sigh, now you've gotten the topic moved.
MF Forum: 170,000+ posts. iPhone forum: 423 in total.
Well done.

[BJL]

Sigh, now you've gotten the topic moved.
MF Forum: 170,000+ posts. iPhone forum: 423 in total.
Well done.

No problem: soon most posts will be divided between "Mirrorless Cameras" and the forthcoming "Computational Photography", with the rest on a level with "The Wet Darkroom" 

[Alan Smallbone]

Dpreview has posted some early versions of the raw files done with the Pixel3

https://www.dpreview.com/samples/5091061161/google-pixel-3-pixel-3-xl-sample-gallery

Alan

[Jeremy Roussak]

Sigh, now you've gotten the topic moved.
MF Forum: 170,000+ posts. iPhone forum: 423 in total.
Well done.

It's been moved here because this is where it belongs. I don't see the relevance of the post counts, but now there are several more posts on this forum than there were before, and it's thanks to you!

Jeremy

[narikin]

It's been moved here because this is where it belongs. I don't see the relevance of the post counts, but now there are several more posts on this forum than there were before, and it's thanks to you!

Jeremy

Respectfully disagree. It was just posted this week, in MF, about Phase One introducing Frame averaging, to make a composite RAW. This is that, on steroids, again making a composite RAW, so... it's relevant to people who might use and engage with that technique. PLUS many people use their phones as viewfinders/framing guides in MF photography (not Full Frame or Mirrorless) so knowing the best Phone for MF photo users, with composite RAW inside, is pretty relevant.

But... whatever, in the grand scheme of things, it doesn't matter.  Thanks.

[BJL]

Respectfully disagree. It was just posted this week, in MF, about Phase One introducing Frame averaging, to make a composite RAW. This is that, on steroids, again making a composite RAW, so... it's relevant to people who might use and engage with that technique. PLUS many people use their phones as viewfinders/framing guides in MF photography (not Full Frame or Mirrorless) so knowing the best Phone for MF photo users, with composite RAW inside, is pretty relevant.

But... whatever, in the grand scheme of things, it doesn't matter.  Thanks.

Come on, read the title you gave to the thread, and your OP itself: it is about what Google is doing in Android phones, with no mention of how that relates to MF gear. Maybe you had MF on your mind, but nothing in the post made that connection, and nor did any of the responses. If there is any indirect relevance to other type of cameras, it might belong under a more general heading like "Cameras, Lenses and Shooting Gear". Or "Digital Image Processing"? Or "Mirrorless Cameras", since Android phones are mirrorless?!

And I would not worry too much about traffic levels by forum: posts in all forums show up under Recent Unread Topics, and the most recent post in each forum shows up to the right of the forum name in the main forum list, so I think plenty of people will see the thread title even if they do not usually visit the camera-phone forum.