Pages: [1]   Go Down

Author Topic: file size with newer high megapixel cameras  (Read 1198 times)

Eric Brody

  • Full Member
  • ***
  • Offline Offline
  • Posts: 235
    • http://www.ericbrodyphoto.com
file size with newer high megapixel cameras
« on: October 01, 2017, 05:17:30 PM »

I recently rented a Fuji GFX, 50MP, and will probably try out a D850, 45MP, when it's more available. I also rented a Sony A7RII, 42MP, about a year ago. Each time I take one of their RAW files, process it in Lightroom, or Iridient, take it into Photoshop and do some work in layers, I often end up with a file too large for TIFF, e.g. >4GB. While I can just use the psb file to save it, Lightroom does not see these files. It seems to me that Adobe should be able to make this possible in some way. I recall hearing of a workaround but cannot recall it. One of the strengths of Lightroom is the catalog. If this becomes useless for these large files, what are we to do.

It's likely into the future there will be even higher MP cameras e.g. Sony A7RIII? or whatever it will be called. Are those fortunate enough to use these machines doomed to inconvenient kludges just to process an image, be it simple or complex?

Thanks for any help or further speculation.

Eric
Logged

BobShaw

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1005
    • Aspiration Images
Re: file size with newer high megapixel cameras
« Reply #1 on: October 01, 2017, 05:31:17 PM »

Without knowing exactly what you are doing, it sounds like you are saving layered TIFFs.
Don't do that.
TIF and JPG are output formats. Save them flat with no layers or alpha channels etc.
If you need to save the layers then save as a PSD.
Logged
Website - http://AspirationImages.com
Blog - http://AspirationImages.com/blog
Landscape, Portrait, Product Photography - Sydney, Australia

Eric Brody

  • Full Member
  • ***
  • Offline Offline
  • Posts: 235
    • http://www.ericbrodyphoto.com
Re: file size with newer high megapixel cameras
« Reply #2 on: October 01, 2017, 07:25:09 PM »

Thanks BobShaw for your reply. Maybe I'm misunderstanding you or you're misunderstanding me? :-)  I am currently using a Fuji X T-2, which has a 24 megapixel sensor. I use lossless compression and have RAF files in Lightroom of about 24MB. After Lightroom processing, these files are brought into Photoshop and are about 137MB there. Between all my manipulation, often including multiple layers with masks, smart filters, and such things as Silver Efex and Color Efex, I end up with large files even with my "small" camera.

I do choose to save all the layers but as psd's are limited to 2GB, I end up needing to use tiffs, which can go up to 4GB.

I do save my master files with all the layers as tiffs. I flatten a copy of my master file as a print file, specific to the size of the print. I may have one large master file and several smaller flattened print files each for a different sized print. By saving my master files, I am able to go back to them, sometimes years later, and with better skills (?) and improved software, I am able to improve the image. Do other folks flatten their master files before saving them? That would seem to be sacrificing flexibility to save storage space which currently is relatively inexpensive.

I hope I'm making myself clear with this.

Eric

Logged

Doug Gray

  • Sr. Member
  • ****
  • Online Online
  • Posts: 742
Re: file size with newer high megapixel cameras
« Reply #3 on: October 01, 2017, 08:00:13 PM »

Thanks BobShaw for your reply. Maybe I'm misunderstanding you or you're misunderstanding me? :-)  I am currently using a Fuji X T-2, which has a 24 megapixel sensor. I use lossless compression and have RAF files in Lightroom of about 24MB. After Lightroom processing, these files are brought into Photoshop and are about 137MB there. Between all my manipulation, often including multiple layers with masks, smart filters, and such things as Silver Efex and Color Efex, I end up with large files even with my "small" camera.

I do choose to save all the layers but as psd's are limited to 2GB, I end up needing to use tiffs, which can go up to 4GB.

I do save my master files with all the layers as tiffs. I flatten a copy of my master file as a print file, specific to the size of the print. I may have one large master file and several smaller flattened print files each for a different sized print. By saving my master files, I am able to go back to them, sometimes years later, and with better skills (?) and improved software, I am able to improve the image. Do other folks flatten their master files before saving them? That would seem to be sacrificing flexibility to save storage space which currently is relatively inexpensive.

I hope I'm making myself clear with this.

Eric

Make sure you save your tif files using zip compression. It's also lossless and LZH is broken (much bigger files) for 16 bit tifs.
Logged

Eric Brody

  • Full Member
  • ***
  • Offline Offline
  • Posts: 235
    • http://www.ericbrodyphoto.com
Re: file size with newer high megapixel cameras
« Reply #4 on: October 01, 2017, 09:25:39 PM »

Thanks Doug, I cannot recall why or when, but I was "told" not to use compression on my "precious" master files. I just played with zip compression a bit with some huge, almost 5GB, psb files from my time with the Fuji GFX and they saved nicely to tiffs with a significant decrease in file size. They do take longer to save than uncompressed files but I guess I can just wait to save with zip until I'm done with the file. 
Logged

Schewe

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6062
    • http:www.schewephoto.com
Re: file size with newer high megapixel cameras
« Reply #5 on: October 02, 2017, 12:57:44 AM »

Do other folks flatten their master files before saving them? That would seem to be sacrificing flexibility to save storage space which currently is relatively inexpensive.

One should always be careful that when asking for advice, you consider the advice you get carefully...what you are doing–saving the master file with all layers as TIFF files (far better than PSD since TIFF files are publicly documented while PSD files are proprietary and TIFF files can contain everything a PSD file can except for Duotone) and if need be, PSB files is fine and it's what i do.

I often need to use PSB files because of panos using high rez captures (from IQ 180 files and Nikon D810). These files are way to large for saving as TIFF so I save as PSB with Ma as the last letters of the file name. The Xxxxxx-Ma.psb means Master file. For working files, I use the letters Xxxxxx-Wo.tif for the Working file. The Xxxxxx-Wo.tif is imported into LR.

When I need to edit the layered files which are PSB, I go to the Library panel and use the Show File in finder option in the Context menu while the Xxxxxx-Wo.tif is selected that will take me to the folder where the tif file is located. Since I save the The Xxxxxx-Ma.psb in the same folder, the Master file with layers is right next to the Working file when the folder is sorted by file name.

Then I'll open the PSB file in Photoshop, do the work I need to do and save it as Xxxxxx-Ma.psb then do a Save As and add the Wo.tif to the file and save over the Xxxxxx-Wo.tif file in the folder. Back in Lightroom I'll make sure I select the image and use the command to Read metadata from file to update the LR version to have the newly edited file back in LR.

Since I use LR for printing and exporting for other uses, that dual file practice works ok. I do wish LR supported PSB files :~(

Oh, and I view TIFF files as the optimal file format to use (when possible) because it's documented and is therefore a better archival digital object file format than PSB...TIFF is not merely an output file format is like sRGB.
Logged

kers

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1625
    • Pieter Kers
Re: file size with newer high megapixel cameras
« Reply #6 on: October 02, 2017, 06:57:22 AM »

I use PSB files too a lot, for i work with very large images...

But my problem with PSB is  the thumbnails;  some program i use cannot show them ( media pro for instance)
Very strange that even Lightroom does not deal with them.
Since my workflow has only produced flat tifs during the years, i do not like to use them as a master format, also it has a 4Gb limitation so would only work for some images.

All these size limitation are a bit antique; with the modern 100MP camera's and more to come you cross the boundaries before you know it. Thumbnails should also be about 1000px at least.


« Last Edit: October 02, 2017, 07:06:24 AM by kers »
Logged
Pieter Kers
www.beeld.nu

nirpat89

  • Full Member
  • ***
  • Offline Offline
  • Posts: 143
    • Photography by Niranjan Patel
Re: file size with newer high megapixel cameras
« Reply #7 on: October 02, 2017, 08:03:56 AM »

Are those fortunate enough to use these machines doomed to inconvenient kludges just to process an image, be it simple or complex?

Thanks for any help or further speculation.

Eric

My speculation is Yes.  But no more or less than what you are dealing with now.  Megapixels will scale up, so would the computing power, memory, storage capacities etc.  Everything will cost less per unit.  But you will require more units.  So your level of frustration will remain the same... :)

I want to buy a higher grade camera but I can not do so until I can justify nearly as much in cost as the camera, if not more, it would require to upgrade my current computing system to handle it. 
Logged

Eric Brody

  • Full Member
  • ***
  • Offline Offline
  • Posts: 235
    • http://www.ericbrodyphoto.com
Re: file size with newer high megapixel cameras
« Reply #8 on: October 02, 2017, 11:41:56 AM »

I appreciate all the great responses.

I still would like to know if folks are using ZIP compression on their tiff's, the ones small enough not to need conversion to psb.

Thanks.

Eric
Logged

BartvanderWolf

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 6674
Re: file size with newer high megapixel cameras
« Reply #9 on: October 02, 2017, 11:54:27 AM »

I appreciate all the great responses.

I still would like to know if folks are using ZIP compression on their tiff's, the ones small enough not to need conversion to psb.

Hi Eric,

Yes, ZIP on TIFFs, PSB if too large.

The only use I have for PSDs is for displacement maps (I don't do Duo-tones).

Cheers,
Bart
Logged
== If you do what you did, you'll get what you got. ==

Eric Brody

  • Full Member
  • ***
  • Offline Offline
  • Posts: 235
    • http://www.ericbrodyphoto.com
Re: file size with newer high megapixel cameras
« Reply #10 on: October 02, 2017, 04:45:31 PM »

I'm beginning to get this. One more picky specific question...

In the Photoshop save dialog box there are choices for "image compression," they are none/LZW/ZIP.

There are also choices for "layer compression," they are RLE/ZIP/discard layers (I'd never discard my layers)

Do people use both image compression and layer compression, or just one, and if so, why?

Thanks, my hard drives thank you.

Eric
Logged

nirpat89

  • Full Member
  • ***
  • Offline Offline
  • Posts: 143
    • Photography by Niranjan Patel
Re: file size with newer high megapixel cameras
« Reply #11 on: October 02, 2017, 06:47:26 PM »

I'm beginning to get this. One more picky specific question...

In the Photoshop save dialog box there are choices for "image compression," they are none/LZW/ZIP.

There are also choices for "layer compression," they are RLE/ZIP/discard layers (I'd never discard my layers)

Do people use both image compression and layer compression, or just one, and if so, why?

Thanks, my hard drives thank you.

Eric

Whether or not you do image compression, you still have to do some kind of layer compression - unless you are not making layers or saving the layers.  Then the choice is either RLE which saves a bigger file but does it faster or the ZIP compression.  I guess the question which do you have less of - storage space or time.  I prefer to save time so I use the RLE.  But then I do not have a 50MP camera. 

:Niranjan.
Logged

BobShaw

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1005
    • Aspiration Images
Re: file size with newer high megapixel cameras
« Reply #12 on: October 02, 2017, 07:41:28 PM »

I do save my master files with all the layers as tiffs. I flatten a copy of my master file as a print file, specific to the size of the print. I may have one large master file and several smaller flattened print files each for a different sized print. By saving my master files, I am able to go back to them, sometimes years later, and with better skills (?) and improved software, I am able to improve the image. Do other folks flatten their master files before saving them? That would seem to be sacrificing flexibility to save storage space which currently is relatively inexpensive.
Do you really need to save them layered? I would only save a shot that was a real hero as a layered file, or one that I thought I would come back to.
As for saving multiple copies for different print sizes, I never do that. One file at maximum size and use a print programme such as Mirage Print to print any size.
Storage may be comparatively cheap, but it is still the biggest challenge for photographers going forward. How many backups do you have?
Logged
Website - http://AspirationImages.com
Blog - http://AspirationImages.com/blog
Landscape, Portrait, Product Photography - Sydney, Australia

Eric Brody

  • Full Member
  • ***
  • Offline Offline
  • Posts: 235
    • http://www.ericbrodyphoto.com
Re: file size with newer high megapixel cameras
« Reply #13 on: October 03, 2017, 12:18:49 AM »

BobShaw, I do back up my photo files to five separate drives one of which is off site. I use Carbon Copy Cloner, an excellent program that will make a bootable clone if used to back up a boot drive. I have multiple backups of my boot drive too. We all know there are two kinds of people, those who back up and those who will eventually lose data.

The point of my original question was to find out how other folks process the huge files generated by medium format digital cameras, e.g. the 50MP Fuji GFX that I rented, as well as the newer full frame cameras such as the Sony A7RII and the new Nikon D850. I value my time as well as my hard drive space and agree that it is a trade off between the two. The issue with psb files is that Lightroom will not catalog them, thus for those, such as the recent panoramas I did even with my "tiny" 24MP Fuji, it would be nice to compress them without data loss into a format Lightroom will catalog and "see."

And yes, I really do choose to save my files layered.

Another thanks to all who contributed, I've learned a lot and would love to hear more if anyone has other thoughts. I highly value the members of this forum who are well informed educated people willing to share their expertise.
Logged

engardeknave

  • Newbie
  • *
  • Offline Offline
  • Posts: 7
    • Jason Jared Photography
Re: file size with newer high megapixel cameras
« Reply #14 on: October 10, 2017, 12:08:48 PM »

Working with IQ180 files, I find that the PSD already crosses the 2GB threshold with just four layers that are variants of the original. I have not yet bothered trying to solve this issue. So far I've just been merging a few layers I don't really need until it's small enough. I suppose I'll have to work with PSBs soon. However, this issue did wake me up to the fact that I'm saving redundant data (the original raw file as the base layer) in nearly every PSD. Getting rid of that can help.
Logged
FS: Hasselblad/Phase One Kit - H4X/HC 80mm 2.8/IQ180
http://forum.luminous-landscape.com/index.php?topic=120960.0

Martin Kristiansen

  • Full Member
  • ***
  • Offline Offline
  • Posts: 230
    • http://www.mkdp.co.za
Re: file size with newer high megapixel cameras
« Reply #15 on: October 10, 2017, 12:28:34 PM »

Geoff Schewe gave the best reply and only real working solution to this problem.
Logged

rdonson

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2566
Re: file size with newer high megapixel cameras
« Reply #16 on: October 10, 2017, 06:18:36 PM »

2 thumbs up!!!
Logged
Regards,
Ron

BobShaw

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1005
    • Aspiration Images
Re: file size with newer high megapixel cameras
« Reply #17 on: October 10, 2017, 10:31:34 PM »

Geoff Schewe gave the best reply and only real working solution to this problem.
I am sure Geoff's solution works great.
If we were to line up 10 photographers and ask the solution you would have 11 solutions and they probably all work.
The OP needs to work out his own solution.
As regards to TIF vs PSD being documented/undocumented and proprietary/non-proprietary I don't think anyone needs to worry if you are going to use Photoshop to open a PSD.

I have a 50MP Canon and a 39MP Hassy. My solution is that most photos never go past raw. They are adjusted in the Aperture (read Lightroom if you will) database. There are about 100,000 there. They are just exported (but not printed) as required using a mass of presets (web, Facebook, Instagram etc).
The good ones are worked in Photoshop and when finished are flattened as a 16bit ProPhoto TIF for printing. If I am trying different things or still working they stay as a layered PSD. The database, print files and a few PSD are the only ones I keep on backed up storage other than the raw.
Logged
Website - http://AspirationImages.com
Blog - http://AspirationImages.com/blog
Landscape, Portrait, Product Photography - Sydney, Australia

Wayne Fox

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 3858
    • waynefox.com
Pages: [1]   Go Up