Luminous Landscape Forum
Raw & Post Processing, Printing => Digital Image Processing => Topic started by: Eric Brody on October 20, 2019, 03:48:31 pm
-
I have a Sony A7RIV and while I have a decent computer (Mac Pro 2013 {trashcan} with 64GB of RAM,) the file sizes of my uncompressed, many layered master files are daunting and will not save within the 4GB limit for TIFF's even when using ZIP in layer compression in the TIFF options box in Photoshop. I have no problem using large document format, but Lightroom does not see such files.
I shot some compressed RAW files and looked at them with Raw Digger and they are 14 bit, even compressed, and are a mere 59MB vs 117MB for the uncompressed 14 bit files. Has anyone looked at data (quality) loss with compressed raw on the R4? I want to have maximum quality files. I assume with Sony's "lossy" compression, something has to give, but wonder what it is and whether I'll ever notice it.
Thanks for any responses.
Eric Brody
-
Actually, please let me make a minor correction, when using zip in both image and layer compression, I'm able to get the files to be under 4GB, but it's SLOW. Zip is supposed to be lossless, is it?
-
Actually, please let me make a minor correction, when using zip in both image and layer compression, I'm able to get the files to be under 4GB, but it's SLOW. Zip is supposed to be lossless, is it?
Both ZIP and LZW compression are lossless. If compressed file still over 4 GB or too slow, you can save PSB as working file. And flatted all layers then save TIFF for print or other usage. BTW, a 60 MP un-compressed 16 bits TIFF is just under 350 MB. Maybe you should delete some unnecessary layers.
-
Both ZIP and LZW compression are lossless. If compressed file still over 4 GB or too slow, you can save PSB as working file. And flatted all layers then save TIFF for print or other usage. BTW, a 60 MP un-compressed 16 bits TIFF is just under 350 MB. Maybe you should delete some unnecessary layers.
If you are talking about LZW compression on 16 bit tiff files: Do not do it.
It takes ages and files do not get any smaller. ( 8 bits is OK-, layered tifs i do not know)
-
I shot some compressed RAW files and looked at them with Raw Digger and they are 14 bit, even compressed, and are a mere 59MB vs 117MB for the uncompressed 14 bit files. Has anyone looked at data (quality) loss with compressed raw on the R4? I want to have maximum quality files. I assume with Sony's "lossy" compression, something has to give, but wonder what it is and whether I'll ever notice it.
Thanks for any responses.
Eric Brody
The compressed RAW files of the Sony a7Riv are either 13 bit (single shooting) or 12 bit (continuous shooting) according to Jim Kasson in this article (https://blog.kasson.com/a7riv/sony-a7riv-edr-in-continuous-compressed-mode/).
This can lead to two potential issues compared to the uncompressed RAW files:
- The image will be noisier if the exposure has to be lifted in post, say more than 3 stops of exposure. This is not a problem if exposure is mostly done right.
- Hard edges with black and white transitions might suffer from visible compression artifacts. This can be a deal breaker for some astrophotography pictures or night scenes.
Due to point 2 above, I switch to uncompressed files for night scenes and use compressed files the rest of the time (90%).
In an ideal world, lossless file compression would be handled transparently by the operating system of the computer to make the use of uncompressed files in camera a non-issue. Sadly, most mainstream filesystems perform poorly (NTFS) or have no such compression exposed to users (APFS). Only servers fare better (with ZFS).
Also note that RAW file compression has no effect on the size of demosaiced TIFFs so this would not solve your other problem.
Cheers,
Fabien
-
Photoshop is slow at saving large ZIP/Deflate compressed TIFF files, but in return it compresses them down to very small sizes. Other software, like ACDSee, compresses much less strong in much less time. Unfortunately Photoshop does not offer any option to change compression strength, this would be very useful.
Overall it's a shame that image compression is still done on just a single CPU core (single-threaded), despite the fact that data compression in general is very efficient at being multi-threaded. Winrar manages to use at least some multi-threading on a single TIFF file using ZIP/Deflate and Winzip even uses all CPU cores when "legacy" ZIP is used, so it *is* possible with ZIP/Deflate.
-
I did some performance tests on an uncompressed 3.75 gb TIF file, 8 bit, single layer.
Photoshop PSB (compressed): 0:15 min, 3.03 gb
Photoshop TIF (ZIP): 40 min (!), 1.16 gb
Gimp TIF (ZIP): 4 min, 1.2 gb
Irfanview TIF (ZIP): 2 min, 2.67 gb
All of these only use a single thread/core, but Photoshop's ZIP compression time is seriously bugged. Especially compared to Gimp, which is very close in file size, but ten times faster.
-
There is no free lunch with TIFF compression; smaller size (yes lossless), slower save and opening:
http://digitaldog.net/files/TIFFvsPSD.pdf
-
Yes, but Photoshop taking 10x as long as Gimp for virtually the same file-size is rather a bug than a limitation.
-
Yes, but Photoshop taking 10x as long as Gimp for virtually the same file-size is rather a bug than a limitation.
It's not a bug: it works albeit slower. There can be reasons for the speed difference but calling it a bug is disingenuous and wrong. If you personally have an issue with the speed, use some other products or don't save with compression because again, there is no free lunch doing so.
Drive space is also dirt cheap. Your time?
-
The original poster rightfully states that you can run into the 4 gb limit when saving uncompressed TIFF files. Furthermore the average NAS is connected via Gigabit Ethernet that maxes out at about 110 mb/s. So it makes a difference whether you are loading 3.75 gb or just 1.2 gb over the line, which then bottlenecks load times more than CPU load for decompression of such large files. Things are different on a local SSD and with smaller files, especially with the single-threaded limitation of (un)modern image software.
Some software creates TIFF files that cannot be read by other software. For example, FastStone Image Viewer cannot read very large uncompressed TIFF files created by Photoshop, but it can read LZW and ZIP compressed ones. On the other hand it cannot read uncompressed *and* ZIP compressed TIFF files from ACDSee and Gigapixel, but LZW does work again. So the choice of compression can depend on the combination of software being used by a user.
Coming back to Photoshop's very long save times: There is no reasonable explanation other than bad programming. All other software creates ZIP/Deflate compressed TIFF files faster and GIMP does it in 1/10th the time even while maintaining the same very small file size. Photoshop's LZW also is very considerably faster.
Last but not least: "Use some other product" is of no use to the original poster. He is working on layered files in Photoshop and then wants to get that data out of PS to an"other product". Since hardly any other product is able to read PSB files the only choice is TIFF files and then we are back to the original problem of hitting the 4 gb limit with uncompressed files.
-
My scans of medium format film (6x7cm) run around 225mb. I have no problem of course with these with Lightroom. How do people deal with scans of 8x10" film?
-
This is very simple. IF the file exceeds the limitation of TIFF, no options, you save as PSB. Doesn't matter what other products can or can't read it; that's the solution other than reduce layers and file size and save as PSD or better, TIFF. Such layers are proprietary anyway! That means Photoshop (If PS layers) and you save as PSB if the size exceeds that which can be saved as a TIFF.
This is also very simple: If a file can't read or write a TIFF, that's a bug, not how long another product that can read or write a TIFF takes to do so. IF FastStone can't read a TIFF, it's their fault. This is a software bug. As simple as that.
-
This is very simple. IF the file exceeds the limitation of TIFF, no options, you save as PSB. Doesn't matter what other products can or can't read it
Seriously? When you know that a compressed TIFF would work and can be read by another product while PSB cannot? How can the solution "don't use it" be the right choice when another solution allows to use it?
I'd rather say: If the file exceeds the limitation of TIFF and you need to export it to another product then use compressed TIFF. Then file a bug-report to Adobe about Photoshop taking ages to create said file, bordering on becoming a useless function. Unfortunately the "bug-report" part is not that easy with Adobe.
The pragmatic way of dealing with the reality of software limitations is to get around them. If a program you want to use cannot read a certain TIFF format then the reasonable choice is to use another format until the program is fixed (if at all).
Photoshop itself is not able to read various compressed TIFF formats created by other software, even though other programs open the very same files without issues. So in this case your best bet is to either use uncompressed TIFF or PNG (for single layer files). It's all a question of pragmatic choices.
-
My Irfanview software (free) allows storage as a TIFF. There are options with this storage. You can select one of the following for compression: None, LZW, Packbits, JPEG, ZIP and for BW Huffman RLE, CCITT Fax3 , CCITT FAx 4. Also you can check: save all pages from original image and/or save palette for grayscale images (default on).
1. What is the point of compressing a TIFF image? I thought the whole point was not to so you can avoid any loses.
2. What do the various compressions mean and why use one over the other? I always store TIFFs uncompressed. I have no problem with space. However, would there be another reason why these other compression selection might be used?
-
Seriously?
“If my answers frighten you then you should cease asking scary questions.”
― Quentin Tarantino, Pulp Fiction
You seem to have little experience with software development or understanding about bugs. You incorrectly state Photoshop has bugs due to the speed in which it correctly reads and writes TIFFs with or without compression but don't seem to be able to connect the dots that the various software products you state don't write TIFF correctly don't appear to be buggy.
I'd rather say: If the file exceeds the limitation of TIFF and you need to export it to another product then use compressed TIFF. Then file a bug-report to Adobe about Photoshop taking ages to create said file, bordering on becoming a useless function. Unfortunately the "bug-report" part is not that easy with Adobe.
Photoshop correctly writes TIFFs with or without compression. Photoshop and no other product can write a TIFF that exceeds it's own limitations and therefore, there's a simple answer for that; PSB.
Photoshop itself is not able to read various compressed TIFF formats created by other software, even though other programs open the very same files without issues.
No, it reads TIFF's correctly and doesn't read incorrectly written TIFFs if that's what you're implying. If so say so; you've got software that writes TIFFs with bugs and Photoshop apparently can't read them. Don't use software that has bugs writing TIFFs. Simple.
So in this case your best bet is to either use uncompressed TIFF or PNG
But you stated the OP's issue (your exact text):rightfully states that you can run into the 4 gb limit when saving uncompressed TIFF files.
Doesn't matter then, he can't use TIFF or PNG. HE HAS TO USE PSB. And he should.
It's all a question of pragmatic choices.
No, it clearly isn't. Because the choice of what format to save when the 4 GIG limitation is reached isn't pragmatic; it's PSB.
-
1. What is the point of compressing a TIFF image?
The URL article answers that, smaller files, but slower saving and opening, for older products (or products with Bugs like those used by Timur), potential problems. Otherwise, stick with TIFF uncompressed.
There's also what's been written here on LuLa forums in the past:
https://forum.luminous-landscape.com/index.php?topic=18965.msg134830#msg134830 (https://forum.luminous-landscape.com/index.php?topic=18965.msg134830#msg134830)
Jeff Schewe wrote:
Wrong...PSD is now a bastardized file format that is NOT a good idea to use. Even the Photoshop engineers will tell you that PSD is no longer the Photoshop "native" file format. It has no advantages and many disadvantages over TIFF.
TIFF is publicly documented, PSD is not. That makes TIFF a preferred file format for the long term conservation of digital files.
TIFF uses ZIP compression for max compression, PSD uses RLE which if you save without the Max compatibility will be a bit smaller, but at the risk of not being able to be used by apps, like Lightroom.
TIFF can save EVERYTHING a PSD can save including layers, paths, channels, transparency, annotations and can go up to 4 GIGS in file size. TIFF can save all the color spaces PSD can. The ONLY thing I can think of that PSD can save that currently TIFF can't save is if you Save out of Camera Raw a cropped PSD, you can uncrop the PSD in Photoshop CS, CS2 or 3. That's one tiny obscure thing that PSD can do that TIFF currently doesn't. How many people even knew that let alone use it?
PSD used to be the preferred file format back before Adobe bastardized it for the Creative Suite. The moment that happened, PSD ceased to be a Photoshop "native" file format. PSB is the new Photoshop "native" file format for images beyond 30,000 pixels. And , at the moment, only Photoshop can open a PSB.
Getting back to the fist point, Adobe can do anything including stopping support for PSD because it's a proprietary file format. TIFF is public, even if it's owned by Adobe (by virtue of the Aldus purchase). Even if Adobe went belly up tomorrow, TIFF would continue.
And, let me be blunt, anybody who thinks PSD is "better" than TIFF is ignorant of the facts. If Adobe would let them, the Photoshop engineers would tell you to quit using PSD. Lightroom for the first beta did NOT support PSD and Hamburg fought tooth and nail to prevent having to accept PSD. He blinked, but you still can't import a PSD without Max compat enabled-which basically makes it a TIFF with a PSD extension.
Look, I'll make it REAL simple...
TIFF = Good
PSD = Bad
Ok?
-
The original poster stated:
the file sizes of my uncompressed, many layered master files are daunting and will not save within the 4GB limit for TIFF's
So this is why he needs to use compressed TIFF files, in order to get smaller than 4 gb files. Compressed TIFF files can be dramatically smaller and thus not break the 4 gb limit. Of course PSB files can be used, but these are limited to only be usable in Photoshop. The original poster himself wrote about this point:
I have no problem using large document format, but Lightroom does not see such files.
For Photoshop (exclusively) he could use PSB files ("large document format"), but he wants to open those files in other software (Lightroom) and PSB files mostly can only be read by Photoshop. At least Lightroom cannot read PSB files, so he chose TIFF.
Currently the only choices to do this all is to either wait for Photoshop to painfully slowly create ZIP/Deflate compressed TIFF files, or to flatten layers (decrease file-size) just for importing into Lightroom.
IrfanView saves ZIP compressed TIFF files considerably faster than Photoshop (1/20th the time) or Gimp (1/2 the time), but the resulting file-size is much larger due to lower Deflate compression ratios being used. And for the problem at hand third party software cannot be used. There are some programs that can read/convert PSB files, but that does not mean that they can successfully work with 4+ gb files.
PS: I do not see any benefit in discussing semantics instead of intention.
-
Good. Less things to worry about. I'll keep storing images as tiffs, uncompressed.
-
Good. Less things to worry about. I'll keep storing images as tiffs, uncompressed.
Indeed stick with uncompressed and uncomplicated TIFF.
-
Look, I'll make it REAL simple...
TIFF = Good
PSD = Bad
very interesting.
Now i have a layered tif that exceeds the 4Gb limit: is the 'solution' PSB?
Or are there any alternatives?
PS is .tif the same as .tiff ?
-
layered master files are daunting and will not save within the 4GB limit for TIFF's even when using ZIP in layer compression in the TIFF options box in Photoshop. I have no problem using large document format, but Lightroom does not see such files.
Eric Brody
Why save layered TIFFs? Save layered PSD and flattened TIFFs. Before you flatten them remove any extra channels and paths that you created. That is often a reason that files can't be read.
-
very interesting.
Now i have a layered tif that exceeds the 4Gb limit: is the 'solution' PSB?
Yes, the "solution" is PSB. It is a format that uses larger variables to store information, thus the limits are higher. Unfortunately hardly any software other than Photoshop can read this format.
Or are there any alternatives?
Yes, the alternatives are compressed TIFF files that decrease the size of the file down to less than 4 gb.
PS is .tif the same as .tiff ?
Yes, usually extensions are three characters long, but this is not mandatory anymore (it was in MS DOS days).
-
Now i have a layered tif that exceeds the 4Gb limit: is the 'solution' PSB?
All already explained.
PS is .tif the same as .tiff ?
PS is Photoshop, no it's not a TIFF. TIFF is TIFF. PSD and PSB are 'native' Photoshop files and as outlined, with rare exception, there's nothing a PSD provides a TIFF doesn't.
-
PS is Photoshop
I think he meant "post scriptum", as in "PS: Is .tif the same as .tiff". And then he meant file extensions, which is why the "." is in front of .tif and .tiff.
So he forgot the ":", but you overlooked the two "." in your answer. If you stack the latter on top of each other this makes you even.
-
I think he meant "post scriptum", as in "PS: Is .tif the same as .tiff". And then he meant file extensions, which is why the "." is in front of .tif and .tiff.
Again, this is a simple fact: In Photoshop, TIFF is TIFF, opening or saving:
-
Why save layered TIFFs?
Because it's a far more supported format than PSD and the why's are explained by Schewe.
Why save a layered PSD and a flattened TIFF? One is plenty.
-
PS is .tif the same as .tiff ?
In terms of the extension, Photoshop will provide one for you as you toggle the format options while allowing the name to be edited, it uses .tif but depending on the OS, doesn't matter whatsoever.
-
In terms of the extension, Photoshop will provide one for you as you toggle the format options while allowing the name to be edited, it uses .tif but depending on the OS, doesn't matter whatsoever.
thanks!
and now is understand the communication problem:
i used PS in the sense of Post Scriptum - it had nothing to do with Photoshop:)
-
I did some performance tests on an uncompressed 3.75 gb TIF file, 8 bit, single layer.
Photoshop PSB (compressed): 0:15 min, 3.03 gb
Photoshop TIF (ZIP): 40 min (!), 1.16 gb
Gimp TIF (ZIP): 4 min, 1.2 gb
I tested a 1.35 gb source file of the same image at lower resolution and both Photoshop and Gimp take 3 minutes to compress the file to TIF ZIP.
Then I duplicated the single layer to get two identical layers and saved as PSB and uncompressed TIFF. The PSB file came out at double the size (2.69 gb), quite expected. The uncompressed TIFF with RLE layer compression could not be created, though, with PS claiming to hit the 4 gb file size limit?! Using uncompressed TIFF with ZIP layer compression came out at 2.75 gb.
Next was ZIP compressed TIFF with RLE layer compression. This came out at 3.37 gb and only took 3:20 min even while a compressed image backup of my system was done in the background (SSD to NAS). Last was ZIP compressed TIFF with ZIP layer compression, resutling in 2.1 gb and tool 4:20 min (backup was mostly done by that time).
So what about the 40 (fourty) minutes ZIP TIFF save I reported earlier? That was at higher image dimensions (pixels). Turns out that Photoshop's ZIP compression performance drops *dramatically* when pixel count increases. And by that I do not mean that it takes longer to write more data, but data write speed drops significantly.
For example, my 19200 x 25200 px test image writes at 3.5 to 6 mb/s average. Changing dimensions to 25200 x 25200 px decreases write-rate to below 1.5 mb/s average. And it drops further in big steps the higher the resolution. The 40 min test file was 32000 x 42000 and write speed dropped below 0.5 mb/s average.
The same large resolution file exported from Gimp writes at about the same 3.5 - 6 mb/s average that the smaller file did, thus Gimp only takes longer for the larger file, because more data has to be written, not because it calculates slower.
So ZIP compressing layers to get around the 4 gb TIFF limit should work, but you have to watch your image resolution.
-
I tested a 1.35 gb source file of the same image at lower resolution and both Photoshop and Gimp take 3 minutes to compress the file to TIF ZIP.
Took a 9 layered document (image and text layers), size being 1.69 gb, saved as TIFF with Zip, WITH layers; took 110 seconds. No big deal not that I'd ever save a TIFF with ZIP because Photoshop is smart enough to advise not doing so:
-
I tried saving a 1.6gb tiff as a tiff ZIP and took around one minute with Irfanview.
-
Irfanview saves TIFF ZIP faster, because it compresses considerably less (matching ACDSee). Curiously it uses stronger compression for LZW (matching Photoshop).
-
Andrew, I see the same dialog but do not use "older" TIFF viewers and time is not a factor. I usually save frequently without compression with complex multilayered images until I get the dreaded dialog of over 4GB then usually use everything available, ZIP image compression AND ZIP layer compression. What is the problem with this? It seems to give me the smallest file size on disk. Please help me understand why this is a bad idea.
-
Andrew, I see the same dialog but do not use "older" TIFF viewers and time is not a factor. I usually save frequently without compression with complex multilayered images until I get the dreaded dialog of over 4GB then usually use everything available, ZIP image compression AND ZIP layer compression. What is the problem with this? It seems to give me the smallest file size on disk. Please help me understand why this is a bad idea.
There is no problem per se. There could be issues with compatible software as outlined but nothing really to worry about with BUG free modern software that correctly follows the TIFF published standards. If future archive of data is a concern, a non compressed TIFF is safer but then a compressed TIFF should be more accessible than a PSB which would be the other alternative. So again:
TIFF = Good
PSD = Bad
PSB = Bad if there's an alternative
-
PSB = Great if Adobe would finally support it in LR and lift the old file size restrictions for LR and ACR.
-
Why save a layered PSD and a flattened TIFF? One is plenty.
Because you can print a TIFF using Mirage Print or whatever and catalogue and keyword it in catalogue programmes. All of my print files are TIFF.
I only keep a layered PSD if I think it may need more work or can be used for something else. They become too huge.
-
Because you can print a TIFF using Mirage Print or whatever and catalogue and keyword it in catalogue programmes. All of my print files are TIFF.
I only keep a layered PSD if I think it may need more work or can be used for something else. They become too huge.
And you can keep the layered TIFFs. Doesn't Mirage support that? It should.
-
I only keep a layered PSD if I think it may need more work or can be used for something else. They become too huge.
Who cares when 5TB is 100$ ? i choose for a compressed files for that saves me a lot of time saving and opening... i don't care about the size.
My problem is that PSB files do not have a Thumbnail...
(while at te same time they show a preview in column-view on my mac ??)
PSB = Great if Adobe would finally support it in LR and lift the old file size restrictions for LR and ACR.
+1 if Adobe internally does not support then who does?
With 100MP camera's you need PSB.
-
There is a $10 PSD Quick Look Plugin that adds PSB thumbnails to Finder. The plugin homepage also mentions that Catalina brings its own thumbnail support for PSBs, but is slower than said plugin.
-
There is a $10 PSD Quick Look Plugin that adds PSB thumbnails to Finder. The plugin homepage also mentions that Catalina brings its own thumbnail support for PSBs, but is slower than said plugin.
No need to spend anything, certainly under Mojave, the PSB thumbnail shows up just fine in the Mac Finder: