LR CC - Catalog doesn't sort by Capture Time correctly

[EvanConway]

Hi,

I have been using Lightroom since the first version as it was my preferred software for a majority of my workflow for the better part of a decade (not sure how long its actually been at this point since the original LR) Whether it was importing, key wording, cataloging and basic batch editing I was always able to get most of my work organized into a convenient DAM system that also aloud me to prep photos for finalizing in photoshop in a much more efficient fashion. I loved LR and was its biggest advocate for the longest time, but now I have an issue I cannot explain and I was hoping that somebody here may be able to help me out with this.

Since a little bit of background on my DAM system seems relevant... I keep separate catalogs for specific trips,weeks or months where I plan to shoot a lot, then also for the current year (when this problem originally occurred it was 2016) and then I have my Archive catalog which at the end of each year I import that years catalog into it. The Archive cat has now got all my work from 2007 - 2017 which is something like 350,000 images and its because of the amount of photos in that catalog that I find it a necessity to make smaller working catalogs so that (i.e 2016 cat) I can actually work on the images without LR basically crashing. I have never really heard any confirmation what the absolute limit is for catalogs in LR but once it gets to around 150,000 images performance diminishes greatly and I am running a 6 core mac pro with SSD everything running Thunderbolt II and 32Gb RAM so its not the computer or its peripherals slowing things down.

Sooooo last October I had shot probably 1200 images one morning as the waves were incredible and a lot of those images made up sequences. I was testing the Canon 200-400mm f/4L through Canons professional service program so I was trying to really shoot as much as I could so I could compare the files through the ever changing lighting situations at dawn on the ocean. I imported these files later and and while I was working on them my sequences were out of order which was odd so I naturally went to sort them by Capture time and to no avail. It has basically been ever since then that my sort by Capture Time stopped working appropriately. Its weird because even though the files are not ordered  by the appropriate capture time the files metadata still reads their original capture date in the files as it should but it does not correlate with the sorting order. I have searched the internet to see if anyone else has had this issue but have been unsuccessful and I keep thinking its got to be something stupid/simple but I am all out of ideas so hopefully somebody has an idea that might be able to relieve me of this major annoyance. A lot of my work contains sequences so when your working on images and they are always out of order it really does hinder my performance when rating images and picking the best shot of the sequences. After this happened in my small catalog which I think was only a months worth of images, the problem started showing up in the rest of my catalogs without me ever merging the smaller catalogs into the archives. One thing to note is that I was also using a CPS loaned canon 5dsr at the time and am skeptical that perhaps some sort of setting in that camera itself is what triggered this chain reaction? Or maybe it is somehow the LRMobile app that is the culprit? I am really at a loss here and its pretty much pushing me towards having to migrate my entire archive of 350K to Capture One which I am not in the least bit excited about nor do I know if Capture One will handle a large catalog any better...

If anybody has had this mis-sorting issue in LR or has any ideas as to why this has happened to all of my catalogs please help!


In this screenshot you can see that the sequence is out of order

[MBehrens]

Evan,

What are the actual timestamps for the images?

Looking at the metadata in LR the time is displayed in HH:MM:SS format. I'm not sure if milliseconds are captured. But if you are shooting multiple frames a second, and milliseconds are not captured or considered in sorting, then you will have things out of sequence. When the sort criteria has the same value, the order will be random. One of the rules of database management.

I avoid this by using the same file naming convention for all images: YYYYMMDD_HHMMSS_CameraImageNumber. I always have the correct sequencing with sorts by File Name. Always.

BTW, nice wave shots!

Morey

[fdisilvestro]

I don't think this is an issue with LR. The catalog can store times up to a millisecond and most modern cameras record up to 1/100 second in the capture time field. Unfortunately, LR does not show these values in the user interface.

One possibility is that the precision in the time values was lost when converting to DNG. Does the same thing happen with your camera raw files? Have you tried importing to C1 (you can use a free trial).

The only way to tell what is going on with precision is to check directly the data in the catalog or DNGs

[FabienP]

I have seen other strange sorting issues in Lightroom, but these always happened with video files being copied to the wrong folder (when importing by date in dedicated subfolders) or appearing in the wrong sorting order compared to RAW files (if sorted by capture date).

However, this one eludes me since it only involves one class of objects, RAW files, which should all read the same date attribute. If the capture time of the DNGs is recopied as is in the catalogue and if the granularity of the date format is sufficient, there should be no reason for this behaviour. Can you reproduce the problem if you reimport the files in a newly created catalogue?

Cheers,

Fabien

[mdijb]

I had the same problem in past.  What was irritating was that it came out of nowhere,and the sortingcould not be arranged according to my desires.  What I see in you naming sequence is that you changed the name of the image and ommitted the number attached to by the camera.  WHen I left the camera assigned number intact and just added other items after this item, the images came into LR in the proper sequence and I have had less issues.  Give that a tary.

MDIJB

[MBehrens]

I don't think that milliseconds are stored in the raw file. Here is Jeffrey Friedl's online Exif viewer
http://exif.regex.info/exif.cgi
If anyone is going to display every detailed bit of data it is Jeffrey and only seconds are displayed.
Without milliseconds in the Exif, LR has no way to order shots within the same second.

Here is the header data right out of a DNG file
xmp:MetadataDate="2017-01-16T16:45:46-08:00"
No milliseconds, -08:00 is the timezone offset. Laso, not verified that this is what LR is using, just an example from the file.

Best bet is to include the camera image number in the filename and sort by it.

[EvanConway]

I cant recreate this issue no, its very random and I havent really experimented specifically with DNG or CR2 being the main variables and potentially the problem so I will definitely try that out.

Hey Morey, Thank you!  but in this situation I am pretty positive this is not an issue of milliseconds that its miss-sorting because some of the images are several seconds apart and it will just shuffle them, sometimes its a majority of images in an entire import other times its only a few. But one thing I do know, is the original file names and the time stamps represent the right order and LR doesnt seem to be reading the timestamp

So I normally do leave the original image number from the camera in my naming as my last 4 numbers in my file and you can see it says Econ_bowl_october_3489. I usually do that no matter what and I tend to name the entire shoot something more memorable as 350K photo file names organized by number dates wouldnt get me anywhere. I have the files arranged in folders which I have a hierarchy by date and year.

So that first image in the sequence shown in the screenshot is econ_bowl_october_3489 and it should be followed by the 4th image in the top row. If you notice the third image to the left in the top row is pretty much the second to last image of that entire sequence and it arranged it as the 3rd image in the seq. Those original 4 file numbers at the end of each file still represents what the proper order in the sequence of images is suppose to be and if you were to sort according to those last 4 numbers it correlates with the time stamp perfectly and pretty much every time.

So that being said I am pretty certain it is not a milisecond mis read and more of a LR un willing to read.

I cant really sort by the original file number because a lot of times I will have shot with 2 camera bodies and those file numbers will be close and then every other picture is from 2 different cameras and thats worse than the orginal problem!

I have however only heard of people experiencing this with video files and not to the extent that im experiencing it.

Another thing I dont know if I had mentioned was this started on my macbook pro originally and somehow migrated like a flu through my LR catalogs and eventually into my main Archive..

What a nightmare!

[fdisilvestro]

LR can definitely store milliseconds and some cameras store up to 1/100 of a second, at least to my knowledge. In the attached image, you can see a screenshot of the table "Adobe_images" inside a Lightroom catalog that I just created for a test.
The first images are from a Nikon D300, which does store 1/100 of a second, as you can see in the image, and converting them to DNG kept the same precision.

The last row contains the data from a JPG created elsewhere (not direct from camera) and the capture time goes up to milliseconds plus the time zone.

Unless the OP views the actual data in the catalog, we can only suppose what is happening.

I don't think that milliseconds are stored in the raw file. Here is Jeffrey Friedl's online Exif viewer
http://exif.regex.info/exif.cgi
If anyone is going to display every detailed bit of data it is Jeffrey and only seconds are displayed.
Without milliseconds in the Exif, LR has no way to order shots within the same second.

Here is the header data right out of a DNG file
xmp:MetadataDate="2017-01-16T16:45:46-08:00"
No milliseconds, -08:00 is the timezone offset. Laso, not verified that this is what LR is using, just an example from the file.

Best bet is to include the camera image number in the filename and sort by it.

Regards

[fdisilvestro]

You will need to view the actual data in the catalog to determine if the problem is the application or the data itself. There have been versions of LR that did not manage the data correctly. For instance, a few versions ago LR was not reading the lens model when using Nikon.

I cant recreate this issue no, its very random and I havent really experimented specifically with DNG or CR2 being the main variables and potentially the problem so I will definitely try that out.

Hey Morey, Thank you!  but in this situation I am pretty positive this is not an issue of milliseconds that its miss-sorting because some of the images are several seconds apart and it will just shuffle them, sometimes its a majority of images in an entire import other times its only a few. But one thing I do know, is the original file names and the time stamps represent the right order and LR doesnt seem to be reading the timestamp

So I normally do leave the original image number from the camera in my naming as my last 4 numbers in my file and you can see it says Econ_bowl_october_3489. I usually do that no matter what and I tend to name the entire shoot something more memorable as 350K photo file names organized by number dates wouldnt get me anywhere. I have the files arranged in folders which I have a hierarchy by date and year.

So that first image in the sequence shown in the screenshot is econ_bowl_october_3489 and it should be followed by the 4th image in the top row. If you notice the third image to the left in the top row is pretty much the second to last image of that entire sequence and it arranged it as the 3rd image in the seq. Those original 4 file numbers at the end of each file still represents what the proper order in the sequence of images is suppose to be and if you were to sort according to those last 4 numbers it correlates with the time stamp perfectly and pretty much every time.

So that being said I am pretty certain it is not a milisecond mis read and more of a LR un willing to read.

I cant really sort by the original file number because a lot of times I will have shot with 2 camera bodies and those file numbers will be close and then every other picture is from 2 different cameras and thats worse than the orginal problem!

I have however only heard of people experiencing this with video files and not to the extent that im experiencing it.

Another thing I dont know if I had mentioned was this started on my macbook pro originally and somehow migrated like a flu through my LR catalogs and eventually into my main Archive..

What a nightmare!

[John Caldwell]

Rapid shutter bursts from our 1Dx II cameras often have out-of-chronology time stamps. LR accurately reads those stamps, and so files appear out of order.

[fdisilvestro]

If the camera does not record the time stamp properly (I cannot think of a justifiable cause) then it is a bug/limitation of the camera, not LR