Luminous Landscape Forum

Equipment & Techniques => Digital Asset Management => Topic started by: rogerxnz on December 29, 2012, 03:20:24 pm

Title: Worries about having all image files in one database/catalogue/portfolio
Post by: rogerxnz on December 29, 2012, 03:20:24 pm
I have decided to try to bring some order to my collections of negatives, transparencies and digital files from cameras and scanners.

I gather most DAM apps store original image files inside "containers" which are variously called databases, catalogues, or similar. I am concerned that, over time, with 60MP shots and 6x12 scans, the size of the container will become very large and unwieldy.

I know I could use external hard drives to avoid filling up my computer's own hard drive but managing a collection over several hard drives will also be difficult.

What about using a database program containing tiffs or jpegs of the original images at a smaller size than the originals to reference where the original is stored (such as, DVD#123)? That way, you would have more chance of fitting your entire collection on one hard drive.

Filemaker Pro v11 has a sample Photo Catalog which I found easily accepted a tiff of a 6x12 scan and you can customise the data fields to suit your own purposes.

Does anyone use this database approach?

On the other hand, a 1TB drive should be able to hold a large number of files. Even if you allow 100MB for a digital camera file and do reference scans which are about the same size, a 1TB drive could store 10,000 images and multi-TB hard drives are available to store more.

So, maybe, with cheap large capacity hard drives being readily available, my concerns about storing original files in a catalogue are not worth worrying about and I should just use catalogues in Capture One to store everything?

Your comments, please.
Roger

Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on December 29, 2012, 06:23:26 pm
Roger my experiences have all been with Lightroom but the overall principles are much the same across other DAM applications.

In Lightroom the core of the application is a SQLite database.
The database stores information about the image collection including metadata, keywords, editing instructions, etc, and the image's location.
Images are NOT stored directly in the database but the pointers pointing to the location of the images are.

There is no reason why images cannot be stored across multiple internal and external hard drives. (I and many others do this).
Obviously if an external hard drive is offline then Lightroom cannot directly access those images but it still knows where they should be. Bringing that external HD back on line instantaneously makes those images available again.

The terminology used in Lightroom calls the database a catalog.
In principle only one catalog is required.
Many individuals citing organizational needs or performance issues break their image collections into multiple catalogs.
If you do this then know that you are creating a rod for your own back.
It is far better to keep everything in one catalog so the entire collection is instantaneously accessible and searchable.

Back up the catalog religiously.
To repeat the catalog is not your image files it is the database.
Backing up the database regularly means that you have a very recent fallback should your catalog become corrupted.
You can also backup the catalog to multiple physical locations as well.
Murphy's law of computing tells you that database corruption is inevitable sooner of later.

In Lightroom there is the ability to optimise the catalog.
This helps immensely with performance as the image collection grows.

Back up you images.
Create an online backup of all your images files exactly as they are in your primary storage HD.
Create two redundant offline backups of your images again with the folder structures exactly as they are on your primary storage HD.
You will also ideally need two remotely located archives of your image collection that are also updated regularly.
The recommendation here is to use HD's that can store the entire collection on one drive and then obviously to duplicate that.

If you have been following this then you will see that in fact one ends up with multiple backups of both the database and the actual image files.
The fact that a single catalog (database) is the primary organizational tool in no way implies susceptibility to data loss as long as the precautions mentioned above are taken.
I can easily accomplish these tasks using Lightroom but other applications that are truly DAM capable should give the same overall utility.

There are several resources that I would like to recommend to you:
Peter Krogh, The DAM book. (the definitive overview of DAM);
Seth Resnick, D65's Lightroom 4 Workbook: Workflow, Not Workslow in Lightroom 4. (DAM at work with Lr4) and;
Luminous Landscape Guide to Asset Management 'Where the #%*! Are My Pictures?' with Seth Resnick and Michael Reichman http://www.luminous-landscape.com/videos/wamp.shtml (the best of the lot to get you started practically).

Feel free to sharpen up the questions as you get going in your quest.
Good luck!

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Les Sparks on February 12, 2013, 03:14:57 pm
Check out George Jardine's Lightroom Library tutorials. He discusses many of your concerns including having the photos on multiple hard disks. Get the Location Workflow & Lightroom Catalog Management tutorial and you get access to the basic library tutorial free. I don't have any relation with George Jardine except as a user of his tutorials. The url for the Location Workflow & Lightroom Catalog Management tutorial is http://mulita.com/blog/?page_id=4130 (http://mulita.com/blog/?page_id=4130).
Les
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: rogerxnz on February 13, 2013, 03:17:03 am
Hi Jay
I never thanked you before for your very helpful and detailed advice and I now do so!

I guess I would like to avoid the expense of Lightroom as it does more than catalogue images. I understand that it allows users to process images like Capture One, and so on. I already have Photoshop and Capture One, so, I consider I do not need those capabilities repeated in the app I will only use to catalogue my pics.

What can users recommend on the Mac platform just for cataloguing images?
Roger
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 13, 2013, 03:34:24 am
Roger, I have to say that Lightroom at $140 roughly is excellent value for money.

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 25, 2013, 04:12:55 pm
Lightroom is not a DAM, so this is not a solution for what you're looking for.

If I could recommend a few:

AssetBank - http://www.assetbank.co.uk/digital-asset-management-software/

Celum - http://www.celum.com/products/celum-synergy/top-features/

Extensis - http://www.extensis.com/digital-asset-management/

filecamp - http://filecamp.com

immersify - http://www.immersify.co.uk

intelligencebank - http://www.intelligencebank.com/features

netexposure - http://netx.net

seefile - http://www.seefile.com/product.html

web.archives - http://www.webarchives.com/digital-asset-management-software.aspx

webdam - http://www.webdamsolutions.com


There are essentially two types of DAMs: 1. On-premise and 2. SaaS (cloud-based)

Both have advantages and disadvantages. The biggest factor should be workflow. Let me know if you have any questions as I've setup quite a few, all with over a million images each!

Graham

Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: RFPhotography on February 26, 2013, 02:46:33 pm
Lightroom is not a DAM, so this is not a solution for what you're looking for.


This would seem an odd statement to many.  Could you expand on it please?

WRT some of the options, particularly the cloud-based, I'm also curious how practical those are.  Filecamp, for example, offers a mere 15GB of storage for $61/month.  An additional 800GB of storage is $668 PER MONTH.  How much would one of those million + image catalogues you mention cost at Filecamp? 
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 26, 2013, 03:17:19 pm
Lightroom is not a DAM, so this is not a solution for what you're looking for.
This is plain wrong.

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 26, 2013, 06:01:00 pm
DAM vendor list: http://digitalassetmanagementnews.org/dam-vendors/

Although Aperture and Lightroom use databases on the backend, it does not mean they are DAMs in the industry standard sense of the acronym. The industry purpose of a DAM is for company or team-based collaboration, ingestion, organization and retrieval of media assets that include documents, videos, images, other filetypes such as .CAD, .PSD, .AI etc. Aperture and LR are machine-centric solutions designed with photography in mind. A DAM is intended to be all-encompassing with a few key requirements:


In the typical sense of what a DAM is, there needs to be a taxonomy for keyboarding and a common workflow that image reviewers adhere to. If you and I (as reviewers) have different keywords then the DAM breaks immediately, as the integrity of a DAM is only as good as the data you put into it. Keywords can never be ambiguous.

For example, the company I built a DAM for a little bit ago had the following requirements:


The type of data that was to be part of this DAM looked something like:

From my experience the above is a pretty standard (if not light?) set of requirements for DAM customers. For landscape shooters this is of course overkill : )

Graham
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 26, 2013, 06:04:21 pm
I would recommend an inexpensive RAID-1 (drive mirroring - redundant drives) for this purpose. If one drive fails, you replace it with a bare drive and it rebuilds.

http://eshop.macsales.com/shop/firewire/usb/raid_1/Gmax

Graham
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 26, 2013, 06:18:36 pm
Hello Tony,

I think we're talking about two different things: 1. Digital asset management, and 2. DAM the acronym which encompasses more than simply photography.

http://digitalassetmanagementnews.org/dam-vendors/

Graham

Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: RFPhotography on February 26, 2013, 07:15:27 pm
This is a photography forum.  While your definition of a DAM application may technically be correct, it's largely irrelevant in this context.  From a photography standpoint, the vast majority of us are individual owner/operators so a solution like those you note is beyond overkill.  In this context and in the context of this forum section, LR is a DAM application.  It does all of the things the multi-user applications do in terms of cataloguing, search, file management.  The only thing photography-specific applications like Lightroom don't do, or at least not too easily, is manage licenses.
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 26, 2013, 07:17:47 pm
Hey Bob,

Got it! Photography teams often utilize DAMs, so perhaps it's not beyond the scope of this forum! : )

But I agree, for this post, it's overkill.

Graham
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 26, 2013, 07:19:26 pm
Hello Tony,

I think we're talking about two different things: 1. Digital asset management, and 2. DAM the acronym which encompasses more than simply photography.

http://digitalassetmanagementnews.org/dam-vendors/

It very much occurred to me that you were using a different definition of DAM.
The fact that in your definition of DAM a multi-user requirement is non-negotiable does not exclude Lightroom in my estimation.

I also very much doubt that Roger requires the functionality that you are driving at.
In percentage terms not a lot of people need a digital asset management environment that requires a simultaneous multi user environment and the rest.

I also have a tiny suspicion that you have esentially hijacked this thread.

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 26, 2013, 07:23:18 pm
The non-ambigious keyboarding that's so important to a multi-tenant workflow is what's difficult to get in Aperture or Lightroom. If there's a way that would be awesome!

Graham
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: RFPhotography on February 26, 2013, 09:17:29 pm
Lightroom really isn't intended as multi-user platform, Tony.  It's not meant to sit on a server and be accessed by multiple users simultaneously.  I don't even think it can be used that way.

Graham, true about photography teams, but those are probably in the minority.  I recall a post on Chase Jarvis' blog and I think he and his team just use Aperture.
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 27, 2013, 01:38:12 am
The fact that in your definition of DAM a multi-user requirement is non-negotiable does not exclude Lightroom in my estimation.

Hi Bob.
Just to clarify I was not trying to intimate that Lightroom could be used as a multi user application - it can't.
Rather I was challenging the rather casual dismissal of Lightroom as a digital asset management capable application and the hijacking of the thread in order to somewhat crudely advertise (might I be overcalling this?).

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 27, 2013, 02:38:19 am
... rather casual dismissal of Lightroom as a digital asset management capable application and the hijacking of the thread in order to somewhat crudely advertise ...

Really? Advertise?

Graham
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 27, 2013, 02:39:56 am
I recall a post on Chase Jarvis' blog and I think he and his team just use Aperture.

I saw that! Really interesting (and very modular) workflow for such a high throughput of images.

Graham
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 27, 2013, 05:28:02 am
Really? Advertise?

Circumstantial evidence admittedly:
Thread quiet for 13 days.
From the OP's initial question clear that Lightroom would (could) the solution.
Sudden post from Graham dismissing Lightroom as a DAM option, certainly gives his reasons.
More posts from Graham who happens to mention that he designs and builds DAM solutions.
Graham sort of admits to a hijack of the thread as it takes a sudden dive to the left.

Could I be overreading this (I have already admitted to the possibilty)?
I have no axe to grind but I am somewhat suspicious - perhaps Graham would like to clarify the context of his posts.

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 27, 2013, 11:59:02 am
perhaps Graham would like to clarify the context of his posts.

Hello Tony,

No trolling please, it dilutes the quality of the content here!

We're discussing DAMs. Lightroom isn't an industry standard DAM: http://digitalassetmanagementnews.org/dam-vendors/

Have a great day!

Graham

Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: rogerxnz on February 27, 2013, 12:30:02 pm
Gentlemen!
I found all the discussion interesting and helpful. For my purposes, Lightroom sounds the best solution and it is not nearly as expensive as I thought.

What I am not enjoying is the war that is developing based on one member's suspicions. I don't think members should get to the stage of querying a member's motives based on suspicions. If a member mentions a suspicion and admits in the same post that they could be wrong, it would be more responsible in my opinion not to raise the suspicion in the first place.
Roger
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 27, 2013, 03:46:03 pm
Roger, sadly I do still have concerns but I am not interested in a war in the way that you describe.
As an analogy your post asked a question where clearly a one-tone pickup was the answer.
Suddenly Graham has come in to the conversation dismissing that option and instead suggesting the software equivalent of a mining truck with tyres the size of your house and an engine big enough to generate enough electricity for the South Island. It doesn't fit in your road never mind in your garage.
So, with regard to DAM solutions almost all of us are single operators for which Graham's industry standard version of DAM solutions are ridiculously overspecified (not because they would not work well). Were I the director of a press agency, museum, government agency etc then Lightroom as a DAM solution is very limiting indeed and the capabilities of applications that Graham is advocating become useful indeed for multiples of reasons (pun intended).
The fact that Graham seems apparently so oblivious of how inappropriate his suggestions are in the context of your requirements allied to the fact that he develops (apparently) DAM software prompted the suspicions.

I am certainly no troll as a review of my posts on LuLa will indicate.
I have invited Graham to explain himself: so far he has only sheltered behind the term "industry standard" while accusing me of being a troll.
I have no problem with the idea of apologising and shaking hands with Graham (literally and metaphorically) should my concerns be unfounded.
I am still trusting that Graham will contextualize his suggestions as to why a two hundred ton mining truck is required for a one ton pickup solution.

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Graham Clark on February 27, 2013, 08:10:30 pm
... for which Graham's industry standard version of DAM solutions are ridiculously overspecified...

Somehow the discussion went beyond the initial use-case into what a 'DAM' is. After all, this is the DAM forum.

Graham

Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Tony Jay on February 27, 2013, 08:38:05 pm
Somehow the discussion went beyond the initial use-case into what a 'DAM' is. After all, this is the DAM forum.

Graham

Fair enough.
Apology and handshake offered.

Tony Jay
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: Phil Indeblanc on February 28, 2013, 03:33:50 am
There are many DAM apps:

PhotoMechanic
Extensis
ACDSee
IDImager / PhotoSupreme
iMatch

I avoid clouds. For my purpose it is inconvenient, money that can go to new large drives, and slow.

Lightroom or CaptureOnev7 are maybe DAM's for RAW files of cameras, but not Digital Asset Managers. In fact the biggest request users make for LR is to support more filetype, such as their own Adobe types like PDF, IDesign, PSB, and common ones, PNG, and others.

I use lightroom with 2 catalogs so far and have about 400K images. I did slow down. You have to raise the cache size. There are more speed issues when displaying on large screens and large raw files..

So...No LR is not a DAM as it can't allocate its resources to just being a DAM. It is also a RAW processor with lots of functions that slow each others functions down as you switch from one module to the other. With 2 apps running along side, ther would be no switching. In fact if ACRaw had the functions of the Develop module on LR, I would not use LR at all.
 But, ACR isn't like LR, sooooo, I use it, and it is easy to use. But I would be blind of my actual content without ACDSee, which is a DAM as it can catalog. I use it as a super Explorer to find files.  Hope this helps.


I should add to the above...perhaps if LightRooms Library module grew into Bridge, I would surely use it. In the past I found Bridge to be unstable, and never liked the 2 files it places in each folder(not a huge deal). If they can make Bridge and LR Library merge, and do this by the Bridge folder ability look up changes made in a folder , It would prompt you and show the new files available to add to the Library/catalog. This would be the direction I could see very possible.
Title: Re: Worries about having all image files in one database/catalogue/portfolio
Post by: rogerxnz on March 02, 2013, 04:29:14 pm
OK, I am trialing Lightroom.

I have a desktop computer and a laptop. I guess if I have one catalog, I can only have it on one computer unless the catalog is in the cloud?

How reliable is it to have a cloud-based catalog?
Roger

I have since posted this question in the Lightroom forum and I ask any responses to be made there. Thank you.