Pages: [1] 2   Go Down

Author Topic: Worries about having all image files in one database/catalogue/portfolio  (Read 18507 times)

rogerxnz

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 526
    • Hayman Lawyers

I have decided to try to bring some order to my collections of negatives, transparencies and digital files from cameras and scanners.

I gather most DAM apps store original image files inside "containers" which are variously called databases, catalogues, or similar. I am concerned that, over time, with 60MP shots and 6x12 scans, the size of the container will become very large and unwieldy.

I know I could use external hard drives to avoid filling up my computer's own hard drive but managing a collection over several hard drives will also be difficult.

What about using a database program containing tiffs or jpegs of the original images at a smaller size than the originals to reference where the original is stored (such as, DVD#123)? That way, you would have more chance of fitting your entire collection on one hard drive.

Filemaker Pro v11 has a sample Photo Catalog which I found easily accepted a tiff of a 6x12 scan and you can customise the data fields to suit your own purposes.

Does anyone use this database approach?

On the other hand, a 1TB drive should be able to hold a large number of files. Even if you allow 100MB for a digital camera file and do reference scans which are about the same size, a 1TB drive could store 10,000 images and multi-TB hard drives are available to store more.

So, maybe, with cheap large capacity hard drives being readily available, my concerns about storing original files in a catalogue are not worth worrying about and I should just use catalogues in Capture One to store everything?

Your comments, please.
Roger

Logged
Roger Hayman
Wellington, New Zealand

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #1 on: December 29, 2012, 06:23:26 pm »

Roger my experiences have all been with Lightroom but the overall principles are much the same across other DAM applications.

In Lightroom the core of the application is a SQLite database.
The database stores information about the image collection including metadata, keywords, editing instructions, etc, and the image's location.
Images are NOT stored directly in the database but the pointers pointing to the location of the images are.

There is no reason why images cannot be stored across multiple internal and external hard drives. (I and many others do this).
Obviously if an external hard drive is offline then Lightroom cannot directly access those images but it still knows where they should be. Bringing that external HD back on line instantaneously makes those images available again.

The terminology used in Lightroom calls the database a catalog.
In principle only one catalog is required.
Many individuals citing organizational needs or performance issues break their image collections into multiple catalogs.
If you do this then know that you are creating a rod for your own back.
It is far better to keep everything in one catalog so the entire collection is instantaneously accessible and searchable.

Back up the catalog religiously.
To repeat the catalog is not your image files it is the database.
Backing up the database regularly means that you have a very recent fallback should your catalog become corrupted.
You can also backup the catalog to multiple physical locations as well.
Murphy's law of computing tells you that database corruption is inevitable sooner of later.

In Lightroom there is the ability to optimise the catalog.
This helps immensely with performance as the image collection grows.

Back up you images.
Create an online backup of all your images files exactly as they are in your primary storage HD.
Create two redundant offline backups of your images again with the folder structures exactly as they are on your primary storage HD.
You will also ideally need two remotely located archives of your image collection that are also updated regularly.
The recommendation here is to use HD's that can store the entire collection on one drive and then obviously to duplicate that.

If you have been following this then you will see that in fact one ends up with multiple backups of both the database and the actual image files.
The fact that a single catalog (database) is the primary organizational tool in no way implies susceptibility to data loss as long as the precautions mentioned above are taken.
I can easily accomplish these tasks using Lightroom but other applications that are truly DAM capable should give the same overall utility.

There are several resources that I would like to recommend to you:
Peter Krogh, The DAM book. (the definitive overview of DAM);
Seth Resnick, D65's Lightroom 4 Workbook: Workflow, Not Workslow in Lightroom 4. (DAM at work with Lr4) and;
Luminous Landscape Guide to Asset Management 'Where the #%*! Are My Pictures?' with Seth Resnick and Michael Reichman http://www.luminous-landscape.com/videos/wamp.shtml (the best of the lot to get you started practically).

Feel free to sharpen up the questions as you get going in your quest.
Good luck!

Tony Jay
« Last Edit: December 29, 2012, 06:27:49 pm by Tony Jay »
Logged

Les Sparks

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 305
    • http://www.ncsparks.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #2 on: February 12, 2013, 03:14:57 pm »

Check out George Jardine's Lightroom Library tutorials. He discusses many of your concerns including having the photos on multiple hard disks. Get the Location Workflow & Lightroom Catalog Management tutorial and you get access to the basic library tutorial free. I don't have any relation with George Jardine except as a user of his tutorials. The url for the Location Workflow & Lightroom Catalog Management tutorial is http://mulita.com/blog/?page_id=4130.
Les
Logged

rogerxnz

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 526
    • Hayman Lawyers
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #3 on: February 13, 2013, 03:17:03 am »

Hi Jay
I never thanked you before for your very helpful and detailed advice and I now do so!

I guess I would like to avoid the expense of Lightroom as it does more than catalogue images. I understand that it allows users to process images like Capture One, and so on. I already have Photoshop and Capture One, so, I consider I do not need those capabilities repeated in the app I will only use to catalogue my pics.

What can users recommend on the Mac platform just for cataloguing images?
Roger
Logged
Roger Hayman
Wellington, New Zealand

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #4 on: February 13, 2013, 03:34:24 am »

Roger, I have to say that Lightroom at $140 roughly is excellent value for money.

Tony Jay
Logged

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #5 on: February 25, 2013, 04:12:55 pm »

Lightroom is not a DAM, so this is not a solution for what you're looking for.

If I could recommend a few:

AssetBank - http://www.assetbank.co.uk/digital-asset-management-software/

Celum - http://www.celum.com/products/celum-synergy/top-features/

Extensis - http://www.extensis.com/digital-asset-management/

filecamp - http://filecamp.com

immersify - http://www.immersify.co.uk

intelligencebank - http://www.intelligencebank.com/features

netexposure - http://netx.net

seefile - http://www.seefile.com/product.html

web.archives - http://www.webarchives.com/digital-asset-management-software.aspx

webdam - http://www.webdamsolutions.com


There are essentially two types of DAMs: 1. On-premise and 2. SaaS (cloud-based)

Both have advantages and disadvantages. The biggest factor should be workflow. Let me know if you have any questions as I've setup quite a few, all with over a million images each!

Graham

Logged
Graham Clark  |  grahamclarkphoto.com

RFPhotography

  • Guest
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #6 on: February 26, 2013, 02:46:33 pm »

Lightroom is not a DAM, so this is not a solution for what you're looking for.


This would seem an odd statement to many.  Could you expand on it please?

WRT some of the options, particularly the cloud-based, I'm also curious how practical those are.  Filecamp, for example, offers a mere 15GB of storage for $61/month.  An additional 800GB of storage is $668 PER MONTH.  How much would one of those million + image catalogues you mention cost at Filecamp? 
« Last Edit: February 26, 2013, 02:54:34 pm by BobFisher »
Logged

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #7 on: February 26, 2013, 03:17:19 pm »

Lightroom is not a DAM, so this is not a solution for what you're looking for.
This is plain wrong.

Tony Jay
Logged

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #8 on: February 26, 2013, 06:01:00 pm »

DAM vendor list: http://digitalassetmanagementnews.org/dam-vendors/

Although Aperture and Lightroom use databases on the backend, it does not mean they are DAMs in the industry standard sense of the acronym. The industry purpose of a DAM is for company or team-based collaboration, ingestion, organization and retrieval of media assets that include documents, videos, images, other filetypes such as .CAD, .PSD, .AI etc. Aperture and LR are machine-centric solutions designed with photography in mind. A DAM is intended to be all-encompassing with a few key requirements:

  • Workflow editor to define workflows
  • Non-ambiguous keywording
  • Platform agnostic; must be accessible to everyone regardless of computer

In the typical sense of what a DAM is, there needs to be a taxonomy for keyboarding and a common workflow that image reviewers adhere to. If you and I (as reviewers) have different keywords then the DAM breaks immediately, as the integrity of a DAM is only as good as the data you put into it. Keywords can never be ambiguous.

For example, the company I built a DAM for a little bit ago had the following requirements:

  • Ability of external contractors to upload content to the system
  • Access / control privileges; view, upload, edit, download (with or without approval)
  • Searching/Indexed; Accessibility for image type, size or format. A keyword taxonomy is essential to these systems
  • Stores files of any type
  • Download presets; end-user control over re-size, crop, convert etc.
  • Browser-based

The type of data that was to be part of this DAM looked something like:
  • 600GB of photographs, mainly RAW and 16BIT TIFF
  • 7TB of uncompressed video footage
  • 70GB of design-related assets, including .PSD, .AI, .TIFF etc.

From my experience the above is a pretty standard (if not light?) set of requirements for DAM customers. For landscape shooters this is of course overkill : )

Graham
« Last Edit: February 26, 2013, 06:17:15 pm by Graham Clark »
Logged
Graham Clark  |  grahamclarkphoto.com

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #9 on: February 26, 2013, 06:04:21 pm »

I would recommend an inexpensive RAID-1 (drive mirroring - redundant drives) for this purpose. If one drive fails, you replace it with a bare drive and it rebuilds.

http://eshop.macsales.com/shop/firewire/usb/raid_1/Gmax

Graham
Logged
Graham Clark  |  grahamclarkphoto.com

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #10 on: February 26, 2013, 06:18:36 pm »

Hello Tony,

I think we're talking about two different things: 1. Digital asset management, and 2. DAM the acronym which encompasses more than simply photography.

http://digitalassetmanagementnews.org/dam-vendors/

Graham

Logged
Graham Clark  |  grahamclarkphoto.com

RFPhotography

  • Guest
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #11 on: February 26, 2013, 07:15:27 pm »

This is a photography forum.  While your definition of a DAM application may technically be correct, it's largely irrelevant in this context.  From a photography standpoint, the vast majority of us are individual owner/operators so a solution like those you note is beyond overkill.  In this context and in the context of this forum section, LR is a DAM application.  It does all of the things the multi-user applications do in terms of cataloguing, search, file management.  The only thing photography-specific applications like Lightroom don't do, or at least not too easily, is manage licenses.
Logged

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #12 on: February 26, 2013, 07:17:47 pm »

Hey Bob,

Got it! Photography teams often utilize DAMs, so perhaps it's not beyond the scope of this forum! : )

But I agree, for this post, it's overkill.

Graham
Logged
Graham Clark  |  grahamclarkphoto.com

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #13 on: February 26, 2013, 07:19:26 pm »

Hello Tony,

I think we're talking about two different things: 1. Digital asset management, and 2. DAM the acronym which encompasses more than simply photography.

http://digitalassetmanagementnews.org/dam-vendors/

It very much occurred to me that you were using a different definition of DAM.
The fact that in your definition of DAM a multi-user requirement is non-negotiable does not exclude Lightroom in my estimation.

I also very much doubt that Roger requires the functionality that you are driving at.
In percentage terms not a lot of people need a digital asset management environment that requires a simultaneous multi user environment and the rest.

I also have a tiny suspicion that you have esentially hijacked this thread.

Tony Jay
Logged

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #14 on: February 26, 2013, 07:23:18 pm »

The non-ambigious keyboarding that's so important to a multi-tenant workflow is what's difficult to get in Aperture or Lightroom. If there's a way that would be awesome!

Graham
« Last Edit: February 27, 2013, 11:55:50 am by Graham Clark »
Logged
Graham Clark  |  grahamclarkphoto.com

RFPhotography

  • Guest
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #15 on: February 26, 2013, 09:17:29 pm »

Lightroom really isn't intended as multi-user platform, Tony.  It's not meant to sit on a server and be accessed by multiple users simultaneously.  I don't even think it can be used that way.

Graham, true about photography teams, but those are probably in the minority.  I recall a post on Chase Jarvis' blog and I think he and his team just use Aperture.
Logged

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #16 on: February 27, 2013, 01:38:12 am »

The fact that in your definition of DAM a multi-user requirement is non-negotiable does not exclude Lightroom in my estimation.

Hi Bob.
Just to clarify I was not trying to intimate that Lightroom could be used as a multi user application - it can't.
Rather I was challenging the rather casual dismissal of Lightroom as a digital asset management capable application and the hijacking of the thread in order to somewhat crudely advertise (might I be overcalling this?).

Tony Jay
Logged

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #17 on: February 27, 2013, 02:38:19 am »

... rather casual dismissal of Lightroom as a digital asset management capable application and the hijacking of the thread in order to somewhat crudely advertise ...

Really? Advertise?

Graham
Logged
Graham Clark  |  grahamclarkphoto.com

Graham Clark

  • Full Member
  • ***
  • Offline Offline
  • Posts: 179
    • grahamclarkphoto.com
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #18 on: February 27, 2013, 02:39:56 am »

I recall a post on Chase Jarvis' blog and I think he and his team just use Aperture.

I saw that! Really interesting (and very modular) workflow for such a high throughput of images.

Graham
Logged
Graham Clark  |  grahamclarkphoto.com

Tony Jay

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 2965
Re: Worries about having all image files in one database/catalogue/portfolio
« Reply #19 on: February 27, 2013, 05:28:02 am »

Really? Advertise?

Circumstantial evidence admittedly:
Thread quiet for 13 days.
From the OP's initial question clear that Lightroom would (could) the solution.
Sudden post from Graham dismissing Lightroom as a DAM option, certainly gives his reasons.
More posts from Graham who happens to mention that he designs and builds DAM solutions.
Graham sort of admits to a hijack of the thread as it takes a sudden dive to the left.

Could I be overreading this (I have already admitted to the possibilty)?
I have no axe to grind but I am somewhat suspicious - perhaps Graham would like to clarify the context of his posts.

Tony Jay
Logged
Pages: [1] 2   Go Up