Pages: [1] 2   Go Down

Author Topic: Amazon Glacier + Lightroom ???  (Read 18142 times)

Fips

  • Full Member
  • ***
  • Offline Offline
  • Posts: 195
    • some unrelated photos on flickr
Amazon Glacier + Lightroom ???
« on: August 23, 2012, 04:41:02 am »

Amazon just introduced a new online archival service called Glacier. But it's unlike Dropbox, Google Drive and the like, in that it is not meant for real-time access. The idea is that you can quickly upload stuff for free, but retrieval might take a few hours and it costs. Storage itself is much cheaper than anything else with only $0.01 per GB and month. So it's really meant as an archive or last-line-of-defense backup.
AFAIK there is not front end supplied yet, but only some APIs which developers can use to cook up their own applications.

So that makes me wonder if is would be a good idea to integrate Glacier into LR. Maybe as a plugin. I could imagine that during every import an archive of the new images is created and sent to the glacier. Or alternatively this could be done automatically once a month or week.
Thanks to my OCD-like editing I have less than 100GB of image in total including all Photoshop files. That would imply costs of less than $12 per year for me which even in the long run isn't more expensive than buying external disks.

So, what do you think? Any plugin programmers around who are interested?


(as a technical side note: I believe the service is based on magnetic tape. This would explain the wait times for retrieval and why it is not possible to manipulate uploaded files. Even today, when it comes to large volume, long-term storage nothing beats tape)
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: Amazon Glacier + Lightroom ???
« Reply #1 on: August 23, 2012, 08:49:48 am »

This is very interesting compared to what I'm current paying for Mozy home backup.  I guess the delay in being able to access one's data is probably OK for me since I would only need it if there were an absolute crisis where both my computer HD and backup HD failed on the same day which is not all that likely.  I'm going to take a look at this and make sure that there are not any hidden gotchas here.  Thanks for the tip!!!!!

I haven't read the stuff on the site yet so I don't know how the backup time is configured.  Mozy Home automatically backs up things twice a day and you can configure what files are backed up.

Alan
Logged

PeterAit

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4559
    • Peter Aitken Photographs
Re: Amazon Glacier + Lightroom ???
« Reply #2 on: August 23, 2012, 09:10:25 am »

For most of us, who are using cable or DSL connections, wouldn't this be glacially slow for large backups? A cable modem's upload speed is about 700k/sec, and that's under ideal conditions (neighbors not streaming movies, etc.). A 100 GB upload would take 40 hours.
Logged

Fips

  • Full Member
  • ***
  • Offline Offline
  • Posts: 195
    • some unrelated photos on flickr
Re: Amazon Glacier + Lightroom ???
« Reply #3 on: August 23, 2012, 09:14:28 am »

The thing is, right now this is not meant as a service for end user. Amazon provides some APIs for developers to come up with front ends or to integrate glacier into existing programs as a service (where the user might not even be aware that he's using Amazon).
So, I really hope that soon some body comes up with a LR plugin or something. Actually I have little hope, that Adobe will integrate it directly in LR as they have their own cloud based service Revel which unfortunately is not very useful for me and it's far too expensive.
Logged

Fips

  • Full Member
  • ***
  • Offline Offline
  • Posts: 195
    • some unrelated photos on flickr
Re: Amazon Glacier + Lightroom ???
« Reply #4 on: August 23, 2012, 09:18:48 am »

Peter, that's right. But you would only have to do this once. If you then keep on updating once a month or so, it will go much faster. Also, this is meant as an long(er)-term archival service, so hopefully in a couple of years time, upload speeds will have increased.

But one idea would be to have to uploads done automatically according to some user defined schedule. Say every sunday night, starting at 11pm or so. Then it would be less distracting (that is when 11pm is not in the middle of your working hours  ;) )
Logged

Alan Goldhammer

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4344
    • A Goldhammer Photography
Re: Amazon Glacier + Lightroom ???
« Reply #5 on: August 23, 2012, 11:55:55 am »

The thing is, right now this is not meant as a service for end user. Amazon provides some APIs for developers to come up with front ends or to integrate glacier into existing programs as a service (where the user might not even be aware that he's using Amazon).
I did some reading on the site and this is correct.  To use it as is, you would need to upload individual packets of data with each use and the manage them.  It's pretty cumbersome to do it this way and until someone writes an API to manage this better I'll stick with what Mozy does for me with is fully automated.  To Peter's point about upload speeds, yes the initial upload takes a long time but as noted, you only need to do it once.  I think the way the Amazon service is presently set up without an API, you are going to run into problems in this regard.  I suspect they are going after the business user who has an Internet connection much faster than those of us who are home users.
Logged

Samotano

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 88
    • http://flickr.com/photos/97538742@N00/sets/72157603392490862/
Re: Amazon Glacier + Lightroom ???
« Reply #6 on: August 28, 2012, 11:20:13 am »

You can send them a hard drive and they will upload the content for you.  In comparing this with their S3 service, I assumed Glacier was the natural end user version of it.  Given its pricing, I find it quite attractive for my photo library backup and the pain of downloading/uploading is a small price to pay for offsite backup (I always have my own onsite backup). 

A much bigger problem though is to figure out a sustainable workflow for it.  Update it once a year or once every few months? How? What about files that have been modified? I am curious to hear in the following months from people who have tried it in the context of photo backup.
Logged

stormyboy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 94
Re: Amazon Glacier + Lightroom ???
« Reply #7 on: August 28, 2012, 01:01:27 pm »

I saw this link on another site.  It's a Windows client for Amazon Glacier.  I haven't tried it.
http://fastglacier.com/
Logged

Samotano

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 88
    • http://flickr.com/photos/97538742@N00/sets/72157603392490862/
Re: Amazon Glacier + Lightroom ???
« Reply #8 on: August 28, 2012, 01:14:11 pm »

Nice!  Thanks for sharing this.
Does anyone know how this would work to update files?  Say I have uploaded my library.  Then I modify 100 files.  Now I want the backup to reflect these updates.  What do you do? 
Logged

Fips

  • Full Member
  • ***
  • Offline Offline
  • Posts: 195
    • some unrelated photos on flickr
Re: Amazon Glacier + Lightroom ???
« Reply #9 on: August 28, 2012, 01:41:20 pm »

In Glacier, you can only upload and delete files. There is no such thing as modification or overwriting. It's due to the fact, that (most likely) magnetic tape is used on which you can only write data serially. You would have to delete the old files and uploaded the modified ones again.
However, that's not really an issue as there is no need to write changes into the raw file. Even when working with DNG where the XMP is embedded in the file itself. All information is stored in the catalog.
Personally would be to convert to DNG during import and write the most important metadata, like copyright information, geo data, Keywords, and such into the DNG file before uploading.

EDIT: Forgot to mention, that there's a charge of $0.03 per GB for data deleted prior to 90 days.
« Last Edit: August 28, 2012, 02:23:15 pm by Fips »
Logged

Snoopy Lane

  • Newbie
  • *
  • Offline Offline
  • Posts: 6
Re: Amazon Glacier + Lightroom ???
« Reply #10 on: August 28, 2012, 07:33:45 pm »

It's 2012.  Most likely not tape.
Logged

howardm

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 1984
Re: Amazon Glacier + Lightroom ???
« Reply #11 on: August 28, 2012, 07:40:24 pm »

There is no other storage medium that has a lower cost per GB and as low power requirements.

At somewhere between 1.5TB and 3TB per tape capacity and a price of well under $50, it's pretty hard
to beat good ol' tape (ala LTO5)


It's 2012.  Most likely not tape.

cybis

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 283
    • http://www.lucbusquin.com
Logged

stormyboy

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 94
Re: Amazon Glacier + Lightroom ???
« Reply #13 on: August 28, 2012, 10:39:10 pm »

Hi, cybis.  I haven't tried it yet, but I did find some discussion about it:
https://forums.aws.amazon.com/thread.jspa?threadID=102829
Logged

Fips

  • Full Member
  • ***
  • Offline Offline
  • Posts: 195
    • some unrelated photos on flickr
Re: Amazon Glacier + Lightroom ???
« Reply #14 on: August 29, 2012, 03:15:39 am »

Quote
[...] it's pretty hard to beat good ol' tape (ala LTO5)

Exactly. And don't forget the 15-30 years durability.


Regarding clients. There will be a lot of programs hacked together in short time now to make use of Glacier. But as it is meant to be really, really secure, I'd give it some time and wait for a proven-to-be-reliable solution, hopefully with good LR integration.
Logged

cybis

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 283
    • http://www.lucbusquin.com
Re: Amazon Glacier + Lightroom ???
« Reply #15 on: August 29, 2012, 10:01:14 am »

Hi, cybis.  I haven't tried it yet, but I did find some discussion about it:
https://forums.aws.amazon.com/thread.jspa?threadID=102829

Thanks @stormyboy, I gave it a go today and it worked just fine and is easy to use. Now I'm just waiting on Amazon to publish the tool to transfer files from S3 to Glacier...

Also FastGlacier is free only for non-commercial use. They have a pro version for $29.95 that they claim is faster.
Logged

Samotano

  • Jr. Member
  • **
  • Offline Offline
  • Posts: 88
    • http://flickr.com/photos/97538742@N00/sets/72157603392490862/
Re: Amazon Glacier + Lightroom ???
« Reply #16 on: August 29, 2012, 11:18:08 am »

So one possible workflow is:
1. Upload data to Amazon Glacier (say only RAW files).
  Question: can the folder structure be preserved on Glacier?

2. One month later (or 6 month later - depending on user's needs) user has new RAWs files to upload.
  I seem to understand that the user cannot populate the directory structure created in 1.  What then?

My library is organized as follows:
Photo_Library>Digital>YYYY_MM>YYYY_MM_eventname>Originals>YYYYMMDD_XX_XXXX.dng

Also, one would need a way to keep track of what was backed-up and what wasn't

« Last Edit: August 29, 2012, 11:20:32 am by Samotano »
Logged

john beardsworth

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 4755
    • My photography site
Re: Amazon Glacier + Lightroom ???
« Reply #17 on: August 29, 2012, 11:20:27 am »

Exactly. And don't forget the 15-30 years durability.
Regarding clients. There will be a lot of programs hacked together in short time now to make use of Glacier. But as it is meant to be really, really secure, I'd give it some time and wait for a proven-to-be-reliable solution, hopefully with good LR integration.
If you've got a general client, I wonder if you can roll your own integration in LR using Publish services. You'd create a new service using the Hard Drive option, probably choosing DNG as the output format, and a smart collection which picks up images edited in the last x hours.
Logged

Fips

  • Full Member
  • ***
  • Offline Offline
  • Posts: 195
    • some unrelated photos on flickr
Re: Amazon Glacier + Lightroom ???
« Reply #18 on: August 30, 2012, 12:43:05 pm »

Yes, that's more or less it. But as uploading large amounts of data can take quite a while, I would split the process in two parts: One LR plugin and one external program.

The plugin one has to write would provide a new publish service and create a database which keeps track of what's already uploaded and when that happened. All the files are then prepared, i.e. the metadata is saved to those files. If they are proprietary raw file, a DNG copy is created somewhere.
Then comes the external program into play which handles the upload according to a time schedule set in the plugin dialogue. The advantage of this is that LR does not need to run during uploading.

In principle, this shouldn't be too complicated. I had a look at the LR SKD and it's all done in Lua which provides modules for HTTP and SQL (so in principle one could even do in all within a single plugin!).
Logged

cybis

  • Sr. Member
  • ****
  • Offline Offline
  • Posts: 283
    • http://www.lucbusquin.com
Re: Amazon Glacier + Lightroom ???
« Reply #19 on: August 30, 2012, 11:13:10 pm »

So I played a little with FastGlacier today. To understand Glacier’s limitations it may help to imagine that Amazon’s servers are located on Neptune. A round trip to Neptune takes about 9 hours at light speed. One can have a high bandwidth connection to Neptune but latency will always be what it is - long.

Every file operation that requires two way communications will take that long. For example in dealing with duplicate files, it would be impractical to warn the user and ask what to do. So Glacier gives each file a unique id; therefore multiple files with the same name and path can coexist (yes glacier keeps paths intact). This makes incremental backups impractical.

In the same fashion, downloading files will be as fast as your bandwidth permits but will only reach you hours after a request.

The usual file browser user interface is impractical. FastGlacier user interface simply shows one long list of files with their path, i.e. no collapsing folders, no drag and dropping files between folders, etc.

As already mentioned Glacier is for long term archival, not for last month files that you might still be working with next week. Since it’s at least just as important to backup yesterday’s files as it is for last year’s files, I would use S3-jungle disk or similar for recent files and Glacier for the rest.

I’m not sure yet what the best way to organize files for archiving in Glacier is. If you first group files together in zip or rar archives, it seems to me it would make the whole thing even less flexible.

Thoughts?
 
Logged
Pages: [1] 2   Go Up