Storage in the Cloud

In these last few weeks, I've been diving deeper into cloud-based storage.  Using Amazon's S3, I've created a few different file shares - a few to test and now 1 in active production.  My initial reaction is pretty positive, which is saying something!  File transfer times were something that I was initially concerned with, and while they do generally suck, for the average user who deals in files generally <250 kb, this really is not an issue.

A few weeks ago, a colleague approached me and wanted to create a shared storage directory that can be accessed from all of our different office locations.  Now, in with our existing infrastructure that is provided for us, this is not possible.  So, I went off to Amazon and built a bucket.  Then, using s3fs (a fantastic utility for mounting S3 buckets to *nix systems - http://code.google.com/p/s3fs/), you can mount your S3 share via FUSE.

In order to share your new FUSE-mounted directory via SMB, however, you need to ensure that your mount command allows users other than root to access it.  Here's an example of my mount string:

sudo /usr/bin/s3fs BUCKETNAME -o allow_other -o use_cache=/tmp /mnt/FUSEDIRECTORY

There is an inherent delay involved, as you could imagine, as your are providing access to a non-local share, so in order to side-step some of the frustration, I defined a local cache for the system to use.

This may have all sounded fantastic in my head... but from the user end, the latency issue is too frustrating.  So, today I'm working on having the shares at all of our offices write locally and then sync to Amazon.... either via raw rsync or Duplicity.  I'll publish my results, after I figure out which one works best for us.


On another note... Google seems to be stepping into the cloud-based storage game a little further today, after announcing that Google Apps users can purchase up to 16TB of storage.  Now, I use Google Apps for one of the organizations that I manage, but their storage requirements are small (for the time being, at least).  But, this is something that I'm defiantly going to look into....