Cloud Storage with Amazon’s SimpleDB

Part of the Amazon Web Services family, Amazon’s SimpleDB is a massively scalable and reliable key/value datastore that is exposed via a web interface and can be accessed using the Java language.

Get started with Amazon’s SimpleDB in this two part series exploring SimpleDB’s unique approach to schemaless data storage, including a demonstration of one of the datastore’s most unusual features: lexicographic searching.

Part 1:
http://www.ibm.com/developerworks/java/library/j-javadev2-9/

Part 2:
http://www.ibm.com/developerworks/java/library/j-javadev2-10.html

Amazon Dynamo

Here’s a nice white paper from Werner Vogels, CTO of Amazon, describing Dynamo, Amazon’s Highly Available Key-Value Store:

http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf

!SQL Architectures

There are some great presentations on InfoQ regarding architectures involving No-SQL implementations. These include companies such as Linked In, Reddit, the GILT Group, and Heroku.

http://www.infoq.com/presentations/Project-Voldemort-Scaling-Simple-Storage

http://www.infoq.com/presentations/Project-Voldemort-at-Gilt-Groupe

http://www.infoq.com/news/2010/05/7-Lessons-Reddit

http://www.infoq.com/presentations/Horizontal-Scalability

A couple of these architectures use Project Voldemort which is a key/value store that has the following attributes:

• Uses a Hash Ring for Consistent Hashing
• Key space is partitioned into many small partitions
• Keys are mapped to partitions, and partitions are mapped to machines
• Replication – Each partition is stored by n nodes
• Multiple Reads – there are multiple reads (across several nodes) to be sure you get the latest value for a key
• Multiple Writes – updates made to each node that holds the data for your key
• Vector Clocks – to handle versioning of your data across nodes and to allow for data conflict resolution

Free Book on CouchDB

CouchDB: The Definitive Guide by O’Reilly books is freely available online. So if you’re interested in finding out more about NoSQL or non-relational databases this is a great start.

From the CouchDB website:

Apache CouchDB is a document-oriented database that can be queried and indexed in a MapReduce fashion using JavaScript. CouchDB also offers incremental replication with bi-directional conflict detection and resolution.

CouchDB provides a RESTful JSON API than can be accessed from any environment that allows HTTP requests. There are myriad third-party client libraries that make this even easier from your programming language of choice. CouchDB’s built in Web administration console speaks directly to the database using HTTP requests issued from your browser.

CouchDB is written in Erlang, a robust functional programming language ideal for building concurrent distributed systems. Erlang allows for a flexible design that is easily scalable and readily extensible.

The NoSQL Trend

NoSQL (or non-relational) databases seem to be one of the hot trends these days in the world of web development and cloud computing. Practically all of the big names are already using it or are rapidly moving towards it, and many of these companies have been pioneering the way by developing these technologies.

According to wikipedia:

NoSQL is a movement promoting a loosely defined class of non-relational data stores that break with a long history of relational databases. These data stores may not require fixed table schemas, usually avoid join operations and typically scale horizontally. Academics and papers typically refer to these databases as structured storage.

Many of these databases are implemented as a kind of distributed hash table and queries against them are implemented using Google’s MapReduce algorithm. These databases are highly performant and can be scaled up easily by adding more servers.

There are many options in the non-relational database space, some proprietary and many open source. These include some of the following:

• Google’s Bigtable
• Amazon’s Dynamo and Amazon’s SimpleDB
• CouchDB
• MongoDB
• HBase (Hadoop)
• Cassandra
• Redis
• Riak
• MemcacheDB
• Voldemort
• Neo4j
• Tokyo Cabinet / Tokyo Tyrant
• Dynomite
• Hypertable
• ThruDB
• Jackrabbit