NoSQL: Cassandra, HBase, RocksDB

转自: http://www.linkedin.com/pulse/nosql-cassandra-hbase-rocksdb-siddharth-anand

I've had the pleasure of working in Data Infrastructure for more than 12 years now  at companies like eBay, Etsy, Netflix, and LinkedIn. If you are unfamiliar with the term Data Infrastructure, the term refers to a collection of services that receive, persist, transfer, and provide custom access to data. This includes databases, caches, messaging systems, data streams (for ingress, replication, and egress), ETL, data warehouses, search engines, blob & object stores, filers, graph engines, and other specialty data systems.

Every popular web site invests in building data infrastructure that is performant, scalable, and available. Though the extent of these investments may vary, the problems these companies face often do not. These companies must invariably answer the same class of questions : How do I write a lot of data relatively quickly, access the data in flexible and interesting ways (again, at scale and with low latency), and allow for changes to my system without any interruption to service. In some cases, companies must also defend against forces beyond their control (e.g. power outages in their hosting providers) while providing uninterrupted service to end users.

One technology trend that emerged about 5 years ago and reached a level of maturity (at least in Silicon Valley) is NoSQL. NoSQL is really a counter-culture term that signaled a revolutionary approach to database problems. Instead of trying to solve all or most problems in a single database, the NoSQL movement decided to solve a small set of problems well. NoSQL databases did not pretend to be the inevitable replacement for their general-purpose RDBMS counterparts, but they did provide a better solution for certain use-cases. Akin to the ASIC (application-specific integrated circuits) revolution of 2 decades earlier, NoSQL databases came to live alongside their RDBMS counterparts in a data center, just as ASICs came to live alongside general purpose CPUs on a motherboard.

However, there are still those who are confused by NoSQL DBs (or more-aptly named ASDBs). They try to reconcile giving up their RDBMS installations for the narrower-functioning NoSQL DBs or they get confused when comparing 2 NoSQL technologies for the same use-case, when each was clearly designed for different purposes.

I'd like to clear up this confusion.

Firstly, it is important to recognize that the NoSQL revolution was sparked by 2 seminal papers : Amazon's Dynamo DB & Google's Big Table. These 2 papers solve very different problems. The former (Dynamo) attempted to build a distributed, fault-tolerant OLTP store with a very simple data schema (key-value). Dynamo's goal was to reliably carry out shopping-cart actions related to persistence and lookup in face of data center failures. The latter (Big Table) was built for a company in which web crawlers were constantly inserting large amounts of data and in which search indexers were constantly building new indexes. Search indexes were built using Google's Map-Reduce system, so the Map phase needed an efficient way to scan data. Data was flowing in as a result of crawling and it was being read as a result of Map-related scans that were used to build search indexes or generate analytics.

From these papers came two of the most widely-adopted NoSQL Databases today: Cassandra and HBase. Cassandra leverages elements of the Dynamo distribution model and the Big-table data model. HBase is influenced by Big Table and hence relies of the GFS equivalent in the open-source realm : HDFS.

Cassandra & HBase : Cousins, not Siblings

A common question that I am asked is which one is better. This is the wrong question to ask. They are built for different purposes.

There are some similarities between HBase and Cassandra. Both leverage append-only semantics in data persistence in order to achieve high-speed data ingestion. This is used in writing both the commit logs and core data files (a.k.a. region files in HBase or sstables in Cassandra). Both leverage in-memory data structures to land the writes, choosing to flush the writes out as immutable region files or sstables in a background process. The region files and sstables are opened as append-only for writing. In both systems, reads must reconcile stale data with new data. Since read operations become more expensive as the number of region files or sstables grows, another background process is needed to consolidate or compact the files. Finally, both systems are written in Java.

The similarities end there. Whereas it may appear that these systems should have similar performance and operational profiles, they do not.

Cassandra for OLTP, HBase for Map-Reduce

The Difference SSDs Make

HBase is co-deployed with Hadoop distributions. This allows Map-Reduce jobs and HBase to share the same HDFS cluster. The hardware profile of the cluster is typically oriented towards the needs of map-reduce jobs : high data throughput workloads that don't require tight time SLAs. As such, the Hadoop software distribution is usually installed on a large but cheap fleet of commodity hardware running magnetic disks. Although, this might be reasonable for map-reduce, this is terrible for a database (a.k.a. HBase). Every commit log fsync and every region file flush incurs disk head scheduling penalties.

Cassandra deployments typically leverage SSDs. Cassandra is simply faster thanks to the use of SSDs

Why Scans Hurt OLTP Performance

HBase is range-partitioned. This makes sense since it was designed to be used by mappers in map-reduce. Mapper scans are very expensive when run in Java with generational garbage collection enabled. Scans are like snow-plows pushing objects out of eden and into the survivor or tenured spaces, even if those force-promoted objects would have died in young gen. This force promotion means that the more expensive old-gen collections are needed to clean our garbage. Though CMS is pretty good about limiting pause times during old gen collections, it doesn't compact, so fragmentation will eventually become an issue. When running mixed loads (OLTP and Map-reduce) on HBase, people often see increased OLTP response times caused by the mapper scan-elevated pause times.

Cassandra is mod-partitioned and does not support range scans on contiguous row keys. This makes map-reduce against a running Cassandra cluster less efficient than it would be on HBase (and not viable). However, companies like Netflix and Coursera carry out Map-reduce using the SSTable files directly, by-passing this problem. This project is called Aegisthus

The key take-away is to not use HBase for OLTP use-cases. On the flip side, HBase is better supported in the map-reduce eco-system than Apache Cassandra. Netflix's Aeghistus project is the only way to run map-reduce over Cassandra sstables.

RocksDB, Learning from HBase Issues

A couple of years ago, the same team at Facebook that spearheaded the HBase Messaging work unveiled RocksDB. What is RocksDB? It's HBase with the following differences :

• Does not use HDFS
• Simple Key-Value store
• Runs on the local file system (typically with SSD)
• No write distribution or multi-node reliability

In other words, it is similar to a single Cassandra node, except it is purely Key --> Value, instead of (rowKey, columnKey) --> Value

Document-oriented DBs

I would be remiss to exclude MongoDB as the other popular NoSQL DB. It is yet a 3rd type of NoSQL DB. It is document-oriented and hence supports nested data types. It has gained a lot of popularity but I have not seen it used at scale. LinkedIn has developed its own document-oriented DB, Espresso. Google similarly wrote a paper a few years ago on F1, its document-oriented DB.

Recap

NoSQL is not one type of DB. It's a term used to classify application-specific DBs. There are trade-offs among them by definition as they are not built to be general-purpose. Among these, Cassandra and MongoDB have found a place as replacements for OLTP RDBMS where businesses can work-around the trade-offs. HBase does not perform well as an OLTP replacement, but has found a niche in the world of map-reduce, where it has its roots.