Loft (Blue)

"NoSQL is awesome! I need to use it on my next project!" ... [hours later] ... "How the heck do I get my data out of this thing?!"

Sound familiar? Non-relational data storage solutions (NoSQL) promise all kinds of benefits -- scalability, flexibility, fault-tolerance -- but (by the nature of the moniker) don't have SQL to query with. Riak is one such solution, a distributed key-value store that implements MapReduce for querying, and has some awesome client libraries for Javascript and Node.js.

HBase can easily store terabytes of data, but how do you scale your search mechanism to sift through these mountains of bits and retrieve large result sets in a matter of milliseconds? We used a combination of Solr sharding, careful index creation, and result pruning to meet these strict requirements in our production environment. Come see how we handle millions of rapid fire queries from dozens of parallel search clients against many terabytes of data while addressing high availability through load balancing and replication.

Obviously, there’s a huge amount of interest and use around Hadoop for processing large amounts of data given its scalability and cost/performance. Its great for a lot of Big Data needs but it was never designed to replace RDBMS systems. Hadoop is batch oriented, it doesn’t supports queries and its environment can best be described as “ideal” for programmers but abysmal for business users.

Hadoop, until recently, would trust any user based on who he says he is. This is clearly not enough in large companies where they have hadoop instances storing sensitive data (like financial, revenue, etc.), and where these instances are being used by many users and from potentially different groups. In this talk, I will cover the security threats in Hadoop in the various communication paths (in Hadoop Distributed File System, MapReduce, and the client components). I will present the solutions we designed for each of them.