Hadoop and Structured Data — Things to Keep In Mind

From an earlier circa when processing a couple of petabytes of data in a traditional data warehouse to a time when number crunching Petaflop sized data is a breeze using Hadoop. Hadoop leverages a network of processors to process massive sets of data that overwhelm traditional data processing systems. It also provides a robust framework for storing and processing big data. Hadoop offers big benefits to businesses if implemented correctly to improve and support core operations.

In a world where data is constantly generated from innumerable sources, using Hadoop can help a business structure and make sense of patterns and trends hidden in the data. While Hadoop’s superior capabilities for processing unstructured data cannot be contested, it can also be deployed for tackling structured data.

Working on structured data with Hadoop

When considering Hadoop for handling structured data, the primary thing to keep in mind is that it is a data storage and processing platform designed to scale out to petabytes of data; data is stored as raw files on the Hadoop cluster. Because Hadoop is a general-purpose data storage platform, it can be customized for highly specific purposes. For processing and storing structured data storage and processing, the following Hadoop projects are the most commonly used:

• Hive: A data warehousing framework that catalogs data in structured files. It offers a query interface with a language similar to SQL — HiveQL. Hive is ideal for heavy ETL/ELT workloads where reliability is a priority. In Hive 2.0, Hive LLAP is meant for performing Low Latency Analytical Processing (LLAP) on Hadoop data. While this serves as a good option for interactive workloads, it can be extended for speeding up data integration workloads too
• HBase: A distributed database that uses multiple computers instead of a single CPU. HBase is built on top of Hadoop
• Giraph: A graph processing engine for data on Hadoop

Best practices for transforming data using Hive

Practices like data partitioning, joining optimization, and vectorized query execution are ideal for data processing in the Hive. Data partitioning can be leveraged for improving query performance on large tables, allowing for data storage in separate sub-directories. When partitioning a table in Hive, users should do so on a relatively high cardinality column and use time or region attributes as partition keys.

In ETL workloads, it is a common practice to join a large fact table source with smaller dimension tables. In such cases, it is the cost of the join operation that is critical to the overall query performance. Size parameter controls the size of data in-memory and can be deployed when there is a fact load with a snapshot source joined to a smaller dimension.

Vectorization in query execution refers to processing a batch of rows instead of a row at a time. While Hive typically processes one row at a time, vectorized execution of queries is helpful in reducing CPU time during aggregation, joins, and filters.

Good Practices for HBase Architecture

A perfectly designed HBase cluster follows some best practices. For example, using a key prefix that distributes load well based on the use case. If the prefix is timestamped when sorted, stored or queried in a batch then this will result it in overloading of each region server instead of evenly distributing the load.

Things to keep in mind

Hadoop’s relatively low storage expense makes it a good option for storing structured data instead of a relational database system. However, Hadoop is not ideal for transactional data since it is highly complex and needs quick implementation. Hadoop is also not recommended for structured data sets that require minimal latency.

Because of its batch processing capabilities, Hadoop should be deployed for pattern recognition, creating recommendation engines, index building, and sentiment analysis — functions that generate data at a high volume. These can be easily stored in Hadoop and later queried using MapReduce functions.

However, Hadoop shouldn’t be used as a replacement for your existing data center. In fact, it must be integrated with the existing IT infrastructure to augment the organization’s data management and storage capabilities. Using several easily available tools, organizations can connect their existing systems with Hadoop and process data irrespective of size and scale.