Thoughts on analytics, data management, visualization and collaboration

–Hadoop Examples

Posted by Brett Sheppard on July 31, 2011

Organizations in multiple industries and the public sector are using Hadoop as one part of their integrated data architectures to obtain the highest value from their data. Hadoop is an increasingly popular option to process, store and analyze huge volumes of semi-structured, unstructured or raw data, often from disparate data sources.

The following are 10 public-domain examples of Hadoop production clusters, with URL links if you would like additional information:

  • AOL Advertising pairs Hadoop’s capability for handling large, complex data volumes with Membase’s support for sub-millisecond latency to make optimized decisions for real-time ad placement. (Blog at Cloudera.com by Couchbase co-founder James Phillips, February 16, 2011).
  • Hadoop helps quantitative analytic staff for Bank of America’s cross-enterprise lines of business – such as Home Loans and Insurance, Consumer and Small Businesses, Online Banking, Risk Management and Card Products – study entire data sets with billions of records versus the limitations of looking only at sample data, to assess the market, credit and/or operational risk, and revenue lift, of new and existing financial products (Author interview and 2010 Hadoop Summit talk by Tresata co-founder Abhishek Mehta).
  • Hadoop powers DNA sequencing analysis and other bioinformatics research, starting with the CloudBurst application at the University of Maryland and continuing with the Crossbow, Contrail and Myrna genomic research projects (Ronald Taylor, BMC Bioinformatics, December 2010).
  • At Disney, the Technology Shared Services Group uses Hadoop for an integration mashup for diverse departmental data to analyze patterns across different but connected customer activities, such as attendance at a theme park, purchases from Disney stores, and viewership of Disney’s cable television programming (PwC Technology Forecast case study, 2010).
  • eBay extended a Teradata enterprise data warehouse with Hadoop for image processing and deep data mining along with a Teradata offshoot named Singularity for behavioral analysis and clickstream semi-relational data (author interview 2011 and video presentation by Anil Madan).
  • In addition to using Hadoop for web analytics and to back up MySQL databases, Facebook uses HBase as a back-end for materialized views to support real-time analytics (Facebook Engineering Notes on Facebook).
  • LinkedIn uses Hadoop together with a Project Voldemort key-value store, Kafka distributed messaging system and Azkaban work-flow tool to power analytics-driven features such as “People You May Know” and “Jobs You May Be Interested In” (LinkedIn Engineering).
  • Hadoop forms part of the Motorola architecture to process and understand ever-growing volumes of mobile-device usage data. (Big Data Cloud meetup, August 2011).
  • Twitter stores log data in Hadoop to run analytics and hit its FlockDB graph database in parallel to assemble social-graph aggregates (Twitter Engineering).
  • Yahoo! runs 42,000 Hadoop nodes storing about 200 petabytes of data as an integral part of the Yahoo! cloud infrastructure, and joined Benchmark Capital to fund a Hadoop startup, Hortonworks (Yahoo! Developer Network, Hadoop Summit 2011).

Cloudera Customer Solutions VP Omer Trajman counts 16 organizations that operate Hadoop production clusters with one petabyte or more of data stored in each cluster. (DBMS2, July 6, 2011).

5 Responses to “–Hadoop Examples”

  1. I needed to thank you for this excellent read!! I certainly enjoyed every little bit of
    it. I have got you bookmarked to check out new stuff you post…

  2. Great post! We will be linking to this particularly great
    post on our site. Keep up the good writing.

  3. unquestionably like your web site but you have got to check the punctuation upon many of one’s blogposts. Quite a few tend to be rife having punctuation concerns i still find it really irritating to express to the certainty having said that I am going to certainly return all over again.

  4. In 2010 he became one of the millions of people who are planning to perform saudi
    arabia Hajj at least once in their lifetime. So in the Grand Mosque in Mecca, but was
    alarmed about the situation in Medina, which is also known as the hajj.

  5. shopping said

    This sale mostly takes place around the time when the retailer
    is ready to put up fresh stock and needs to clear
    the old unsold stock. With online shopping you will instantly know if you can find what you are looking for, or if
    you can’t, potentially saving you hours looking for what you want.
    E-commerce hosting has reduced the interference of middlemen or retailers in terms paying

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: