Archive for the ‘Solutions’ Category

Hadoop on a shared infrastructure with Isilon and VMware – a report from a recent POC in Switzerland

Johannes Geissler

Johannes Geissler

Senior Specialized Solutions Engineer at EMC Computer Systems AG
Johannes Geissler
RT @PureStorage: The Pure Storage #FlashArray//X received @CRN's 2017 Tech Innovator Award for storage! Read more: https://t.co/vIyicaublR… - 16 hours ago

Traditionally customers deploy Hadoop on dedicated servers with a dedicated Hadoop file system on top of direct attached storage. Loading data into this Hadoop silo can be a challenge and time consuming. This is an outdated approach and dedicated hardware is not needed anymore.

With VMware you already have a server infrastructure to run Hadoop and Isilon allows you to leverage HDFS directly on your scale-out NAS.

I had the chance to test this setup at University of Fribourg and run a couple of Hadoop jobs on a small 4 Node ESX Cluster (VSPEX Blue) pointed to a 3 Node Isilon system for shared HDFS.

To get started you need to invest 0$. Just download the Software;

  • VMware Big Data Extension 2.3
  • existing Isilon NAS or IsilonSD (Software Isilon for ESX)
  • Hortonworks, Cloudera or PivotalHD
  • EMC Isilon Hadoop Starter Kit (documentation and scripts)

Preparation

VMware Big Data Extension

VMware Big Data Extension helps to quickly roll out Hadoop clusters. BDE is a virtual appliance based on Serengenti and integrated as a plug-in to vCenter. https://www.vmware.com/support/pubs/vsphere-big-data-extensions-pubs.html

The vApp deploys a management VM and a template to clone Hadoop VMs from;

BDE_VMs_template

One can manage BDE through CLI or in vCenter.  For consistency it’s recommended to do either CLI or GUI management only. From the new management-server VM you can start Serengeti CLI and connect it to vCenter. Add networks and datastores to be used by BDE and you’re ready to roll out the first cluster:

cluster_create

This command creates a cluster with 1 master node and 16 workers. A name node is not needed. It runs as a clustered service directly on all Isilon nodes. The vspex-bde.json file contains the configuration for the cluster nodes. Like how many nodes, memory and vCPU per node, master / worker node, etc.

VMs_created

In vCenter the cluster is named accordingly and the VMs are visible;

vsphere

Now you have basically 17 naked VMs for Hadoop without a distribution installed. This is important to understand. BDE only automates the deployment of VMs and makes it easy to adjust vCPU, Memory count and add or remove worker VMs.

In the next step you can start to install your preferred Hadoop distribution. I tested Hortonworks and Cloudera utilizing an existing 3 Node Isilon Cluster as HDFS Store.

HDFS on Isilon scale-out NAS

For HDFS we have an Isilon which is a multiprotocol NAS platform. This means the data can be stored through any protocol like NFS, CIFS and directly analyzed by Hadoop nodes through HDFS as a protocol. If you don’t have an Isilon cluster, you can download the software only version for free use.  www.emc.com/getisilon

With HDFS on Isilon one can benefit from all Enterprise NAS features like replication, snapshots and backup integration. Plus the usable capacity on Isilon is 80% of raw.  Instead of < 33% with HDFS on DAS.

Quote from Cloudera; ”In the decade and a half since Google originally designed that architecture, though, the industry has made some progress. A few years back we worked with EMC to integrate the Isilon storage system with Hadoop, and we have customers running Cloudera today against large-scale Isilon stores. Separating the storage grid from the compute grid turns out to have some administrative and cost advantages for enterprise deployment at scale.”

 

EMC Hadoop Start Kits

Before installing your preferred Hadoop distribution, have a look at EMC Hadoop start kits. It includes scripts to prepare your VMs for Hadoop deployment.  Like scripts to enable SSH, mount a common NFS share as repository and install prerequisite packages on all VMs

https://github.com/claudiofahey/hsk-docs

The start kit is available for Pivotal, Cloudera and Hortonworks

Hadoop Distribution Setup

Hortonworks

As preparation for Hortonworks we need to enable the Ambari agent on Isilon and create the required users. During HDP Setup, when registering your hosts, just add the name of the Isilon cluster as one of the hosts; and check to use Isilon as the DataNode only;

hortonworks_install

Cloudera Setup

The Setup with Cloudera Manager is even straighter forward. In the list of services to install one can just choose Isilon as the HDFS Layer;

Cloudera_install_smallPerformance Tests

With the Hadoop cluster ready it’s finally time for some performance tests.

The compute nodes are four nodes with an E5-2620 each all in one 2U chassis and I’ve deployed 16 VMs as Hadoop worker nodes. Each worker VM has 2 vCPU and 16GByte of Memory.

The HDFS Layer is running on a small Isilon Cluster with 3 physical Nodes.

Teragen

Teragen is not a compute intensive workload and the bottleneck is typically the storage layer. I’ve generated 100GByte of random data and wrote it to HDFS on Isilon.

30 tasks, about 2 tasks per VM, have shown to give decent performance.

[root@cloudera-master-0 ~]# time hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen 1000000000 /hadoop/teragen/cloudera/30task-100GB

Isilon GUI shows that inbound throughput jumps to 15-19Gbit/s. That’s a pretty decent number for writes on 3*23 Disks protected with FEC on a distributed files system. Each Isilon Node processes about 700-800 MByte/s ingest.

ISILON_GUI

The teragen job of 100GByte completed in 1 minute and 11 seconds.

Terasort

Next up is the terasort benchmark which is more a compute bound job. For testing sake I run the terasort against the 100GB of random data generated in the last test;

[root@cloudera-master-0 ~]# time hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /hadoop/teragen/cloudera/30task-100GB /hadoop/teragen/cloudera/30task-100GB-out

Terasort by default does mapping first and starts the reduce tasks when mapping is past 85%. After 7 minutes all data was mapped and reduce was at 25%… After 10 minutes the job completed.

Interestingly the load on Isilon was very little during terasort and the utilization of the nodes remained <10%.  Only between 0-500 MByte/s were read (while the 3 Isilon Nodes could easily deliver >3GByte/s of reads.) This is because the bottleneck was on the compute side as witnessed in Cloudera Manager and in vCenter;

At the beginning of the terasort job, the cluster CPU utilization jumped to 100%;

Cloudera_Manager

And in vCenter alerts of vCPU usage popped up showing all 16VMs consuming around 50GHz. That’s about half of the power available to the 4 ESX hosts.

vsphere

Summary

This POC and many other customer examples have shown that with today’s virtualization, networking and storage technology also a Hadoop cluster can run on shared infrastructure leveraging enterprise grade availability, efficiency and reliability. The use of Isilon provides the unique capability to analyze the data directly where it was created on the multi-protocol scale-out NAS platform. Virtualizing Hadoop means increased efficiency; you can start very small and offer Hadoop as a service. Multi tenancy can be implemented end-to-end.

A BIG THANK YOU to www.datastore.ch to provide the equipment and support for setting up this POC @ University of Fribourg.

What’s the next hot thing?

If you need more, much more performance and plan to do real-time analytics, check out the new release of DSSD D5.

Cloudera is also excited about it, check out their view; https://vision.cloudera.com/when-andy-bechtolsheim-starts-a-company-you-pay-attention/

BREAKING NEWS – VIPR CONTROLLER GETS OPEN SOURCED

Sascha

Sascha

Field CTO Office Switzerland at EMC Computer Systems AG
Sascha

@vSaschaMeier

Field CTO ¦ Sr. Manager Systems Engineering @ DellEMC Switzerland Passionate Technologist
RT @JClarkeatDell: Fantastic review of the @DellEMC PowerEdge R640 "The PowerEdge R640 packs a lot into its compact 1U chassis and deliver… - 5 days ago

CoprHDEMC plans to release an open source project based on EMC ViPR Controller named Project CoprHD into the open source community.
The project will make the code for ViPR Controller — the storage automation and control functionality — open for community-driven development. The code will be available on GitHub in June 2015 and it will be licensed under the Mozilla Public License 2.0 (MPL 2.0).
Project CoprHD APIs are engineered to provide developers a single, vendor-neutral control point for storage automation with deep integration into Cloud automation frameworks from VMware, Microsoft and OpenStack. Just as with the EMC ViPR Controller, with the commercial version of Project CoprHD, customers, partners, service providers and system integrators can develop new service catalog offerings with automated workflows to meet their customers specific needs. ViPR Controller is designed to provide simple storage management for EMC and third-party arrays, and we will continue selling ViPR Controller as a commercial offering. ViPR Controller and Project CoprHD share the same core features and functionality. However ViPR Controller customers benefit from access to EMC’s support and professional services.
(more…)

What a first Day at EMC World 2015!

Sascha

Sascha

Field CTO Office Switzerland at EMC Computer Systems AG
Sascha

@vSaschaMeier

Field CTO ¦ Sr. Manager Systems Engineering @ DellEMC Switzerland Passionate Technologist
RT @JClarkeatDell: Fantastic review of the @DellEMC PowerEdge R640 "The PowerEdge R640 packs a lot into its compact 1U chassis and deliver… - 5 days ago

EMCWUpdate

EMC World 2015 starts with an impressive first General Session. David Goulden EMC II CEO on stage is talking about our EMC strategy based on Storage platforms, Converged Infrastructures and Federation solutions, all designed to improve the IT agility and to drive down the costs to be able to invest in new technologies and solutions supporting the Digital Business Strategy of a company.

The fastest way to improve the IT Agility and to drive down costs is the deployment of a Converged Infrastructure Solution.

EMC provided a technology outlook how we will expand our CI solution portfolio with a next generation Hyper Converged Rack Scale Infrastructure Solution called VXRACK!

VXRACK
(more…)

Products are dead, long live solutions!

Jean-Paul Nussbaumer

Jean-Paul Nussbaumer

Sales Director Region West at EMC Computer Systems SA
Jean-Paul Nussbaumer
Accenture, Pivotal Spin Up Cloud Migration Business Group https://t.co/kbGeyvuubs - 16 hours ago
Jean-Paul Nussbaumer
Jean-Paul Nussbaumer

Latest posts by Jean-Paul Nussbaumer (see all)

Now that’s a pretty bold statement from a company that’s mainly known for its excellent products, isn’t it? EMC will always be a technology company, but we are definitely seeing a shift in the buying patterns of our customers. The challenges that companies face today are so daunting that a standalone product can never solve all of the issues. Products and related services need to be glued together in total solutions that fit in – and actually enable – a broad IT strategy.

What I found out when talking to customers since joining EMC last spring is that not one company or industry can escape the changes that are affecting our entire society. Even the most traditional and conservative industries have to take into account that the new expectations of their clients are high. Very high. Whether you are in banking, insurance, manufacturing or running a city or government, regardless of whether you serve consumers or other businesses, the people using your products or services expect you to be always online. They want to be able to consult their investment portfolio, place and track their orders at the tip of their fingers, adapt their subscriptions to fit their exact needs. And they demand the possibility of doing that wherever they are and from whatever device they have at hand at that moment. Platform- and location independent instant gratification, that is what makes your – and our – customers happy.

(more…)