There they go again, HPE misleading customers about Superdome X futures
When the acquisition of SGI by HPE was announced last year, I was openly skeptical about HPE’s motives and wrote a blog post to discuss the potential reasons and implications behind this decision. I felt pretty certain that HPE would not retain both Superdome X (SD-X) and the high end SGI system, now called MC990 X as they play in the same space. I speculated that the MC990 X would not be the winner because it has inferior memory reliability technology and uses a hand wired matrix to interconnect different nodes compared to SD-X which uses a backplane to connect nodes to switches for connectivity to other nodes.
At SapphireNow this past week, I learned, from a couple of different sources, that Intel’s next generation platform, “Purely” using “Skylake” processors include so many significant changes that SD-X will very likely not be able to support them. Those changes include a new chip level interconnect technology “UltraPath” (UPI), as a follow on to QPI, faster memory and a larger footprint which, according to pictures available on the internet, appears to be at least 25% larger. This makes sense since the high end Skylake processor will have up to 28 cores versus Broadwell’s 24 cores and includes 25% more L3 cache but uses the same 14nm lithography. Though specs are limited, my sources also believe this new platform will require more power and cooling than the “Broadwell” chips used in SD-X today. SD-X cell boards are already pretty compact nodes, which made sense when they were trying to deliver a “converged” solution, but which means that any increase in footprint of any component could push these boards beyond their limits. Clearly, the UPI change alone would require HPE’s XNC2 cell controller to be redesigned even if no power or cooling limits were exceeded. Considering how few of these chips are used per year, it would be impractical for HPE to maintain a development program for both XNC2 (and the associated SX3000 switch) in addition to the SGI acquired NUMALink7 cell controllers. The speculation is therefore that SD-X will, in fact, be the loser and HPE will utilize the MC990 X as the go forward, high end platform.
If this turns out to be a correct conclusion, then a lot of very large customers are likely to be in for a major shock since they have purchased SD-X with the expectation of being able to upgrade it and add on like architecture systems for their SAP HANA workloads as they grow over time as well as for other large memory applications. Had they been informed about the upcoming obsolescence of SD-X, many of these customers may have made a different purchasing decision. I am not convinced they would have purchased the MC990 X instead for a variety of reasons, not the least of which is the lack of robust memory protection, available with SD-X, known as RAS, Lockstep or DDDC+ mode, but not with MC990 X. Lacking this, the failure of a chip on a memory DIMM could result in a HANA failure, system failure or the need to immediately shut down the system to diagnose and fix the failed DIMM. Most SAP customers running Suite on HANA or S/4HANA are unlikely to find this level of protection to be acceptable, especially when they are running massive HANA transactional systems.
On the subject of reliability, one of the highest exposures that any system has are the physical connections and the MC990 X has a lot of them to form the ccNUMA mesh. Each cable has a connector on each end which are then inserted into the appropriate port on each pair of node controllers. Any time a cable is inserted into any sort of receptacle, there is a possibility that it will fail due to the pressure required to insert it or corrosion. A system with 2 nodes has 8 such cables, 2 between the pair of controllers on each node and 4 between the nodes. A system with 5 nodes has approximately 50 such cables. While not necessarily catastrophic, a failure of one of these cables would require a planned outage as soon as possible as there would be a performance impact from the loss of the connection.
Another limitation of the MC990 X is the lack of virtualization technology. As each node in a MC990 X contains 4 sockets, the level of granularity of this system is so coarse that very poor utilization is likely to result and a massive increase in footprint will be required to handle the same set of workloads that might have been previously handled by SD-X. Physical partitioning is available, but based on one or more 4 socket nodes, such a waste as to be completely irrelevant.
If this speculation proves to be true, and HPE has not been sharing this roadmap with customers, then those customers have been convinced to invest in an obsolete platform and they should be furious. For those in the process of making a decision, they should be asking HPE the right questions about SD-X product futures and then taking that into account as well as the product weaknesses of MC990 X.
I should note that IBM has but one go forward architecture for Power Systems based on the on-chip interconnect technology that has been included and evolving since POWER4. Customers that invested in Power technology have seen a consistent 3 to 4 year cycle to the next generation with those who purchased high end models often having the option to upgrade from one generation to the next.
If you read my blog post last week, you may recall that I pointed out how HP was lying to customers about HANA on IBM Power Systems. Now we learn that they may also be doing the same sort of thing about their own products. Me thinks a trend is emerging here!?!?
HPE acquisition of SGI implications to HANA customers
Last week, HP announced their intention to acquire SGI for $275M, a 30% premium over its market cap prior to the announcement. This came as a surprise to most people, including me, both that HP would want to do this and how little SGI was worth, a fact that eWeek called “Embarrasing”, http://www.eweek.com/innovation/hpe-buys-sgi-for-275-million-how-far-the-mighty-have-fallen.html. This raises a whole host of questions that HANA customers might want to consider.
First and foremost, there would appear to be three possible reasons why HP made this acquisition, 1) eliminate a key competitor and undercut Dell and Cisco at the same time, 2) acquire market share, 3) obtain access to superior technology, resources and/or keep them out of the hands of a competitor or some combination of the above.
If HP considered SGI a key competitor, it means that HP was losing a large number of deals to SGI, a fact that has not been released publicly but which would imply that customers are not convinced of Superdome X’s strength in this market. As many are aware, both Dell and Cisco have resell agreements with SGI for their high end UV HANA systems. It would seem unlikely that either Dell or Cisco would continue such a relationship with their arch nemesis and as such, this acquisition will seriously undermine the prospects of Dell and Cisco to compete for scale-up HANA workloads such as Suite on HANA and S/4HANA among customers that may need more than 4TB, in the case of Dell, and 6TB, in the case of Cisco.
On the other hand, market share is a reasonable goal, but SGI’s total revenue in the year ending 6/26/15 was only $512M, which would barely be a rounding error on HP’s revenue of $52B for the year ending 10/31/15. Hard to imagine that HP could be this desperate for a potential 1% increase in revenue, assuming they had 0% overlap in markets. Of course, they compete in the same markets, so the likely revenue increase is considerably less than even that paltry number.
That brings us to the third option, that the technology of SGI is so good that HP wanted to get their hands on it. If that is the case, then HP would be admitting that the technology in Superdome X is inadequate for the demands of Big Data and Analytics. I could not agree more and made such a case in a recent post on this blog. In that post, I noted the latency inherent in HP’s minimum 8-hop round trip access to any off board resources (remote memory accesses adds another two hops), remember there are only two Intel processors per board in a Superdome X system which can accommodate up to 16 processors. Scale-up transactional workloads typically access data in rows dispersed across NUMA aligned columns, i.e. will constantly be traversing this high latency network. Of course, this is not surprising since the architecture used in this system is INCREDIBLY OLD having been developed in the early 2000s, i.e. way before the era of Big Data. But the surprising thing is that this would imply that HP believes SGI’s architecture is better than their own. Remember, SGI’s UV system uses a point to point, hand wired, 4-bit wide mesh of wires between every two NUMA ASICs in the system, which, for the potential 32-sockets in a single cabinet system means 16 ASICS and 136 wires, if I have done the math correctly. HP has been critical of the memory protection employed in systems like SGI’s UV which is based on SDDC (Single Device Data Correction). In HP’s own words about their DDDC+1 memory protection: “This technology delivers up to a 17x improvement in the number of DIMM replacements versus those systems that use only Single-Chip Sparing technologies. Furthermore, DDDC +1 significantly reduces the chances of memory related crashes compared to systems that only have Single-Chip Sparing capabilities” HP Integrity Superdome X system architecture and RAS Whitepaper. What is really interesting is that SDDC does not imply even Single-Chip Sparing.
The only one of those options which makes sense to me is the first one, but of course, I have no way of knowing which of the above is correct. One thing is certain; customers considering implementing HANA on a high-end scale-up architecture from either HP or SGI, and Dell and Cisco by extension, are going to have to rethink their options. HP has not stated which architecture will prevail or if they will keep both, hard to imagine but not out of the question either. Without concrete direction from HP, it is possible that a customer decision for either architecture could result in almost immediate obsolescence. I would not enjoy being in a board room of a company or meeting with my CFO to explain how I made a decision for a multi-million dollar solution which is dead-ended, worth 1/2 or less overnight and for which a complete replacement will be required sooner than later. Likewise, it would be hard to imagine an upcoming decision for a strategic system to run the company’s entire business based on a single flip of a coin.
Now I am going to sound like a commercial for IBM, sorry. There is an alternative which is rock solid, increasing, not decreasing in value, and has a strong and very clear roadmap, IBM Power Systems. POWER8 for HANA has been seeing one of the fastest market acceptance rates of any solution in recent memory with hundreds of customers implementing HANA on Power Systems and/or purchasing systems in preparation for such an implementation, ranging from medium sized businesses in high growth markets to huge brand names. The roadmap was revealed earlier this year at the OpenPower Summit, http://www.nextplatform.com/2016/04/07/ibm-unfolds-power-chip-roadmap-past-2020/. This roadmap was further backed up with Google’s announcement of their plans for POWER9, http://www.nextplatform.com/2016/04/06/inside-future-google-rackspace-power9-system/, the UK’s SFTC plans, http://www.eweek.com/database/ibm-powers-uk-big-data-research.html worth £313 (roughly $400M based on current exchange rates) and the US DOE’s decision to base its Summit and Sierra supercomputers on IBM POWER9 systems, http://www.anandtech.com/show/8727/nvidia-ibm-supercomputers, a $375M investment, either of which is worth more than the entire value of SGI interestingly enough. More importantly, these two major wins mean IBM is now contractually obligated to deliver POWER9 thereby ensuring the Power roadmap for a long time to come. And, of course, IBM has a long history of delivering mission critical systems to customers, evolving and improving the scaling and workload handling characteristics over time while simultaneously improving systems availability.
Lintel for SAP App Servers – The right choice
Or is it? Running SAP application servers on IBM Power Systems with Linux results in a lower TCA than using x86 systems with Linux and VMware. Usually, I don’t start a blog post with the conclusion, but was so amazed by the results of this analysis, that I could not help myself.
For several years now, I have seen many customers move older legacy app servers to x86 systems using Linux and VMware as well as implementing new SAP app servers on the same. When asked why, the answers boil down to cost, skills and standards. Historically, Lintel servers were not just perceived to cost less, but could the cost differences could be easily demonstrated. Students emerging from colleges have worked with Linux far more often than with UNIX and despite the fact that learning UNIX and how it is implemented in actual production environments is very little different in real effort/cost, the perception of Linux skills being more plentiful and consequently less expensive persists. Many companies and government entities have decided to standardize on Linux. For some legacy IBM Power Systems customers, a complicating factor, or perhaps a compelling factor in the analysis, has compared partitions on large enterprise class systems against low cost 2-socket x86 servers. And so, increasingly, enterprises have defaulted to Lintel as the app server of choice.
Something has changed however and is completely overturning the conventional wisdom discussed above. There is a new technology which takes advantage of all of those Linux skills on the market, obeys the Linux standards mandate and costs less than virtualized Lintel systems. What is this amazing new technology? Surprise, it is a descendent of the technology introduced in 1990 by IBM called the RS/6000, with new Linux only POWER8 systems. (OK, since you know that I am an IBM Power guy and I gave away the conclusion at the start of this post, that was probably not much of a surprise.) At least, this is what the marketing guys have been telling us and they have some impressive consultant studies and internal analyses that back up their claims.
For those of you who have been following this blog for a while, you know that I am skeptical of consultant analyses and even more so of internal analyses. So, instead of depending on those, I set out to prove, or disprove, this assertion. The journey began with setting reasonable assumptions. Actually, I went a little overboard and gave every benefit of the doubt to x86 and did the opposite for Power.
Overhead – The pundits, both internal and external, seem to suggest that 10% or more overhead for VMware is reasonable. Even VMware’s best practices guide for HANA suggests an overhead of 10%. However, I have heard some customers claim that 5% is possible. So, I decided to use the most favorable number and settled on 5% overhead. PowerVM does have overhead, but it is already baked into all benchmarks and sizings since it is built into the embedded hypervisor, i.e. it is there even when you are running a single virtual machine on a system.
Utilization – Many experts have suggested that average utilization of VMware systems range in the 20% to 30% range. I found at least one analyst that said that the best run shops can drive their VMware systems up to 45%. I selected 45%, once again since I want to give all of the benefit of the doubt to Lintel systems. By comparison, many experts suggest that 85% utilization is reasonable for PowerVM based systems, but I selected 75% simply to not give any of the benefit of the doubt to Power that I was giving to x86.
SAPS – Since we are talking about SAP app servers, it is logical to use SAP benchmarks. The best result that I could find for a 2 socket Linux Intel Haswell-EP system was posted by Dell @ 90,120 SAPS (1). A similar 2-socket server from IBM was posted @ 115,870 SAPS (2).
IBM has internal sizing tables, as does every vendor, in which it estimates the SAPS capacity of different servers based on different OSs. One of those servers, the Power S822L, a 20-core Linux only system, is estimated to be able to attain roughly 35% less SAPS than the benchmark result for its slightly larger cousin running AIX, but this takes into consideration differences in MHz, number of cores and small differences due to the compilers used for SAP Linux binaries.
For our hypothetical comparison, let us assume that a customer needs approximately the SAPS capacity as can be attained with three Lintel systems running VMware including the 5% overhead mentioned above, a sustained utilization of 45% and 256GB per server. Extrapolating the IBM numbers, including no additional PowerVM overhead and a sustained utilization of 75%, results in a requirement of two S822L systems each with 386GB.
Lenovo, HP and Dell all offer easy to use configurators on the web. I ran through the same configuration for each: 2 @ Intel Xeon Processor E5-2699 v3 18C 2.3GHz 45MB Cache 2133MHz 145W, 16 @ 16GB x4 DIMMS, 1 @ Dual-port 10GB Base-T Ethernet adapter, 2 @ 300GB 10K RPM disk (2 @ 1TB 7200 RPM for Dell) and 24x7x4 hour 3-year warranty upgrades (3). Both the Lenovo and HP sites show an almost identical number for RedHat Enterprise Linux with unlimited guests (Dell’s was harder to decipher since they apply discounts to the prices shown), so for consistency, I used the same price for RHEL including 3-yr premium subscription and support. VMware also offers their list prices on the web and the same numbers were used for each system, i.e. Version 5.5, 2-socket, premium support, 3yr (4).
The configuration for the S822L was created using IBM’s eConfig tool: 2 @ 10-core 3.42 GHz POWER8 Processor Card, 12 @ 32GB x4 DIMMS, 1 @ Dual-port 10GB Base-T Ethernet adapter, 2 @ 300GB 10K RPM disk and a 24x7x4 hour 3-year warranty upgrade, RHEL with unlimited guests and 3yr premium subscription and support and PowerVM with unlimited guests, 3yr 24×7 support (SWMA). Quick disclaimer; I am not a configuration expert with IBM’s products much less those from other companies which means there may be small errors, so don’t hold me to these numbers as being exact. In fact, if anyone with more expertise would like to comment on this post and provide more accurate numbers, I would appreciate that. You will see, however, that all three x86 systems fell in the same basic range, so small errors are likely of limited consequence.
The best list price among the Lintel vendors came in at $24,783 including the warranty upgrade. RHEL 7 came in at $9,259 and VMware @ $9,356 with a grand total for of $43,398 and for 3 systems, $130,194. For the IBM Power System, the hardware list was $33,136 including the warranty upgrade, PowerVM for Linux $10,450 and RHEL 7 $6,895 for a grand total of $51,109 and for 2 systems, $102,218.
So, for equivalent effective SAPS capacity, Lintel systems cost around $130K vs. $102K for Power … and this is before we consider the reliability and security advantages not to mention scalability, peak workload handling characteristics, reduced footprint, power and cooling. Just to meet the list price of the Power System, the Lintel vendors would have to deliver a minimum of 22% discount including RHEL and VMware.
Conclusions:
For customers making HANA decisions, it is important to note that the app server does not go away and SAP fully support heterogeneous configurations, i.e. it does not matter if the app server is on a different platform or even a different OS than the HANA DB server. This means that Linux based Power Boxes are the perfect companion to HANA DB servers regardless of vendor.
For customers that are refreshing older Power app servers, the comparisons can be a little more complicated in that there is a reasonable case to be made for running app servers on enterprise class systems potentially also housing database servers in terms of higher effective utilization, higher reliability, the ability to run app servers in an IFL (Integrated Facility for Linux) at very attractive prices, increased efficiencies and improved speeds through use of virtual Ethernet for app to DB communications. That said, any analysis should start with like for like, e.g. two socket scale-out Linux servers, and then consider any additional value that can be gained through the use of AIX (with active memory expansion) and/or enterprise class servers with or without IFLs. As such, this post makes a clear point that, in a worst case scenario, scale-out Linux only Power Systems are less expensive than x86. In a best case scenario, the TCO, reliability and security advantages of enterprise class Power Systems make the value proposition of IBM Power even more compelling.
For customers that have already made the move to Lintel, the message is clear. You moved for sound economic, skills and standards based reasons. When it is time to refresh your app servers or add additional ones for growth or other purposes, those same reasons should drive you to make a decision to utilize IBM Power Systems for your app servers. Any customer that wishes to pursue such an option is welcome to contact me, your local IBM rep or an IBM business partner.
Footnotes:
1. Dell PowerEdge R730 – 2 Processors / 36 Cores / 72 Threads 16,500 users, Red Hat Enterprise Linux 7, SAP ASE 16, SAP enhancement package 5 for SAP ERP 6.0, Intel Xeon Processor E5-2699 v3, 2.3 Ghz, 262,144MB, Cert # 2014033, 9/10/2014
2. IBM Power System S824, 4 Processors / 24 Cores / 192 Threads, 21,212 Users, AIX 7.1, DB2 10.5, SAP enhancement package 5 for SAP ERP 6.0, POWER8, 3.52 Ghz, 524,288MB, Cert # 2014016, 4/28/2014
3. https://www-01.ibm.com/products/hardware/configurator/americas/bhui/flowAction.wss?_eventId=launchNIConfigSession&CONTROL_Model_BasePN=5462AC1&_flowExecutionKey=_cF5B38036-BD56-7C78-D1F7-C82B3E821957_k34676A10-590F-03C2-16B2-D9B5CE08DCC9
http://configure.us.dell.com/dellstore/config.aspx?c=us&cs=04&fb=1&l=en&model_id=poweredge-r730&oc=pe_r730_1356&s=bsd&vw=classic
http://h71016.www7.hp.com/MiddleFrame.asp?view=std&oi=E9CED&BEID=19701&SBLID=&AirTime=False&BaseId=45441&FamilyID=3852&ProductLineID=431
High end Power Systems customers have a new option for SAP app servers that is dramatically less expensive than x86 Linux solutions
Up until recently, if you were expanding the use of your SAP infrastructure or have some older Power Systems that you were considering replacing with x86 Linux systems, I could give you a TCO argument that showed how you could see roughly equivalent TCO using lower end Power Servers. Of course, some people might not buy into all of the assumptions or might state that Linux was their new standard such that AIX was no longer an acceptable option. Recently, IBM made an announcement which has changed the landscape so dramatically that you can now obtain the needed capacity using high end server “dark cores” with Linux, not at an equivalent TCO, but at a dramatically lower TCA.
The new offering is called IFL which stands for Integrated Facility for Linux. This concept originated with System Z (aka mainframe) several years ago. It allows customers that have existing Power 770, 780 or 795 servers with capacity on demand “dark cores”, i.e. for which no workload currently runs and the license to use the hardware, virtualization and OS software have not been activated, to turn on a group of cores and memory specifically to be used for Linux only workloads. A Power IFL is composed of 4 cores with 32GB of memory and has a list price of $8,591.
In the announcement materials provided by IBM Marketing, an example is provided of a customer that would need to add the equivalent of 16 cores @ 80% utilization and 128GB of memory to an existing Power 780 4.4GHz system or would need the equivalent capacity using a 32-core HP DL560 2.7GHz system running at 60% utilization. They used SPECint_rate as the basis of this comparison. Including 3 year license for PowerVM, Linux subscription and support, 24×7 hardware maintenance and the above mentioned Power activations, the estimated street price would be approximately $39,100. By comparison, the above HP system plus Linux subscription and support, VMware vSphere and 24×7 hardware maintenance would come in at an estimated street price of approximately $55,200.
Already sounds like a good deal, but I am a skeptic, so I needed to run the numbers myself. I find SPECint_rate to be a good indicator of performance for almost no workloads and an incredibly terrible indicator of performance for SAP workloads. So, I took a different approach. I found a set of data from an existing SAP customer of IBM which I then used to extrapolate capacity requirements. I selected the workloads necessary to drive 16 cores of a Power 780 3.8GHz system @ 85% utilization. Why 85%? Because we, and independent sources such as Solitaire Interglobal, have data from many large customers that report routinely driving their Power Systems to a sustained utilization of 85% or higher. I then took those exact same workloads and modeled them onto x86 servers assuming that they would be virtualized using VMware. Once again, Solitaire Interglobal reports that almost no customers are able to drive a sustained utilization of 45% in this environment and that 35% would be more typical, but I chose a target utilization of 55% instead to make this as optimistic for the x86 servers as possible. I also applied only a 10% VMware overhead factor although many sources say that is also optimistic. It took almost 6 systems with each hosting about 3 partitions to handle the same workload as the above 16-core IFL pool did.
Once again, I was concerned that some of you might be even more optimistic about VMware, so I reran the model using a 65% target utilization (completely unattainable in my mind, but I wanted to work out the ultimate, all stars aligned, best admins on the planet, tons of time to tune systems, scenario) and 5% VMware overhead (I don’t know anyone that believes VMware overhead to be this low). With each system hosting 3 to 4 partitions, I was able to fit the workloads on 5 systems. If we just go crazy with unrealistic assumptions, I am sure there is a way that you could imagine these workloads fitting onto 4 systems.
Next, I wanted to determine the accurate price for those x86 systems. I used HP’s handy on-line ordering web site to price some systems. Instead of the DL560 that IBM Marketing used, I chose the DL360e Gen8 system, with 2@8-core 1.8GHz processors, 64GB of memory, a pair of 7200rpm 500GB hard drives, VMware Enterprise for 2 processors with 3 yr subscription, RH Enterprise Linux 2 socket/4 guest with 3 yr subscription, 3yr 24×7 ProCare Service and HP installation services. The total price comes to $27,871 which after an estimated discount of 25% on everything (probably not realistic), results in a street price of $20,903.
Let’s do the math. Depending on which x86 scenario you believe is reasonable, it either takes 6 systems at a cost of $125,419, 5 systems @ $104,515 or 4 systems @ $83,612 to handle the same load as a 4 IFL/16-core pool of partitions on a 780 at a cost of $39,100. So, in the most optimistic case for x86, you would still have to pay $44,512 more. It does not take a rocket scientist to realize that using Power IFLs would result in a far less expensive solution with far better reliability and flexibility characteristics not to mention better performance since communication to/from the DB servers would utilize the radically faster backplane instead of an external TCP/IP network.
But wait, you say. There is a better solution. You could use bigger x86 systems with more partitions on each one. You are correct. Thanks for bringing that up. Turns out, just as with Power Systems, if you put more partitions on each VMware system, the aggregate peaks never add up to the sum of the individual peaks. Using 32-core, DL560s @ 2.2GHz, 5% VMware overhead and 65% target utilization, you would only need 2 systems. I priced them on the HP web site with RH Linux 4 socket/unlimited guests 3yr subscription, VMware Enterprise 4 socket/3yr, 24×7 ProCare and HP installation service and found the price to be $70,626 per system, i.e. $141,252 for two systems, $105,939 after the same, perhaps unattainable 25% discount. Clearly, 2 systems are more elegant than 4 to 6, but still, this solution is still $66,839 more expensive than the IFL solution.
I started off to try and prove that IBM Marketing was being overly optimistic and ended up realizing that they were highly conservative. The business case for using IFLs for SAP app servers on an existing IBM high end system with unutilized dark cores compared to net new VMware/Linux/x86 systems is overwhelming. As many customers have decided to utilize high end Power servers for DB due to their reliability, security, flexibility and performance characteristics, the introduction of IFLs for app servers is almost a no-brainer.
Configuration details:
HP ProLiant DL360e Gen8 8 SFF Configure-to-order Server – (Energy Star)661189-ESC $11,435.00
HP ProLiant DL360e Gen8 Server
HP DL360e Gen8 Intel® Xeon® E5-2450L (1.8GHz/8-core/20MB/70W) Processor FIO Kit x 2
HP 32GB (4x8GB) Dual Rank x4 PC3L-10600 (DDR3-1333) Reg CAS-9 LP Memory Kit x 2
HP Integrated Lights Out 4 (iLO 4) Management Engine
HP Embedded B120i SATA Controller
HP 8-Bay Small Form Factor Drive Cage
HP Gen8 CPU1 Riser Kit with SAS Kit + SAS License Kit
HP 500GB 6G SATA 7.2K rpm SFF (2.5-inch) SC Midline 1yr Warranty Hard Drive x 2
HP 460W Common Slot Platinum Plus Hot Plug Power Supply
HP 1U Small Form Factor Ball Bearing Gen8 Rail Kit
3-Year Limited Warranty Included
3yr, 24×7 4hr ProCare Service $1,300.00
HP Install HP ProLiant $225.00
Red Hat Enterprise Linux 2 Sockets 4 Guest 3 Year Subscription 24×7 Support No Media Lic E-LTU $5,555.00
VMware vSphere Enterprise 1 Processor 3 yr software $4,678.00 x 2 = $9,356.00
DL360e Total price $27,871.00
ProLiant DL560 Gen8 Configure-to-order Server (Energy Star) 686792-ESC $29,364.00
HP ProLiant DL560 Gen8 Configure-to-order Server
HP DL560 Gen8 Intel® Xeon® E5-4620 (2.2GHz/8-core/16MB/95W) Processor FIO Kit
HP DL560 Gen8 Intel® Xeon® E5-4620 (2.2GHz/8-core/16MB/95W) Processor Kit x3
HP 16GB (2x8GB) Dual Rank x4 PC3L-10600 (DDR3-1333) Reg CAS-9 LP Memory Kit x 4
ENERGY STAR® qualified model
HP Embedded Smart Array P420i/2GB FBWC Controller
HP 500GB 6G SAS 7.2K rpm SFF (2.5-inch) SC Midline 1yr Warranty Hard Drive x 2
HP iLO Management Engine(iLO 4)
3 years parts, labor and onsite service (3/3/3) standard warranty. Certain restrictions and exclusions apply.
HP 3y 4h 24×7 ProCare Service $3,536.00
Red Hat Enterprise Linux 4 Sockets Unlimited Guest 3 Yr Subscription 24×7 Support No Media Lic E-LTU $18,519.00
VMware vSphere Enterprise 1 Processor 3 yr software $4,678.00 x 4 = $18,712.00
HP Install DL560 Service $495.00
DL560 Total price: $70,626.00
SAP performance report sponsored by HP, Intel and VMware shows startling results
Not often does a sponsored study show the opposite of what was intended, but this study does. An astute blog reader alerted me about a white paper sponsored by HP, VMware and Intel by an organization called Enterprise Strategy Group (ESG). The white paper is entitled “Lab Validation Report – HP ProLiant DL980, Intel Xeon, and VMware vSphere 5 SAP Performance Analysis – Effectively Virtualizing Tier-1 Application Workloads – By Tony Palmer, Brian Garrett, and Ajen Johan – July 2011.” The words that they used to describe the results are, as expected, highly complementary to HP, Intel and VMware. In this paper, ESG points out that almost 60% of the respondents to their study have not virtualized “tier 1” applications like SAP yet but expect a rapid increase in the use of virtualization. We can only assume that they only surveyed x86 customers as 100% of Power Systems customers are virtualized since the PowerVM hypervisor is baked into the hardware and firmware of every system and can’t be removed. Nevertheless, it is encouraging that customers are moving in the right direction and that there is so much potential for the increased use of virtualization.
ESG provided some amazing statistics regarding scalability. ESG probably does not realize just how bad this makes VMware and HP look, otherwise, they probably would not have published it. They ran an SAP ECC 6.0 workload which they describe as “real world” but for which they provide no backup as to what this workload was comprised of, so it is possible that a given customer’s workload may be even more intensive than the one tested. They ran a single VM with 4 vcpu, then 8, 16 and 32. They show both the number of users supported as well as the IOPS and dialog response time. Then, in their conclusions, they state that scaling was nearly linear. This data shows that when scaling from 4 to 32 cores, an 8x increase, the number of users supported increased from 600 to 3,000, a 5x increase. Put a different way, 5/8 = .625 or 62.5% scalability. Not only is this not even remotely close to linear scaling, but it is an amazing poor level of scalability. IOPS, likewise, increased from 140 to 630 demonstrating 56.3% scalability and response time went from .2 seconds to 1 second, which while respectable, was 5 times that of the 4 vcpu VM.
ESG also ran a non-virtualized test with 32 physical cores. In this test, they achieved only 4,400 users/943 IOPS. Remember, VMware is limited to 32 vcpu which works out to the equivalent of 16 cores. So, with twice the number of effective physical cores, they were only able to support 46.7% more users and 49.7% more IOPS. To make matters much worse, response time almost doubled to 1.9 seconds.
ESG went on to make the following statement: “Considering that the SAP workload tested utilized only half of the CPU and one quarter of the available RAM installed in the DL980 tested, it is not unreasonable to expect that a single DL980 could easily support a second virtualized SAP workload at a similarly high utilization level and/or multiple less intensive workloads driven by other applications.” If response time is already borderline poor with VMware managing only a single workload, is it reasonable to assume that response time will go up or down if you add a second workload? If IOPS are not even keeping pace with the poor scalability of vcpu, it is reasonable to assume that IOPS will all of a sudden start improving faster? If you have not tested the effect of running a second workload, is it reasonable to speculate what might happen under drastically different conditions? This is like saying that on a hot summer day, an air conditioner was able to maintain a cool temperature in a sunny room with half of the chairs occupied and therefore it is not “unreasonable” to assume that it could do the same with all chair occupied. That might be the case, but there is absolutely no supporting evidence to support such a speculation.
ESG further speculates that because this test utilized default values for BIOS, OS, SAP and SQL Server, performance would likely be higher with tuning. … And my car will probably go faster if I wash it and add air to the tires, but by how much?? In summary and I am paraphrasing, ESG says that VMware, Intel processors and HP servers are ready for SAP primetime providing reliability and performance while simplifying operations and lowering costs. Interesting that they talk about reliability yet they, once again, provide no supporting evidence and did not mention a single thing about reliability earlier in the paper other than to say that the HP DL980 G7 delivers “enhanced reliability”. I certainly believe every marketing claim that a company makes without data to back it up, don’t you?
There are three ways that you can read this white paper.
- ESG has done a thorough job of evaluating HP x86 systems, Intel and VMware and has proven that this environment can handle SAP workloads with ease
- ESG has proven that VMware has either incredibly poor scalability or high overhead or both
- ESG has limited credibility as they make predictions for which they have no data to support their conclusions
While I might question how ESG makes predictions, I don’t believe that they do a poor job at performance testing. They seem to operate like an economist, i.e. they are very good at collecting data but make predictions based on past experience, not hard data. When is the last time that economists correctly predicted market fluctuations? If they did, they would all be incredibly rich!
I think it would be irresponsible to say that VMware based environments are incapable of handling SAP workloads. On the contrary, VMware is quite capable, but there are significant caveats. VMware does best with small workloads, e.g. 4 to 8 vcpu, not with larger workloads e.g. 16 to 32 vcpu. This means if a customer utilizes SAP on VMware, they will need more and smaller images than they would on excellent scaling platforms like IBM Power Systems, which drives up management costs substantially and reduces flexibility. By way of comparison, published SAP SD 2-tier benchmark results for IBM Power Systems utilizing POWER7 technology show 99% scalability when comparing the performance of a 16-core to a 32-core system at the same MHz, 89.3% scalability when comparing a 64-core to a 128-core system with a 5% higher MHz, which when normalized to the same MHz shows 99% scalability even at this extremely high performance level.
The second caveat for VMware and HP/Intel systems is in that area that ESG brushed over as if it was a foregone conclusion, i.e. reliability. Solitaire Interglobal examined data from over 40,000 customers and found that x86 systems suffer from 3 times or more system outages when comparing Linux based x86 systems to Power Systems and up to 10 times more system outages when comparing Windows based x86 systems to Power Systems. They also found radically higher outage durations for both Linux and Windows compared to Power and much lower overall availability when looking at both planned and unplanned outages in general: http://ibm.co/strategicOS and specifically in virtualized environments: http://ibm.co/virtualizationplatformmatters. Furthermore, as noted in my post from late last year, https://saponpower.wordpress.com/2011/08/29/vsphere-5-0-compared-to-powervm/, VMware introduces a number of single points of failure when mission critical applications demand just the opposite, i.e. the elimination of single points of failure.
I am actually very happy to see this ESG white paper, as it is has proven how poor VMware scales for large workloads like SAP in ways that few other published studies have ever exposed. Power Systems continues to set the bar very high when it comes to delivering effective virtualization for large and small SAP environments while offering outstanding, mission critical reliability. As noted in https://saponpower.wordpress.com/2011/08/15/ibm-power-systems-compared-to-x86-for-sap-landscapes/, IBM does this while maintaining a similar or lower TCO when all production, HA and non-production systems, 3 years of 24x7x365 hardware maintenance, licenses and 24x7x365 support for Enterprise Linux and vSphere 5.0 Enterprise Plus … and that analysis was done back when I did not have ESG’s lab report showing how poorly VMware scales. I may have to revise my TCO estimates based on this new data.
HP Itanium takes another step down
HP Integrity family has taken hits from a wide range of software vendors ranging from RedHat, Microsoft, Oracle and now it appears that SAP is also getting on that bandwagon at last. In a blog called People, Process and Technology, http://peopleprocesstech.com/2012/06/05/why-the-hp-superdome-is-as-dead-as-a-dodo/, John Appleby wrote about the fact that SAP Business Objects is no longer supported on HP’s Integrity systems. As I don’t spend that much time with Business Objects, I was not aware of this, but was able to quickly verify that this was in fact correct by checking SAP’s PAM (Product Availability Matrix) which clearly states support for a variety of OSs including Windows, Red Hat and Suse Linux, AIX and Solaris with no mention of HP/UX. There is no such issue with Netweaver as of yet, but at this point, it is a matter of when, not if, SAP will also pull support in this area.
Systems based on HP/UX systems have provided very strong infrastructure solutions for SAP customer landscapes over many years. Those customers have come to expect high end features such as capacity on demand, partition isolation at a hardware level, very high scalability, a proven track record for handling very large database environments, reliability where availability in measured in the 99.9%+, robust high availability packages, the ability to add or remove processor boards without having to take down partitions or the system, a strong ecosystem of systems support software from the vendor and third party suppliers and a very extensive community of peer companies. Considering the inevitability of HP UNIX systems demise, current HP customers will have to figure out the best path going forward. As Solaris is declining almost as fast as HP/UX and Exadata is a very poor solution for most SAP environments, as noted in my previous blog entries, customers are left with only a few choices. They could consider Microsoft, but few large customers do which means they would be in an exclusive and largely untested large systems environment. Linux is another option, but once again the population of customers at the high end in the space is extremely small and many of the above characteristics are not available on the Intel platform. Only IBM Power Systems and its big brother, IBM System z, offer the sort of characteristics that HP customers have taken for granted for a long time.
In fact, for each of the above, Power Systems provide “plus” versions, e.g. instead of designing for 99.9% availability, they are designed for 99.99% availability. Instead of offering scalability to 128 cores, Power Systems scales to 256 cores and with DB2 PureScale, Power Systems can scale out well beyond those numbers. Partition isolation is available in the world of Power, not just for CPU level partitioning, i.e. npar/vpar. This is just a few of the areas in which IBM Power Systems deliver “plus” capabilities.
Although my peers from System z would say they offer “plus” capabilities over Power Systems, an assertion that I will not deny, most HP/UX customers feel that either Linux or another UNIX solution such as AIX are the only choices that they are willing to consider. In other words, the most logical and lowest risk path for large SAP customers moving forward from HP/UX systems is with IBM Power Systems.
IBM’s HANA solution
IBM’s implementation of an SAP HANA appliance is nothing short of a technological coup de grace over the competition. These are words that you would never normally see me write as I am not one to use superlatives, but in this case, they are appropriate. This is not to say that I am contradicting anything that I said in my posting last year where I was positioning when and where HANA is the right choice: https://saponpower.wordpress.com/2011/09/18/get-all-the-benefits-of-sap-hana-2-0-today-with-no-risk-and-at-a-fraction-of-the-cost/
As expected, HANA has been utilized for more applications and is about to GA for BW in the very near future. SAP reports brisk sales of systems and our experience echoes this, especially for proof of concept systems. Even though all of the warts of a V1.0 product have not been overcome, the challenges are being met by a very determined development team at SAP following a corporate mandate. It is only a matter of time before customers move into productive environments with more and more applications of HANA. That said, conventional databases and systems are not going away any time soon. Many systems such as ERP, CRM and SCM run brilliantly with conventional database systems. This is a tribute to the way that SAP delivered a strong architecture in the past. There are pieces of those systems which are “problem” areas and SAP is rapidly deploying solutions to fix those problems, often with HANA based point solutions, e.g. COPA. SAP envisions a future in which HANA replaces conventional databases, but not only are there challenges to be overcome, but there are simply not that many problems with databases based on DB2, Oracle, SQLserver or Sybase, not a compelling business case for change, as of yet.
Of course, I am not trying to suggest that HANA is not appropriate for some customers or that it does not deliver outstanding results in many cases. In many situations, the consolidation of a BW and BWA set of solutions makes tremendous sense. As we evolve into the future, HANA will make progressively more sense to more and more customers. And this brings me back to my original “superlative” laden statement. How does IBM deliver what none of the competitors do?
Simply put, (yes I know, I rarely put anything simply), IBM’s HANA appliance utilizes a single, high performance stack, regardless of how small or large a customer’s environment is. And with IBM’s solution, as demonstrated at Sapphire two weeks ago, 100TB across 100 nodes, is neither a challenge nor a limit of its architecture. By the way, Hasso Platner unveiled this solution in his key note speech at Sapphire and, surprisingly, he called out and commended IBM. Let us delve a little deeper and get a little less simple.
At the heart of the IBM solution is GPFS, the General Parallel File System. This is, strangely enough, a product of IBM Power Systems. It allows for striping of data among local SSD and HDD as well as spanning of data across clustered systems, replication of data, high availability and disaster recovery. On a single node system, utilizing the same number of drives, up to twice the IOPS of an ext3 file system can be expected with GPFS. When, not if, a customer grows and needs either larger systems or multiple systems, the file system stays the same and simply adapts to its different requirements. GFPS owes its roots to high performance computing. Customers trying to find the answer (not 42 for those of you geeky enough to understand what that means) to complex problems often required dozens, hundreds or even thousands of nodes and had to connect all of those nodes to a single set of data. As these solutions would run for often, ridiculous, period of time, sometimes counted in months or years even, the file system upon which they relied simply could not break no matter what the underlying hardware did. ASCI, the US Accelerated Strategic Computing Initiative focused on, among other things, simulating the effect of nuclear weapons storage decay, drove this requirement. The same requirements exist in many other “grand challenge” problems, whether they be Deep Blue or Watson and their ability to play games better than humans, or “simple” problems of unfolding DNA or figuring out how weather systems work or how airplane wings can fly more efficiently. More modestly, allowing thousands of scientists and engineers to collaborate using a single file system or thousands of individuals to access a file system without regard to location or the boundaries of the underlying storage technology, e.g. IBM SONAS, are results of this technology. Slightly less modestly, GPFS is the file system used by DB2 PureScale and, historically, Oracle RAC and even though Oracle ASM is now supported on all platforms, GPFS is still frequently used despite the fact that GPFS is an additional cost over ASM which is included with the RAC license. The outcome is an incredibly robust, fault resilient and consistent file system.
Why did I go down that rabbit hole? Because, it is important to understand that whether a customer utilizes one HANA node with a total of 128GB of memory or a thousand nodes with 2PB of memory, the technology does not have to be developed to support this, it has already been done with GPFS.
Now for the really amazing part of this solution: All drives, whether HDD or SSD are located within each of the nodes but through the magic of GPFS, are available to all nodes within the system. This means there is no SAN, no NAS, no NFS, no specialized switches, no gateways, just a simple 10GB/s Ethernet for GPFS to communicate amongst its nodes. Replication is built in so that the data and logs physically located on each node can be duplicated to one or more other nodes. This provides for an automatic HA of the file systems where, even if a node fails, all data can be accessed from other nodes and HANA can restart the failed node on a standby system.
Some other features of IBM’s implementation are worth note. IBM offers two different types of systems for HANA, the x3690 X5, a 1 or 2 socket system and the x3950 X5 system, a 2, 4 or 8 socket system. Either system may be used standalone or for scale up, but currently, the x3690 only is certified to scale to 4 nodes where the x3950 is certified to scale to 16 nodes. While it is not possible to mix and match these nodes in a scale out configuration, it is possible to utilize one in production and another in non-production, for example, as they have identical stacks. It is interesting to note another valuable feature. The x3950 is capable of scaling to 8 sockets, but customers don’t have to purchase an 8 socket system up front. This is because each “drawer” of a x3950 supports up to 4 sockets and a second drawer may be added, at any time, to upgrade a system to 8 sockets. Taken all together, an 8 socket standalone system costs roughly twice what a 4 socket standalone system does and a two node scale out implementation costs roughly twice what a single node standalone system does. For that matter, a 16 node scale out implementation costs 8 times what a 2 node scale out implementation does.
How does this compare with implementations from HP, Dell, Cisco, Fujitsu, Hitachi and NEC? For HP, a customer must choose whether to utilize a 4 or 8 socket systems and consequently must pay for a 4 or 8 node system up front as there is no path from the DL580 4 socket system to the DL980 8 socket system. Many more drives are required, e.g. a 128GB configuration requires 24@15K HDDs compared to an equivalent IBM solution which only requires 8@10K. Next, if one decides to start standalone and then move into a parallel implementation, one must move from a ext3 and xfs file systems to NFS. Unfortunately, HP has not certified either of those systems for scale out, so customers must move to the BL680c for scale out implementations, but the BL680c is only available as 4 socket/512GB nodes. A standalone implementation utilizes internal disks plus, frequently, a disk drawer in order to deliver the high IOPS that HANA requires, but a scale out implementation requires one HP P6500EVA for each 4 nodes and one HP X9300 Network Storage Gateway for every 2 nodes. The result is that not only does the stack change from standalone to scale out, but the systems change, the enclosure and interconnections change and as more nodes are added, complexity grows dramatically. Also, cost is not proportional but instead grows at an ever increasing rate as more nodes are added.
The other vendors’ implementations all share similar characteristics with HP’s with, of course, different types of nodes, file systems and storage architecture, e.g. Cisco uses EMC’s VNX5300 and MPFS parallel file system for scale-out.
At the time of writing this, only Cisco and Dell were not certified for the 1TB HANA configuration as SAP requires 8 sockets for this size system, based on SAP sizing rules, not based on limitations of the systems to support 1TB. Also, only IBM was certified for a scale out configuration utilizing 1TB nodes and up to 16 of those nodes.
The lead that IBM has over the competition is almost unprecedented. In fact, in the x86 space, I am not aware of this wide a gap at any time in IBM’s history. If you would like to get the technical details behind IBM’s HANA implementation, please visit http://IBM-SAP.com/HANA and to see a short video on the subject from Rich Travis, one of IBM’s foremost experts on HANA and the person that I credit with most of my knowledge of HANA, but none of the mistakes in this blog post, please visit: http://www.youtube.com/watch?v=Au-P28-oZvw&feature=youtu.be
SAPonPower 