I was delighted to read Hasso Plattner’s recent blog on the strengths of HANA on platforms using the Haswell-EX chip from Intel: https://blogs.saphana.com/2015/06/29/impact-of-haswell-on-hana/ In that blog, he did an excellent job of explaining how technical enhancements at a processor and memory subsystem level can result in dramatic improvement in the way that HANA operates, Now, I know what you are thinking; he likes what Dr. Plattner has to say about a competitor’s technology? Strange as it may seem, yes … in that he has pointed out a number of relevant features that, as good as Haswell-EX might be, POWER8 surpassed, even before Haswell-EX was announced.
All of these technical features and discussion are quite interesting to us propeller heads. Most business people, on the other hand, would probably prefer to discuss how to improve HANA operational characteristics, deliver flexibility to respond to changing business demands and meet end user SLAs including response time and continuous availability. This is where POWER8 really shines. With PowerVM at its core, Power Systems can be tailored to deliver capacity for HANA production to ensure consistent response time and peak load capacity during high demand times and allow other applications and partitions to utilize capacity unused by the HANA production partition. It can easily mix production with other production and non-production partitions. It features the ability to utilize shared network and SAN resources, if desired, to reduce datacenter cost and complexity. POWER8 delivers unmatched reliability by default, not as an option or a tradeoff against performance.
Regarding the technical features, Herr Dr. Plattner points out that Haswell-EX systems:
- Support up to 144 cores per system with 12TB of memory. POWER8 supports up to 196 cores and 16TB. Actually, this under estimates that actual memory on a POWER8 system which is actually up to 17.7TB but IBM includes the extra 1.7TB at no extra cost as hot spare chips, not available with Haswell-EX systems.
- Deliver L1, L2 and L3 cache size increases, which though he does not state, are, in fact, 32KB (16KB in enterprise RAS mode), 256KB and 45MB respectively, compared to POWER8’s 64KB, 512KB and 96MB respectively plus 128MB L4, not available with Haswell-EX systems.
- Introduces enhancements to vector processing via the new AVX2 instruction unit compared to POWER8’s dual VMX instruction units.
- Rely on local memory access for HANA performance which is absolutely true and underlines why POWER8, with up to 4 times more bandwidth to memory, is such a good fit for HANA.
- Feature TSX, Transactional Synchronization Extensions, to improve lock synchronization, an area that Power Systems has excelled at for decades. POWER8 was actually a bit earlier in the whole transactional memory area but was actually preceded by IBM Blue Gene/Q, another PowerPC based technology.
He concludes by pointing out that internal benchmarks are of limited value but then explains what they achieved with Haswell-EX. As these are not externally audited nor even published, it is hard to comment on their validity.
By comparison, SAP has only one certified benchmark for which HANA systems have been utilized called BW-EML. Haswell-EX cpus were used in the 2B row Dell PowerEdge 930 benchmark and delivered an impressive 172,450 Ad-hoc Navigation Steps/Hr . This is impressive in that it surpassed the previous IvyBridge based benchmark of 137,010 Ad-hoc Navigation Steps/Hr on the Dell PowerEdge R920, an increase of almost 26% which would normally be impressive if it weren’t for the fact that the system includes 20% more cores and 50% more memory. By comparison, POWER8 delivered 192,750 Ad-hoc Navigation Steps/Hr with the IBM Power Enterprise System 870 or 12% more performance with 45% fewer cores and 33% less memory resulting in twice the performance per core.
It would be ideal to run the SAP SD 3-tier benchmark against a system running Suite on HANA as that would do away with discussions of benchmarks that can’t be verified and/or may have limited relevance to a transactional environment typical of Suite on HANA. From what I understand, the current SD benchmark depends on an older version of SAP code which is not supported on HANA. I hope that SAP is able to update the benchmark test kit to enable this benchmark to be run on HANA as that would be far better than any sort of speculation. In the meantime, we can only rely on assertions without detail and external review or on decades of proven experience handling large scaling transactional environments with mission critical levels of availability not to mention a wide variety of audited benchmarks demonstrating this ability. Power Systems stands alone in this respect.
Dell PowerEdge 930: 172,450 Ad-hoc Navigation Steps/Hr using 4 processors / 72 cores / 144 threads, Intel Xeon Processor E7-8890 v3, 2.50 GHz, 1.5TB main memory, Certification #: 2015014
Dell PowerEdge R920: 137,010 Ad-hoc Navigation Steps/Hr on the, 4 processors / 60 cores / 120 threads, Intel Xeon Processor E7-4890 v2, 2.80 GHz, 1TB main memory, Certification #: 2014044
the IBM Power Enterprise System 870: 192,750 Ad-hoc Navigation Steps/Hr with, 4 processors / 40 cores / 320 threads, POWER8, 4.19 GHz, 1TB main memory, Certification #: 2015024
Almost two years ago, I speculated about the potential value of a HANA on Power solution. In June, 2014, SAP announced a Test and Evaluation program for Scale-up BW HANA on Power. That program shifted into high gear in October, 2014 and roughly 10 customers got to start kicking the tires on this solution. Those customers had the opportunity to push HANA to its very limits. Remember, where Intel systems have 2 threads per core, POWER8 has up to 8 threads per core. Where the maximum size of most conventional Intel systems can scale to 240 threads, the IBM POWER E870 can scale to an impressive 640 threads and the E880 system can scale to 1536 threads. This means that IBM is able to provide an invaluable test bed for system scalability to SAP. As SAP’s largest customers move toward Suite “4” HANA (S4HANA), they need to have confidence in the scalability of HANA and IBM is leading the way in proving this capability.
A Ramp-up program began in March with approximately 25 customers around the world being given the opportunity to have access to GA level code and start to build out BW POC and production environments. This brings us forward to the announcement by SAP this week @ SapphireNow in Orlando of the GA of HANA on Power. SAP announced that customers will have the option of choosing Power for their BW HANA platform, initially to be used in a scale-up mode and plans to support scale-out BW, Suite on HANA and the full complement of side-car applications over the next 12 to 18 months.
Even the most loyal IBM customer knows the comparative value of other BW HANA solutions already available on the market. To this end, IBM announced new “solution editions”. A solution edition is simply a packaging of components, often with special pricing, to match expectations of the industry for a specific type of solution. “Sounds like an appliance to me” says the guy with a Monty Python type of accent and intonation (no, I am not making fun of the English and am, in fact, a huge fan of Cleese and company). True, if one were to look only at the headline and ignore the details. In reality, IBM is looking toward these as starting points, not end points and most certainly not as any sort of implied limitation. Remember, IBM Power Systems are based on the concept of Logical Partitions using Power Virtualization Manager (PVM). As a result, a Power “box” is simply that, a physical container within which one or multiple logical systems reside and the size of each “system” is completely arbitrary based on customer requirements.
So, a “solution edition” simply defines a base configuration designed to be price competitive with the industry while allowing customers to flexibly define “systems” within it to meet their specific requirements and add incremental capability above that minimum as is appropriate for their business needs. While a conventional x86 system might have 1TB of memory to support a system that requires 768GB, leaving the rest unutilized, a Power System provides for that 768GB system and allows the rest of the memory to be allocated to other virtual machines. Likewise, HANA is often characterized by periods of 100% utilization, in support of instantaneous response time demanded of ad-hoc queries, followed by unfathomably long periods (in computer terms) of little to no activity. Many customers might consider this to be a waste of valuable computing resource and look forward to being able to harness this for the myriad of other business purposes that their businesses actually depend on. This is the promise of Power. Put another way, the appliance model results in islands of automation like we saw in the 1990s where Power continues the model of server consolidation and virtualization that has become the modus operandi of the 2000s.
But, says the pitchman for a made for TV product, if you call right now, we will double the offer. If you believe that, then you are probably not reading my blog. If a product was that good, they would not have to give you more for the same price. Power, on the other hand, takes a different approach. Where conventional BW HANA systems offer a maximum size of 2TB for a single node, Power has no such inherent limitations. To handle larger sizes, conventional systems must “scale-out” with a variety of techniques, potentially significantly increased costs and complexity. Power offers the potential to simply “scale-up”. Future IBM Power solutions may be able to scale-up to 4TB, 8TB or even 16TB. In a recent post to this blog, I explained that to match the built in redundancy for mission critical reliability of memory in Power, x86 systems would require memory mirroring at twice the amount of memory with an associated increase in CPU and reduction in memory bandwidth for conventional x86 systems. SAP is pushing the concepts of MCOS, MCOD and multi-tenancy, meaning that customers are likely to have even more of their workloads consolidated on fewer systems in the future. This will result in demand for very large scaling systems with unprecedented levels of availability. Only IBM is in position to deliver systems that meet this requirement in the near future.
Details on these solution editions can be found at http://www-03.ibm.com/systems/power/hardware/sod.html
In the last few days, IBM and other organizations have published information about the solution editions and the value of HANA on Power. Here are some sites worth visiting:
Press Release: IBM Unveils Power Systems Solutions to Support SAP HANA
Video: The Next Chapter in IBM and SAP Innovation: Doug Balog announces SAP HANA on POWER8
Case study: Technische Universität München offers fast, simple and smart hosting services with SAP and IBM
Video: Technische Universität München meet customer expectations with SAP HANA on IBM POWER8
Analyst paper: IBM: Empowering SAP HANA Customers and Use Cases
Article: HANA On Power Marches Toward GA
Selected SAP Press
ComputerWorld: IBM’s new Power Systems servers are just made for SAP Hana
eWEEK, IBM Launches Power Systems for SAP HANA
ExecutiveBiz, IBM Launches Power Systems Servers for SAP Hana Database System; Doug Balog Comments
TechEYE.net, IBM and SAP work together again
ZDNet: IBM challenges Intel for space on SAP HANA
Data Center Knowledge: IBM Stakes POWER8 Claim to SAP Hana Hardware Market
Enterprise Times: IBM gives SAP HANA a POWER8 boost
The Platform: IBM Scales Up Power8 Iron, Targets In-Memory
Also a planning guide for HANA on Power has been published at http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102502 .
Or is it? Running SAP application servers on IBM Power Systems with Linux results in a lower TCA than using x86 systems with Linux and VMware. Usually, I don’t start a blog post with the conclusion, but was so amazed by the results of this analysis, that I could not help myself.
For several years now, I have seen many customers move older legacy app servers to x86 systems using Linux and VMware as well as implementing new SAP app servers on the same. When asked why, the answers boil down to cost, skills and standards. Historically, Lintel servers were not just perceived to cost less, but could the cost differences could be easily demonstrated. Students emerging from colleges have worked with Linux far more often than with UNIX and despite the fact that learning UNIX and how it is implemented in actual production environments is very little different in real effort/cost, the perception of Linux skills being more plentiful and consequently less expensive persists. Many companies and government entities have decided to standardize on Linux. For some legacy IBM Power Systems customers, a complicating factor, or perhaps a compelling factor in the analysis, has compared partitions on large enterprise class systems against low cost 2-socket x86 servers. And so, increasingly, enterprises have defaulted to Lintel as the app server of choice.
Something has changed however and is completely overturning the conventional wisdom discussed above. There is a new technology which takes advantage of all of those Linux skills on the market, obeys the Linux standards mandate and costs less than virtualized Lintel systems. What is this amazing new technology? Surprise, it is a descendent of the technology introduced in 1990 by IBM called the RS/6000, with new Linux only POWER8 systems. (OK, since you know that I am an IBM Power guy and I gave away the conclusion at the start of this post, that was probably not much of a surprise.) At least, this is what the marketing guys have been telling us and they have some impressive consultant studies and internal analyses that back up their claims.
For those of you who have been following this blog for a while, you know that I am skeptical of consultant analyses and even more so of internal analyses. So, instead of depending on those, I set out to prove, or disprove, this assertion. The journey began with setting reasonable assumptions. Actually, I went a little overboard and gave every benefit of the doubt to x86 and did the opposite for Power.
Overhead – The pundits, both internal and external, seem to suggest that 10% or more overhead for VMware is reasonable. Even VMware’s best practices guide for HANA suggests an overhead of 10%. However, I have heard some customers claim that 5% is possible. So, I decided to use the most favorable number and settled on 5% overhead. PowerVM does have overhead, but it is already baked into all benchmarks and sizings since it is built into the embedded hypervisor, i.e. it is there even when you are running a single virtual machine on a system.
Utilization – Many experts have suggested that average utilization of VMware systems range in the 20% to 30% range. I found at least one analyst that said that the best run shops can drive their VMware systems up to 45%. I selected 45%, once again since I want to give all of the benefit of the doubt to Lintel systems. By comparison, many experts suggest that 85% utilization is reasonable for PowerVM based systems, but I selected 75% simply to not give any of the benefit of the doubt to Power that I was giving to x86.
SAPS – Since we are talking about SAP app servers, it is logical to use SAP benchmarks. The best result that I could find for a 2 socket Linux Intel Haswell-EP system was posted by Dell @ 90,120 SAPS (1). A similar 2-socket server from IBM was posted @ 115,870 SAPS (2).
IBM has internal sizing tables, as does every vendor, in which it estimates the SAPS capacity of different servers based on different OSs. One of those servers, the Power S822L, a 20-core Linux only system, is estimated to be able to attain roughly 35% less SAPS than the benchmark result for its slightly larger cousin running AIX, but this takes into consideration differences in MHz, number of cores and small differences due to the compilers used for SAP Linux binaries.
For our hypothetical comparison, let us assume that a customer needs approximately the SAPS capacity as can be attained with three Lintel systems running VMware including the 5% overhead mentioned above, a sustained utilization of 45% and 256GB per server. Extrapolating the IBM numbers, including no additional PowerVM overhead and a sustained utilization of 75%, results in a requirement of two S822L systems each with 386GB.
Lenovo, HP and Dell all offer easy to use configurators on the web. I ran through the same configuration for each: 2 @ Intel Xeon Processor E5-2699 v3 18C 2.3GHz 45MB Cache 2133MHz 145W, 16 @ 16GB x4 DIMMS, 1 @ Dual-port 10GB Base-T Ethernet adapter, 2 @ 300GB 10K RPM disk (2 @ 1TB 7200 RPM for Dell) and 24x7x4 hour 3-year warranty upgrades (3). Both the Lenovo and HP sites show an almost identical number for RedHat Enterprise Linux with unlimited guests (Dell’s was harder to decipher since they apply discounts to the prices shown), so for consistency, I used the same price for RHEL including 3-yr premium subscription and support. VMware also offers their list prices on the web and the same numbers were used for each system, i.e. Version 5.5, 2-socket, premium support, 3yr (4).
The configuration for the S822L was created using IBM’s eConfig tool: 2 @ 10-core 3.42 GHz POWER8 Processor Card, 12 @ 32GB x4 DIMMS, 1 @ Dual-port 10GB Base-T Ethernet adapter, 2 @ 300GB 10K RPM disk and a 24x7x4 hour 3-year warranty upgrade, RHEL with unlimited guests and 3yr premium subscription and support and PowerVM with unlimited guests, 3yr 24×7 support (SWMA). Quick disclaimer; I am not a configuration expert with IBM’s products much less those from other companies which means there may be small errors, so don’t hold me to these numbers as being exact. In fact, if anyone with more expertise would like to comment on this post and provide more accurate numbers, I would appreciate that. You will see, however, that all three x86 systems fell in the same basic range, so small errors are likely of limited consequence.
The best list price among the Lintel vendors came in at $24,783 including the warranty upgrade. RHEL 7 came in at $9,259 and VMware @ $9,356 with a grand total for of $43,398 and for 3 systems, $130,194. For the IBM Power System, the hardware list was $33,136 including the warranty upgrade, PowerVM for Linux $10,450 and RHEL 7 $6,895 for a grand total of $51,109 and for 2 systems, $102,218.
So, for equivalent effective SAPS capacity, Lintel systems cost around $130K vs. $102K for Power … and this is before we consider the reliability and security advantages not to mention scalability, peak workload handling characteristics, reduced footprint, power and cooling. Just to meet the list price of the Power System, the Lintel vendors would have to deliver a minimum of 22% discount including RHEL and VMware.
For customers making HANA decisions, it is important to note that the app server does not go away and SAP fully support heterogeneous configurations, i.e. it does not matter if the app server is on a different platform or even a different OS than the HANA DB server. This means that Linux based Power Boxes are the perfect companion to HANA DB servers regardless of vendor.
For customers that are refreshing older Power app servers, the comparisons can be a little more complicated in that there is a reasonable case to be made for running app servers on enterprise class systems potentially also housing database servers in terms of higher effective utilization, higher reliability, the ability to run app servers in an IFL (Integrated Facility for Linux) at very attractive prices, increased efficiencies and improved speeds through use of virtual Ethernet for app to DB communications. That said, any analysis should start with like for like, e.g. two socket scale-out Linux servers, and then consider any additional value that can be gained through the use of AIX (with active memory expansion) and/or enterprise class servers with or without IFLs. As such, this post makes a clear point that, in a worst case scenario, scale-out Linux only Power Systems are less expensive than x86. In a best case scenario, the TCO, reliability and security advantages of enterprise class Power Systems make the value proposition of IBM Power even more compelling.
For customers that have already made the move to Lintel, the message is clear. You moved for sound economic, skills and standards based reasons. When it is time to refresh your app servers or add additional ones for growth or other purposes, those same reasons should drive you to make a decision to utilize IBM Power Systems for your app servers. Any customer that wishes to pursue such an option is welcome to contact me, your local IBM rep or an IBM business partner.
1. Dell PowerEdge R730 – 2 Processors / 36 Cores / 72 Threads 16,500 users, Red Hat Enterprise Linux 7, SAP ASE 16, SAP enhancement package 5 for SAP ERP 6.0, Intel Xeon Processor E5-2699 v3, 2.3 Ghz, 262,144MB, Cert # 2014033, 9/10/2014
2. IBM Power System S824, 4 Processors / 24 Cores / 192 Threads, 21,212 Users, AIX 7.1, DB2 10.5, SAP enhancement package 5 for SAP ERP 6.0, POWER8, 3.52 Ghz, 524,288MB, Cert # 2014016, 4/28/2014
Companies that plan on running Business Suite on HANA (SoH) require systems that are at least as fault tolerant as their current mission critical database systems. Actually, the case can be made that these systems have to exceed current reliability design specifications due to the intrinsic conditions of HANA, most notably, but not limited to, extremely large memory sizes. Other factors that will further exacerbate this include MCOD, MCOS, Virtualization and the new SPS09 feature, Multi-Tenancy.
A customer with 5TB of data in their current uncompressed Suite database will most likely see a reduction due to HANA compression (SAP note 1793345, and the HANA cookbook²) bringing their system size, including HANA work space, to roughly 3TB. That same customer may have previously been using a database buffer of 100GB +/- 50GB. At a current buffer size of 100GB, their new HANA system will require 30 times the amount of memory as the conventional database did. All else being equal, 30x of any component will result in 30x failures. In 2009, Google engineers wrote a white paper in which they noted that 8% of DIMMS experienced errors every year with most being hard errors and that when a correctable error occurred in a DIMM, there was a much higher chance that another would occur in that same DIMM leading, potentially, to uncorrectable errors.¹ As memory technology has not changed much since then, other than getting denser which could lead to even more likelihood of errors due to cosmic rays and other sources, the risk has likely not decreased. As a result, unless companies wish to take chances with their most critical asset, they should elect to use the most reliable memory available.
IBM provides exactly that, the best of breed open systems memory reliability, not as an option at a higher cost, but included with every POWER8 system, from the one and two socket scale-out systems to even more advanced capabilities with the 4 & 8-socket systems, some of which will scale to 16-sockets (announced as a Statement of Direction for 2015). This memory protection is represented in multiple discreet features that work together to deliver unprecedented reliability. The following gets into quite a bit of technical detail, so if you don’t have your geek hat on, (mine can’t be removed as it was bonded to my head when I was reading Heinlein in 6th grade; yes, I know that dates me), then you may want to jump to the conclusions at the end.
Chipkill – Essentially a RAID like technology that spans data and ECC recovery information across multiple memory chips such that in the event of a chip failure, operations may continue without interruption. Using x8 chips, Chipkill provides for Single Device Data Correction (SDDC) and with x4 chips, provides Double Device Data Correction (DDDC) due to the way in which data and ECC is spread across more chips simultaneously.
Spare DRAM modules – Each rank of memory (4 ranks per card on scale-out systems, 8 ranks per card on enterprise systems) contains an extra memory chip. This chip is used to automatically rebuild the data that was held, previously, on the failed chip in the above scenario. This happens transparently and automatically. The effect is two-fold: One, once the recovery is complete, no additional processing is required to perform Chipkill recovery allowing performance to return to pre-failure levels; Two, maintenance may be deferred as desired by the customer as Chipkill can, yet again, allow for uninterrupted operations in the event of a second memory chip failure and, in fact, IBM does not even make a call out for repair until a second chip fails.
Dynamic memory migration and Hypervisor memory mirroring – These are unique technologies only available on IBM’s Enterprise E870 and E880 systems. In the event that a DIMM experiences errors that cannot be permanently corrected using sparing capability, the DIMM is called out for replacement. If the ECC is capable of continuing to correct the errors, the call out is known as a predictive callout indicating the possibility of a future failure. In such cases, if an E870 or E880 has unlicensed or unassigned DIMMS with sufficient capacity to handle it, logical memory blocks using memory from a predictively failing DIMM will be dynamically migrated to the spare/unused capacity. When this is successful this allows the system to continue to operate until the failing DIMM is replaced, without concern as to whether the failing DIMM might cause any future uncorrectable error. Hypervisor memory mirroring is a selective mirroring technology for the memory used by the hypervisor which means that even a triple chip failure in a memory DIMM would not affect the operations of the hypervisor as it would simply start using the mirror.
L4 cache – Instead of conventional parity or ECC protected memory buffers used by other vendors, IBM utilizes special eDRAM (a more reliable technology to start with) which not only offers dramatically better performance but includes advanced techniques to delete cache lines for persistent recoverable and non-recoverable fault scenarios as well as to deallocate portions of the cache spanning multiple cache lines.
Extra memory lane – the connection from memory DIMMs or cards is made up of dozens of “lanes” which we can see visually as “pins”. POWER8 systems feature an extra lane on each POWER8 chip. In the event of an error, the system will attempt to retry the transfer, use ECC correction and if the error is determined by the service processor to be a hard error (as opposed to a soft/transient error), the system can deallocate the failing lane and allocate the spare lane to take its place. As a result, no downtime in incurred and planned maintenance may be scheduled at a time that is convenient for the customer since all lanes, including the “replaced” one are still fully protected by ECC.
L2 and L3 Caches likewise have an array of protection technology including both cache line delete and cache column repair in addition to ECC and special hardening called “soft latches” which makes these caches less susceptible to soft error events.
As readers of my blog know, I rarely point out only one side of the equation without the other and in this case, the contrast to existing HANA capable systems could not be more dramatic making the symbol between the two sides a very big > symbol; details to follow.
Intel offers a variety of protection technologies for memory but leaves the decision as to which to employ up to customers. This ranges from “performance mode” which has the least protection to “RAS mode” which has more protection at the cost of reduced performance.
Let’s start with the exclusives for IBM: eDRAM L4 cache with its inherent superior protection and performance over conventional memory buffer chips, dynamic memory migration and hypervisor memory mirroring available on IBM Enterprise class servers, none of which are available in any form on x86 servers. If these were the only advantages for Power Systems, this would already be compelling for mission critical systems, but this is only the start:
Lock step – Intel included similar technology to Chipkill in all of their chips which they call Lock step. Lock step utilizes two DIMMs behind a single memory buffer chip to store a 64-byte cache line + ECC data instead of the standard single DIMM to provide 1x or 2x 8-bit error detection and 8-bit error correction within a single x8 or x4 DRAM respectively (with x4 modules, this is known as Double Device Data Correction or DDDC and is similar to standard POWER Chipkill with x4 modules.) Lock Step is only available in RAS mode which incurs a penalty relative to performance mode. Fujitsu released a performance white paper³ in which they described the results of a memory bandwidth benchmark called STREAM in which they described Lock step memory as running at only 57% of the speed of performance mode memory.
Lock step is certainly an improvement over standard or performance mode in that most single device events can be corrected on the fly (and two such events serially for x4 DIMMS) , but correction incurs a performance penalty above and beyond that incurred from being in Lock step mode in the first place. After the first such failure, for x8 DIMMS, the system cannot withstand a second failure in that Lockstep pair of DIMMS and a callout for repair (read this as make a planned shutdown as soon as possible) be made to prevent a second and fatal error. For x4 DIMMS, assuming the performance penalty is acceptable, the planned shutdown could be postponed to a more convenient time. Remember, with the POWER spare DRAMS, no such immediate action is required.
Memory sparing – Since taking an emergency shutdown is unacceptable for a SoH system, Lock Step memory is therefore insufficient since it handles only the emergency situation but does not eliminate the need for a repair action (as the POWER memory spare does) and it incurs a performance penalty due to having to “lash” together two cards to act as one (as compared to POWER that achieves superior reliability with a single memory card). Some x86 systems offer memory sparing in which one rank per memory channel is configured as a spare. For instance, with the Lenovo System x x3850, each memory channel supports 3 DIMMs or ranks. If sparing is used, the effective memory throughput of the system is reduced by 1/3 since one of every 3 DIMMs is no longer available for normal operations and the memory that must be purchased is increased by 50%. In other words, 1TB of usable memory requires 1.5TB of installed memory. The downsize of sparing is that it is a predictive failure technology, not a reactive one. According to the IBM X6 Servers: Technical Overview Redbook- “Sparing provides a degree of redundancy in the memory subsystem, but not to the extent of mirroring. In contrast to mirroring, sparing leaves more memory for the operating system. In sparing mode, the trigger for failover is a preset threshold of correctable errors. When this threshold is reached, the content is copied to its spare. The failed rank is then taken offline, and the spare counterpart is activated for use.” In other words, this works best when you can see it coming, not after a part of the memory has failed. When I asked a gentleman manning the Lenovo booth at TechEd && d-code about sparing, he first looked at me as if I had a horn sticking out of my head and then replied that almost no one uses this technology. Now, I think I understand why. This is a good option, but at a high cost and still falls short of POWER8 memory protection which is both predictive and reactive and dynamically responds to unforeseen events. By comparison, memory sparing requires a threshold to be reached and then enough time to be available to complete a full rank copy, even if only a single chip is showing signs of imminent failure.
Memory mirroring – This technology utilizes a complete second set of memory channels and DIMMs to maintain a second copy of memory at all times. This allows for a chip or an entire DIMM to fail with no loss of data as the second copy immediately takes over. This option, however, does require that you double the amount of memory in the system, utilize plenty of system overhead to keep the pairs synchronized and take away ½ of the memory bandwidth (the other half of which goes to the copy). This option may perform better than the memory sparing option because reads occur from both copies in an interleaved manner, but writes have to occur to both synchronously.
Memory mirroring for x86 systems is the closest option to the continuous memory availability that POWER8 delivers. Of course, having to purchase 2TB of memory in order to have proper protection of 1TB of effective memory adds a significant cost to the system and takes away substantial memory bandwidth. HANA utilizes memory as few other systems do.
The problem is that x86 vendors won’t tell customers this. Why? Now, I can only speculate, but that is why I have a blog. The x86 market is extremely competitive. Most customers ask multiple vendors to bid on HANA opportunities. It would put a vendor at a disadvantage to include this sort of option if the customer has not required it of all vendors. In turn, x86 vendors don’t won’t to even insinuate that they might need such additional protection as that would imply a lack of reliability to meet mission critical standards.
So, let’s take this to the next logical step. If a company is planning on implementing SoH using the above protection, they will need to double their real memory. Many customers will need 4TB, 8TB or even some in the 12TB to 16TB range with a few even larger. For the 4TB example, an 8TB system would be required which, as of the writing of this blog post, is not currently certified by SAP. For the 8TB example, 16TB would be required which exceeds most x86 vendor’s capabilities. At 12TB, only two vendors have even announced the intention of building a system to support 24TB and at 16TB, no vendor has currently announced plans to support 32TB of memory.
Oh, by the way, Fujitsu, in the above referenced white paper, measured the memory throughput of a system with memory mirroring and found it to be 69% that of a performance optimized system. Remember, HANA demands extreme memory throughput and benchmarks typically use the fastest memory, not necessarily the most reliable meaning that if sizings are based on benchmarks, they may require adjustment when more reliable memory options are utilized. Would larger core counts then be required to drive the necessary memory bandwidth?
Clearly, until SAP writes new rules to accommodate this necessary technology or vendors run realistic benchmarks showing just how much cpu and memory capacity is needed to support a properly mirrored memory subsystem on an x86 box, customers will be on their own to figure out what to do.
That guess work will be removed once HANA on Power GAs as it already includes the mission critical level of memory protection required for SoH and does so without any performance penalty.
Many thanks to Dan Henderson, IBM RAS expert extraordinaire, from whom I liberally borrowed some of the more technically accurate sentences in this post from his latest POWER8 RAS whitepaper¹¹ and who reviewed this post to make sure that I properly represented both IBM and non-IBM RAS options.
A curious reader just posed some questions that I suspect many more of you also have. Here are the questions posed, and my answers:
“I remember at a trade show some years ago asking on the IBM stand how you ran Linux on a mainframe. I was told that whilst the SLES distribution had to be recompiled you could actually run compiled x86 Linux binaries on the mainframe. I thought that was pretty clever as getting all the ISVs to recompile for Linux on a mainframe would be a nightmare.
Coming to Linux on Power the web site is unclear whether you can run compiled x86 Linux binaries on Power. I suspected that the PowerVM hypervisor may be able to emulate the Intel instructions and run the x86 binaries, but is isn’t very clear.”
Both RedHat and SUSE Linux are operating systems which have been compiled to run on the Power platform. The vast majority of the operating system code is identical across Power and x86 versions with only the low level code that directly interacts with the hardware being unique to the specific platform. Currently, both run in big endian (BE) mode (byte order on disk and memory as opposed to x86 systems which run in little endian (LE); no positive or negative effect for either, simply a design choice). As such, most applications running in those environments today require are compiled natively, not running with any emulation. IBM did offer a “binary translation” function in 2011 called Lx86 which allowed x86 Linux applications to run unmodified on Linux on Power, but it was not widely adopted and was later removed from marketing in 2013. In May, 2014, IBM announced that it would support the software based KVM as an alternative to the hardware/firmware hypervisor. This allows customers that want to have a single set of administrative tools and skills to utilize KVM on both Power and x86 environment. It also enables more OSs to run on Power as with KVM, the system may be run in LE mode. Canonical Ubuntu is now available on Power and is the first Linux OS to run in LE mode. Both RedHat and SUSE are also available to run under KVM, however they currently run only in BE mode with SUSE planning on delivering a LE version (SLES 12) in the near future. Debian and openSUSE are also reportedly working on LE versions for the Power platform. Currently, LE is only supported under KVM and the entire system must run in the same mode. In the future, IBM plans on supporting mixed mode allowing some partitions to run in one mode and others to run in the other mode, as well as allowing LE partitions to run under PowerVM. Please read Jeff Scheel’s blog if you would like to know more about this subject: https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/just_the_faqs_about_little_endian?lang=en
“And that brings me nicely to HoP. Is the HANA code recompiled or can it take advantage of some form of emulation? ”
One of the key attributes of HANA is its incredible performance. As such, even if it were possible to run with emulation, it would defeat the purpose of HANA to run in any sort of degraded mode. One of the ways that HANA delivers its speed is through the use of SIMD, (Single Instruction Multiple Data – http://www.saphana.com/community/blogs/blog/2014/03/19/hana-meets-the-sims-simd-simplicity-and-simulation). On the Intel platform, SIMD is called by the SSE instruction set and is a single pipeline vector unit within each processor core. IBM offers a similar vector unit within each of the Power cores, called Altivec, and which now supports dual pipeline vector operations. Each type of unit is utilized by HANA in the same way, but requires platform specific code. As such, emulation would not allow SSE based code to work even in emulation mode on an Alitvec based system. HANA was originally coded for SSE based operations in LE mode on the Intel platform. SAP has modified their code to support Altivec based operations in BE mode on the Power platform which was subsequently compiled to run on the Power platform under PowerVM natively.
IBM usually has a significant presence at TechEd && d-code and this year is not different. Of course, there will be the usual array of experts ranging from Systems and Software to HANA, Cloud and Consulting services at our booth. In addition, IBM as well as some of our best customers will be hosting many sessions:
Thursday, October 23, 10:30 AM – 11:30 AM
Session ID MOB105: – Bellini 2105 Level 2
Title: Apple + IBM: Evolving to the SAP Enabled Individual Enterprise
Speaker: Scott Geddes
Description: What’s next, now that you’ve done your first waves of transformation with SAP? How do you empower end users in ways never possible before and unleash the power of our SAP implementation? In this session we will explore how Apple + IBM are working together to change the way people work and create new, never before seen capabilities.
EXPERT NETWORKING SESSION:
Thursday, October 23
12:00pm – 1:00pm Lounge #3
Apple + IBM: Evolving to the SAP Enabled Individual Enterprise (IBM and Apple alliance discussion cont’d)
Scott Geddes, IBM SAP Global Business Services – Mobility
Chuck Kichler, IBM SAP iCoC CTO
Tuesday, October 21, 2:00 PM – 3:00 PM
Session ID: DMM137
Title: IBM’s Recommended Approach for Optimizing Different Kinds of SAP Workloads
Speaker: Guersad Kuecuek
Description: Today, customers face various requirements to effectively deal with different kinds of workloads. Key aspects are high Service Level Agreements while maintaining optimal performance for analytical (OLAP) and transactional (OLTP) workloads. Find out how customers like Audi, Balluff, and Coca-Cola have mastered these challenging requirements.
Tuesday, October 21, 3:15 PM – 4:15 PM
Session ID: DMM142
Title: SAP HANA on IBM Power – Value, Performance and Experience
Speaker: Alfred Freudenberger
Description: With the announcement of the testing and evaluation program for SAP HANA on IBM Power Systems at SAPPHIRE NOW in 2014, a new option for SAP HANA deployments will soon be available. Why should SAP clients consider this option? For which environments is it well-suited? What have IBM and SAP learned during development, testing, and evaluation?
EXPERT NETWORKING SESSION:
Wednesday, October 22
11:30am – 12:00pm Lounge #4
SAP HANA on IBM Power – Value, Performance and Experience
Alfred Freudenberger, IBM Leader NA SAP on Power
Tuesday, October 21, 4:30 PM – 5:30 PM
Session ID: DMM145
Title: Simplify IBM Database Performance Tuning with the DBA Cockpit
Speaker: Thomas Rech
Description: In today’s IT world, it is crucial to maintain high SAP system performance to meet demanding Service Level Agreements. The DBA Cockpit for IBM DB2 Linux, Unix, and Windows is an easy, fully integrated solution for database monitoring and administration with SAP. Learn about the design concept, the capabilities, and discuss customer use cases.
Wednesday, October 22, 11:45 AM – 12:45 PM
Session ID ITM220
Title: Business Continuity for SAP HANA-Based Applications – Shared Experiences
Speaker: Irene Hopf
Description: Learn about the options to keep business continuously running when you migrate SAP application landscapes to SAP HANA. High availability and disaster recovery are essential for business-critical applications. Discuss experiences with your peers and learn how other customers have implemented it.
Wednesday, October 22, 5:45 PM – 6:45 PM
Session ID INT206
Title: Integrating Shop-Floor with Enterprise in Real-Time – SAP MII In Action
Speaker: Dipankar Saha
Description: How to integrate heterogeneous shop-floor systems with SAP ERP by SAP Manufacturing Integration and Intelligence (SAP MII) using custom frameworks with various industry case-studies. This includes: manufacturing integration use cases, real-time integration using SAP MII, and architecture and case studies of integration using the frameworks.
Thursday, October 23, 8:00 AM – 9:00 AM
Session ID UXP117
Title: Experience with Google Glass and Business Applications
Speaker: Markus van Kempen
Description: Google Glass presents a mobile form-factor which allows for new possibilities. This session discusses examples of user experiences, including the disconcerting experience of “wearing” a camera all the time, reactions from others, and navigation challenges. We show how to design for Google Glass and demonstrate business applications.
Thursday, October 23, 10:45 AM – 11:45 AM
Session ID ITM235
Title: Establishing Architectural Patterns for SAP in the Cloud at CokeOne +
Speaker: Michael Ryan
Description: The CokeOne + migration to cloud for their non-production SAP environments included the establishment of architectural patterns to take advantage of the services provided by cloud computing. This session focuses on establishing the architectural patterns needed to transform businesses by moving business systems and processes to a cloud model.
Thursday, October 23, 2:00 PM – 3:00 PM
Session ID DMM127
Title: Streamline SAP HANA Solution with Near-Line Storage Solution by PBS and IBM
Speaker: Elke Hartmann-Bakan
Description: Streamline your SAP HANA solution by keeping only hot data in memory and moving warm data to near-line storage (NLS). This allows you to maintain a lean SAP HANA database and sustain high performance. The PBS and IBM NLS solution offers near real-time speed on NLS, ultra fast load time from the online database to the NLS, and extreme compression.
Thursday, October 23, 4:30 PM – 5:30 PM
Session ID ITM123
Title: Planning Your Company’s SAP Systems Migration to the Cloud
Speaker: Michael Ryan
Description: The opportunity to move the SAP infrastructure to cloud is a game changer. Businesses are offered a level of speed and agility that has not been available in the past. However, moving to cloud does not solve basic issues that we experience in the IT world. We take a look at some of the key issues and think about the impact across enterprises.
EXPERT NETWORKING SESSION
Tuesday, October 21
2:30pm – 3:00pm Lounge #4
SAP Applications on IBM Cloud – from self-service to fully managed
Keith Murray, Global Offerings Manager SAP on SoftLayer, IBM SmartCloud Services
Wolfgang Knobloch, IBM GTS Global Offering Manager, SAP Cloud Offerings
I look forward to seeing you there.
IBM had a by-invitation only IBM i day prior to the start of TechEd in years past. This year, the event has been expanded to all Power customers and is no longer by invitation only. Please read on for the details on this event.
Register now and reserve your seat for our IBM POWER Technology for SAP Day (Session and Dinner) at TechEd && d-code 2014 in Las Vegas.
IBM and SAP experts will provide the latest news and solution updates for AIX, IBM i and POWER Linux for SAP environments. There will also be an overview of the new SAP HANA on POWER solution which was announced at SAPPHIRE NOW.
Don’t miss this opportunity to have your questions answered by experts and to engage in lively discussions with your peers. The business meeting will be followed by a group dinner.
Please note: You do not need a badge to attend.
The Venetian/Palazzo Congress Center
3355 Las Vegas Boulevard South
Las Vegas, NV 89109
Room # Veronese 2404
Monday, October 20, 2014
Refreshments & Registration: 1:00pm – 1:30pm
Welcome: 1:30pm – 2:00pm
IBM POWER Update: 2:00pm – 3:00pm
Refreshment break: 3:00pm – 3:15pm
AIX/POWER Linux breakout session: 3:15pm – 4:45pm (Veronese meeting room # 2404)
IBM i breakout session: 3:15pm – 4:45pm (Titian meeting room #2201B)
Refreshment break: 4:45pm – 5:00pm
Preview of “IBM Hot Topics”
at SAP TechEd && d-code: 5:00pm – 5:15pm
Q & A: 5:15pm – 5:45pm
Reception/Dinner: 6:00pm – 9:00pm
Register today to reserve your seat! Seating is limited.
At SAPPHIRE NOW 2014 in Orlando, Bernd Leukert, (member of the SAP Executive Board with global responsibility for development and delivery of all products) made an announcement for which most of the community of IBM Power Systems customers, sellers and integrators have been waiting for a long time. He announced the testing and evaluation program for HANA on IBM Power Systems (HoP). As SAP has done with virtually every product they have introduced in the past, they will work with a select group of customers that have the interest, skills, experience and willingness to dedicate the time to thoroughly put a product through its paces to ensure that it will deliver the value necessary to customers, operate efficiently and without error and perform at an acceptable level. Often, this is called Ramp-up and usually, unless critical problems are found, it is followed by an announcement of General Availability (GA). During this announcement, Mr. Leukert mentioned that customers are welcome to request a full disclosure of SAP’s plans to support HANA on Power.
Many people came through our booth, joined us in meetings and used other channels to ask one simple question: “Once HoP reaches GA, why should a customer prefer this solution over one from the plethora of x86 based vendors. I will boil down this subject into 5 key points: 1) Performance, 2) Reliability, 3) Flexibility, 4) Efficiency and 5) Cost
1) Performance – HANA, as most know, is an in-memory database. It replaces complex and cumbersome indexes and aggregates with highly compressed columns of data which can be traversed very quickly (full column scans, in effect) and in ways that don’t have to be considered ahead of time by architects and DBAs. The speed comes at the cost of memory and can be limited by the bandwidth of the memory subsystem in addition to the throughput and number of processor cores in a system. IBM recently announced 1 and 2 socket POWER8 systems capable of supporting a maximum of 1TB of memory (with both sockets populated). A radically faster backplane and up to 128MB of L4 cache (with all memory slots populated) result in a maximum bandwidth of 384GB/s. This compares to a two socket IvyBridge EX system (with 30 cores) which has a maximum bandwidth of 68GB/s in full RAS mode (85GB/s in the benchmark only, no sane customer would ever run HANA with partial RAS mode). In other words, the 2-socket Power System can deliver 5.6 times the memory bandwidth. Not bad, but not the whole story. High memory throughput without high CPU performance results in a simple shifting of bottlenecks.
Each POWER8 core supports a robust cache hierarchy with twice the L1, L2 and L3 compared to Ivy Bridge systems (in full RAS mode), up to 8 simultaneous threads (SMT8) and dual SIMD (Single Instruction Multiple Data) pipelines, compared to 2 threads (Hyperthreading) on IvyBridge EX and a single SIMD pipeline. HANA takes full advantage of lots of cache, threads and SIMD pipelines.
As HoP is not yet GA, no formal benchmarks are available. That said, the Power S824 (POWER8, 2 socket with 24 cores, 4U system, Cert # 2014016) running AIX achieved 21,212 Users, 116,727 SAPS on the SD 2-tier benchmark. Not bad, especially when you consider that the best IvyBridge EX 2-socket , 30-core system SD 2-tier benchmark result to date achieved 12,280 Users, 67,020 SAPS (Cisco UCS B260 M4, Cert # 2014018). On this benchmark, the Power System delivered almost 73% more performance per systems, 116% more performance per core. While performance with a conventional DB and app servers cannot be correlated with predicted HANA performance, it is a reasonable measurement to consider relative performance of the two types of systems.
2) Reliability – As I have written about in previous blog entries, IBM Power Systems have a long history of providing mission critical reliability. Data analysis companies have evaluated extensive outage data from customers and determined that Power Systems, on average, have experienced up to 60% fewer outages than x86 systems with up to 5 times faster recovery from outages when they do occur. That said, Intel has made many strides forward in reliability features. If we were to assume equivalent reliability at the processor core level (Intel claims, not backed up by customer data yet), there are still a whole host of components that can result in outages, not to mention a major difference in the underlying RAS architecture for problem detection and resolution. As this has been a discussion in previous blog entries, let me instead focus on the new improvements in POWER8 which leap frogs x86 systems. As mentioned earlier, HANA loves, eats and breathes memory. So, let’s consider memory reliability. POWER8 memory cards on these new scale-out systems include features previously available only on Enterprise class systems such as the Power 770. Specifically, each rank of memory features an inactive chip that is included at no extra cost and can take the place of a failing chip on the fly. This is in addition to ChipKill (ECC on steroids) or a recovery mechanism across chips, not only within each one. By comparison, x86 systems may offer ChipKill or even double ChipKill, but to avoid “recovery” after a chip failure, x86 systems would require memory mirroring (available on most high end x86 systems) but this effectively reduces memory capacity by 50% as ALL memory is mirrored. This means that a 1.5TB, 30-core IvyBridge EX system would only be able to deliver 768GB of memory with similar memory redundancy to POWER8 with a full complement of 1TB of memory.
Lacking this huge penalty, most customers will stick with non-mirrored memory at a considerably lower reliability level. But the story does not end there. POWER8 adds extra protection on the memory bus with built in redundancy and automatic failover in the event of a memory lane failure.
Taken together, some estimates suggest that POWER8 systems will experience 2.5 to 8 times fewer system outages caused by the memory subsystem than similarly configured x86 systems. Put another way, if we were to assume that a 1TB x86 system would experience one outage every 10 years (just a random number for comparison purposes, not based on any actual data), then 10 such systems would result in 1 outage every year based on memory alone. Maybe not a huge problem for BW systems with scale-out/HA configs, non-mission critical analytics systems or non-prod systems, but could be a big problem for mission critical business suite systems on separate nodes much less larger ones that place multiple SAP components on one node (yes, SAP seems to be envisioning a future of MCOD or Multiple Components on One DB). Remember, however, that this is just the memory subsystem associated failure and there are many other potential areas of system failure for which Power has its long track record of superior protection and recovery.
3) Flexibility – POWER has virtualization built in. It is included by default at the hardware/firmware level (unless one were to remove the PowerVM hypervisor and replace it with PowerKVM, not the intended hypervisor for HANA). At SAPPHIRE, SAP announced GA support for HANA production on VMware vSphere 5.5. Just because you CAN do something does not mean that you SHOULD. You could run across a busy highway without the assistance of lights or overpasses, but most people would consider you crazy to do so. Likewise, vSphere 5.5 still utilizes the same I/O infrastructure that it has in the past resulting in all I/O writes, having to pass from a VM through its virtual device driver to the memory space of the vSphere hypervisor and then on to the I/O subsystem through another “real” device driver and the same, albeit reverse, path on reads. If that was not bad enough to dissuade all but the most ardent VMware proponents from placing any large scale transactional DB on VMware, VMware’s own published study shows only 65% scaling on 32 virtual processors, which when extrapolated to 64 or 128 vps shows no incremental benefit at all. Can you then image an incredibly memory intensive HANA system running with a ton of memory but only being able to utilize 32 threads/16 cores effectively? Would this break the KPIs for HANA?
Another area of flexibility comes from configuration options. We have recommended a TDI approach (Tailored Datacenter Integration: http://www.saphana.com/docs/DOC-4380) where SAP defines the KPIs for HANA and allows customers to determine how they will satisfy those KPIs. In other words, if a customer wants to use an LPAR on a POWER7+ or POWER8 system, their choice of storage based on their current standards and install base (e.g. EMC, HDS, NetApp, IBM), an existing network infrastructure, on-board or off-board flash drives, XFS or GPFS, etc., a TDI approach would allow for the flexibility to select the components and configurations that work best for them.
4) Efficiency – If running VMware is not a good option, as suggested above, then customers would be forced to employ islands of automation, multiple different HANA implementations on separate servers each with its own dedicated HA and DR servers, ABAP and JAVA app servers on their own systems, probably virtualized, but still separate from the HANA system. Yes, you might virtualize non-prod, but if you virtualize QA or a stress test system, then you have violated one of the fundamental rules in system architecture, i.e. never utilize a different stack in the critical QA path that is not used in production.
By comparison, Power has no such limitations. Anything can be mixed and matched as desired, with obvious constraints of CPU, Memory and I/O considered, of course. In other words, customers will have the flexibility to put together anything in almost any combination based on their unique needs. Higher performance and comprehensive virtualization may result in fewer systems being required since it allows customers to place many different partitions (VMs) together, including HANA, app servers, non-prod and HA, into a single cluster using shared, virtualized resources. As a result, data center footprint, power, cooling, administration, problem resolution, etc. can all see substantial reductions.
5) Cost – With the introduction of POWER8 and its Linux only models, IBM entered a new era on pricing. Full disclosure: I am not a wizard at configurations so I am certain that these would not pass a proper design review but should suffice for this comparison. I selected the IBM Power S822L, a Linux only system that includes 2 sockets, 24 cores @ 3.026 GHz, 1TB of memory, PowerVM, a pair of 146GB drives (just for boot and paging), a pair of dual-port 40GB Ethernet adapters, 3 years of 24/7 warranty and Linux support. The price came out a little over $90K. I then went to the HP web site and selected a DL580 Gen8 with a pair of E7-4480v2 2.8GHz/15 core processors, 1TB of memory, a couple of 120GB disk drives and 4 dual port 10GB Ethernet adapters, since they apparently don’t support the newer 40GB ones. Before I even added OS and Virtualization, this config was already close to $62K. Add VMware vSphere 5.5 and a 4 Guest, 2 socket RedHat license with 3 yr 24/7 support and the total is closing on $78K. So, for a little over 15% higher cost, at list, you could get a system with over 50% more performance, radically better virtualization and far better reliability. The reliability advantage alone, especially for Suite on HANA customers, would justify a far higher price tag. Of course, this is just the first wave of systems and many customers will need far larger configurations. As IBM has not announced those, as of yet, I can’t speculate where they will come in, price wise, but if you consider how far things have come in a very short time for Power, the future looks awfully bright.
One thing that I did not discuss above, other than in passing, is GPFS. Not that GPFS is to be ignored, but remember, this will hopefully be an option, not a requirement. This amazing, scale-out technology, is shared with our System x brethren. It is a game changer for customers, offering not only outstanding performance but built in redundancy for data and log volumes for HANA. It has been extensively tested by SAP, IBM and customers. It already certified to scale-out to 56 nodes with System x and SAP has tested it in configurations with over 100 nodes. If GPFS were added to the mix, for scale-out configurations, this would add substantial additional value over similar x86 configurations, other than those from IBM System x.
HANA on Power will enable not just freedom of choice for customers, but the mission critical reliability and performance that may have been holding them back from trying Suite on HANA. We are looking forward to working with SAP and our customers to explore this exciting new offering.
Many thanks to my colleague, Bob Wolf, for his outstanding editing.
I was intrigued by a recent blog post, entitled: Part 1: SAP on VMware : Why choose x86. https://communities.vmware.com/blogs/walkonblock/2014/02/06/part-1-sap-on-vmware-why-choose-x86. I will get to the credibility of the author in just a moment. First, however, I felt it might be interesting to review the points that were made and discuss these, point by point.
- No Vendor Lock-in: “When it comes to x86 world, there is no vendor lock-in as you can use any vendor and any make and model as per your requirements”. Interesting that the author did not discuss the vendor lock-in on chip, firmware or hypervisor. Intel, or to a very minor degree, AMD, is required for all x86 systems. This would be like being able to choose any car as long as the engine was manufactured by Toyota (a very capable manufacturer but with a lock on the industry, might not offer the best price or innovation). As any customer knows, each x86 system has its own unique BIOS and/or firmware. Sure, you can switch from one vendor to another or add a second vendor, but lacking proper QA, training, and potentially different operational procedures, this can result in problems. And then there is the hypervisor with VMware clearly the preference of the author as it is for most SAP x86 virtualization customers. No lock-in there?
SAP certifies multiple different OS and hypervisor environments for their code. Customers can utilize one or more at any given time. As all logic is written in 3rd and 4th GL languages, i.e. ABAP and JAVA, and is contained within the DB server, customers can move from one OS, HW platform and/or hypervisor to another and only have to, wait for it, do proper QA, training and modify operational procedures as appropriate. So, SAP has removed lock-in regardless of OS, HW or hypervisor.
Likewise, Oracle, DB2 and Sybase support most OS’s, HW and hypervisors (with some restrictions). Yes, a migration is required for movement between dissimilar stacks, but this could be said for moving from Windows to Linux and any move between different stacks still requires all migration activities to be completed with the potential exception of data movement when you “simply” change the HW vendor.
- Lower hardware & maintenance costs: “x86 servers are far better than cheaper than non-x86 servers. This also includes the ongoing annual maintenance costs (AMC) as well.” Funny, however, that the author only compared HW and maintenance costs and conveniently forgot about OS and hypervisor costs. Also interesting that the author forgot about utilization of systems. If one system is ½ the cost of another, but you can only drive, effectively, ½ the workload, then the cost is the same per unit of work. Industry analysts have suggested that 45% utilization is the maximum sustained to be expected out of VMware SAP systems with most seeing far less. By the same token, those analysts say that 85% or higher is to be expected of Power Systems. Also interesting to note that the author did not say which systems were being compared as new systems and options from IBM Power Systems offer close to price parity with x86 systems when HW, OS, hypervisor and 3 years of maintenance are included.
- Better performance: “Some of the models of x86 servers can actually out-perform the non-x86 servers in various forms.” Itanium is one of the examples, which is a no-duh for anyone watching published benchmarks. The other example is a Gartner paper sponsored by Intel which actually does not quote a single SAP benchmark. Too bad the author suggested this was a discussion of SAP. Last I checked (today 2/10/14), IBM Power Systems can deliver almost 5 times the SAPS performance of the largest x86 server (as measured by the 2-tier SD benchmark). On a SAPS/core basis, Power deliver almost 30% more SAPS/core compared to Windows systems and almost 60% more than Linux/x86 systems. Likewise, on the 3-tier benchmark, the latest Power result is almost 4.5 times that of the latest x86 result. So, much for point 3.
- Choice of OS: “You have choice of using any OS of your choice and not forced to choose a specific OS.” Yes, it really sucks that with Power, you are forced to choose, AIX … or IBM I for Business … or SUSE Linux … or RedHat Linux which is so much worse than being forced to choose Microsoft Windows … or Oracle Solaris … or SUSE Linux … or RedHat Linux.
- Disaster Recovery: “You can use any type of hardware, make and model when it comes to disaster recovery (DR). You don’t need to maintain hardware from same vendor.” Oh, really? First, I have not met any customers that use one stack for production and a totally different one in DR, but that is not to say that it can’t be done. Second, remember the discussion about BIOS and firmware? There can be different patches, prerequisites and workarounds for different stacks. Few customers want to spend all of the money they “saved” by investing in a separate QA cycle for DR. Even fewer want to take a chance of DR not working when they can least afford it, i.e. when there is a disaster. Interestingly, Power actually supports this better than x86 as the stack is identical regardless of which generation, model, mhz is used. You can even run in Power6 mode on a Power7+ server further enabling complete compatibility regardless of chip type meaning you can use older systems in DR to back up brand new systems in production.
- Unprecedented scalability: “You can now scale the x86 servers the way you want, TB’s of RAM’s , more than 64 cores etc is very much possible/available in x86 environment.” Yes, any way that you want as long as you don’t need more capacity than is available with the current 80 core systems. Any way that you want as long as you are not running with VMware which limits partitions to 128 threads which equates to 64 cores. Any way that you want but that VMware suggests that you contain partitions within a NUMA block which means a max of 40 cores. http://blogs.vmware.com/apps/sap Any way that you want as long as you recognize that VMware partitions are further limited in terms of scalability which results in an effective limit of 32 threads/16 cores as I have discussed in this blog previously.
- Support from Implementation Vendor: “If you check with your implementation vendor/partner, you will find they that almost all of them can certify/support implementation of SAP on x86 environment. The same is the case if you are thinking about migrating from non-x86 to x86 world.” No clue what point is being made here as all vendors on all supported systems and OSs support SAP on their systems.
The author referred to my blog as part of the proof of his/her theories which is the only reason why I noticed this blog in the first place. The author describes him/herself as “Working with Channel Presales of an MNC”. Interesting that he/she hides him/herself behind “MNC” because the “MNC” that I work for believes that transparency and honesty are required in all internet postings. That said, the author writes about nothing but VMware, so you will have to draw your own conclusions as to where this individual works or with which “MNC” his/her biases lie.
The author, in the reference to my posting, completely misunderstood the point that I made regarding the use of 2-tier SAP benchmark data in projecting the requirements of database only workloads and apparently did not even read the “about me” which shows up by default when you open my blog. I do not work for SAP and nothing that I say can be considered to represent them in any way.
Fundamentally, the author’s bottom line comment, “x86 delivers compelling total cost of ownership (TCO) while considering SAP on x86 environment” is neither supported by the facts that he/she shared nor by those shared by others. IBM Power Systems continues to offer very competitive costs with significantly superior operational characteristics for SAP and non-SAP customers.
Virtualizing SAP HANA with VMware in productive environments is not supported at this time, but according to Arne Arnold with SAP, based on his blog post of Nov 5, http://www.saphana.com/community/blogs/blog/2013/11/05/just-a-matter-of-time-sap-hana-virtualized-in-production, they are working hard in that direction.
Clearly memory can be assigned to individual partitions with VMware and to a more limited extent, CPU resources may also be assigned although this may be a bit more limited in its effectiveness. The issues that SAP will have to overcome, however, are inherent limitations in scalability of VMware partitions, I/O latency and potential contention between partitions for CPU resources.
As I discussed in my blog post late last year, https://saponpower.wordpress.com/2012/10/23/sap-performance-report-sponsored-by-hp-intel-and-vmware-shows-startling-results/, VMware 5.0 was proven, by VMware, HP and Intel, to have severe performance limitations and scalability constraints. In the lab test, a single partition achieved only 62.5% scalability, overall, but what is more startling is the scalability between each measured interval. From 4 to 8 threads, they were able to double the number of users, thereby demonstrating 100% scalability, which is excellent. From 8 to 16 threads, they were only about to handle 66.7% more users despite doubling the number of threads. From 16 to 32 threads, the number of users supported increased only 50%. Since the time of the study being published, VMware has released vSphere 5.1 with an architected limit of 64 threads per partition and 5.5 with an architected limited of 128 threads per partitions. Notice my careful wording of architected limit not the official VMware wording, of “scalability”. Scaling implies that with each additional thread, additional work can be accomplished. Linear scaling implies that each time you double the number of threads, you can accomplish twice the amount of work. Clearly, vSphere 5.0 was unable to attain even close to linear scaling. But now with the increased number of threads supported, can they achieve more work? Unfortunately, there are no SAP proof points to answer this question. All that we can do is to extrapolate the results from their earlier published results assuming the only change was the limitation in the number of architected threads. If we use the straight forward Microsoft Excel “trendline” function to project results using a Polynomial with an order of 2, (no, it has been way to long since I took statistics in college to explain what this means but I trust Microsoft (lol)) we see that a VMware partition is unlikely to ever achieve much more throughput, without a major change in the VMware kernel, than it achieved with only 32 threads. Here is a graph that I was able to create in Excel using the data points from the above white paper.
Remember, at 32 threads, with Intel Hyperthreading, this represents only 16 cores. As a 1TB BW HANA system requires 80 cores, it is rather difficult to imagine how a VMware partition could ever handle this sort of workload much less how it would respond to larger workloads. Remember, 1TB = 512GB of data space which, at a 4 to 1 compression ratio, equal 2TB of data. VMware starts to look more and more inadequate as data size increases.
And if a customer was misled enough by VMware or one of their resellers, they might think that using VMware in non-prod was a good idea. Has SAP or a reputable consultant ever recommended using use one architecture and stack in non-prod and a completely different one in prod?
So, in which case would virtualizing HANA be a good idea? As far as I can tell, only if you are dealing with very small HANA databases. How small? Let’s do the math: assuming linear scalability (which we have already proven above is not even close to what VMware can achieve) 32 threads = 16 cores which is only 20% of the capacity of an 80 core system. 20% of 2TB = 400GB of uncompressed data. At the 62.5% scalability described above, this would diminish further to 250GB. There may be some side-car applications for which a large enterprise might replicate only 250GB of data, but do you really want to size for the absolute maximum throughput and have no room for growth other than chucking the entire system and moving to newer processor versions each time they come out? There might also be some very small customers which have data that can currently fit into this small a space, but once again, why architect for no growth and potentially failure? Remember, this was a discussion only about scalability, not response time. Is it likely that response time also degrades as VMware partitions increase in size? Silly me! I forgot to mention that the above white paper showed response time increasing from .2 seconds @ 4 threads to 1 second @ 32 threads a 400% increase in response time. Isn’t the goal of HANA to deliver improved performance? Kind of defeats the purpose if you virtualize it using VMware!