An ongoing discussion about SAP infrastructure

Update – SAP HANA support for VMware, IBM Power Systems and new customer testimonials

The week before Sapphire, SAP unveiled a number of significant enhancements.  VMware 6.0 is now supported for a production VM (notice the lack of a plural); more on that below.  Hybris Commerce, a number of apps surrounding SoH and S/4HANA is now supported on IBM Power Systems.  Yes, you read that right.  The Holy Grail of SAP, S/4 or more specifically, 1511 FPS 02, is now supported on HoP.  Details, as always, can be found in SAP note:   2218464 – Supported products when running SAP HANA on IBM Power Systems .   The importance of this announcement, or should I say non-announcement as you had to be watching the above SAP note as I do on almost a daily basis because it changes so often, was the only place where this was mentioned.  This is not a dig at SAP as this is their characteristic way of releasing updates on availability to previously suggested intentions and is consistent as this was how they non-announced VMware 6.0 support as well.  Hasso Plattner, various SAP executives and employees, in Sapphire keynotes and other sessions, mentioned support for IBM Power Systems in almost a nonchalant manner, clearly demonstrating that HANA on Power has moved from being a niche product to mainstream.

Also, of note, Pfizer delivered an ASUG session during Sapphire including significant discussion about their use of IBM Power Systems.  I was particularly struck by how Joe Caruso, Director, ERP Technical Architecture at Pfizer described how Pfizer tested a large BW environment on both a single scale-up Power System with 50 cores and 5TB of memory and on a 6-node x86 scale-out cluster (tested on two different vendors, not mentioned in this session but probably not critical as their performance differences were negligible), 60-cores on each node with 1 master node and 4 worker nodes plus a not-standby.  After appropriate tuning, including utilizing table partitioning on all systems, including Power, the results were pretty astounding; both environments performed almost identically, executing Pfizer’s sample set, composed of 75+ queries, in 5.7 seconds, an impressive 6 to 1 performance advantage on a per core basis, not including the hot-standby node.  What makes this incredible is that the official BW-EML benchmark only shows an advantage of 1.8 to 1 vs. the best of breed x86 competitor and another set of BW-EML benchmark results published by another x86 competitor shows scale-out to be only 15% slower than scale-up.  For anyone that has studied the Power architecture, especially POWER8, you probably know that it has intrinsics that suggest it should handle mixed, complex and very large workloads far better than x86, but it takes a customer executing against their real data with their own queries to show what this platform can really do.  Consider benchmarks to be the rough equivalent of a NASCAR race car, with the best of engineering, mechanics, analytics, etc, vs. customer workloads which, in this analogy, involves transporting varied precious cargo in traffic, on the highway and on sub-par road conditions.  Pfizer decided that the performance demonstrated in this PoC was compelling enough to substantiate their decision to implement using IBM Power Systems with an expected go-live later this year.  Also, of interest, Pfizer evaluated the reliability characteristics of Power, based in part on their use of Power Systems for conventional database systems over the past few years, and decided that a hot-standby node for Power was unnecessary, further improving the overall TCO for their BW project.  I had not previously considered this option, but it makes sense considering the rarity of Power Systems to be unable to handle predictable, or even unpredictable faults, without interrupting running workloads.  Add to this, for many, the loss of analytical environments is unlikely to result in significant economic loss.

Also in a Sapphire session, Steve Parker, Director Application Development, Kennametal, shared a very interesting story about their journey to HANA on Power.  Though they encountered quite a few challenges along the way, not the least being that they started down the path to Suite on HANA and S/4HANA prior to it being officially supported by SAP, they found the Power platform to be highly stable and its flexibility was of critical importance to them.  Very impressively, they reduced response times compared to their old database, Oracle, by 60% and reduced the run-time of a critical daily report from 4.5 hours to just 45 minutes, an 83% improvement and month end batch now completes 33% faster than before.  Kennametal was kind enough to participate in a video, available on YouTube at: as well as a write up on their experience at:

As I mentioned earlier, SAP snuck in a non-announcement about VMware and how a single production VM is now supported with VMware 6.0 in the week prior to Sapphire.  SAP note 2315348 – describes how a customer may support a single SAP HANA VM on VMware vSphere 6 in production.  One might reasonably question why anyone would want to do this.  I will withhold any observations on the mind set of such an individual and instead focus on what is, and is not, possible with this support.  What is not possible: the ability to run multiple production VMs on a system or to mix production and non-prod.  What is possible: the ability to utilize up to 128 virtual processors and 4TB of memory for a production VM, utilize vMotion and DRS for that VM and to deliver DRAMATICALLY worse performance than would be possible with a bare-metal 4TB system.  Why?  Because 128 vps with Hyperthreading enabled (which just about everyone does) utilizes 64 cores.  To support 6TB today, a bare-metal Haswell-EX system with 144 cores is required.  If we extrapolate that requirement to 4TB, 96 cores would be required.  Remember, SAP previously explained a minimum overhead of 12% was observed with a VM vs. bare-metal, i.e. those 64 cores under VMware 6.0 would operate, at best, like 56 cores on bare-metal or 42% less capacity than required for bare-metal.  Add to this the fact that you can’t recover any capacity left over on that system and you are left with a hobbled HANA VM and lots of leftover CPU resources.  So, vMotion is the only thing of real value to be gained?  Isn’t HANA System Replication and a controlled failover a much more viable way of moving from one system to another?  Even if vMotion might be preferred, does vMotion move memory pages from source to target system using the EXACT same layout as was implemented on the source system?  I suspect the answer is no as vMotion is designed to work even if other VMs are currently running on the target system, i.e. it will fill memory pages based on availability, not based on location.  As a result, this would mean that all of the wonderful CPU/memory affinity that HANA so carefully established on the source system would likely be lost with a potentially huge impact on performance.

So, to summarize, this new VMware 6.0 support promises bad performance, incredibly poor utilization in return for the potential to not use System Replication and suffer even more performance degradation upon movement of a VM from one system to another using vMotion.  Sounds awesome but now I understand why no one at the VMware booth at Sapphire was popping Champagne or in a celebratory mood.  (Ok, I just made that up as I did not exactly sit and stare at their booth.)

May 26, 2016 Posted by | Uncategorized | , , , , , , , , | 2 Comments

Certified vs. supported HANA solutions; what’s the difference and why should you care

There seems to be a lot of confusion about the terms “Certified” and “Supported” in the HANA context.  Those are not qualitative terms but more of a definition of how solutions are put together and delivered.  SAP recognized that HANA was such a new technology, back in 2011, and had so many variables which could impact performance and support, that they asked technology vendors to design appliances which SAP could review, test and ensure that all performance characteristics met SAP’s KPIs.  Furthermore, with a comprehensive understanding of what was included in an appliance, SAP could offer a one-stop-shop approach to support, i.e. if a customer has a problem with a “Certified” appliance, just call SAP and they will manage the problem and work with the technology vendor to determine where the problem is, how to fix it and drive it to full resolution.

Sounds perfect, right?  Yes … as long as you don’t need to make any modifications as business needs change.   Yes … as long as you don’t mind the system running at low utilization most of the time.   Yes … as long as the systems, storage and interconnects that are included in the “certified” solution match the characteristics that you consider important, are compatible with your IT infrastructure and allow you to use the management tools of your choice.

So, what is the option?  SAP introduced TDI, the Tailored Datacenter Integration approach.  It allows customers to put together HANA environments in a more flexible manner using a customer defined set of components (with some restrictions) which meet SAP’s performance KPIs.  What is the downside?  Meeting those KPIs and problem resolution are customer responsibilities.  Sounds daunting, but it is not.  Fortunately, SAP doesn’t just say, go forward and put anything together that you want.  Instead, they restrict servers and storage subsystems to those for which internal or external performance tests have been completed to SAP standards.  This allows reasonable ratios to be derived, e.g. the memory to core ratio for various types of systems and HANA implementation choices.  Some restrictions do apply, for example Intel Haswell-EX environments must utilize systems which have been approved for use in appliances and Haswell-EP and IBM Power Systems environments must use systems listed on the appropriate “supported” tabs of the official HANA infrastructure support guide.[i]   Likewise, Certified Enterprise storage subsystems are also listed, but this does not rule out the use of internal drives for TDI solutions.

Any HANA solution, whether an appliance or a TDI defined system, is equally capable of handling the HANA workload which falls within the maximums that SAP has identified.  SAP will support HANA on any of the above.  As to the full solution support, as mentioned previously, this is a customer responsibility.  Fortunately, vendors, such as IBM, offer a one-stop-shop support package.  IBM calls its package Custom Technical Support (US) or Total Solution Service (outside of US).  Similar to the way that SAP supports an appliance, with this offering, a customer need call only one number for support.  IBM’s support center will then work with the customer to do problem determination and problem source identification.  When problems are determined to be caused by IBM, SAP or SUSE products, warm transfers are made to those groups.  The IBM support center stays engaged even after the warm transfer occurs to ensure the problem is resolved and delivered to the customer.  In addition, customers may benefit from optional proactive services (on-site or remote) to analyze the system in order to receive recommendations to keep the system up to date and/or to perform necessary OS, firmware or hardware updates and upgrades.  With these proactive support offerings, customers can ensure that the HANA systems are maintained in line with SAP’s planned release calendar and are fully prepared for future upgrades.

There are a couple of caveats however.  Since TDI permits the use of the customer’s preferred network and storage vendors, these sorts of support offerings typically encompass only the vendors’ products that are within the scope of the warm transfer agreements of each offering vendor.  As a result, a problem with a network switch or a third-party storage subsystem for which the proactive support vendor does not have a warm transfer support agreement would still be the responsibility of the customer.

So, should a customer choose a “certified” solution or a TDI supported solution?  The answer depends on the scope of the HANA implementation, the customer’s existing standards, skills and desire to utilize them, the flexibility with resource utilization and instance placement desired and, of course, cost.

Scope – If HANA is used as a side-car, a small BW environment, or perhaps for Business One, an appliance can be a viable option, especially if the HANA solution will be located in a setting for which local skilled personnel are not readily available.  If, however, the HANA environment is more complex, e.g. BW scale-out, SoH, S/4, large, etc, and located in a company’s main data centers with properly skilled individuals, then a TDI supported approach may be more desirable.

Standards – Many customers have made large investments in network infrastructure, storage subsystems and the tools and skills necessary to manage them.  Appliances that include components which are not part of those standards not only bring in new devices that are unfamiliar to the support staff, but may be largely invisible to the tools currently in use.

Flexibility – Appliances are well defined, single purpose devices.  That definition includes a fixed amount of memory, CPU resources, I/O adapters, SSD and/or HDD devices.  Simple to order, inflexible to change.  If a different amount of any of the above resources is desired, in theory, any change permitted by the offering vendor results in the device moving from a SAP supported appliance to a TDI supported configuration instantly requiring the customer to accept responsibility for everything just as quickly.  By comparison, TDI supported solutions start out as a customer responsibility meaning it has been tailored around the customer’s standards and can be modified as desired at any time.  All that is required for support is to run SAP’s HWCCT (Hardware Configuration Check Tool) to ensure that the resulting configuration still meets all SAP KPIs.  As a result, if a customer desires to virtualize, mixing multiple production and non-production or even non-SAP workloads (when supported by the chosen virtualization solution, see my blog post on VMware and HANA recently published), a TDI solution, vendor and technology dependent, supports this by definition; an appliance does not.  Likewise, a change in capacity, e.g. physical addition/removal of components, logical change of capacity, often called Capacity on Demand and VM resizing, are fully supported with TDI, not with appliances.  As a result, once a limit is reached with an appliance, either a scale-out approach much be utilized, in the case of analytics that support scale-out, or the appliance must be decommissioned and replaced with a larger one.  A TDI solution will available additional capacity or the ability to add additional gives the customer the ability to upgrade in place, thereby providing greater investment protection.

Cost – An appliance trades simplicity with potentially higher TCO as it is designed to meet the above mentioned KPIs without a comprehensive understanding of what workload will be handled by said appliance, often resulting in dramatic over-capacity, e.g. uses dozens of HDDs to meet disk throughput requirements.  By comparison, customers with existing enterprise storage subsystems, may need only a few additional SSD and/or HDDs to meet those KPIs with limited incremental cost to infrastructure, environmentals and support.  Likewise, the ability to use fewer, potentially larger systems with the ability to co-reside production, non-prod, app servers, non-SAP VMs, HA, etc., can result in significant reductions of systems footprints, power, cooling, management and associated costs.

IBM Power Systems has chosen to take the TDI-only approach as a direct result of feedback received from customers, especially enterprise customers, that are used to managing their own systems, have available skills, have prevalent IT standards, etc.  HANA on Power is based on virtualization, so is designed, by default, to be a TDI based solution.  HANA on Power allows for one or many HANA instances, a mixture of prod and potentially non-prod or non-HANA to share the system.  HA and DR can be mixed with other prod and non-prod instances.

I am often asked about t-shirt sizes, but this is a clear indication that the individual asking the question has a mindset based on appliance sizing.  TDI architecture involves landscape sizing and encompasses all of the various components required to support the HANA and non-HANA systems, so a t-shirt sizing would end up being completely misleading unless a customer only needs a single system, no HA, no DR, no non-prod, no app servers, etc.


May 11, 2016 Posted by | Uncategorized | , , , , , , , , , , , | Leave a comment

SAP support for multiple production HANA VMs with VMware

Recently, SAP updated their SAP notes regarding the ability to run multiple production HANA VMs with VMware.  On the surface, this sounds like VMware has achieved parity with IBM’s PowerVM, but the reality could not be much farther away from that perception.  This is not to say that users of VMware for HANA will see no improvement.  For a few customers, this will be a good option, but as always, the devil is in the details and, as always, I will play the part of the devil.

Level of VMware supported: 5.5 … still.  VMware 6.0 is supported only for non-production.[i]  If VMware 6.0 is so wonderful and they are such “great” partners with SAP, it seems awfully curious why a product announced on Feb 3, 2015 is still not supported by SAP.

Maximum size of each production HANA instance: 64 virtual processors and 1TB of memory, however this translates to 32 physical processors with Hyperthreading enabled and sizing guidelines must still be followed.  Currently, BW HANA is sized @ 2TB for 4 Haswell chips, i.e. 28.4GB/core which translates to a maximum size of 910GB for a 32 core VM/64 vp, so slightly less than the 1TB supported.  Suite on HANA on Intel is supported at 1.5x higher memory ratio than BW, but since the size of the VM is limited to 1TB, this point is largely moot.

Performance impact: At a minimum, SAP estimates a 12% performance degradation compared to bare metal (upon which most benchmarks are run and from which most sizings are based), so one would logically conclude that the memory/cpu ratio should be reduced by the same level.  The 12% performance impact, but not the reduced sizing effect that I believe should be the result, are detailed in a SAP note.[ii]  It goes on to state “However, there are around 100 low-level performance tests in the test suite exercising various HANA kernel components that exhibit a performance degradation of more than 12%. This indicates that there are particular scenarios which might not be suited for HANA on VMware.Only 100?  When you consider the only like-for-like published benchmarks using VMware and HANA[iii], which showed a 12% degradation (coincidence, I think not) for a single VM HANA system vs. bare metal, it leaves one to wonder what sort of degradation might occur in a multiple VM HANA environment.  There is no guidance provided on this which might make anything less than a bleeding edge customer with no regard for SLA’s to be VERY cautious.  Another SAP note[iv] goes on to state, “For optimal VM performance, VMware recommends to size the VMs within the NUMA node boundaries of the specific server system (CPU cores and local NUMA node memory).” How much impact?  Not provided here.  So, either you size your VMs to fit within NUMA building blocks, i.e. a single 18-core socket or you suffer an undefined performance penalty.  It is also interesting to note what VMware said in Performance Best Practices for VMware vSphere® 5.5Be careful when using CPU affinity on systems with hyper-threading. Because the two logical processors share most of the processor resources, pinning vCPUs, whether from different virtual machines or from a single SMP virtual machine, to both logical processors on one core (CPUs 0 and 1, for example) could cause poor performance.”  That certainly gives me the warm and fuzzy!

Multiple VM support: Yes, you can now run multiple production HANA VMs on a system[v].  HOWEVER, “The vCPUs of a single VM must be pinned to physical cores, so the CPU cores of a socket get exclusively used by only one single VM. A single VM may span more than one socket, however. CPU and Memory overcommitting must not be used.”  This is NOT the value of virtualization, but of physical partitioning, a wonderful technology if we were living in the 1990s.  So, if you have an 8-socket system, you can run up to 8 simultaneous production VMs as long as al VMs are smaller than 511GB for BW, 767GB for SoH.  Need 600GB for BW, well that will cost you a full socket even though you only need a few cores thereby reducing the maximum number of VMs you can support on the system.  And this is before we take the 12% performance impact detailed above into consideration which could further limit the memory per core and number of VMs supported.

Support for mixed production HANA VMs and non-production: Not included in any of the above mentioned SAP notes.  One can infer from the above notes that this is not permitted meaning that there is no way to harvest unused cycles from production for the use of ANY other workload, whether non-prod or non-HANA DB.

Problem resolution: SAP Note 1995460 details the process by which problems may be resolved and while they are guided via SAP’s OSS system, there is transfer of ownership of problems when a known VMware related fix is not available.  The exact words are: “For all other performance related issues, the customer will be referred within SAP’s OSS system to VMware for support. VMware will take ownership and work with SAP HANA HW/OS partner, SAP and the customer to identify the root cause. Due to the abstraction of hardware that occurs when using virtualization, some hardware details are not directly available to SAP HANA support.” and of a little more concern “SAP support may request that additional details be gathered by the customer or the SAP HANA HW partner to help with troubleshooting issues or VMware to reproduce the issue on SAP HANA running in a bare metal environment.”

My summary of the above: One of more production HANA instances may be run under VMware 5.5 with a maximum of 64 vp/32 pp (assuming Hyperthreading is enabled) and a minimum of 12% performance degradation with potential proportionate impact on sizing, with guidance about “some scenarios might be suited for HANA with VMware”, with potential performance issues when VMs cross socket boundaries, with physical partitioning at the socket level, no sharing of CPU resources, no support for running non-production on the same system to harvest unused cycles and potential requirement to reproduce issues on a bare metal system if necessary.

Yes, that was a long, run-on sentence.  But it begs the question of just when would VMware be a good choice for hosting one or more production HANA instances?  My take is that unless you have very small instances which are unsuitable for HANA MDC (multitenancy) or are a cloud provider for very small companies, there is simply no value in this solution.  For those potential cloud providers, their target customer set would include companies with very small HANA requirements and the willingness to accept an SLA that is very flexible on performance targets while using a shared infrastructure for which there a wide variety of issues for which a problem in one VM could result in the whole system to fail with multiple customers impacted simultaneously.

And in case anyone is concerned that I am simply the bearer of bad news, let me remind the reader that IBM Power Systems with PowerVM is supported by SAP with up to 4 production HANA VMs (on the E870 and E880, 3 on all other HANA supported Power Systems) with granularity at the core level, no restrictions on NUMA boundaries, the ability to have a shared pool in place of one of the above production VMs with any number of non-production HANA VMs up to the limits of PowerVM which can utilize unused cycles from the production VMs, no performance penalties, no caveats about what types of workloads are well suited for PowerVM, excellent partition isolation preventing the vast majorities that could happen in one VM from affecting any other ones and no problem resolution handoffs or ownership changes.

In other words, if customers want to continue the journey around virtualization and server consolidation that they started in the early 2000s, want to have a very flexible infrastructure which can grow as they move to SoH, shrink as they move to S/4, grow as they consolidate more workloads into their primary instance, shrink again as they roll off data using data tiering, data aging or perhaps Hadoop and all without having to take significant system outages or throw away investment and purchase additional systems, IBM Power Systems with PowerVM can support this; VMware cannot.



[i] 1788665 – SAP HANA Support for virtualized / partitioned (multi-tenant) environments

[ii] 1995460 – Single SAP HANA VM on VMware vSphere in production

[iii] Benchmark detail for bare metal and VMware 5.5 based runs from

06/02/2014 HP 2,000,000,000 111,850 SuSE Linux Enterprise Server 11 on VMWARE ESX 5.5 SAP HANA 1.0 SAP NetWeaver 7.30 1 database server: HP DL580 Gen8, 4 processors / 60 cores / 120 threads, Intel Xeon Processor E7-4880 v2, 2.50 GHz, 64 KB L1 cache and 256 KB L2 cache per core, 30 MB L3 cache per processor, 1024 GB main memory

03/26/2014 HP 2,000,000,000 126,980 SuSE Linux Enterprise Server 11 SAP HANA 1.0 SAP NetWeaver 7.30 1 database server: HP DL580 Gen8, 4 processors / 60 cores / 120 threads, Intel Xeon Processor E7-4880 v2, 2.50 GHz, 64 KB L1 cache and 256 KB L2 cache per core, 30 MB L3 cache per processor, 1024 GB main memory

[iv] 2161991 – VMware vSphere configuration guidelines

[v] 2024433 – Multiple SAP HANA VMs on VMware vSphere in production

April 5, 2016 Posted by | Uncategorized | , , , , , , , , , | 4 Comments

SAP HANA on Power support expands dramatically

SAP’s release of HANA SPS11 marks a critical milestone for SAP/IBM customers. About a year ago, I wrote that there was Hope for HoP, HANA on Power.  Some considered this wishful thinking, little more than a match struck in the Windy City.  In August, that hope became a pilot light with SAP’s announcement of General Availability of Scale-up BW HANA running on the Power Systems platform.  Still, the doubters questioned whether Power could make a dent in a field already populated by dozens of x86 vendors with hundreds of supported appliances and thousands of installed customers.  With almost 1 new customer per business day deciding to implement HANA on Power since that time, the pilot light has quickly evolved into a nice strong flame on a stove.

In November, 2015, SAP unleashed a large assortment of support for HoP.  First, they released a first of a kind support for running more than 1 production instance using virtualization on a system.[1]  For those that don’t recall, SAP limits systems running HANA in production on VMware to one[2], count that as 1, total VMs on the entire system.  Yes, non-prod can utilize VMware to its heart’s content, but is it wise to mess with best practices and utilize different stacks for prod and non-prod, much less deal with restrictions that limit the number of vps to 64, i.e. 32 real processors not counting VMware overhead and 1TB of memory?  Power now supports up to 4 resource pools on E870 and E880 systems and 3 on systems below this level.  One of those resource pools can be a “shared pool” supporting many VMs of any kind and any supported OS as long as none of them run production HANA instances.  Any production HANA instance must run in a dedicated or dedicated-donating partition in which when production HANA needs CPU resources, it gets it without any negotiation or delay, but when it does not require all of the resources, it allows partitions in the shared pool to utilize unused resources.   This is ideal for HANA as it is often characterized by wide variations in loads, often low utilization and very low utilization on non-prod, HA and DR systems, resulting in the much better flexibility and resource utilization (read that as reduced cost).

But SAP did not stop there.  Right before the US Thanksgiving holiday, SAP released support for running HANA on Power with Business Suite, specifically ERP 6.0 EHP7, CRM 7.0 EHP3 and SRM 7.0 EHP3, SAP Landscape Transformation Replication Server 2.0, HANA dynamic tiering, BusinessObjects Business Intelligence platform 4.1 SP 03, HANA smart data integration 1.0 SP02, HANA spatial SPS 11 and controlled availability of BPC[3], scale-out BW[4] using the TDI model with up to 16-nodes.  SAP plans to update the application support note as each additional application passes customer and/or internal tests, with support rolling out rapidly in the next few months.

Not enough?  Well, SAP took the next step and increased the memory per core ratio on high end systems, i.e. the E870 and E880, to 50GB/core for BW workloads thereby increasing the total memory supported in a scale-up configuration to 4.8TB.[5]

What does this mean for SAP customers?  It means that the long wait is over.  Finally, a robust, reliable, scalable and flexible platform is available to support a wide variety of HANA environments, especially those considered to be mission critical.  Those customers that were waiting for a bet-your-business solution need wait no more.  In short order, the match jumped to a pilot light, then a flame to a full cooktop.  Just wait until S/4HANA, SCM and LiveCache are supported on HoP, likely not a long wait at this rate, and the flame will have jumped to one of those jet burners used for crawfish boiling from my old home town of New Orleans!  Sorry, did I push the metaphor to far? :-)


[1] 2230704 – SAP HANA on IBM Power Systems with multiple – LPARs per physical host

[2] 1995460 – Single SAP HANA VM on VMware vSphere in production

[3] 2218464 – Supported products when running SAP HANA on IBM Power Systems  and

[4] BW Scale-out support restriction that was previously present has been removed from 2133369 – SAP HANA on IBM Power Systems: Central Release Note for SPS 09 and SPS 10

[5] 2188482 – SAP HANA on IBM Power Systems: Allowed Hardware

December 8, 2015 Posted by | Uncategorized | , , , , , , , , , | 8 Comments

Haswell-EX for HANA looks good on paper, POWER8 for HANA looks even better in real life

I was delighted to read Hasso Plattner’s recent blog on the strengths of HANA on platforms using the Haswell-EX chip from Intel:  In that blog, he did an excellent job of explaining how technical enhancements at a processor and memory subsystem level can result in dramatic improvement in the way that HANA operates,   Now, I know what you are thinking; he likes what Dr. Plattner has to say about a competitor’s technology?   Strange as it may seem, yes … in that he has pointed out a number of relevant features that, as good as Haswell-EX might be, POWER8 surpassed, even before Haswell-EX was announced.

All of these technical features and discussion are quite interesting to us propeller heads.   Most business people, on the other hand, would probably prefer to discuss how to improve HANA operational characteristics, deliver flexibility to respond to changing business demands and meet end user SLAs including response time and continuous availability.  This is where POWER8 really shines.  With PowerVM at its core, Power Systems can be tailored to deliver capacity for HANA production to ensure consistent response time and peak load capacity during high demand times and allow other applications and partitions to utilize capacity unused by the HANA production partition.   It can easily mix production with other production and non-production partitions.  It features the ability to utilize shared network and SAN resources, if desired, to reduce datacenter cost and complexity.  POWER8 delivers unmatched reliability by default, not as an option or a tradeoff against performance.

Regarding the technical features, Herr Dr. Plattner points out that Haswell-EX systems:

  • Support up to 144 cores per system with 12TB of memory.  POWER8 supports up to 196 cores and 16TB.  Actually, this under estimates that actual memory on a POWER8 system which is actually up to 17.7TB but IBM includes the extra 1.7TB at no extra cost as hot spare chips, not available with Haswell-EX systems.
  • Deliver L1, L2 and L3 cache size increases, which though he does not state, are, in fact, 32KB (16KB in enterprise RAS mode), 256KB and 45MB respectively, compared to POWER8’s 64KB, 512KB and 96MB respectively plus 128MB L4, not available with Haswell-EX systems.
  • Introduces enhancements to vector processing via the new AVX2 instruction unit compared to  POWER8’s dual VMX instruction units.
  • Rely on local memory access for HANA performance which is absolutely true and underlines why POWER8, with up to 4 times more bandwidth to memory, is such a good fit for HANA.
  • Feature TSX, Transactional Synchronization Extensions, to improve lock synchronization, an area that Power Systems has excelled at for decades.  POWER8 was actually a bit earlier in the whole transactional memory area but was actually preceded by IBM Blue Gene/Q, another PowerPC based technology.

He concludes by pointing out that internal benchmarks are of limited value but then explains what they achieved with Haswell-EX.  As these are not externally audited nor even published, it is hard to comment on their validity.

By comparison, SAP has only one certified benchmark for which HANA systems have been utilized called BW-EML.  Haswell-EX cpus were used in the 2B row Dell PowerEdge 930 benchmark and delivered an impressive 172,450 Ad-hoc Navigation Steps/Hr . This is impressive in that it surpassed the previous IvyBridge based benchmark of 137,010 Ad-hoc Navigation Steps/Hr on the Dell PowerEdge R920, an increase of almost 26% which would normally be impressive if it weren’t for the fact that the system includes 20% more cores and 50% more memory.  By comparison, POWER8 delivered 192,750 Ad-hoc Navigation Steps/Hr with the IBM Power Enterprise System 870 or 12% more performance with 45% fewer cores and 33% less memory resulting in twice the performance per core.

It would be ideal to run the SAP SD 3-tier benchmark against a system running Suite on HANA as that would do away with discussions of benchmarks that can’t be verified and/or may have limited relevance to a transactional environment typical of Suite on HANA.  From what I understand, the current SD benchmark depends on an older version of SAP code which is not supported on HANA.  I hope that SAP is able to update the benchmark test kit to enable this benchmark to be run on HANA as that would be far better than any sort of speculation.  In the meantime, we can only rely on assertions without detail and external review or on decades of proven experience handling large scaling transactional environments with mission critical levels of availability not to mention a wide variety of audited benchmarks demonstrating this ability.  Power Systems stands alone in this respect.

Benchmark details:

Dell PowerEdge 930: 172,450 Ad-hoc Navigation Steps/Hr using 4 processors / 72 cores / 144 threads, Intel Xeon Processor E7-8890 v3, 2.50 GHz, 1.5TB main memory, Certification #: 2015014

Dell PowerEdge R920: 137,010 Ad-hoc Navigation Steps/Hr on the, 4 processors / 60 cores / 120 threads, Intel Xeon Processor E7-4890 v2, 2.80 GHz,  1TB main memory, Certification #: 2014044

the IBM Power Enterprise System 870: 192,750 Ad-hoc Navigation Steps/Hr with, 4 processors / 40 cores / 320 threads, POWER8, 4.19 GHz,  1TB main memory, Certification #: 2015024

July 22, 2015 Posted by | Uncategorized | , , , , , , , , , , , | Leave a comment

HoP keeps Hopping Forward – GA Announcement and New IBM Solution Editions for SAP HANA

Almost two years ago, I speculated about the potential value of a HANA on Power solution.  In June, 2014, SAP announced a Test and Evaluation program for Scale-up BW HANA on Power.  That program shifted into high gear in October, 2014 and roughly 10 customers got to start kicking the tires on this solution.  Those customers had the opportunity to push HANA to its very limits.  Remember, where Intel systems have 2 threads per core, POWER8 has up to 8 threads per core.  Where the maximum size of most conventional Intel systems can scale to 240 threads, the IBM POWER E870 can scale to an impressive 640 threads and the E880 system can scale to 1536 threads.  This means that IBM is able to provide an invaluable test bed for system scalability to SAP.  As SAP’s largest customers move toward Suite “4” HANA (S4HANA), they need to have confidence in the scalability of HANA and IBM is leading the way in proving this capability.

A Ramp-up program began in March with approximately 25 customers around the world being given the opportunity to have access to GA level code and start to build out BW POC and production environments.  This brings us forward to the announcement by SAP this week @ SapphireNow in Orlando of the GA of HANA on Power.  SAP announced that customers will have the option of choosing Power for their BW HANA platform, initially to be used in a scale-up mode and plans to support scale-out BW, Suite on HANA and the full complement of side-car applications over the next 12 to 18 months.

Even the most loyal IBM customer knows the comparative value of other BW HANA solutions already available on the market.  To this end, IBM announced new “solution editions”.  A solution edition is simply a packaging of components, often with special pricing, to match expectations of the industry for a specific type of solution.  “Sounds like an appliance to me” says the guy with a Monty Python type of accent and intonation (no, I am not making fun of the English and am, in fact, a huge fan of Cleese and company).  True, if one were to look only at the headline and ignore the details.  In reality, IBM is looking toward these as starting points, not end points and most certainly not as any sort of implied limitation.  Remember, IBM Power Systems are based on the concept of Logical Partitions using Power Virtualization Manager (PVM).  As a result, a Power “box” is simply that, a physical container within which one or multiple logical systems reside and the size of each “system” is completely arbitrary based on customer requirements.

So, a “solution edition” simply defines a base configuration designed to be price competitive with the industry while allowing customers to flexibly define “systems” within it to meet their specific requirements and add incremental capability above that minimum as is appropriate for their business needs.  While a conventional x86 system might have 1TB of memory to support a system that requires 768GB, leaving the rest unutilized, a Power System provides for that 768GB system and allows the rest of the memory to be allocated to other virtual machines.   Likewise, HANA is often characterized by periods of 100% utilization, in support of instantaneous response time demanded of ad-hoc queries, followed by unfathomably long periods (in computer terms) of little to no activity.  Many customers might consider this to be a waste of valuable computing resource and look forward to being able to harness this for the myriad of other business purposes that their businesses actually depend on.  This is the promise of Power.  Put another way, the appliance model results in islands of automation like we saw in the 1990s where Power continues the model of server consolidation and virtualization that has become the modus operandi of the 2000s.

But, says the pitchman for a made for TV product, if you call right now, we will double the offer.  If you believe that, then you are probably not reading my blog.  If a product was that good, they would not have to give you more for the same price.  Power, on the other hand, takes a different approach.  Where conventional BW HANA systems offer a maximum size of 2TB for a single node, Power has no such inherent limitations.  To handle larger sizes, conventional systems must “scale-out” with a variety of techniques, potentially significantly increased costs and complexity.  Power offers the potential to simply “scale-up”.  Future IBM Power solutions may be able to scale-up to 4TB, 8TB or even 16TB.   In a recent post to this blog, I explained that to match the built in redundancy for mission critical reliability of memory in Power, x86 systems would require memory mirroring at twice the amount of memory with an associated increase in CPU and reduction in memory bandwidth for conventional x86 systems.  SAP is pushing the concepts of MCOS, MCOD and multi-tenancy, meaning that customers are likely to have even more of their workloads consolidated on fewer systems in the future.  This will result in demand for very large scaling systems with unprecedented levels of availability.  Only IBM is in position to deliver systems that meet this requirement in the near future.

Details on these solution editions can be found at
In the last few days, IBM and other organizations have published information about the solution editions and the value of HANA on Power.  Here are some sites worth visiting:

Press Release: IBM Unveils Power Systems Solutions to Support SAP HANA
Video: The Next Chapter in IBM and SAP Innovation: Doug Balog announces SAP HANA on POWER8
Case study: Technische Universität München offers fast, simple and smart hosting services with SAP and IBM 
Video: Technische Universität München meet customer expectations with SAP HANA on IBM POWER8 
Analyst paper: IBM: Empowering SAP HANA Customers and Use Cases 
Article: HANA On Power Marches Toward GA

Selected SAP Press
ComputerWorld: IBM’s new Power Systems servers are just made for SAP Hana
eWEEK, IBM Launches Power Systems for SAP HANA
ExecutiveBiz, IBM Launches Power Systems Servers for SAP Hana Database System; Doug Balog Comments
TechEYE.netIBM and SAP work together again
ZDNet: IBM challenges Intel for space on SAP HANA
Data Center Knowledge: IBM Stakes POWER8 Claim to SAP Hana Hardware Market
Enterprise Times: IBM gives SAP HANA a POWER8 boost
The Platform: IBM Scales Up Power8 Iron, Targets In-Memory

Also a planning guide for HANA on Power has been published at .

May 7, 2015 Posted by | Uncategorized | , , , , , , , , , , | 3 Comments

Lintel for SAP App Servers – The right choice

Or is it? Running SAP application servers on IBM Power Systems with Linux results in a lower TCA than using x86 systems with Linux and VMware. Usually, I don’t start a blog post with the conclusion, but was so amazed by the results of this analysis, that I could not help myself.

For several years now, I have seen many customers move older legacy app servers to x86 systems using Linux and VMware as well as implementing new SAP app servers on the same. When asked why, the answers boil down to cost, skills and standards. Historically, Lintel servers were not just perceived to cost less, but could the cost differences could be easily demonstrated. Students emerging from colleges have worked with Linux far more often than with UNIX and despite the fact that learning UNIX and how it is implemented in actual production environments is very little different in real effort/cost, the perception of Linux skills being more plentiful and consequently less expensive persists. Many companies and government entities have decided to standardize on Linux. For some legacy IBM Power Systems customers, a complicating factor, or perhaps a compelling factor in the analysis, has compared partitions on large enterprise class systems against low cost 2-socket x86 servers. And so, increasingly, enterprises have defaulted to Lintel as the app server of choice.

Something has changed however and is completely overturning the conventional wisdom discussed above. There is a new technology which takes advantage of all of those Linux skills on the market, obeys the Linux standards mandate and costs less than virtualized Lintel systems. What is this amazing new technology? Surprise, it is a descendent of the technology introduced in 1990 by IBM called the RS/6000, with new Linux only POWER8 systems. (OK, since you know that I am an IBM Power guy and I gave away the conclusion at the start of this post, that was probably not much of a surprise.) At least, this is what the marketing guys have been telling us and they have some impressive consultant studies and internal analyses that back up their claims.

For those of you who have been following this blog for a while, you know that I am skeptical of consultant analyses and even more so of internal analyses. So, instead of depending on those, I set out to prove, or disprove, this assertion. The journey began with setting reasonable assumptions. Actually, I went a little overboard and gave every benefit of the doubt to x86 and did the opposite for Power.

Overhead – The pundits, both internal and external, seem to suggest that 10% or more overhead for VMware is reasonable. Even VMware’s best practices guide for HANA suggests an overhead of 10%. However, I have heard some customers claim that 5% is possible. So, I decided to use the most favorable number and settled on 5% overhead. PowerVM does have overhead, but it is already baked into all benchmarks and sizings since it is built into the embedded hypervisor, i.e. it is there even when you are running a single virtual machine on a system.

Utilization – Many experts have suggested that average utilization of VMware systems range in the 20% to 30% range. I found at least one analyst that said that the best run shops can drive their VMware systems up to 45%. I selected 45%, once again since I want to give all of the benefit of the doubt to Lintel systems. By comparison, many experts suggest that 85% utilization is reasonable for PowerVM based systems, but I selected 75% simply to not give any of the benefit of the doubt to Power that I was giving to x86.

SAPS – Since we are talking about SAP app servers, it is logical to use SAP benchmarks. The best result that I could find for a 2 socket Linux Intel Haswell-EP system was posted by Dell @ 90,120 SAPS (1). A similar 2-socket server from IBM was posted @ 115,870 SAPS (2).

IBM has internal sizing tables, as does every vendor, in which it estimates the SAPS capacity of different servers based on different OSs. One of those servers, the Power S822L, a 20-core Linux only system, is estimated to be able to attain roughly 35% less SAPS than the benchmark result for its slightly larger cousin running AIX, but this takes into consideration differences in MHz, number of cores and small differences due to the compilers used for SAP Linux binaries.

For our hypothetical comparison, let us assume that a customer needs approximately the SAPS capacity as can be attained with three Lintel systems running VMware including the 5% overhead mentioned above, a sustained utilization of 45% and 256GB per server. Extrapolating the IBM numbers, including no additional PowerVM overhead and a sustained utilization of 75%, results in a requirement of two S822L systems each with 386GB.

Lenovo, HP and Dell all offer easy to use configurators on the web. I ran through the same configuration for each: 2 @ Intel Xeon Processor E5-2699 v3 18C 2.3GHz 45MB Cache 2133MHz 145W, 16 @ 16GB x4 DIMMS, 1 @ Dual-port 10GB Base-T Ethernet adapter, 2 @ 300GB 10K RPM disk (2 @ 1TB 7200 RPM for Dell) and 24x7x4 hour 3-year warranty upgrades (3). Both the Lenovo and HP sites show an almost identical number for RedHat Enterprise Linux with unlimited guests (Dell’s was harder to decipher since they apply discounts to the prices shown), so for consistency, I used the same price for RHEL including 3-yr premium subscription and support. VMware also offers their list prices on the web and the same numbers were used for each system, i.e. Version 5.5, 2-socket, premium support, 3yr (4).

The configuration for the S822L was created using IBM’s eConfig tool: 2 @ 10-core 3.42 GHz POWER8 Processor Card, 12 @ 32GB x4 DIMMS, 1 @ Dual-port 10GB Base-T Ethernet adapter, 2 @ 300GB 10K RPM disk and a 24x7x4 hour 3-year warranty upgrade, RHEL with unlimited guests and 3yr premium subscription and support and PowerVM with unlimited guests, 3yr 24×7 support (SWMA). Quick disclaimer; I am not a configuration expert with IBM’s products much less those from other companies which means there may be small errors, so don’t hold me to these numbers as being exact. In fact, if anyone with more expertise would like to comment on this post and provide more accurate numbers, I would appreciate that. You will see, however, that all three x86 systems fell in the same basic range, so small errors are likely of limited consequence.

The best list price among the Lintel vendors came in at $24,783 including the warranty upgrade. RHEL 7 came in at $9,259 and VMware @ $9,356 with a grand total for of $43,398 and for 3 systems, $130,194. For the IBM Power System, the hardware list was $33,136 including the warranty upgrade, PowerVM for Linux $10,450 and RHEL 7 $6,895 for a grand total of $51,109 and for 2 systems, $102,218.

So, for equivalent effective SAPS capacity, Lintel systems cost around $130K vs. $102K for Power … and this is before we consider the reliability and security advantages not to mention scalability, peak workload handling characteristics, reduced footprint, power and cooling. Just to meet the list price of the Power System, the Lintel vendors would have to deliver a minimum of 22% discount including RHEL and VMware.

For customers making HANA decisions, it is important to note that the app server does not go away and SAP fully support heterogeneous configurations, i.e. it does not matter if the app server is on a different platform or even a different OS than the HANA DB server. This means that Linux based Power Boxes are the perfect companion to HANA DB servers regardless of vendor.

For customers that are refreshing older Power app servers, the comparisons can be a little more complicated in that there is a reasonable case to be made for running app servers on enterprise class systems potentially also housing database servers in terms of higher effective utilization, higher reliability, the ability to run app servers in an IFL (Integrated Facility for Linux) at very attractive prices, increased efficiencies and improved speeds through use of virtual Ethernet for app to DB communications. That said, any analysis should start with like for like, e.g. two socket scale-out Linux servers, and then consider any additional value that can be gained through the use of AIX (with active memory expansion) and/or enterprise class servers with or without IFLs. As such, this post makes a clear point that, in a worst case scenario, scale-out Linux only Power Systems are less expensive than x86. In a best case scenario, the TCO, reliability and security advantages of enterprise class Power Systems make the value proposition of IBM Power even more compelling.

For customers that have already made the move to Lintel, the message is clear. You moved for sound economic, skills and standards based reasons. When it is time to refresh your app servers or add additional ones for growth or other purposes, those same reasons should drive you to make a decision to utilize IBM Power Systems for your app servers. Any customer that wishes to pursue such an option is welcome to contact me, your local IBM rep or an IBM business partner.

1. Dell PowerEdge R730 – 2 Processors / 36 Cores / 72 Threads 16,500 users, Red Hat Enterprise Linux 7, SAP ASE 16, SAP enhancement package 5 for SAP ERP 6.0, Intel Xeon Processor E5-2699 v3, 2.3 Ghz, 262,144MB, Cert # 2014033, 9/10/2014

2. IBM Power System S824, 4 Processors / 24 Cores / 192 Threads, 21,212 Users, AIX 7.1, DB2 10.5, SAP enhancement package 5 for SAP ERP 6.0, POWER8, 3.52 Ghz, 524,288MB, Cert # 2014016, 4/28/2014



January 23, 2015 Posted by | Uncategorized | , , , , , , , , , , , , | 4 Comments

How to ensure Business Suite on HANA infrastructure is mission critical ready

Companies that plan on running Business Suite on HANA (SoH) require systems that are at least as fault tolerant as their current mission critical database systems.  Actually, the case can be made that these systems have to exceed current reliability design specifications due to the intrinsic conditions of HANA, most notably, but not limited to, extremely large memory sizes.  Other factors that will further exacerbate this include MCOD, MCOS, Virtualization and the new SPS09 feature, Multi-Tenancy.

A customer with 5TB of data in their current uncompressed Suite database will most likely see a reduction due to HANA compression (SAP note 1793345, and the HANA cookbook²) bringing their system size, including HANA work space, to roughly 3TB.  That same customer may have previously been using a database buffer of 100GB +/- 50GB.  At a current buffer size of 100GB, their new HANA system will require 30 times the amount of memory as the conventional database did.  All else being equal, 30x of any component will result in 30x failures.  In 2009, Google engineers wrote a white paper in which they noted that 8% of DIMMS experienced errors every year with most being hard errors and that when a correctable error occurred in a DIMM, there was a much higher chance that another would occur in that same DIMM leading, potentially, to uncorrectable errors.¹  As memory technology has not changed much since then, other than getting denser which could lead to even more likelihood of errors due to cosmic rays and other sources, the risk has likely not decreased.  As a result, unless companies wish to take chances with their most critical asset, they should elect to use the most reliable memory available.

IBM provides exactly that, the best of breed open systems memory reliability, not as an option at a higher cost, but included with every POWER8 system, from the one and two socket scale-out systems to even more advanced capabilities with the 4 & 8-socket systems, some of which will scale to 16-sockets (announced as a Statement of Direction for 2015).  This memory protection is represented in multiple discreet features that work together to deliver unprecedented reliability.  The following gets into quite a bit of technical detail, so if you don’t have your geek hat on, (mine can’t be removed as it was bonded to my head when I was reading Heinlein in 6th grade; yes, I know that dates me), then you may want to jump to the conclusions at the end.

Chipkill – Essentially a RAID like technology that spans data and ECC recovery information across multiple memory chips such that in the event of a chip failure, operations may continue without interruption.   Using x8 chips, Chipkill provides for Single Device Data Correction (SDDC) and with x4 chips, provides Double Device Data Correction (DDDC) due to the way in which data and ECC is spread across more chips simultaneously.

Spare DRAM modules – Each rank of memory (4 ranks per card on scale-out systems, 8 ranks per card on enterprise systems) contains an extra memory chip.  This chip is used to automatically rebuild the data that was held, previously, on the failed chip in the above scenario.  This happens transparently and automatically.  The effect is two-fold:  One, once the recovery is complete, no additional processing is required to perform Chipkill recovery allowing performance to return to pre-failure levels; Two, maintenance may be deferred as desired by the customer as Chipkill can, yet again, allow for uninterrupted operations in the event of a second memory chip failure and, in fact, IBM does not even make a call out for repair until a second chip fails.

Dynamic memory migration and Hypervisor memory mirroring – These are unique technologies only available on IBM’s Enterprise E870 and E880 systems.  In the event that a DIMM experiences errors that cannot be permanently corrected using sparing capability, the DIMM is called out for replacement.  If the ECC is capable of continuing to correct the errors, the call out is known as a predictive callout indicating the possibility of a future failure.  In such cases, if an E870 or E880 has unlicensed or unassigned DIMMS with sufficient capacity to handle it, logical memory blocks using memory from a predictively failing DIMM will be dynamically migrated to the spare/unused capacity. When this is successful this allows the system to continue to operate until the failing DIMM is replaced, without concern as to whether the failing DIMM might cause any future uncorrectable error.  Hypervisor memory mirroring is a selective mirroring technology for the memory used by the hypervisor which means that even a triple chip failure in a memory DIMM would not affect the operations of the hypervisor as it would simply start using the mirror.

L4 cache – Instead of conventional parity or ECC protected memory buffers used by other vendors, IBM utilizes special eDRAM (a more reliable technology to start with) which not only offers dramatically better performance but includes advanced techniques to delete cache lines for persistent recoverable and non-recoverable fault scenarios as well as to deallocate portions of the cache spanning multiple cache lines.

Extra memory lane – the connection from memory DIMMs or cards is made up of dozens of “lanes” which we can see visually as “pins”.  POWER8 systems feature an extra lane on each POWER8 chip.  In the event of an error, the system will attempt to retry the transfer, use ECC correction and if the error is determined by the service processor to be a hard error (as opposed to a soft/transient error), the system can deallocate the failing lane and allocate the spare lane to take its place.  As a result, no downtime in incurred and planned maintenance may be scheduled at a time that is convenient for the customer since all lanes, including the “replaced” one are still fully protected by ECC.

L2 and L3 Caches likewise have an array of protection technology including both cache line delete and cache column repair in addition to ECC and special hardening called “soft latches” which makes these caches less susceptible to soft error events.

As readers of my blog know, I rarely point out only one side of the equation without the other and in this case, the contrast to existing HANA capable systems could not be more dramatic making the symbol between the two sides a very big > symbol; details to follow.

Intel offers a variety of protection technologies for memory but leaves the decision as to which to employ up to customers.  This ranges from “performance mode” which has the least protection to “RAS mode” which has more protection at the cost of reduced performance.

Let’s start with the exclusives for IBM:  eDRAM L4 cache with its inherent superior protection and performance over conventional memory buffer chips, dynamic memory migration and hypervisor memory mirroring available on IBM Enterprise class servers, none of which are available in any form on x86 servers.  If these were the only advantages for Power Systems, this would already be compelling for mission critical systems, but this is only the start:

Lock step – Intel included similar technology to Chipkill in all of their chips which they call Lock step.  Lock step utilizes two DIMMs behind a single memory buffer chip to store a 64-byte cache line + ECC data instead of the standard single DIMM to provide 1x or 2x 8-bit error detection and 8-bit error correction within a single x8 or x4 DRAM respectively (with x4 modules, this is known as Double Device Data Correction or DDDC and is similar to standard POWER Chipkill with x4 modules.)  Lock Step is only available in RAS mode which incurs a penalty relative to performance mode.  Fujitsu released a performance white paper³ in which they described the results of a memory bandwidth benchmark called STREAM in which they described Lock step memory as running at only 57% of the speed of performance mode memory.

Lock step is certainly an improvement over standard or performance mode in that most single device events can be corrected on the fly (and two such events serially for x4 DIMMS) , but correction incurs a performance penalty above and beyond that incurred from being in Lock step mode in the first place.  After the first such failure, for x8 DIMMS, the system cannot withstand a second failure in that Lockstep pair of DIMMS and a callout for repair (read this as make a planned shutdown as soon as possible) be made to prevent a second and fatal error.  For x4 DIMMS, assuming the performance penalty is acceptable, the planned shutdown could be postponed to a more convenient time.  Remember, with the POWER spare DRAMS, no such immediate action is required.

Memory sparing – Since taking an emergency shutdown is unacceptable for a SoH system, Lock Step memory is therefore insufficient since it handles only the emergency situation but does not eliminate the need for a repair action (as the POWER memory spare does) and it incurs a performance penalty due to having to “lash” together two cards to act as one (as compared to POWER that achieves superior reliability with a single memory card).  Some x86 systems offer memory sparing in which one rank per memory channel is configured as a spare.  For instance, with the Lenovo System x x3850, each memory channel supports 3 DIMMs or ranks.  If sparing is used, the effective memory throughput of the system is reduced by 1/3 since one of every 3 DIMMs is no longer available for normal operations and the memory that must be purchased is increased by 50%.  In other words, 1TB of usable memory requires 1.5TB of installed memory.  The downsize of sparing is that it is a predictive failure technology, not a reactive one.  According to the IBM X6 Servers: Technical Overview Redbook-  “Sparing provides a degree of redundancy in the memory subsystem, but not to the extent of mirroring. In contrast to mirroring, sparing leaves more memory for the operating system. In sparing mode, the trigger for failover is a preset threshold of correctable errors. When this threshold is reached, the content is copied to its spare. The failed rank is then taken offline, and the spare counterpart is activated for use.”  In other words, this works best when you can see it coming, not after a part of the memory has failed.    When I asked a gentleman manning the Lenovo booth at TechEd && d-code about sparing, he first looked at me as if I had a horn sticking out of my head and then replied that almost no one uses this technology.  Now, I think I understand why.  This is a good option, but at a high cost and still falls short of POWER8 memory protection which is both predictive and reactive and dynamically responds to unforeseen events.  By comparison, memory sparing requires a threshold to be reached and then enough time to be available to complete a full rank copy, even if only a single chip is showing signs of imminent failure.

Memory mirroring – This technology utilizes a complete second set of memory channels and DIMMs to maintain a second copy of memory at all times.  This allows for a chip or an entire DIMM to fail with no loss of data as the second copy immediately takes over.  This option, however, does require that you double the amount of memory in the system, utilize plenty of system overhead to keep the pairs synchronized and take away ½ of the memory bandwidth (the other half of which goes to the copy).  This option may perform better than the memory sparing option because reads occur from both copies in an interleaved manner, but writes have to occur to both synchronously.


Memory mirroring for x86 systems is the closest option to the continuous memory availability that POWER8 delivers.  Of course, having to purchase 2TB of memory in order to have proper protection of 1TB of effective memory adds a significant cost to the system and takes away substantial memory bandwidth.  HANA utilizes memory as few other systems do.

The problem is that x86 vendors won’t tell customers this.  Why?  Now, I can only speculate, but that is why I have a blog.  The x86 market is extremely competitive.  Most customers ask multiple vendors to bid on HANA opportunities.  It would put a vendor at a disadvantage to include this sort of option if the customer has not required it of all vendors.  In turn, x86 vendors don’t won’t to even insinuate that they might need such additional protection as that would imply a lack of reliability to meet mission critical standards.

So, let’s take this to the next logical step.  If a company is planning on implementing SoH using the above protection, they will need to double their real memory.  Many customers will need 4TB, 8TB or even some in the 12TB to 16TB range with a few even larger.  For the 4TB example, an 8TB system would be required which, as of the writing of this blog post, is not currently certified by SAP.  For the 8TB example, 16TB would be required which exceeds most x86 vendor’s capabilities.  At 12TB, only two vendors have even announced the intention of building a system to support 24TB and at 16TB, no vendor has currently announced plans to support 32TB of memory.

Oh, by the way, Fujitsu, in the above referenced white paper, measured the memory throughput of a system with memory mirroring and found it to be 69% that of a performance optimized system.  Remember, HANA demands extreme memory throughput and benchmarks typically use the fastest memory, not necessarily the most reliable meaning that if sizings are based on benchmarks, they may require adjustment when more reliable memory options are utilized.  Would larger core counts then be required to drive the necessary memory bandwidth?

Clearly, until SAP writes new rules to accommodate this necessary technology or vendors run realistic benchmarks showing just how much cpu and memory capacity is needed to support a properly mirrored memory subsystem on an x86 box, customers will be on their own to figure out what to do.

That guess work will be removed once HANA on Power GAs as it already includes the mission critical level of memory protection required for SoH and does so without any performance penalty.

Many thanks to Dan Henderson, IBM RAS expert extraordinaire, from whom I liberally borrowed some of the more technically accurate sentences in this post from his latest POWER8 RAS whitepaper¹¹ and who reviewed this post to make sure that I properly represented both IBM and non-IBM RAS options.


November 19, 2014 Posted by | Uncategorized | , , , , , , , , | Leave a comment

Questions about Linux on Power and HANA on Power

A curious reader just posed some questions that I suspect many more of you also have.  Here are the questions posed, and my answers:


“I remember at a trade show some years ago asking on the IBM stand how you ran Linux on a mainframe. I was told that whilst the SLES distribution had to be recompiled you could actually run compiled x86 Linux binaries on the mainframe. I thought that was pretty clever as getting all the ISVs to recompile for Linux on a mainframe would be a nightmare.

Coming to Linux on Power the web site is unclear whether you can run compiled x86 Linux binaries on Power. I suspected that the PowerVM hypervisor may be able to emulate the Intel instructions and run the x86 binaries, but is isn’t very clear.”

Both RedHat and SUSE Linux are operating systems which have been compiled to run on the Power platform.  The vast majority of the operating system code is identical across Power and x86 versions with only the low level code that directly interacts with the hardware being unique to the specific platform.  Currently, both run in big endian (BE) mode (byte order on disk and memory as opposed to x86 systems which run in little endian (LE); no positive or negative effect for either, simply a design choice).  As such, most applications running in those environments today require are compiled natively, not running with any emulation.  IBM did offer a “binary translation” function in 2011 called Lx86 which allowed x86 Linux applications to run unmodified on Linux on Power, but it was not widely adopted and was later removed from marketing in 2013.  In May, 2014, IBM announced that it would support the software based KVM as an alternative to the hardware/firmware hypervisor.  This allows customers that want to have a single set of administrative tools and skills to utilize KVM on both Power and x86 environment.  It also enables more OSs to run on Power as with KVM,  the system may be run in LE mode.  Canonical Ubuntu is now available on Power and is the first Linux OS to run in LE mode.  Both RedHat and SUSE are also available to run under KVM, however they currently run only in BE mode with SUSE planning on delivering a LE version (SLES 12) in the near future.  Debian and openSUSE are also reportedly working on LE versions for the Power platform.   Currently, LE is only supported under KVM and the entire system must run in the same mode.  In the future, IBM plans on supporting mixed mode allowing some partitions to run in one mode and others to run in the other mode, as well as allowing LE partitions to run under PowerVM.  Please read Jeff Scheel’s blog if you would like to know more about this subject:

“And that brings me nicely to HoP. Is the HANA code recompiled or can it take advantage of some form of emulation? ”

One of the key attributes of HANA is its incredible performance.  As such, even if it were possible to run with emulation, it would defeat the purpose of HANA to run in any sort of degraded mode. One of the ways that HANA delivers its speed is through the use of SIMD, (Single Instruction Multiple Data –  On the Intel platform, SIMD is called by the SSE instruction set and is a single pipeline vector unit within each processor core.  IBM offers a similar vector unit within each of the Power cores, called Altivec, and which now supports dual pipeline vector operations.  Each type of unit is utilized by HANA in the same way, but requires platform specific code.  As such, emulation would not allow SSE based code to work even in emulation mode on an Alitvec based system.  HANA was originally coded for SSE based operations in LE mode on the Intel platform. SAP has modified their code to support Altivec based operations in BE mode on the Power platform which was subsequently compiled to run on the Power platform under PowerVM natively.

October 2, 2014 Posted by | Uncategorized | Leave a comment

IBM @ TechEd && d-code

IBM usually has a significant presence at TechEd && d-code and this year is not different.  Of course, there will be the usual array of experts ranging from Systems and Software to HANA, Cloud and Consulting services at our booth.  In addition, IBM as well as some of our best customers will be hosting many sessions:


Thursday, October 23, 10:30 AM – 11:30 AM

Session ID MOB105: – Bellini 2105 Level 2

Title:  Apple + IBM: Evolving to the SAP Enabled Individual Enterprise

Speaker:  Scott Geddes

Description:  What’s next, now that you’ve done your first waves of transformation with SAP? How do you empower end users in ways never possible before and unleash the power of our SAP implementation?  In this session we will explore how Apple + IBM are working together to change the way people work and create new, never before seen capabilities.


Thursday, October 23

12:00pm – 1:00pm  Lounge #3

Apple + IBM: Evolving to the SAP Enabled Individual Enterprise (IBM and Apple alliance discussion cont’d)

Scott Geddes, IBM SAP Global Business Services – Mobility

Chuck Kichler, IBM SAP iCoC CTO

Tuesday, October 21, 2:00 PM – 3:00 PM

Session ID: DMM137

Title:  IBM’s Recommended Approach for Optimizing Different Kinds of SAP Workloads

Speaker: Guersad Kuecuek

Description:  Today, customers face various requirements to effectively deal with different kinds of workloads. Key aspects are high Service Level Agreements while maintaining optimal performance for analytical (OLAP) and transactional (OLTP) workloads. Find out how customers like Audi, Balluff, and Coca-Cola have mastered these challenging requirements.

Tuesday, October 21, 3:15 PM – 4:15 PM

Session ID:  DMM142

Title:   SAP HANA on IBM Power – Value, Performance and Experience

Speaker:  Alfred Freudenberger

Description:  With the announcement of the testing and evaluation program for SAP HANA on IBM Power Systems at SAPPHIRE NOW in 2014, a new option for SAP HANA deployments will soon be available. Why should SAP clients consider this option? For which environments is it well-suited? What have IBM and SAP learned during development, testing, and evaluation?


Wednesday, October 22

11:30am – 12:00pm  Lounge #4

SAP HANA on IBM Power – Value, Performance and Experience

Alfred Freudenberger, IBM Leader NA SAP on Power

Tuesday, October 21, 4:30 PM – 5:30 PM

Session ID: DMM145
Title:  Simplify IBM Database Performance Tuning with the DBA Cockpit

Speaker:  Thomas Rech

Description:  In today’s IT world, it is crucial to maintain high SAP system performance to meet demanding Service Level Agreements. The DBA Cockpit for IBM DB2 Linux, Unix, and Windows is an easy, fully integrated solution for database monitoring and administration with SAP. Learn about the design concept, the capabilities, and discuss customer use cases.

Wednesday, October 22, 11:45 AM – 12:45 PM

Session ID ITM220

Title:  Business Continuity for SAP HANA-Based Applications – Shared Experiences

Speaker:  Irene Hopf

Description:  Learn about the options to keep business continuously running when you migrate SAP application landscapes to SAP HANA. High availability and disaster recovery are essential for business-critical applications. Discuss experiences with your peers and learn how other customers have implemented it.

Wednesday, October 22, 5:45 PM – 6:45 PM

Session ID INT206

Title:  Integrating Shop-Floor with Enterprise in Real-Time – SAP MII In Action

Speaker:  Dipankar Saha

Description:  How to integrate heterogeneous shop-floor systems with SAP ERP by SAP Manufacturing Integration and Intelligence (SAP MII) using custom frameworks with various industry case-studies. This includes: manufacturing integration use cases, real-time integration using SAP MII, and architecture and case studies of integration using the frameworks.

Thursday, October 23, 8:00 AM – 9:00 AM

Session ID UXP117

Title: Experience with Google Glass and Business Applications

Speaker:  Markus van Kempen

Description:  Google Glass presents a mobile form-factor which allows for new possibilities. This session discusses examples of user experiences, including the disconcerting experience of “wearing” a camera all the time, reactions from others, and navigation challenges. We show how to design for Google Glass and demonstrate business applications.

Thursday, October 23, 10:45 AM – 11:45 AM

Session ID ITM235

Title:  Establishing Architectural Patterns for SAP in the Cloud at CokeOne +

Speaker:  Michael Ryan

Description:  The CokeOne + migration to cloud for their non-production SAP environments included the establishment of architectural patterns to take advantage of the services provided by cloud computing. This session focuses on establishing the architectural patterns needed to transform businesses by moving business systems and processes to a cloud model.

Thursday, October 23, 2:00 PM – 3:00 PM

Session ID DMM127

Title:  Streamline SAP HANA Solution with Near-Line Storage Solution by PBS and IBM

Speaker:  Elke Hartmann-Bakan

Description:  Streamline your SAP HANA solution by keeping only hot data in memory and moving warm data to near-line storage (NLS). This allows you to maintain a lean SAP HANA database and sustain high performance. The PBS and IBM NLS solution offers near real-time speed on NLS, ultra fast load time from the online database to the NLS, and extreme compression.

Thursday, October 23, 4:30 PM – 5:30 PM

Session ID ITM123

Title: Planning Your Company’s SAP Systems Migration to the Cloud

Speaker:  Michael Ryan

Description:  The opportunity to move the SAP infrastructure to cloud is a game changer. Businesses are offered a level of speed and agility that has not been available in the past. However, moving to cloud does not solve basic issues that we experience in the IT world. We take a look at some of the key issues and think about the impact across enterprises.


Tuesday, October 21

2:30pm – 3:00pm  Lounge #4

SAP Applications on IBM Cloud – from self-service to fully managed

Keith Murray, Global Offerings Manager SAP on SoftLayer, IBM SmartCloud Services

Wolfgang Knobloch,  IBM GTS Global Offering Manager, SAP Cloud Offerings

I look forward to seeing you there.

October 1, 2014 Posted by | Uncategorized | Leave a comment


Get every new post delivered to your Inbox.

Join 185 other followers