Scale-up vs. scale-out architectures for SAP HANA – part 1
Dozens of articles, blog posts, how-to guides and SAP notes have been written about this subject. One of the best was by John Appleby, now Global Head of DDM/HANA COEs @ SAP.[i] Several others have been written by vendors with a vested interest in the proposed option. The vendor for which I work, IBM, offers excellent solutions for both options, so my perspective is based on both my and the experiences of our many customers, some that have chosen one or the other option, or both, in some cases.
Scale-out for BW is well established, understood, fully supported by SAP and can be cost effective from the perspective of systems acquisition costs. Scale-out for S/4HANA, by comparison, is in use by very few customers, not well understood, yet is support by SAP for configurations up to 4 nodes. Does this mean that a scale-out architecture should always be used for BW and a scale-up architecture for S/4HANA the only viable choice? This blog post will discuss only BW and similar analytical environments including BW/4HANA, data marts, data lakes, etc. The next will discuss S/4HANA and the third in the series will discuss vendor selection and where one might have an advantage over the others.
Scale-out has 3 key advantages over scale-up:
- Every vendor can participate therefore competitive bidding of “commodity” level systems can result in optimal pricing.
- High availability, using host auto-failover requires nothing more than n+1 systems as the hot standby node can take over the role of any other node (some customers chose n+2 or group nodes and standby nodes).
- Some environments are simply too large to fit in even the largest supported scale-up systems.
Scale-up, likewise, has 3 key advantages over scale-out:
- Performance is, inevitably, better as joins across memory are always faster than joins across a network
- Management is much simpler as query analysis and data distribution decisions need not be performed on a regular basis plus fewer systems are involved with the corresponding decrease in monitoring, updating, connectivity, etc.
- TCO can be lower when the costs of systems, storage, network and basis management are included.
Business requirements, as always, should drive the decision as to which to use. As mentioned, when an environment is simply too large, unless a customer is willing to ask for an exception from SAP (and SAP is willing to grant it), then scale-out may be the only option. Currently, SAP supports BW configurations of up to 6TB on many 8-socket Intel Skylake based systems (up to 12TB on HPE’s 16-socket system) and up to 16TB on IBM Power Systems.
The next most important issue is usually cost. Let’s take a simple example of an 8TB BW HANA requirement. With scale-out, 4 @ 2TB nodes may be used with a single 2TB node for hot standby for a total of 10TB of memory. If scale-up is used, the primary system must be 8TB and the hot-standby another 8TB for a total of 16TB of memory. Considering that memory is the primary driver of the cost of acquisition, 16TB, from any vendor, will cost more than 10TB. If the analysis stops there, then the decision is obvious. However, I would strongly encourage all customers to examine all costs, not just TCA.
In the above example, 5 systems are required for the scale-out configuration vs. 2 for scale-up. The scale-out config could be reduced to 4 systems if 3TB nodes are used with 1TB left unused although the total memory requirement would go up to 12TB. At a minimum, twice the management activities, trouble-shooting and connectivity would be required. Also, remember, prod rarely exists on its own with some semblance of the configuration existing in QA, often DR and sometimes other non-prod instances.
The other set of activities is much more intensive. To distribute load amongst the systems, first data must be distributed. Some data must reside on the master node, e.g. all row-store tables, ABAP tables, general operations tables. Other data such as Fact, DataStore Object (DSO), Persistent Staging Area (PSA) is distributed evenly across the slave nodes based on the desired partitioning specification, e.g. hash, round robin or range. There are also more complex options where specifications can be mixed to get around hash or range limitations and create a multi-level partitioning plan). And, of course, you can partition different tables using different specifications. Which set of distribution specifications you use is highly dependent on how data is accessed and this is where it gets really complicated. Most customers start with a simple specification, begin monitoring placement using the table distribution editor and performance using STO3N plus getting feedback from end users (read that as complaints to the help desk). After some period of time and analysis of performance, many customers elect to redistribute data using a better or more complex set of specifications. Unfortunately, what is good for one query, e.g. distribute data based on month, is bad for another which looks for data based on zipcode, customer name or product number. Some customers report that the above set of activities can consume part or all of one or more FTEs.
Back to the above example. 10TB vs. 16TB which we will assume is replicated in QA and DR, for sake of argument, i.e. the scale-up solution requires 18TB more memory. If the price per TB is $35,000 then the cost different in TCA would be $630,000. The average cost of a senior basis administrator (required for this sort of complex task) in most western countries is in the $150,000 range. That means that over the course of 5 years, the TCO of the scale-up solution, considering only TCA and basis admin costs would be roughly equivalent to the cost of the scale-out solution. Systems, storage and network administration costs could push the TCO of the scale-out solution up relative to the scale-up solution.
And then there is performance. Some very high performance network adapter companies have been able to drive TCP latency across a 10Gb Ethernet down to 3.6us which sounds really good until you consider memory latency is around 120ns, i.e. 30 times faster. Joining tables across nodes not only is substantially slower, but also results in more CPU and memory overhead.[ii] A retailer in Switzerland, Coop Group, reported 5 times quicker analytics while using 85% fewer cores after migrating from an 8-node x86 scale-out BW HANA cluster with 320 total cores to a single scale-up 96-core IBM Power Systems.[iii] While various benchmarks suggest 2x or better per core performance of Power Systems vs. x86, the results suggest far higher, much of which can, no doubt, be attributed to the effect of using a scale-up architecture.
Of course, performance is relative. BW queries run with scale-out HANA will usually outperform BW on a conventional DB by an order of magnitude or more. If this is sufficient for business purposes, then it may be hard to build a case for why faster is required. But end users have a tendency to soak up additional horsepower once they understand what is possible. They do this in the form of more what-if analyses, interactive drill downs, more frequent mock-closes, etc.
If the TCO is similar or better and a scale-up approach delivers superior performance with many fewer headaches and calls to the help desk for intermittent performance problems, then it would be very worthwhile to investigate this option.
To recap; For BW HANA and similar analytical environments, Scale-out architectures usually offer the lowest TCA and scalability beyond the largest scale-up environment. Scale-up architectures offers significantly easier administration, much better performance and competitive to superior TCO.
[i]https://blogs.saphana.com/2014/12/10/sap-hana-scale-scale-hardware/
[ii]https://launchpad.support.sap.com/#/notes/2044468(see FAQ 8)
[iii]https://www.ibm.com/case-studies/coop-group-technical-reference
TDI Phase 5 – SAPS based sizing bringing better TCO to new and existing Power Systems customers
SAP made a fundamental and incredibly important announcement this week at SAP TechEd in Las Vegas: TDI Phase 5 – SAPS based sizing for HANA workloads. Since its debut, HANA has been sized based on a strict memory to core ratio determined by SAP based on workloads and platform characteristics, e.g. generation of processor, MHz, interconnect technology, etc. This might have made some sense in the early days when much was not known about the loads that customers were likely to experience and SAP still had high hopes for enabling all customer employees to become knowledge workers with direct access to analytics. Over time, with very rare exception, it turned out that CPU loads were far lower than the ratios might have predicted.
I have only run into one customer in the past two years that was able to drive a high utilization of their HANA systems and that was a customer running an x86 BW implementation with an impressively high number of concurrent users at one point in their month. Most customers have experienced just the opposite, consistently low utilization regardless of technology.
For many customers, especially those running x86 systems, this has not been an issue. First, it is not a significant departure from what many have experienced for years, even those running VMware. Second, to compensate for relatively low memory and socket-to-socket bandwidth combined with high latency interconnects, many x86 systems work best with an excess of CPU. Third, many x86 vendors have focused on HANA appliances which are rarely utilized with virtualization and are therefore often single instance systems.
IBM Power Systems customers, by comparison, have been almost universal in their concern about poor utilization. These customers have historically driven high utilization, often over 65%. Power has up to 5 times the memory bandwidth per socket of x86 systems (without compromising reliability) and very wide and parallel interconnect paths with very low latencies. HANA has never been offered as an appliance on Power Systems, instead being offered only using a Tailored Datacenter Infrastructure (TDI) approach. As a result, customers view on-premise Power Systems as a sort of utility, i.e. that they should be able to use them as they see fit and drive as much workload through them as possible while maintaining the Service Level Agreements (SLA) that their end users require. The idea of running a system at 5%, or even 25%, utilization is almost an affront to these customers, but that is what they have experienced with the memory to core restrictions previously in place.
IBM’s virtualization solution, PowerVM, enabled SAP customers to run multiple production workloads (up to 8 on the largest systems) or a mix of production workloads (up to 7) with a shared pool of CPU resources within which an almost unlimited mix of VMs could run including non-prod HANA, application servers, as well as non-SAP and even other OS workloads, e.g. AIX and IBM i. In this mixed mode, some of the excess CPU resource not used by the production workloads could be utilized by the shared-pool workloads. This helped drive up utilization somewhat, but not enough for many.
These customers would like to do what they have historically done. They would like to negotiate response time agreements with their end user departments then size their systems to meet those agreements and resize if they need more capacity or end up with too much capacity.
The newly released TDI Overview document http://bit.ly/2fLRFPb describes the new methodology: “SAP HANA quicksizer and SAP HANA sizing reports have been enhanced to provide separate CPU and RAM sizing results in SAPS”. I was able to verify Quicksizer showing SAPS, but not the sizing reports. An SAP expert I ran into at TechEd suggested that getting the sizing reports to determine SAPS would be a tall order since they would have to include a database of SAPS capacity for every system on the market as well as number of cores and MHz for each one. (In a separate blog post, I will share how IBM can help customers to calculate utilized SAPS on existing systems). Customers are instructed to work with their hardware partner to determine the number of cores required based on the SAPS projected above. The document goes on to state: “The resulting HANA TDI configurations will extend the choice of HANA system sizes; and customers with less CPU intensive workloads may have bigger main memory capacity compared to SAP HANA appliance based solutions using fixed core to memory sizing approach (that’s more geared towards delivery of optimal performance for any type of a workload).”
Using a SAPS based methodology will be a good start and may result in fewer cores required for the same workload as would have been previously calculated based on a memory/core ratio. Customers that wish to allocate more of less CPU to those workloads will now have this option meaning that even more significant reduction of CPU may be possible. This will likely result in much more efficient use of CPU resources, more capacity available to other workloads and/or the ability to size systems with less resources to drive down the cost of those systems. Either way helps drive much better TCO by reducing numbers and sizes of systems with the associated datacenter and personnel costs.
Existing Power customers will undoubtedly be delighted by this news. Those customers will be able to start experimenting with different core allocations and most will find they are able to decrease their current HANA VM sizes substantially. With the resources no longer required to support production, other workloads currently implemented on external systems may be consolidated to the newly, right sized, system. Application servers, central services, Hadoop, HPC, AI, etc. are candidates to be consolidated in this way.
Here is a very simple example: A hypothetical customer has two production workloads, BW/4HANA and S/4HANA which require 4TB and 3TB respectively. For each, HA is required as is Dev/Test, Sandbox and QA. Prior to TDI Phase 5, using Power Systems, the 4TB BW system would require roughly 82-cores due to the 50GB/core ratio and the S/4 workload would require roughly 33 cores due to the 96GB/core ratio. Including HA and non-prod, the systems might look something like:
Note the relatively small number of cores available in the shared pool (might be less than optimal) and the total number of cores in the system. Some customers may have elected to increase to an even larger system or utilize additional systems as a result. As this stood, this was already a pretty compelling TCO and consolidation story to customers.
With SAPS based sizing, the BW workload may require only 70 cores and S/4 21 cores (both are guesses based on early sizing examples and proper analysis of the SAP sizing reports and per core SAPS ratings of servers is required to determine actual core requirements). The resulting architecture could look like:
Note the smaller core count in each system. By switching to this methodology, lower cost CPU sockets may be employed and processor activation costs decreased by 24 cores per system. But the number of cores in the shared pool remains the same, so still could be improved a bit.
During a landscape session at SAP TechEd in Las Vegas, an SAP expert stated that customers will be responsible for performance and CPU allocation will not be enforced by SAP through HWCCT as had been the case in the past. This means that customers will be able to determine the number of cores to allocate to their various instances. It is conceivable that some customers will find that instead of the 70 cores in the above example, 60, 50 or fewer cores may be required for BW with decreased requirements for S/4HANA as well. Using this approach, a customer choosing this more hypothetical approach might see the following:
Note how the number of cores in the shared pool have increased substantially allowing for more workloads to be consolidated to these systems, further decreasing costs by eliminating those external systems as well as being able to consolidate more SAN and Network cards, decreasing computer room space and reducing energy/cooling requirements.
A reasonable question is whether these same savings would accrue to an x86 implementation. The answer is not necessarily. Yes, fewer cores would also be required, but to take advantage of a similar type of consolidation, VMware must be employed. And if VMware is used, then a host of caveats must be taken into consideration. 1) overhead, reportedly 12% or more, must be added to the capacity requirements. 2) I/O throughput must be tested to ensure load times, log writes, savepoints, snapshots and backup speeds which are acceptable to the business. 3) limits must be understood, e.g. max memory in a VM is 4TB which means that BW cannot grow by even 1KB. 4) Socket isolation is required as SAP does not permit the sharing of a socket in a HANA production/VMware environment meaning that reducing core requirements may not result in fewer sockets, i.e. this may not eliminate underutilized cores in an Intel/VMware system. 5) Non-prod workloads can’t take advantage of capacity not used by production for several reasons not the least of which is that SAP does not permit sharing of sockets between VM prod and non-prod instances not to mention the reluctance of many customer to mix prod and non-prod using a software hypervisor such as VMware even if SAP permitted this. Bottom line is that most customers, through an abundance of caution, or actual experience with VMware, choose to place production on bare-metal and non-prod, which does not require the same stack as prod, on VMware. Workloads which do require the same stack as prod, e.g. QA, also are usually placed on bare-metal. After closer evaluation, this means that TDI Phase 5 will have limited benefits to x86 customers.
This announcement is the equivalent of finally being allowed to use 5th gear on your car after having been limited to only 4 for a long time. HANA on IBM Power Systems already had the fastest adoption in recent SAP history with roughly 950 customers selecting HANA on Power in just 2 years. TDI Phase 5 uniquely benefits Power Systems customers which will continue the acceleration of HANA on Power. Those individuals that recommended or made decisions to select HANA on Power will look like geniuses to their CFOs as they will now get the equivalent of new systems capacity at no cost.
HANA on Power hits the Trifecta!
Actually, trifecta would imply only 3 big wins at the same time and HANA on Power Systems just hit 4 such big wins.
Win 1 – HANA 2.0 was announced by SAP with availability on Power Systems simultaneously as with Intel based systems.[i] Previous announcements by SAP had indicated that Power was now on an even footing as Intel for HANA from an application support perspective, however until this announcement, some customers may have still been unconvinced. I noticed this on occasion when presenting to customers and I made such an assertion and saw a little disbelief on some faces. This announcement leaves no doubt.
Win 2 – HANA 2.0 is only available on Power Systems with SUSE SLES 12 SP1 in Little Endian (LE) mode. Why, you might ask, is this a “win”? Because true database portability is now a reality. In LE mode, it is possible to pick up a HANA database built on Intel, make no modifications at all, and drop it on a Power box. This removes a major barrier to customers that might have considered a move but were unwilling to deal with the hassle, time requirements, effort and cost of an export/import. Of course, the destination will be HANA 2.0, so an upgrade from HANA 1.0 to 2.0 on the source system will be required prior to a move to Power among various other migration options. This subject will likely be covered in a separate blog post at a later date. This also means that customers that want to test how HANA will perform on Power compared to an incumbent x86 system will have a far easier time doing such a PoC.
Win 3 – Support for BW on the IBM E850C @ 50GB/core allowing this system to now support 2.4TB.[ii] The previous limit was 32GB/core meaning a maximum size of 1.5TB. This is a huge, 56% improvement which means that this, already very competitive platform, has become even stronger.
Win 4 – Saving the best for last, SAP announced support for Suite on HANA (SoH) and S/4HANA of up to 16TB with 144 cores on IBM Power E880 and E880C systems.ii Several very large customers were already pushing the previous 9TB boundary and/or had run the SAP sizing tools and realized that more than 9TB would be required to move to HANA. This announcement now puts IBM Power Systems on an even footing with HPE Superdome X. Only the lame duck SGI UV 300H has support for a larger single image size @ 20TB, but not by much. Also notice that to get to 16TB, only 144 cores are required for Power which means that there are still 48 cores unused in a potential 192 core systems, i.e. room for growth to a future limit once appropriate KPIs are met. Consider that the HPE Superdome X requires all 16 sockets to hit 16TB … makes you wonder how they will achieve a higher size prior to a new chip from Intel.
Win 5 – Oops, did I say there were only 4 major wins? My bad! Turns out there is a hidden win in the prior announcement, easily overlooked. Prior to this new, higher memory support, a maximum of 96GB/core was allowed for SoH and S/4HANA workloads. If one divides 16TB by 144 cores, the new ratio works out to 113.8GB/core or an 18.5% increase. Let’s do the same for HPE Superdome X. 16 sockets times 24 core/socket = 384 cores. 16TB / 384 cores = 42.7GB/core. This implies that a POWER8 core can handle 2.7 times the workload of an Intel core for this type of workload. Back in July, I published a two-part blog post on scaling up large transactional workloads.[iii] In that post, I noted that transactional workloads access data primarily in rows, not in columns, meaning they traverse columns that are typically spread across many cores and sockets. Clearly, being able to handle more memory per core and per socket means that less traversing is necessary resulting in a high probability of significantly better performance with HANA on Power compared to competing platforms, especially when one takes into consideration their radically higher ccNUMA latencies and dramatically lower ccNUMA bandwidth.
Taken together, these announcements have catapulted HANA on IBM Power Systems from being an outstanding option for most customers, but with a few annoying restrictions and limits especially for larger customers, to being a best-of-breed option for all customers, even those pushing much higher limits than the typical customer does.
[i] https://launchpad.support.sap.com/#/notes/2235581
[ii] https://launchpad.support.sap.com/#/notes/2188482
[iii] https://saponpower.wordpress.com/2016/07/01/large-scale-up-transactional-hana-systems-part-1/
SAPonPower 