The high ground in the server market used to be large-scale SMP and NUMA
machines with 16, 32, 64, or 128 sockets all lashed together to make a
big shared memory machine. But that was back in the days when processors
have one or maybe two cores, and the pace of Moore’s Law increases in
transistor etching technologies has allows processor makers like Intel,
IBM, and Oracle to cram a lot more cores and threads onto a single die.
Enterprise workloads do not grow as fast as hyperscale and HPC parallel
workloads, and that means over time a fairly modest machine from 2015
has the oomph of big iron from a decade ago.
Such is the case with machines based on Intel’s “Haswell-EX” Xeon E7 v3 processors,
announced last week
and scaling up to eight sockets using Intel’s on-chip NUMA links, and
the Power8 midrange and high end, which IBM is updating to go up against
Xeon-based machines, particularly for Linux-based applications.
The Platform covered the expansion of IBM’s high-end Power E880 machines,
which scale out to a maximum of 16 sockets and 16 TB of main memory,
last week. IBM is formally announcing these high-end boxes at its
Edge2015 conference in Las Vegas this week, but details on the largest
of its Power8 machines slipped out a bit early. The final machine to be
added to the Power8-based Power Systems lineup from Big Blue is the
Power E850, and it is a four-socket machine that is aimed squarely at
systems from Hewlett-Packard, Dell, Oracle, Fujitsu, NEC, and others
that employ Intel’s Xeon E7-4800 v3 CPUs, which similarly support
four-way NUMA clustering in their hardware.
The Power E850 is a bit different from four-socket boxes that IBM has
shipped in the past. For one thing, it includes some capacity-on-demand
features that up until now have only been available on larger Power
Systems machines. With capacity on demand, IBM ships a box loaded with
processors and main memory and allows customers to activate it as needed
either permanently or temporarily on a daily or monthly basis with
utility pricing. The base Power E850 system ships with two processors
and a full memory complement (based on 16 GB, 32 GB, or 64 GB memory
sticks) as a base, and customers active Power8 cores and memory in 1 GB
increments.
The engines in the Power E850 are based on IBM’s “Murano” dual-chip
module, which puts two half-cored Power8 chips into a single Power8
socket and links them by a crossbar. IBM uses dual-chip modules for a
number of reasons, and the first is that a smaller chip has a high yield
than a larger chip, generally speaking, and this is important because
IBM’s 22 nanometer copper/SOI process, now controlled by
GlobalFoundries, is nowhere near as high volume as Intel’s 22 nanometer
Tri-Gate process, which is used to make the Haswell-EX Xeon E7 v3
processors. The DCM variants of the Power8 chips have more I/O capacity
on their PCI-Express controllers, at 48 lanes per second instead of the
32 lanes per socket that are used in the single-chip variants of the
Power8 chips used in the high-end Power E870 and Power E880 systems,
which respectively scale to eight and sixteen sockets in a single image.
These SCM Power8 variants do not need as much I/O bandwidth per socket
because they have many more sockets in a system.
Initially, IBM will be supporting up to 2 TB of maximum main memory
across the 32 memory slots, but Steve Sibley, director of worldwide
product management for IBM’s Power Systems line, says that Big Blue will
double it up again with its 128 GB memory modules, maybe later this
year or early next. That will give the Power E850 the same maximum
memory per socket as the top-end Power E880.
Intel supports 6 TB of memory across four sockets using 64 GB DDR3 or
DDR4 memory right now, and IBM only supports DDR3 memory (which
generally runs hotter for a given level of performance). The memory
controllers in the Power8 chips were designed to be protocol agnostic
and can support either DDR3 or DDR4 memory, but IBM tends to lag when it
comes to memory because it likes to keep its memory costs low and its
profit margins high.
Importantly, all of the enterprise-class Power8 machines make use of
IBM’s “Centaur” memory buffer and controller chip, which has a chunk of
16 MB memory on it that is used to make up to 128 MB of L4 cache memory
between the main memory and the L3 cache subsystem on the Power8
processors. The Xeon E7 v3 processors do not have L4 cache memory, and
one of the reasons why IBM has been able to jack up the memory bandwidth
on the Power8s relative to X86 architectures is this distributed L4
cache. Memory bandwidth and higher performance per core are the two key
selling points that IBM is leveraging to promote the Power8 chip over
Xeon alternatives. (No one talks much about AMD Opterons anymore, but
that could change in a few years if AMD revamps its X86 server business as it plans to.)
As you can see, IBM is packing a lot of electronic components into
the 4U chassis of the Power E850 system. The machine has four
processors, and it is very likely that they run in a 190 watt thermal
envelope (like the merchant silicon variants of the Power8 chips that
Google, Tyan, and others are building systems based on) instead of the
hotter 250 watt chips that IBM has used in its other and less densely
packed systems. IBM is supporting three different variants of the Murano
DCM: one with eight cores in the package that run at 3.72 GHz, one with
ten cores that runs at 3.35 GHz, and one with twelve cores that run at
3.02 GHz.
The Power E850 has fans and drives in the front, and specifically,
five large fans on top that blow first across the memory sticks, then
the processors and then the PCI-Express slots in the back. Drive bays
are below this – eight 2.5-inch drives, four 1.8-inch SSDs, and one DVD
drive – and four power supplies fill the bottom of the rack behind the
drives.
Customers with more storage requirements can hang up to four I/O
drawers off the Power E850, which can have as many as 40 PCI-Express
peripheral cards in them. The Power E850 system has eight PCI-Express
3.0 x16 slots and three PCI-Express 3.0 x8 slots internally. It seems
very unlikely, given its dense packaging, that customers could put more
than a couple GPU coprocessors into this machine, and given the
workloads that the Power E850 is aimed at – database, in-memory
processing (particularly SAP HANA and IBM DB2 Blu), analytics, and fat
HPC cluster nodes – it is very unlikely that anyone will add GPUs to
this machine. The Power E850 comes with a dual-port 10 Gb/sec Ethernet
card in one of its PCI-Express slots by default.
Pushing Linux On Power Hard
Another thing that IBM is doing to keep the Power E850 competitive
with systems using Intel’s Xeon E7 v3 processors (and the impending Xeon
E5-4600 v3 variant of the Haswell chip, aimed at lower-cost four-socket
machines) is offering what it calls Integrated Facility for Linux, or
IFL, pricing on the Power8 cores in the Power E850 system. With the IFL
approach, IT shops can restrict cores to running Linux and if they do,
IBM gives them a big price break. The idea is somewhat counterintuitive,
and the net result is IBM ends up charging customers using its own AIX
Unix and IBM i proprietary operating system considerably more on
processors and memory than it does for customers using Linux. This might
be annoying to customers using AIX or IBM i, but they are a lot more
captive (given that IBM is the only system supplier that supports them,
and that probably will not change as the OpenPower systems come to
market until IBM decides to get out of the server hardware design and
manufacturing business) and hence have fewer options than Linux
customers. IBM has to drop prices for Linux systems no matter what
The Platform is putting together an analysis to compare the
compute performance and price/performance for Xeon, Power, and Sparc
processors to try to get a better handle on how these platforms stack
up. Sibley says that a fully loaded, four-socket Power E850 with 48
cores will have somewhere between 5 percent and 10 percent more oomph in
terms of raw performance compared to a four-socket Xeon E7-4800 v3
machine with 72 cores. Pricing for the machines configured with a
hypervisor and Linux will be about the same, he says, and with a 70
percent utilization guarantee – meaning, IBM is promising that customers
can load this machine up to that level of CPU capacity and still have
workloads run with snappy response time – the gap widens up because, at
least according to IBM’s tests, VMware ESXi on Xeons does not handle
multiple workloads as well as IBM’s PowerVM hypervisor on Power8s. The
gap on workloads could be as much as a 30 percent to 40 percent
price/performance advantage favoring the Power E850 over a Xeon E7 box.
Intel can – and does – show its own charts illustrating how it beats Power8 machines.
What IBM is equally focused on is showing how the Power E850 offers a
significant performance boost to its own customers running AIX and Linux
workloads. (The Power E850 does not support the IBM i operating system,
which is sure to annoy a bunch of the company’s customers who need more
than a two-socket machine. The will be encouraged to buy a half-loaded
Power E870 machine, which is more expensive and which is put into a
higher software pricing tier, too, that will make IBM systems software
and third party application software more expensive.)
The Power 750 four-socket machines that IBM announced in April 2010
using its Power7 processors are looking a little long in the tooth now
and are the main targets in the IBM customer base where Big Blue and its
partners are expected to push the new Power E850 four-socket box. Core
for core, this Power E850 offers about twice the performance of the
Power 750. The Power 750 used single-chip variants of the Power7 and
Power7+ processors, but the Power 760 tested out the dual-chip module
idea, and that is why you don’t see a Power E860 in the lineup. In
effect, the Power E850 is the DCM variant; IBM did play around with the
idea of having a SCM variant of the Power8 in a four-socket machine, but
for whatever reason it has decided against the idea, so far. The point
is, the resulting machine has a lot more oomph than the prior two
generations, and customers who use this class of machine for application
serving, database clustering, and in-memory processing will be looking
hard at the Power E850. (Up until when the Chinese government put the
brakes on buying Power Systems iron from IBM nearly two years ago, the
Power 550 and Power 750 machines were hot sellers in the Middle
Kingdom.)
The Power E850 will be available on June 5. IBM’s AIX 6.1 and 7.1
Unix variants are supported on the machine. The Power8 chip supports big
endian byte ordering (used by prior Power chips) and little endian byte
ordering (used by X86 processors), and now Linux variants can run in
either mode. Red Hat Enterprise Linux 6.6 and 7.1 are both supported in
big endian mode, and so is SUSE Linux Enterprise Server 11 SP3. In
little ending mode, RHEL 7.1, SLES 12, and Canonical Ubuntu Server 14.04
and 15.04 are all supported. For Linux and AIX, IBM is also moving the
machine down to the small software tier, rather than the medium one,
which further lowers the price of application software on the box. This
may or may not make software vendors happy. Pricing on the Power E850
was not available at press time, but we are digging.
Link : http://www.theplatform.net/2015/05/11/power8-iron-to-take-on-four-socket-xeons/
Written by :
Timothy Prickett Morgan
May 11, 2015