Leap Day 2016 – VMAX & All Flash

Happy Leap Day; a day every four years I’m reminded how technologists across the globe can manage to work together to ensure our systems all stay on (relatively) the same time, even with odd things like leap day thrown into the mix!

If you haven’t been on social media in the past few weeks, you might have missed today EMC is making quite a few announcements. One of them, the ‘All Flash VMAX’ storage array, is something I want to share some thoughts on. I’ve answered quite a few questions about VMAX over the years, from running it, from technical panels and references calls; so I’m going to put this post in Q/A type format. It’s also a little long, if you feel like the cliff notes, head over to GotDedupe.com for the Podcast on this subject; or hookup your favorite podcast subscription tool to Remain Silent.

Before we start, a few things I want to make clear, as I think it’s germane to the conversation:

  • I do not work for EMC; though I am EMC Elect, and have been on and off their Enterprise Technical Advisory Panel for years.
  • I have been a Symmetrix/VMAX customer for a little over 15 years in some form or fashion; yes I still call it Symmetrix.
  • I’ve been a customer of countless other storage technologies as well.
  • I’m going to refer to ‘other’ storage techniques, when doing so, don’t read between the lines. I’m talking about all products, both EMC and their competitor. This blog isn’t about trash talking the non-VMAX arrays, this is about what makes VMAX great.
  • I’m going to be candid. In many cases, the questions are a little obtuse simply because it’s what I hear.
  • My response here is similar to what I’d say in person; akin to some very conversations I’ve had. Please read in that vein and with the humor I intend.
  • These are my opinions; I’d love to debate them if you don’t share them.

Q: Isn’t EMC late to the game with All Flash VMAX?
A: First, no, they are not. I’m not sure people can argue specific dates, but the Symmetrix line has had flash drives for years. Before even the VMAX product (think DMX days). Yes, it predominantly was as a caching layer for FAST VP tiering, or full LUN tiering, or in small increments to carve up for application use like temp tables; but it’s been in the product for longer than any ‘All-Flash-Array’ even existed to my knowledge. I spoke with people running All Flash DMX arrays years ago for very critical workloads in the finance space; it was, very expensive; but so was the workload they were supporting. So you could buy an all-flash VMAX before, it warranted some special sizing and attention is all.

Q: But there are other ‘all-flash-arrays’ out there that are selling like hot-cakes, so isn’t this EMC just playing catch-up?
A: Still no, besides the previous point and the fact they have analyst leading all-flash-array in their portfolio already, because quite simply, in almost all the scenarios a VMAX properly configured with FAST VP tiering could produce the same performance as those all-flash-arrays; so there wasn’t a large need to create an All Flash version of VMAX before. Those other arrays offset the price of flash with deduplication, which in some scenarios worked incredibly well; but not in all scenarios. In fact, if you didn’t get a good dedupe rate, you not only had to buy more storage but buy at a price higher than you originally thought based on the initial dedupe ratio you were told at pre-sales. VMAX offset the price with tiering; which doesn’t have the unknown dedupe problem, though does have a parallel problem if the IO density skew was not correct. Today, with 3D NAND flash bringing consistent performance at ever lowering prices, and the fact we’re seeing flash drives will outpace spinning disk for capacity per square inch, the economics of flash can now be the price offset.

Q: Matt, you’re avoiding the point, “why even play if you show up late to the game.”
A: Well you didn’t really read my previous points, but fine; the introduction of an ‘All Flash VMAX’ is behind other products that market an ‘all flash array.’ But answer my question now:

If you had a binary choice between bleeding edge, or reliability, which would you choose?

If it were a binary choice, I would choose reliability. To the point, VMAX arrays are sitting behind some of the most mission-critical workloads in the world. VMAX handle transactions for large credit providers that cannot deal with any transaction getting lost or slowing down. VMAX sits in large hospitals that have people lives riding on stable infrastructure. Banks have multiple VMAX arrays with replication in multiple countries. Not just two sites, but three or four; chained replication, star replication, in some cases, I know banks that have ‘bunker VMAX’ array simply receiving a copy of all their data in a super secure facility in other countries to protect against major disasters. By the way, when talking about major disasters, we’re not just discussing hurricanes or ice storms; but great depression era economic crashes, nuclear events, pandemics, or military invasions. These are the scenarios dealt with by some of our IT brethren we only see in blockbuster movies. When I talk to those engineers, they run VMAX arrays.

At one point in my career, I worked for a software vendor in the healthcare space, I know first hand when making decisions about our product, we asked ourselves if we would put our loved ones in a hospital running our product. Personally, I was responsible for reliability and performance testing, so I asked that of myself often when evaluating hardware as well. These are the questions the VMAX engineers ask themselves too because they know all the mission-critical workloads that are running on the product they develop. I’ve bought VMAX over and over again because of that very fact. Because they are worrying about reliability, not just on the latest buzzword technology, but interoperability on Mainframe, AS/400, AIX/Power, Solaris/Sparc, Integrity/HP-UX, Linux, Vmware, Hyper-V and more.

So, I still don’t think they are late to the game, given the economics of flash are just coming to the point all flash is making financial sense without highly space efficient workloads, but even if you do; keep in mind… doing something right, takes some time. Doing it well enough that lives, countries, and world economies ride on it; is worth taking the extra time. I don’t want a surgeon telling me he’s trying a new free POC scalpel a vendor gave him for free right before I go into surgery.

“When VMAX comes out with a feature, I’d say, without a doubt, it’s going to be the most reliable option.”

Q: I get your point, but I don’t run a bank and no ones lives depends on my storage, why does this apply to me?
A: That level of reliability engineering and testing trickles downhill; even if you aren’t that bank or hospital, do you want your systems going down?

Q: No of course not, I don’t want my systems going down, but it’s not like other solutions fail all the time, when is VMAX the right solution?
A: Now we get past the marketing into the real conversation, when VMAX All-Flash vs. all-flash-array with dedupe, or virtual storage, or classic active/passive scale up storage? Of course, it depends on your requirements and technology stack, and I HIGHLY recommend to have that conversation with a VAR or manufacturer that has all those options. Only when you’re toolbox is full, can you pick the right tool for a job. If you’d like, I’d be happy to have the conversation too, feel free to reach out.

That said, this is a blog, so let’s explore the questions… here is how I look at VMAX:

“VMAX is the RIGHT solution for every storage requirement, is it not, however, always the BEST solution for every storage requirement”.

Because of that rock solid operations, because of all the options you have with VMAX; scale up, scale out, ficon, fibre channel, eNAS, iSCSI, large cache, consistency groups, asynchronous replication, synchronous replication, metro-clustered storage, snap-shots, cache partitioning, GUI management, CLI management, (on and on and on); I have not encountered a situation the VMAX will not provide the necessary solution. You might need all flash, you might need several of them, you might need 1 engine or 8, you might need 3.5-inch drives with high-density SATA; but I bet you can use VMAX to solve it.


If you have a VDI environment, you want dedicated disk for, an all-flash-array with deduplication will probably be simpler and more cost effective. For that matter, hyper-configured infrastructure would be even more appropriate as it’s more than just storage but includes compute, software, management and probably supported together.

If you have an enormous amount of file data that’s growing at a high rate; you might want a scale-out NAS solution that allows adding storage at a node level and can allow you to access those objects across a single namespace or through a central API. HCIA is starting to get interesting here as we see software defined NAS solutions that scale-out, as well object specific arrays, or cloud object stores.

If you have a medium environment, you need host interoperability, don’t have a high scale or super high-reliability requirements; active/passive scale-up arrays might be more cost effective. Yet again, HCIA is a good option here too if you are x86-based and virtualization comfortable.

If you’re doing genomic sequencing on a Linux compute farm, you might need a PCI Express based low latent storage array for blistering fast IO.

But if you have a mainframe running a mission critical workload in a CICS-complex across long distance; you’re probably going to choose VMAX.

If you are running a large ERP system on two ‘big-iron’ RISC-based Unix boxes and need synchronous replication to your secondary site across town, and from there asynchronous replication to your third site across the country; you’ll probably choose VMAX.

If you have a large VMware environment under high load and need multi-tenancy options to subdivide the workloads and prevent group of VMs from adversely affecting each other,  down to having dedicated storage array channels, plus control each group differently at the replication layer and fail them over with rich Site Recovery Manager integration, you’ll probably choose VMAX.

If you have a heterogeneous environment with lots of technologies and need something that supports them all; you’ll probably choose VMAX because you have a single pool of storage to leverage across those technologies with one management interface and substantial vendor support for all those platforms.

Or, if you’re like me, and walk into IT organizations that need help across the technology landscape; I pick VMAX because I know it will solve all the storage needs. I know it will be almost transparent to maintain after setting it up and that with everything I have to deal with, I won’t be getting a call at 3 am because of the VMAX. I did this even with an all VMware environment, because even though VMAX still supports ficon on mainframe, it also supports all the VAAI command set and one of the first to support VVOLs, plus has some of the best VMWare plug-ins out there.

Q: Mainframes, big-iron, Unix? Those are all dinosaurs, just like VMAX.
A: So if those are dinosaurs, they are the alligators, that survived not only extinction-level events but the epochs since, to sit at the top of their food chain today. Seriously, stop using that phrase about technology just because you are only exposed to (or sell) the current technology heralded as the second coming, which will only become yet another ‘dinosaur’ when the next recycled idea gets a new spin and big marketing campaign. Do not misread this, I work with the newer technologies daily, I believe in many of them, but there is still a place for some of these more battle-tested technologies; especially when they are still under massive development and used by giants of industry.

If you can’t integrate the bleeding edge with existing technologies, you’re on an island connected to nothing.

Q: Wow, ok Matt, I get it, you like VMAX…
A: Yes, I do, though I like a lot of other storage as well, as I said at the beginning, we’re talking about what makes VMAX great. We’re not talking about anyone else, because there are lots out there I like and have used them.

“Put it this way, if I had to choose one array, it would be VMAX. If I got to choose two, one of them would be VMAX.”

I just think in today’s age with all the new technology getting attention, we often lose sight of all the needs of IT, big and small, across industries. A good technologist will look at their requirements and pick the BEST solution; which again, may not be VMAX; though more use-cases just became in play with the announcement of the All-Flash VMAX.

Q: Speaking of All Flash VMAX, I thought it was the point of this post?
A: Somewhat, so today EMC is releasing the ‘VMAX All-Flash’, the reason the previous points were important, was not only to address some of the questions I get when discussing VMAX but to acknowledge the history of functionality and reliability that the new announcement put in play.

“This is… a VMAX… with, all flash. That is a big deal.”

There are some enhancements to the technology today, and coming, that take advantage of all flash, there is also a new bundling model to simplify purchasing, and a sizing model to ensure it’s configured correctly. But you’re getting SRDF and TimeFinder, and Hypermax (Enginuity). Those things are important if you understand this product.

Q: What do I get with a VMAX All Flash?
A: I’m a little past the TLDR point, and there are going to be some great blogs on the speeds and feed, which I’ll link to when they come out. To wrap this up here is how I look at the product when taking knowledge of the VMAX into account.

This is a VMAX configuration optimized for all flash. Because of that, it will offer the larger drives (think 3.8TB today, and I believe that will grow quickly). These are larger flash drives than available in the VMAX3, because there is less worry about IO density. In a tiered system, it is important the flash tier be designed for the necessary IO, meaning drive count was as important as tier size. Even flash has IO limitations. With all flash, you balance the IO density in one tier so larger drives will not become a bottleneck (put more simply: people skimp on drives in their flash tier when they are so big, and cause that IO skew problem we talked about earlier; with all-flash everything is tier 0 so the larger drives are generally safe, though the sizing configurator for this model will ensure the right drive is selected for IO vs capacity).

One of the distinguishing factors with VMAX in general is the larger DRAM-based cache and associated algorithms. This comes into play with all-flash. While other arrays are starting to tier between SLC and MLC to provide fast write while avoiding cell level wearing; the VMAX can leverage the large cache to coalesce writes, speeding up the write activity, reducing wear and still leveraging more cost efficient flash technology. Those VMAX engineers we talked about, that think mission critical, put about a million lines of code into HyperMax to ensure the best leverage of these drives, not only putting IO to them, but monitoring them.

You get large-scale; think 8 engines with 16 directors, 16TB of DRAM cache, 256 host ports, 1,920 flash drives pushing 4.3PB of usable flash capacity.

SPC-2 numbers like 55+GBps, small block IOPS in the millions. With that entire 4.3PB of space being diamond class.

Simple purchasing:

  • Two hardware model
    • 450 = 4 engine (200K)
    • 850 = 8 engine (400K)
  • Two software packages
    • F = Base+Snapshots (add-ons ala carte)
    • FX = Probably everything you need, like SRDF, Cloud Array, Data at Rest Encryption, eNAS, VIPR (other options ala carte)
  • Base configuration, then you grow in ‘V-Brick’ units, packs of disks that give you a model you know how to grow up front.

You can also run SRDF to a Hybrid VMAX, opening all sorts of possibilities to insert the new All Flash VMAX into muti-array/site implementations. The only thing I’ve seen to be aware of is the lack of FAST.X. This does make sense when you think about it; if you’re going to leverage FAST.X to tier DOWN from the VMAX, then use a VMAX3. That product still is the RIGHT solution, it’s just that maybe the new All Flash VMAX is the BEST solution to your requirements.

“This is what makes VMAX great; the flexibility of choice to make the storage fit your environment, rather than make your environment fit the storage.”VMAX-AFpng