Home > VMware > Impact of memory reservation

Impact of memory reservation

Hi, I have moved this blog to the new site frankdenneman.nl.
The new home of this article is: http://frankdenneman.nl/2009/12/impact-of-memory-reservation/
Apologies for the inconvenience!

  1. December 8, 2009 at 8:00 pm

    Excellent information, and in an easily digestible format. The info about VMs never surrendering their memory reservation once it has been used is particularly useful.

    Something I only recently encountered is the Mem.CtlMaxPercent setting (default of 65%) which controls how much of a VMs memory can be taken away by ballooning.

    My understanding is that if a host is memory constrained and wants to reclaim more than the 65% default limit it’ll stop ballooning and start swapping the VM to disk, even if it potentially has more memory to give up. By setting a reservation higher than 35% of allocated memory, that swapping can be prevented.

    So it’s a bit of a balancing act, with the easy way out being to have plenty of RAM to go round at all times….

    • Frank Denneman
      December 9, 2009 at 9:05 am

      Remember that the VMkernel prefers to balloon when its in soft state, but sometimes it will use swapping. The mem.CtlMaxPercent setting is the maximum value it CAN balloon. Primarily the memory state of the VMkernel determines which memory reclaimation technique will be used.

  2. dboftlp
    December 8, 2009 at 11:11 pm

    Great read, thanks!

  3. nate
    December 9, 2009 at 7:33 pm

    Note that reservations are needed in many cases, myself I set reservations on every VM(running about 300 VMs at the moment), in addition to using resource pools in some cases(the reservations are mainly for “minimum” levels of memory being allocated, when using resource pools I don’t reserve the full memory amount.

    VMware specifically reccomends that at least for java apps that you fully reserve the amount of memory required for java, here’s a PDF from a vmware presentation on performance I was at a few months ago:

    Click to access 09Q3-perf_overview_and_tier1-pac_nw.pdf

    On page 37 you can see the impact on java apps when ballooning or swapping is in use, and page 38 specifically recommends setting full reservation for VMs running java apps(most of mine are). As for page sharing, at least for me it seems to be pretty useless for my VMs, I looked at one vSphere system with 24GB of ram in the field(runs about 12 VMs, two sets of 4 VMs run identical applications), and the amount of memory being shared was *less* than *1 megabyte*, I was pretty shocked. Checking at random one of my 64GB systems which has 22 running VMs on it(56GB of memory used) it is sharing roughly 800MB of ram, still far from impressive at least for the workload that it runs, 800MB of ram on a 64GB box is nothing to me.

    Also watch out for the “idle memory tax” that bit me earlier this year, where I had a VM configured for 20GB of ram, and I set reservations for 15GB min, 22GB max (to try to take into account the VM “overhead” and reserve that as well). On two sequential nights VMware decided to force that VM into 5GB of swap(this was running a Java VM with a heap size of more than 10GB), which slowed the system to a *crawl*. It did this despite the host hardware having more than 20GB of available physical memory, and the memory profile of the system throughout the day had not changed(very steady usage). After escalating a ticket with vmware it turns out it was the “idle memory tax”, which was fixed by hard reserving the full amount of ram to the VM(20GB).

    I run some linux “utility” (SMTP/DNS/syslog/proxy services) in as little as 96MB of memory. And separate NTP server VMs running at 64MB of memory.

    So it all depends on the apps, of course..maybe I’ve been lucky but haven’t had too many issues and manage to get pretty good consolidation ratios, enough to make me happy at least, and very few availability or performance related issues.

    I also suggest if you have a good SAN (or NAS) to setup a dedicated volume for swapping, so you can monitor the performance swap has on your storage separately from your normal VM traffic. Wouldn’t want to go spend tens of thousands of dollars on more disks to handle swap when you can just add more ram to the boxes(or add boxes) for a fraction of the cost.

    Myself not using things like DRS, vMotion etc just yet.

  4. nate
    December 10, 2009 at 9:55 pm

    No I have not but I have it now, looks interesting, thanks for the lin!!

  5. Laurie
    December 16, 2009 at 10:24 am

    This is a wonderful explanation of how this all works. One question, if the reservation is at let’s say 35% and the ballooning is at 65% what happens then?

  6. Zak
    January 8, 2010 at 10:53 am

    Very useful info, thanks.

  7. BC
    January 8, 2010 at 9:37 pm

    Nice read….

    I was told at one of my VMware classes to set all my VM’s memory reservations to 128mb (Eventhough I think I read if set to zero min is 256mb) if I wasn’t using resource pools. Eventhough the .vswp’s are using alot of extra space on my LUNS since most of my VM’s are allocated 2 to 3 gigs of RAM everything seems to work great. What do you usually set?

    • Frank Denneman
      January 11, 2010 at 9:54 pm

      I advice my customers to monitor the active memory usage and set the reservation accordingly. Remember set the reservation to the minimum level which guarantee you a well enough performance “baseline”.

  1. December 30, 2009 at 3:05 pm
  2. January 4, 2010 at 1:24 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: