SQL Server memory configuration
[NOTE: This is an old post that is no longer actively maintained and covers versions of SQL Server that are no longer in support. I’m leaving it up because it’s generated a lot of comments over the years which may contain useful information for those folks having to maintain legacy systems.]
Correctly configuring SQL Server memory settings is critical for server performance, but one of the things that I frequently come across when reviewing SQL Server installations whilst working in CSS for MS, is just how many of them had not been set up with appropriate memory configuration settings, or, as in many cases, not set up in the way the administrators of the system had assumed they were; usually the DBAs thought the system was set up to use all e.g. 8GB of RAM, but no changes had been made to the OS or SQL Server configuration, so their (32-bit) SQL Server would only be accessing 2 GB, and reporting that it was using 1.6 GB.
(Edit 2013-08-26: Since writing this post nearly 5 years ago I’ve noticed an increasing trend in visits to this page via questions on forums relating to the fact that SQL Server is using a lot of memory. The quick answer is because that is how 64-bit SQL Server behaves out of the box if left unconfigured. The rest of this post goes into the how and they why, as well as the resolution. Also, despite the fact that most systems are now 64-bit, I’ve left the 32-bit stuff in this post because I know there’s still a lot of 32-bit SQL Server systems out there).
The problem is due in part to the fact that on 32-bit systems configuration changes usually have to be made both in SQL Server and at the OS level, and in part to the sprawl of documentation available on configuring SQL Server’s memory options, as opposed to a single jumping off point which runs through all the settings and considerations that need to be made.
Add to that the black art of establishing exactly how much memory SQL Server is using (most of the obvious options will only show how much memory the buffer pool is using) and it’s easy to see why it’s such a problem area.
In this post I’ll attempt to clear some of the smog and provide what I hope will be one document which answers most of the questions that arise about configuring SQL Server’s memory usage.
This discussion will cover configuring memory for SQL Server 2000, SQL Server 2005 and SQL Server 2008 (with the exception of the Resource Governor). This blog assumes an edition of SQL Server that is not internally limited in its memory usage.
32-bit or 64-bit SQL Server?
There’s a big difference between the memory configuration settings between 64-bit SQL Server and 32-bit SQL Server, so it’s not possible to start a discussion about SQL Server’s memory management without clarifying whether we are dealing with 32-bit versions or 64-bit versions of the product, as this is key to how much memory SQL Server can address, and (almost as importantly) how it addresses that memory.
Until fairly recently 32-bit software was ubiquitous. The server Windows operating systems were 32-bit, your desktop, usually Windows XP was 32-bit. Therefore, I’ll be focusing a fair bit on 32-bit SQL Server as this is what requires the most configuration, and also where most of the confusion lies.
So, here goes.
Windows memory architecture
A quick aside on Windows memory architecture first.
The amount of memory a 32-bit process can access is 2^32 or 4294967296 bytes, or 4 GB.
On a 32-bit Windows OS this 4 GB of memory is not all addressable by a single process. Instead, it’s partitioned by the OS into two address spaces. 2 GB is kept by the OS (commonly referred to as the kernel) and the remaining 2 GB is the user mode address space, or the area each application (process) will have access to. Whilst each user mode process gets 2 GB as its addressable memory range, the kernel mode area is shared between all processes. SQL Server runs in the user mode address space and is bound by default to the 2 GB of memory limit on a 32-bit OS.
This directly addressable memory will hereon be referred to by what it is more commonly known as, the virtual address space or VAS.
Configuring SQL Server’s memory settings
SQL Server’s default out-of-the-box memory limit is deliberately set to a very high value of 2147483647 MB, which basically means all available memory, but as you should now know, there’s no way it can actually use anywhere near that much memory, particularly on a 32-bit platform.
64-bit operating systems have a far far bigger address space open to them; 8 TB to be exact. Before you run off to your calculator to evaluate 2^64, the answer won’t be 8TB, but 8TB is what each user mode application gets due to current hardware and OS limitations. The kernel also gets 8TB and this kernel address space is shared by all processes, just as in 32-bit Windows.
Having said all that, I should point out that no current MS Windows OS can address more than 2 TB.
What this means for 64-bit SQL Server is that out of the box, it can address all the memory on a server without any special configuration changes either within SQL Server or at the OS level. The only thing you need to look at is a cap on its memory usage; capping memory usage is covered in the ‘max server memory’ and ‘min server memory’ section which is further down.
To /3GB or not to /3GB
So, as 32-bit applications are natively restricted to a 2 GB VAS, OS configuration tweaks are required to allow access to more than 2 GB, and these are covered next.
The first modification is one that, rather ironically, should be used as a last resort. Ideally, it should be used on the advice of Microsoft Support (PSS).
I’m choosing to get it out of the way now because the /3GB setting is probably the most well known and most misused.
/3GB allows a 32-bit process to increase its VAS to 3 GB by taking away 1 GB of address space from the kernel, and this is why it’s a last resort; there’s no such thing as a free lunch as the removal of 1 GB of addressable memory from the OS can introduce instability. More on that shortly.
To allow a 32-bit process to gain a 3 GB VAS you have to add the /3GB switch to the Windows boot.ini file.
As I stated, this can introduce system instability by starving the OS of System Page Table Entries (PTEs). A discussion about PTEs is out of the scope of this blog, but its effects can be dramatic and cause blue-screens. The good news is that this mainly affected Windows 2000 so you should be fine if you’re on a later Windows version.
If you’re still looking after a legacy system, there is some scope for manoeuvre here, by adding the /USERVA switch to the boot.ini it is possible to reduce the VAS increase from a straight 3 GB to a lower user-defined amount which will give the OS room to breathe, and thus resolve any instability issues.
The main reason you will be advised by PSS to use /3GB is if you are suffering VAS starvation issues, such as a bloated procedure cache as it can only reside in VAS memory (also see the next section on MemToLeave) because 32-bit version of SQL Server only allow database pages to reside in the part of the SQL Server memory cache (called the buffer pool) that is utilising awe enabled memory.
MemToLeave
(EDIT 2011-06-30: The correct terminology for this is Virtual Address Space Reservation.)
Because of the inherent address space limitations of a 32-bit process, a certain amount of memory has to be set aside by SQL Server on startup that SQL Server uses for overheads. This memory is set aside in case it all gets used by the buffer pool.
COM objects, extended stored procs, third party backup solutions, some anti-virus apps, memory allocations exceeding 8K and the memory allocated to the threads SQL Server creates to service e.g. user connections come from a section of memory within the VAS but outside the buffer pool which is typically referred to as the MemToLeave area. This is 384 MB by default on an e.g. 2-proc 32-bit SQL. If you want to know more about how it is calculated, check Jonathan Kehayias’s post covering this.
0.5 MB is the default thread stack size for a thread on 32-bit Windows. 64-bit Windows has a default stack size of 2 MB or 4 MB depending on which 64-bit flavour of Windows you are running (AMD x64 or IA64).
SQL Server 2005 and beyond uses a formula to calculate the max worker threads setting which affects the size of the MemToLeave area.
There is a SQL Server startup parameter (-g) which can be used to increase the MemToLeave area, but again, only do this if advised by PSS (it’s ignored on 64-bit as MemToLeave or VAS reservation won’t be an issue on that architecture) as this will reduce the maximum amount of memory the buffer pool can therefore use.
4 GB of RAM and beyond
So, we know 32-bit SQL Server can use 2 GB out of the box, and up to 3 GB (with an OS tweak that is best avoided, if at all possible).
However, 32-bit SQL Server can benefit from much more memory than 3 GB with the help of OS and SQL Server configuration modifications which will be covered next.
/PAE
To address more than 4 GB of RAM on 32-bit Windows, the OS needs to have the /PAE switch added to the boot.ini file, although if your system supports hot-swappable memory you won’t need to add this as Windows should automatically be able to see the additional memory. If you’re not sure, take a look at how much memory the OS can see via System properties; if you have more than 4 GB installed and the OS is only showing 4 GB, review your boot.ini settings. I’m not mentioning specific Windows versions because the /PAE switch applies to all current 32-bit versions of Windows.
(EDIT 2011-06-30: For Windows Server 2008 you have to run ‘BCDEDIT /SET PAE FORCEENABLE’ from a CMD prompt running under administrator privileges).
Both /PAE and /3GB
Some systems have both /3GB and /PAE enabled.
This is fine as long as the system does not have more than 16 GB of RAM. Add any more memory and Windows will not recognise it because of the overhead required to manage the additional memory.
Clusters
No special configuration settings regarding memory settings are required for a cluster, but I thought I better mention clusters specifically because you won’t believe how many installations there are out there where there are different settings on different nodes within the same cluster.
So, make sure any OS setting changes like /3GB or /PAE are consistently applied across all nodes.
Enable AWE
After configuring the OS, you’ll need to configure SQL Server by enabling AWE (Address Windowing Extensions). AWE is in essence a technique for ‘paging’ in sections of memory beyond the default addressable range.
AWE can be enabled in SQL Server using Query Analyzer/SQL Server Management Studio (SSMS) via the following statements:
sp_configure ‘show advanced options’, 1 RECONFIGURE GO sp_configure ‘awe enabled’, 1 RECONFIGURE GO
AWE enablement is not a dynamic option and will require a SQL Server restart, so before you do that make sure the SQL Server service account has the ‘Lock Pages in Memory’ privilege assigned to it.
Once AWE has been enabled within SQL Server and the ‘Lock Pages in Memory’ privilege has been assigned you should be good to go after a restart.
(Edit:SQL 2008 R2 is the last version that will support AWE functionality. As of SQL Server 2012 the awe_enabled option is no longer supported. If you have a 32-bit install of SQL Server 2012 I strongly urge you to read that article and it’s implications on systems with more than 4 GB of RAM).
‘max server memory’ and ‘min server memory’
The more memory you give SQL Server, the greater the need to set an upper limit on how much it uses. When you start SQL Server it’ll ramp up its memory usage until it has used up all the memory it can access, which will either be an internal OS limit or a SQL Server configured limit.
A 32-bit SQL Server instance will therefore grab up to 2 GB if the workload demands it and it is on default settings.
An awe enabled SQL Server instance will go on using up all the memory on the system if the workload is there and an upper limit on its memory usage is not set.
Having said all this, the key thing to remember is that the ‘max server memory’ setting only controls the memory assigned to the buffer pool which is the memory cache SQL Server uses; every database page that the system reads or modifies is read into the buffer pool first. What this means is that there is a substantial area of memory which we cannot directly monitor or control (make sure you read the memtoleave section) but we have to ensure enough memory is available for it. It is from this area of memory that all allocations for e.g. COM objects, CLR functions, linked server queries, OLEDB providers and multi-page allocations (i.e. anything over 8K) are made from. This is what complicates sizing SQL Server’s memory usage correctly as every system will use the memtoleave area differently.
Setting a limit has the double-benefit of not starving the OS of resources and avoiding ‘Out of memory’ errors which can occur on SQL Server systems that may have a lot of memory. The latter (rather contradictory) situation can arise because SQL Server will try and allocate more memory when it is already at the system limit (if no upper limit has been set via the ‘max server memory’ setting) instead of freeing up memory it is already using.
Configuring memory for multiple instances
A third reason to set an upper limit is if you have more than one SQL Server instance installed on a single host, as this will stop the instances competing for memory.
Allocate a high enough ‘max server memory’ limit to each instance to allow it to do its job without running into memory starvation issues, whilst reserving the bulk of the memory for higher priority instances (if any) and the OS.
This is where benchmarking comes in handy.
To set a max server memory limit of 12 GB via Query Analyzer/SSMS:
sp_configure ‘max server memory’, 12288 RECONFIGURE GO
SQL Server ramps up its memory usage because by default it is set to use no memory on startup. This is controlled by the ‘min server memory’ setting. Specifying this to a higher value has the benefit of reserving a set amount of memory for SQL Server from the off which can provide a slight performance benefit, especially on busy systems. It’s actually not uncommon to see ‘min server memory’ and ‘max server memory’ set to the same value, to reserve all of SQL Server’s memory straight away. The downside is SQL Server will take slightly longer to start up than if ‘min server memory’ was set to a low value.
Older SQL versions did not really release memory
This will probably get me into a bit of trouble, as there are KBs that clearly state that SQL Server releases memory when the OS is under pressure.
True, but only under a lot of pressure and SQL Server 2000 and it’s forerunners were notoriously bad at this – often requiring a sytem restart which is why it vital to set an upper limit via the ‘max server memory’ for any system (not just SQL Server 2000).
Memory corruption
Slightly off-topic, but this is an appropriate place to bring this up.
Certain builds of Windows 2000 and Windows Server 2003 contained a potentially serious memory corruption problem which affected SQL Server more than other applications, mainly because there are few applications that run on Windows that can utilise the amount of memory SQL Server does.
It’s difficult to overstate the problems this can cause, so make sure you’re on the appropriate Windows Service Packs if you’re running SQL Server on a PAE enabled system.
Another issue that arose in SQL Server 2000 SP4 was a bug that meant SQL Server only saw half the memory on awe enabled systems, although it was identified quickly and the hotfix for this was placed alongside the SP4 download.
32-bit SQL Server on 64-bit Windows
If you have a 32-bit SQL Server on 64-bit Windows the SQL Server process can access the entire 4 GB VAS.
Checking SQL Server’s memory usage
This is another area where there is lot of confusion, so below is a run-through of the most common methods for confirming SQL Server’s memory usage.
Ignore Task Manager
If you have an awe enabled SQL Server instance, do not rely on Task Manager to display memory usage as it does not show the AWE memory a process is using, so the memory usage figure it presents for the SQL Server process (sqlservr.exe) will be incorrect.
DBCC MEMORYSTATUS
Running the above command outputs the memory usage of SQL Server including how that memory is allocated, so unless you need to know how and where that memory is being used, the output it generates can be a bit bewildering. The important bits of this output pertaining to SQL Server’s total memory usage are as follows:
Buffer Counts Buffers
Committed 3872 Target 65536 Hashed 2485 Stolen Potential 60972 External Reservation 0 Min Free 64 Visible 65536 Available Paging File 702099
The key figures in the above output are committed, target and hashed.
Committed is the amount of memory in use by the buffer pool and includes AWE pages.
Target is how big SQL Server wants the buffer to grow, so you can infer from this whether SQL Server wants more memory or is releasing memory.
There’s an excellent KB on interpreting all the output INF: Using DBCC MEMORYSTATUS to Monitor SQL Server Memory Usage for SQL Server 2000 and How to use the DBCC MEMORYSTATUS command to monitor memory usage on SQL Server 2005.
[EDIT 2009-05:02: Remember the buffer count numbers refer to pages of memory which are 8K in SQL Server.]
System Monitor (perfmon)
Perfect way to get a quick reference on exactly how much memory SQL Server is using at that moment. Start System Monitor and add the SQL Server: Memory Manager: Total Server Memory (KB) counter.
Replace “SQL Server” with MSSQL$ and the name of the named instance if it’s not a default instance, e.g. MSSQL$INSTANCE1.
‘Total’ memory usage
When trying to establish exactly how much memory SQL Server is using it’s not just the buffer pool memory you have look at, but the MemToLeave area as well. The key point to bear in mind here is that it’s not only SQL Server that can make allocations from this latter area of memory but third party processes as well, which can make it impossible to precisely account for SQL Server’s absolute memory usage, contrary to some myths out there about calculating SQL Server’s memory usage via e.g. DBCC MEMORYSTATUS, as such methods can only account for SQL Server’s own memory allocations and not allocations by foreign processes.
[EDIT 2011-06-27: Soft NUMA section removed.]
64-bit
I mentioned at the start of this post that all you have to worry about for 64-bit SQL Server is setting a max memory limit as SQL Server can access all the memory current Windows operating systems can support, and 8 TB in total. That’s mostly true, with the exception of a certain privilege that the SQL Server service account needs, and that’s the ‘Lock Pages in Memory’ (LPIM) privilege.
This privilege is vital as it prevents the OS from paging out SQL Server memory to the swap file.
With the introduction of SQL Server 2005, this right was restricted on 64-bit Windows to only take effect on Enterprise Editions of SQL Server, so if you’re wondering why your huge new multi-gigabyte multi-core 64-bit system is paging like crazy, this might be why. [Edit: This has finally been reversed for both SQL Server 2005 and SQL Server 2008 Standard Editions: http://blogs.msdn.com/psssql/archive/2009/04/24/sql-server-locked-pages-and-standard-sku.aspx
Whilst we’re on the subject of paging on 64-bit SQL Server systems, take a look at the following KB:
How to reduce paging of buffer pool memory in the 64-bit version of SQL Server 2005 which covers issues a number of issues that cause SQL Server’s (Standard or Enterprise editions) memory to be paged out.
[EDIT 2013-09-19: More on LPIM. Memory management is enhanced in every version of SQL Server, and for SQL Server 2008 onwards the prevailing consenus is that this is no longer required, see this article for more information]
If your 64-bit system has hundreds of GB of RAM, don’t assign it all via the ‘max server memory’ setting. The advice if you have a lot of memory changes with each version of SQL Server. In SQL Server 2005 the (very) general recommendation was to leave aside 1 GB for every 16 GB, so if you had a 256 GB system this would mean setting ‘max server memory’ no higher than 240 GB.
For SQL Server 2008 the SQLCAT team posted a best practices article (based on an OLTP system) advising 1 GB to be set aside for every 4 cores, so if our 256 GB system had 32 cores that would be a ‘max server memory’ setting of 248 GB. The usual caveats of assuming this is a system dedicated to SQL Server apply.
Personally, I’d be more cautious and use the former strategy and then baseline the system for a few weeks taking into account the System Monitor (perfmon) counter: Memory > Avalailable MBytes as this will show how much free memory the system has and I’d tweak the max memory setting accordingly, as every system will use CLR functions, linked server queries, 3rd party DLLs differently (i.e. all the stuff that runs in the db but does not use memory from the buffer pool. The buffer pool is the only facet of SQL Server’s memory usage that the ‘max server memory’ setting controls).
In summary…
The table below describes how much memory SQL Server can use, and assumes an edition of SQL Server that has no internal limitations as to how much memory it can use, e.g. Express and Workgroup editions are limited to 1 GB and 3GB respectively.
SQL Server type Installed physical memory —————— ————————————- —————————— Up to 4GB More than 4GB (/PAE enabled 1) 32-bit SQL Server Default memory usage With /3GB 2 All available RAM3 2 GB 3 GB 64-bit SQL Server All available RAM 3
1 Not all 32-bit systems now need to have /PAE explicitly set in boot.ini for the OS to see more than 4 GB of RAM
2 Assuming /USERVA switch has not been used to tune memory usage to between 2 GB and 3 GB
3 Assuming ‘max server memory’ is left on defaults, otherwise SQL Server will use no more memory than that stipulated by the ‘max server memory’setting.
When I started this post I wanted to keep it as short and succinct as possible, but I realised pretty quick that that was never gonna happen, as there’s a lot more to configuring SQL Server’s memory usage than simply setting a ‘max server memory’ limit.
It’s a complex undertaking, especially in a 32-bit environment. It’s not easy to cover all the pertinent points without branching off and describing the different areas of its memory architecture, although I’ve tried to provide the relevant information without going into too much detail.
The key thing to remember is that the ‘max server memory’ setting is a misnomer and only accounts for the memory assigned to the buffer pool. Memory assigned to threads, linked server queries, the CLR and a host of other processes utilise memory from outside the buffer pool.
Hopefully, this post has helped clarify a little of how to configure SQL Server’s memory usage and provided enough information to answer most memory configuration related questions, although, as you might have guessed, there’s no black-and-white way of precisely determining SQL Server’s memory usage as there are so many external processes that can make allocations from within SQL Server’s address space…
Thanks also to Jonathan Keheyias for additional information and comments on this post. Please read his post on this topic if you want to delve a bit deeper.
Related posts
How It Works: DBCC MemoryStatus Locked Pages Allocated and SinglePageAllocator Values
How to use the DBCC MEMORYSTATUS command…
MemToLeave and 64-bit SQL Server
Memory Manager changes is SQL Server 2012
How many CPU cores can SQL Server use?
