As explained my the book “Windows System Performance Through Caching”, the Microsoft operating system cache tries to save memory by allowing multiple processes to share executable file images in RAM. DLL files are commonly used by numerous applications and by having a single instance of them in RAM, considerable memory is saved. Sometimes, however, a dll cannot be loaded into the process memory location that the developer intended, causing the OS to "rebase" the dll on the fly. The OS accomplishes this by using "copy on write" memory and when the rebase fixups are applied, the process gets a new copy of those memory pages.

About Dll Base Address and Rebasing:

When a dll is created, the developer chooses a base address. This base address is used during the linking in various places within the dll file. If an application process loads a dll into it's process memory space, the operating system will look to see if the process memory space needed by the dll is available at that base address. If so, the dll is efficiently loaded into that address.

When this address space is already in use, the dll is loaded into a different process memory location and an on-the-fly rebase is done to modify the ram copy of the dll for use by this process. Here is how that process traditionally works (without ASLR, explained later in this chapter):

  • When a dll is loaded into by the process, the Windows Cache Manger of the kernel allocates Page Table Entries (PTE) for the entire dll as reserved memory. Most/all of those pages will be marked as "copy on write" memory.
  • Next the necessary portions of the dll are read into RAM (this would include things like the PE header, external address table, plus any data initialized memory pages). Only a small portion of the dll is read into RAM at this time.
  • If on-the-fly rebasing is needed, the OS will then read the "fix-up" table portion of the dll into RAM. The fix-up table tells the OS about any memory locations that need changing due to the rebasing. The difference between the dll base address and the address the OS loaded the dll into is applied to each of these fix-up locations. In order to apply the fix-up, the required dll memory page is read into RAM and the change is made. Because the memory page is marked as "copy on write", this change triggers a fault and the OS. To handle the fault, the kernel allocates a new PTE (not marked "copy on write") and RAM page, copies the existing memory page to this new page, and assigns this new page to the process. The write is then allowed to complete to this new page, affecting only this process.
  • Additional memory pages for this dll will only be ready in as needed.
  • If another process also needs this dll while it remains in memory, the other process will be assigned the original PTE pages, and if on-the-fly rebasing will begin again for this process (if needed). In this situation, each new process uses more PTEs and RAM than would be needed if this on-the-fly rebasing was not occurring.

Most of the time, dlls do not have a process memory space conflict. The process has 2GB of user mode memory space to deal with and few processes use much of this space. Generally, the problem occurs because dlls are used by multiple applications, often other companies. The developer has to choose some address and the only guidance by Microsoft to developers is to choose an address based on a scheme of using the first letter of the dll name. Sometimes, though, it happens because the developer did not actually set a base address and the dll gets a default that is often going to cause a conflict.

Recent versions of the Microsoft Windows operating systems have added a feature called "Address Space Location Randomization", a security feature that causes all DLLs to be loaded at a semi-random address. DLLs may opt out of this randomization, and to ensure this feature didn't break older applications, all dlls that were built using linkers before this feature will be exempted.

These all combine to have "interesting" effects on memory usage in certain scenarios.

Usage

This tool provide insight into file memory use and resizing .

MassiveRebase

MassiveRebase is a GUI based tool that allows you to dynamically load two dlls in a specific order in order to view the memory usage results.

The tool comes with a number of prebuilt dlls, with a selection of base addresses allowing you to see conflict and non conflict scenarios. Most of the dlls are built to be ASLR enabled, however two of them are built with the ASLR exemption. In addition, two of the dlls (one with ASLR exemption and one without) are huge. These large dlls will magnify the effects. They are so large that I had to write a program to produce the source code for them.

There is no installer for the program. Just unzip into a folder. If you copy additional dlls into the same folder, these will also become available to the tool.

Start up the tool, select the two dlls and load them. To obtain the full effect, click the "Call Func_0" button to force all of the pages into RAM.

MassiveRebase sample display

What does MassiveRebase cost?

TMurgent believes in giving back to the community.   MassiveRebase is free for use by everyone.

How does this tool work?

The tool makes use of published APIs, when applicable, but also uses some unpublished kernel interfaces to obtain information or request system modifications. The tool understands the differences between versions of the operating system and adjusts to those differences.

What OSs will it run on?

The tool will run on Windows 2000 and above. The 32-bit version only supports 32-bit dlls, but it can be run on x86 or x64 systems. The 64-bit version supports 64 bit dlls and may only be run on x64 systems.

Are there more free tools?

TMurgent has a bunch of tools on this page. Usually they are free, unless you are a Software Vendor. Tools for performance have a segregated list here.

Here are the links to the ZIP packages:
   MassiveRebase_V2x86.zip (12.5MB )
   MassiveRebase_V2x64.zip (14.6MB )

Where Can I buy the book to learn more?

The book, “Windows System Performance Through Caching”, is published through Lulu, and you can find it from this page.