One the big mysteries to me has always been why Shell Extensions are so hard for application virtualization. And I had never really taken a deep look into them to unravel they mystery until this year. But now I have, and it turns out to be less of a mystery novel and more of the Steven King kind.
The Problem
The problem, in a nutshell, is that we use application virtualization to isolate out applications from each other and the operating system. But we need to provide interfaces that are external to the package so that the user can interact with the application, and sometimes so that other applications can interact as well. Typically, this means shortcuts to the start menu, so that the user can launch the applications, and file type associations so that the user can launch the application from a file of certain file types.
But over the many years of application development for the Microsoft platforms, Microsoft has offered developers many other types of integrations, especially with the windows shell, aka “the windows explorer”. Because of the isolation that App-V has provided in the past, some of those integrations worked and some didn’t. When they didn’t work, the integrations just don’t show up, and 99% of the time we can live with it because the functionality is still available if the user just launches the app. In the past we (improperly) called these things shell extensions and never understood why they were so hard to get right.
The Goal
With App-V 5 out, I wanted to see if I could use post-sequencing customization of the App-V package to deliver the missing shell extensions. We have already noticed that just by sequencing packages in App-V 5 that we had sequenced in the past, we are automatically finding that some of these missing integrations now appear in the new virtualized packages, but not that many more.
The Research
As a developer, I have never created shell extensions, so this has been a lacking area of my developer knowledge. So I went digging through the Microsoft documentation to learn more. It does turn out that Microsoft has very little documentation on Shell integrations written for IT folks. Almost all of it is aimed at the developer in MSDN, explaining how to create one certain kind of integration. They never really explain how the shell is supposed to work in complete detail. So after a fair amount of research over the last few months, let me try to provide a simplified definition of how the shell works in IT terms.
About Terminology
So first of all, let’s get our terminology right. Fourterms to talk about. “The Shell“, “Shell-Integrations“, “Built-in Shell Integrations“, and “Shell Extensions“. These will be defined as we go along, but you need to keep them straight.
The Shell
The operating system provides a replaceable and configurable “shell” which is the user interface presented to the user, what we sometimes call “the desktop”. By default, this shell is a copy of the Windows Explorer (explorer.exe) that is launched at the user’s login. By the way, vendors can, and have, provided replacement programs for an alternate shell, but let’s ignore that.
Shell Integrations
The Windows Explorer, out of the box, understands how to display the start menu and file and folder displays. It understands what to do when a user right clicks on a file, double clicks on a file, drags a file somewhere else, and drops it. And a countless list of other things. These default settings are available for any file or folder by default, and provide generic copy/move operations, unless the shell is configured for specific cases.
But the Windows Explorer is configurable in these actions. Especially when it comes to files that have particular file type extensions. We should call this file type associated configuration “shell integrations” as a high level term. So far, I have discovered almost two dozen kinds of configurable and/or programmable shell integrations. But generally we should categorize them as either with support built into the explorer itself, or external extensions.
Built-in Shell Integrations
The Windows Explorer has built-in shell integration support that allows for configuration of the icon display, default action, and right click menus, based on the file extension type. By built-in, I mean that only code that is part of the Windows explorer, as configured by information in the registry is needed, with two exceptions:
- One exception is that for the icon display, the registry will contain a reference to either an icon file, or a file containing an icon; but otherwise it is completely inside the registry. Because the file reference might be inside the isolated package, previously App-V solved this problem by providing an external icon file for this registry configuration. In App-V 5, it actually points to the package file, but the “Publishing Feature Block” of the new AppV file format makes the icon portion of the file available externally.
- These built-in shell integrations may also reference an target executable file (exe) to run, along with command line arguments, when the user requests certain actions. These are always exe files, and they work great with App-V because the virtualization engine knows how to start the executable inside the correct virtualization environment.
Windows pre-configures support for many files types for applications when you install the operating system. For example, a file ending in “.txt” is configured to work with Notepad.exe, and “.rtf” for Wordpad.exe.
The most obvious integrations are the icon displayed for the file, the double click action, and right click menu; but there are a few more specialized ones that we don’t need to go into detail on here.
There are also a few other pre-configured special cases, like for a mounted drive, or a directory. The latter is how an app can integrate a menu item when you right click on a folder or open area (which is a folder also), such as to create a new zip file.
The configuration for file associations is stored in the “classes” portion of the Windows Registry. Feel free to pop open regedit as you read through this (just don’t change anything!). The ones that apply to all users are stored under HKEY_LOCAL_MACHINE\SOFTWARE\classes, and ones that apply only to a specific user live in the HKEY_CURRENT_USER\Software\Classes area. Microsoft makes it easy for explorer by exposing an overlay hive called “HKEY_CLASSES_ROOT” which automatically combines these. Under the classes key, are subkeys for three kinds of things that explorer uses for configuration: File Extension Types (such as “.txt”), ProgIDs, and COM Guids.
The ProgIds you can think of as a link. An application might support multiple file types (.bmp, .png, etc), and can have each point to a shared ProgId that contains the detail.
Under the File Extension (or ProgId) there can be subkeys that are called “verbs”. These verbs are somewhat standardized by practice; words such as “open”, “edit”, or “play”. When a verb is configured with a shell command, it will appear in the right click menu and if the user selects the menu item, the registered target application is launched.
To complicate things, there can also be two special keys, “Open with” and “Open with ProgIDs”, which are further links, so that the user sees the “open with” menu with another program they can launch.
Shell Extensions
There can also be another subkey, called “ShellEx”. It is only the things under this key that we should refer to as “Shell Extensions“. Shell Extensions enhance the integrations available by allowing the explorer shell to load dll assets of an application directly into the explorer.exe for execution.
The way that this works, is that the registration will point to a COM based dll that has specific COM functionality for a Shell Extension. Now there are many kinds of COM objects, and most of them have nothing to do with shell extensions. These are a specific subset of the COM objects that the application might contain. Specifically, these COM objects must be “in-process” COM objects (so that the shell can load them directly into the explorer memory space for execution), and will provide the functionality using one of many specifically named interfaces (like “IContextMenu”) for the type of shell extension functionality.
The Prototype
At TMurgent we (Patrick and I) have been working on a generic technique to post modify application packages produced by App-V 5 and provide the missing Shell Integrations to teach in our training classes. We have a process down and it works really well.
When it works. But sometimes things go bump in the night and we still don’t get the extension. Which leads us to out story…
The Case of the Missing Shell Extension
The main character of our story is Techsmith Camtasia Studio. While a complicated application, it virtualizes quite easy and I use the virtualized version to produce videos all the time. But it has one small shell extension. I live without it, because I never would have known it was there unless App-V alerted me that it wasn’t supported.
The shell extension adds a context menu to the right click of files that end in “.camrec”. A camrec file is the raw output of a captured Camtasia recording. Normally, I just import this file into the Camtasia editor and work with it there. But now I know that if I right click on the file, this shell extension will examine the file and let me extract out the embedded avi or audio files directly from the Windows Explorer. Believe me, I can live without that. But I wanted to make it work because I thought I could.
So we used our process to post-modify the sequencer output, but this time the menu didn’t show up. The good news with the process we are using is that it doesn’t seem to break things, you just don’t get the integration you were looking for.
The Investigation into the Missing
CHAPTER 1: The Backstory
Our post sequencing technique tricks the publishing action of the App-v client to integrate the missing shell integrations, including Shell Extensions that are registered under the ShellEx subkey.
CHAPTER 2: Someone is Missing
Camtasia registers a Shell Extension that adds a ContextMenu COM object, and our process accommodates that. The idea of a ContextMenu type of Shell Extension is that when the user right clicks, the shell will call through iContextMenu function of this ComObject to determine if there are additional menu items to add (along with what to call if the user selects it). So the function supplied looks inside the file and adds any appropriate menu items.
CHAPTER 3: The Detective Arrives at the Scene
When this didn’t work, I pulled out ProcessMonitor to watch the shell (fail to) perform this trick. Using procmon, I could see explorer reading my neatly placed shell extension and properly opening the COM dll.
CHAPTER 4: A Strange Clue is Fount
But then I noticed two funny things.
First, it tried to open the registry for the HKLM based Camtasia Studio.
And then it tried to load an additional Camtasia dll. Both of these are part of the virtual application, but neither appear in any part of the shell registration, even on a natively installed copy.
CHAPTER 5: Deductive Reasoning Solves the Case
It seems that the COM dll has a hard coded string to open the Camtasia registry to determine what folder Camtasia is installed to. It uses this to locate the additional dll.
In App-V 5, the virtual files are available outside the virtual environment, just off in a strange place that is shielded from normal view. But the virtual registry is not. Our process can handle getting the right file reference when we add entries into the classes area, but it can’t make the virtual registry appear.
Since the COM dll running inside of explorer couldn’t open the registry location that was only inside the virtual registry, it didn’t know where to find the dll. So it checked in the explorer’s current working directory at the root of the C: drive. (It didn’t even do a path walk, which is typical behavior for dll loading).
Prolog
And this explains the problem with Shell Extensions in application virtualization (App-V isn’t the only one with this problem). Our goal is to isolate the application, and yet sometimes these Shell Extensions need access inside the virtual environment.
Oh, much of the time the shell extension is very simple and doesn’t need access to the virtual environment to perform it’s magic. You just provide the shell extension COM object and appropriate registration and the extension works.
But you really can’t tell when it won’t appear by looking at the registration or the COM dll file. You can only tell by testing. Of course, most of the time we don’t even know what to look for (vendors don’t document shell extensions).
I have learned enough to know how to make this shell extension work under App-V 5 by extending the integration process even further, but it just isn’t worth it. Unless the missing functionality is critical to the application use, the work to find everything that is needed and exposing more and more of the virtualized assets as part of the publishing processes just is not justified by the reward.
PostScript: A Call for Help to the Authorities
The right solution remains for Microsoft to figure out how to provide the explorer shell with access to all of the virtualized package assets, but only for the purpose of implementing the shell integrations. This might eliminate the need for all of the publishing actions.
Now where is that phone number? Is that missing too…
