File this one under “Things to consider when your app doesn’t work”.
What the customer wanted:
I was working with a large customer that is in the Health Care business. They had an application, well maybe we should call it an integrated suite of applications, that is used to tune hearing aids. One major app and a ton of independently built side apps for each and every model of hearing aids under the sun. These devices generally connect via USB, and a few use a device driver (probably Twain) as part of the install. The side apps seem to generally be built by hardware people that understand just enough software development to talk to their hardware. Installer technology is not their expertise.
The difficulty that caused the customer to consider App-V was that these side app installers many times (especially the smaller ones) did not have any kind of silent install capability, these installers were a form of home-grown exes and the vendor expected a human to click through the options. They had previously spent a huge amount of time trying to repackage into MSIs, but that proved unsustainable. This was a big ongoing problem for the customer because the devices needing this software were spread over a multi-state area with one device per location. Without remote installation capabilities somebody had to be sent to all of the machines for updates. Which could happen every month or so. The customer had determined that they wanted to App-V as much of the mess as they could — install once and deliver it virtually. The belief is that the side apps that did have drivers came with separate, real, installers with silent options; that part could be dealt with via SCCM and we could App-V the rest.
What went wrong with App-V:
When we sequenced the main application and deployed it to a test client, we had a problem that it didn’t talk to the main database. During the installation, I noticed that the software used Sql Compact Edition, but while there is an option to install the database locally we were using a regular SQL server on a centralized machine for the database so we assumed that Sql CE was only adding client side pieces in our install .
The vendor error information led us to a KB that provided a command we could run at the client to repair itself. After a bit a troubleshooting at the client, we determined that two of the files in the package, both ending in the sdf file type extension, were missing from the package. These sdf files are associated with Windows CE, and it seems that while patient data is stored on the Sql server, these files are used to register the side apps with the main app and tell the main app how to talk to the remote database.
Troubleshooting the problem:
Now if you are thinking that a SQL database inside of an App-V package is a no-go, you would be half right. You are correct that you can’t virtualize a Sql Server instance inside an App-V package; this is due to the boot time services involved. The same goes for Sql Express (which replaced Sql CE many years ago). But you should think of Sql CE as more of an Access Database with a SQL syntax interface. It is OK to virtualize. I’ve done it many times and it is fine.
But that darned SQL CE database is still giving us an issue for this app and we need to troubleshoot it.
It is unusual for the App-V sequencer to miss any files. Back in the days of SoftGrid, it happened a bit but improvements in the sequencer have pretty much eliminated this as an issue. I considered whether it was possible that a pre-existing windows process might be responsible for writing the files. In theory this might happen with an out-of-process COM object or a Windows Service. I probably haven’t seen this happen for at least 10 years, but I do know that it is possible.
To troubleshoot this, we would have to run a ProcMon trace of the install and look for who created the missing files. Indeed, the trace showed that it was a service that was added by the installer which created the missing files. Because the msi based installer added the service, we properly capture the service and all of the files associated with it laid down by the installer. But the installer didn’t create those files, it is the running service that created the two sdf files once the service was started.
So why is that a problem for this app and not others with services? Because unlike most services added by applications, the service was not a stand-alone process that lived inside its own exe. This was an “old school” dll-based service that loaded itself as a thread inside of a LocalHost process. Because the LocalHost process was not a new process, the sequencer (properly) ignored the files created by the service.
Feeling uncomfortable that maybe there would be more than just the two files we noticed, we filtered the ProcMon capture to verify. We filtered on just the windows service, we added a filter on SUCCESS for the Result, we added a filter on CreateFile operation, and a filter on the Details column including the word “Write”. The extra check on the Details column is because CreateFile really means “create a file handle” which could mean I want to read a file or open a directory to look at what is inside. This filtering turned 30,000 lines of trace down to about 7 lines to be verified and indeed only those two sdf files were missing.
The solution:
So what could we do about that? We can’t stop the LocalHost process for “Local System” context threads prior to sequencing because the system will crash. We identified three possible solutions:
- After installation, but while in monitoring mode we could run a script. In the early days of sequencing (SoftGrid 2.0 and 3.x) we sometimes had these kind of issues and I have a standalone exe called “touch”, a free tool available here. You run touch against each of these files while in monitoring mode and it re-writes the files. Since touch is a new process writing to these files, they now get captured.
- After installation, in the sequence editor we can right click on the folder and add the files one at a time.
- Every device gets every side app so we could be thinking of one monolithic App-V package, but the customer wants to limit the size of what gets sent each time something updates; so they are planning to add these side apps as separate packages via connection group. This opens up the possibility of having a package in the group that contains the latest most complete sdf files. So their process for adding or updating a side app would consist of just creating a package for the side app, updating the sdf package, and adding the new side app package into the connection group (“virtual environment” since they use SCCM).
We tested by using the sequence editor and it indeed solved the problem. For production they’ll probably go with the third option, possibly using the script using touch as the installer for the sdf package.
