Monday, August 20, 2012

You're Doing It Wrong: IE Protected Mode and WebDriver

There's a common problem most people run into with the Internet Explorer driver when they first start using it with IE 7 and above. Most people start by writing code that looks something like this, expecting it to work on a clean installation of Windows, or at least one with the default settings for Internet Explorer:
WebDriver driver = new InternetExplorerDriver();
driver.get("http://seleniumhq.org");
driver.quit();
Imagine their surprise when they get an exception that looks something like this:
org.openqa.selenium.WebDriverException: Unexpected error launching Internet Explorer. Protected Mode must be set to the same value (enabled or disabled) for all zones. (WARNING: The server did not provide any stacktrace information)
A careful reading of the exception's message tells one exactly what the problem is. People who don't bother to read the exception message then turn to their favorite search engine, and after a quick search, they often blindly modify their code to do something like the following:

DesiredCapabilities caps = DesiredCapabilities.internetExplorer();
caps.setCapability(
    InternetExplorerDriver.INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS,
    true);
WebDriver driver = new InternetExplorerDriver(caps);
driver.get("http://seleniumhq.org");
driver.quit();

While this will certainly get them past the initial exception, and will allow the test to run in most cases without incident, it's patently the Wrong Thing to do. The operative questions then are, "Why is it wrong, and what is the right way?" If you don't care about why it's wrong, and just want to know how to fix it correctly, you can skip the historical background by clicking on this handy tl;dr link.

Why does the IE driver require Protected Mode settings changes anyway?


Way back through the mists of time, before 2006, life was easy for automating Internet Explorer. A browser session was represented by a single instance of the iexplore.exe executable. A framework for driving IE could instantiate the browser as a COM object using CoCreateInstance(), or could easily get the COM interfaces to a running instance by using the presence of ActiveAccessibility and sending a WM_HTML_GETOBJECT message to the appropriate IE window handle. Once the framework had a pointer to the COM interfaces, you could be sure that they'd be valid for the lifetime of the browser. It also meant you could easily attach to the events fired by the browser through the DWebBrowserEvents2 COM interface.

Then along came the combination of IE 7 and Windows Vista. In and effort to reduce the attack surface presented by malicious web sites, IE 7 introduced something called Protected Mode, which leveraged Mandatory Integrity Control in Windows Vista to prevent actions initiated IE, usually initiated by JavaScript, from being able to access the operating system the way it could in prior releases. While this was generally a welcome development for most users of IE, it created all manner of problems for automating IE.

When you cross into or out of Protected Mode by, say, navigating from an internal intranet website to one on the internet, IE has to create a new process, because it can't change the Mandatory Integrity Control level of the existing process. Moreover, in IE versions after 7, it's not always obvious that a Protected Mode boundary has been crossed, since IE tries to present a better user experience by seamlessly merging the browser window of the new process with the already opened browser window. This under-the-covers process switching also means that any references pointing to IE's COM objects before the Protected Mode boundary crossing are left pointing to objects that are no longer used by IE after the boundary crossing.

How do I fix it?


One way to solve the problem is to either turn User Access Control (UAC) off, or elevate your privileges to an administrator when running WebDriver code, because all process started by elevated users have a High Mandatory Integrity Control level. Elevating to administrative privileges was not a great option, because the process of elevating couldn't and shouldn't be automated. Similarly, turning off UAC is unacceptable as it leaves the machine in a vulnerable state. Nevertheless, in pre-2.0 versions of the IE driver, that's exactly what one had to do to get code working, and it was one of the primary motivations for rewriting the IE driver in 2010.

Since the tricky bit to solve is when Protected Mode boundaries are crossed, a design decision was made to eliminate the boundary crossings. The simplest way to do that is to change the Protected Mode settings in the browser to be the same, either enabled or disabled, it doesn't matter which, for all zones. That way, it doesn't matter what navigation occurs, it won't cross the Protected Mode boundary, and won't trigger orphaning of the COM objects the IE driver relies on. Moreover, setting the Protected Mode boundaries for the security zones are per-user settings in Windows, and don't generally require elevated privileges to set them.

So what's with the Capabilities hack?


When the rewritten IE driver was first introduced, it was decided that it would enforce its required Protected Mode settings, and throw an exception if they were not properly set. Protected Mode settings, like almost all other settings of IE, are stored in the Windows registry, and are checked when the browser is instantiated. However, some misguided IT departments make it impossible for developers and testers to set even the most basic settings on their machines.

The driver needed a workaround for people who couldn't set those IE settings because their machine was overly locked down. That's what the capability setting is intended to be used for. It simply bypasses the registry check. Using the capability doesn't solve the underlying problem though. If a Protected Mode boundary is crossed, very unexpected behavior including hangs, element location not working, and clicks not being propagated, could result. To help warn people of this potential problem, the capability was given big scary-sounding names like INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS in Java and IntroduceInstabilityByIgnoringProtectedModeSettings in .NET. We really thought that telling the user that using this setting would introduce potential badness in their code would discourage its use, but it turned out not to be so.

Let me state this now in very clear and unambiguous terms. If you are able to set the Protected Mode settings of IE, and you are still using the capability you are risking the stability of your code. Don't do it. Set the settings. It's not that hard.

How to set Protected Mode settings


In IE, from the Tools menu (or the gear icon in the toolbar in later versions), select "Internet options." Go to the Security tab. At the bottom of the dialog for each zone, you should see a check box labeled "Enable Protected Mode." Set the value of the check box to the same value, either checked or unchecked, for each zone. Here's the dialog for reference:


Note that you don't have to change the slider for security level, and you don't have to disable Protected Mode. I routinely run with Protected Mode turned on for all zones, as I think it provides a more secure browsing experience.


9 comments:

  1. Whoa !! I never knew this nor would I ever have.. I think I will going forward STOP recommending to folks on the Selenium Users forum to use that scary setting "INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS" but instead just on a one time basis try and change the Protected mode as how you have recommended.

    Your blog is definitely a good place to hang around Jim..

    ~Krishnan

    ReplyDelete
  2. Need help. I cannot change the settings for Protected Mode. As its controlled by group policy.

    So i am running tests on a trusted site for which Protected Mode is set to OFF. And after launching the URL, nothing is happening.
    It says No window found exception.

    Setting "INTRODUCE_FLAKINESS_BY_IGNORING_SECURITY_DOMAINS" to true also doesn't help :(

    ReplyDelete
    Replies
    1. Sadly, this isn't going to be your best venue for getting help on something like this. I would recommend posting to the WebDriver user mailing list (https://groups.google.com/forum/?fromgroups#!forum/webdriver). Also, when you do so, please be ready to be asked for an HTML page or a URL to a public site that demonstrates the problem. What's that? You can't provide an HTML page? It'll be a lot harder to help you then. (http://jimevansmusic.blogspot.com/2012/12/not-providing-html-page-is-bogus.html)

      Delete
  3. "This under-the-covers process switching also means that any references pointing to IE's COM objects before the Protected Mode boundary crossing are left pointing to objects that are no longer used by IE after the boundary crossing."

    For what its worth, IE8+ fire a "New Process" event when a new tab process is created to allow automation to follow along. http://msdn.microsoft.com/en-us/library/cc288084(v=vs.85).aspx

    ReplyDelete
    Replies
    1. Thanks for the comment Eric. If anyone would have insights into better ways of making this work, it would be you. :) Nevertheless, I've played around with the NewProcess event, but I must be doing something wrong. According to the documentation, there should be a passed-in reference to "the new process object", but every time I see this event fired, that pointer is null.

      Also, multiple instances of the IE driver may be driving multiple instances of IE, not to mention that the user may have his or her own instance open. Given that, there's no rational way I can see to poll for the new process to know *which* one contains the new browser object that *this* particular instance of IE just created.

      Finally, the event is not available in IE7, which the driver has to continue to support. I could work around that with version checking, but I can't seem to get it to work for me.

      Delete
  4. Hi, I currently working with Selenium and good lord, I watched your video and read this log just in time before I head slam into weird IEDriver behavior. Thank you, a lot.
    But, while running my test, with protected mode on for all zones, I noticed that there's a pop up say "Only secure content is displayed". So is that expected or I'm doing something wrong? And if it's expected, than what would I lose if some of my webpage contents are non-secure?

    ReplyDelete
  5. I set the settings as per suggested by you in this blog but doesn't seems to be working for me.Getting below error
    INFO: I/O exception (java.net.SocketException) caught when processing request: Software caused connection abort: recv failed
    Oct 31, 2013 12:58:50 PM org.apache.http.impl.client.DefaultRequestDirector tryExecute
    INFO: Retrying request
    org.openqa.selenium.TimeoutException: Timed out after 10 seconds waiting for presence of element located by: By.id: txtHQLoginName

    Please help
    Thanks in Advance

    ReplyDelete