Is anyone here using newer versions of Firefox (48+) and Selenium for robot tests? What's your configuration to make it work and avoid "selenium.common.exceptions.WebDriverException: Message: Can't load the profile."?

I've been using older Firefox versions for robot tests (using the ones that are in Plone Jenkins https://github.com/plone/plone.jenkins_node/blob/9a74c2902da653de25020c39c67d0605615f74f4/defaults/main.yml#L3 and https://github.com/plone/jenkins.plone.org/issues/180#issuecomment-230234874) but now because of a corporate decision out of my fingers I have to use a newer Firefox.

Problem is: for Firefox 48+ you need to use GeckoDriver (https://github.com/SeleniumHQ/selenium/issues/2739#issuecomment-249482533) and all the workarounds proposed in this post aren't working. Is anybody here using Firefox 48+ versions for the robot tests? Would you mind sharing the configuration/workaround you made to make it work? Thanks!

I've not been able to make it work, it seems to be in constant flux and the whole Marionette server thingie is just too complicated for me.

Alternatives:

  • The latest phantomjs works well, even for modals. Disadvantage of course is in debugging or devising a new test, in headless mode is pretty hard unless you speak 'robot' fluently...
  • chromedriver works pretty well, both in headless and non-headless. (Also works on Chromium if you have objections to google's chrome version), so with that you can debug, use the devtools to find the right css classes, etc..
1 Like

What I'm doing:

Grabbing nvm for getting a nodejs into userspace.

Installing the latest LTS node with nvm install --lts.

Installing the latest phantomjs user-globally with npm i -g phantomjs-prebuilt

Running the tests with ROBOT_BROWSER='phantomjs'.

The issues?

Nothing is pinned down on the node/phantomjs side.

The tests are either very flaky or consistently actually failing for real reasons. The real reasons could include modern javascript engines making more and more traditionally procedural code actually silently async per default. For loops with branches in them are a good example of a non-obvious gotcha. We had a hiccup with something similar recently.


I'm hoping this is not the case for the Plone tests, but documenting out here this is a thing.

On a grander scheme, should we throw Robot out? It requires developers to both learn an extra unnecessary DSL and also have a very specific and finicky development setup. One cannot just buildout and run the package tests.

I just got a specific configuration that is working so far! Don't know if it's going to have problems along the road but it may be useful to other people.

It's hard to make it work because you need to have a specific Firefox, Geckodriver AND selenium configuration! So how did I check which versions should be used? I started with geckodriver.

They have a changelog but it's not in all releases that they give an insight of which versions of FIrefox/selenium are compatible or not. Since I'm stuck in Firefox 52.6.0 ESR, I searched for "Firefox 53" in their full changelog. In version v0.18.0, there's geckodriver now recommends Firefox 53 and greater so I downloaded v0.17.0 directly into my /usr/bin folder:

sudo wget -c https://github.com/mozilla/geckodriver/releases/download/v0.17.0/geckodriver-v0.17.0-linux64.tar.gz -O - | sudo tar -xz -C /usr/local/bin/

Since they don't tell which Selenium version is compatible with gecko 0.17.0, I kept reading the changelog until I found Note that geckodriver v0.16.0 is only compatible with Selenium 3.4 and greater. and decided to use this version as well.

So, for now, with latest Firefox ESR 52.6.0, these pins are working:

Plone
4.3.15

Firefox
52.6.0

Geckodriver
v0.17.0

Selenium
3.4

The "selenium.common.exceptions.WebDriverException: Message: Can’t load the profile." error is now gone. I don't know if pinning selenium to 3.4 will have bad consequences in really complex test configurations. Don't know if it's possible to automate the geckdriver download in a buildout since I'm using sudo but it would be nice as well.

FYI @gforcada and @mauritsvanrees, since the github issues and commits that I posted here were from you.

1 Like

See https://github.com/plone/buildout.coredev/pull/407 for a PR to bump versions in Plone coredev buildout. But this needs a newer Firefox on the Jenkins servers, which makes it harder.

1 Like

FYi @hvelarde @rodfersou

For me the chromium driver is flaky and finicky as well, though some seem to have had success with running it unsandboxed.

https://groups.google.com/forum/#!topic/chromedriver-users/8yZDv_h8feg

And based on this they're not going to support phantomjs going forward, so I'll have to figure out how to get a geckodriver setup going.

It seems to be a Rust package distributed via cargo? That could make it easier to deploy version-pinnedly.

https://phabricator.wikimedia.org/T137797

Never mind...

So the modern way is to roll a container (like docker) and exec via the container?

We switched to Chrome and chromedriver and that combination is a lot more stable than Firefox. Though, at the beginning we had massive scalability issues with Chrome when we started a new Chrome instance for every single test. As soon as we started Chrome only once for the entire test suite, things were fine...

PhantomJS is dead, isn't it? I recall the main maintainer to announce that he stops working on this. With headless Chrome and Firefox I don't see much future for Phantom.

I'd like Chrome in this case as well, as it has a package in Debian Stable for the driver.

https://packages.debian.org/stretch/chromium-driver

How do you set up your stack?

Here is the base setup:

I am also happy to share our robot test setup if that's interesting. We basically just reduced the "Open browser" calls though.

How do you set up developer workstations? As the driver is just a fetchable binary, I suppose one could make a platform sensitive buildout recipe for also solving that.

Same way on Ubuntu workstations. On OS X I manually download and update the chromedriver bin. A buildout recipe is certainly possible but I am personally ok with keeping those things up to date manually. I usually update both our CI server and my local machine when we run into problems (usually after a Chrome auto-update on my workstation or an update on the CI).

How do you set up developer workstations?

That's my question as well.

Right now, I'm just adding an ugly message stopping the buildout if the user doesn't have the geckodriver binary using plone.recipe.command and it's stop-on-error attribute. In the error message I show "Run the command 'sudo wget -c https://github.com/mozilla/geckodriver/releases/download/v0.17.0/geckodriver-v0.17.0-linux64.tar.gz -O - | sudo tar -xz -C /usr/local/bin/' to download the driver."

/usr/local/bin/ would be more appropriate per FHS and has less of a chance of a collision with a package manager.

1 Like

Indeed.

A buildout could download the driver, put in the ${buildout:directory} itself or somewhere else and add it do $PATH, but it would work only for that session, so is not that reliable.

That didn't work on Travis. :slight_smile:

So, after reading the relevant bits of manuals and playing a few years worth of catchup:

A functional Firefox WebDriver ships in Debian starting with Stretch:
https://packages.debian.org/search?keywords=firefoxdriver

A functional Firefox WebDriver ships in Ubuntu starting with Artful (Bionic for LTS):
https://packages.ubuntu.com/search?keywords=firefoxdriver

A functional Chrome WebDriver ships in Debian starting with Stretch:
https://packages.debian.org/search?keywords=chromium-driver

A functional Chrome WebDriver ships in Ubuntu starting with Trusty (LTS):
https://packages.ubuntu.com/search?keywords=chromium-chromedriver

So, how about headless framebuffer X11? On this I was a bit out of date, it has not been a service for a while. You could still add it to a wrapper script, manually start Xvfb and set the DISPLAY variable. For quite a while now, it has actually shipped its own wrapper, xvfb-run, which does all of this for you, per process, in userspace.

https://packages.debian.org/search?keywords=xvfb
https://packages.debian.org/wheezy/amd64/xvfb/filelist

https://packages.ubuntu.com/search?keywords=xvfb
https://packages.ubuntu.com/trusty/amd64/xvfb/filelist

So, at least on Debian Stretch, I got the Robot tests running with both Firefox and Chromium out of the box just by having installed the appropriate packages from above.

time (
  set -e
  export ROBOT_BROWSER='firefox'
  xvfb-run bin/alltests --all
  xvfb-run bin/alltests-at --all
)
time (
  set -e
  export ROBOT_BROWSER='chrome'
  xvfb-run bin/alltests --all
  xvfb-run bin/alltests-at --all
)

Can someone confirm this on Ubuntu as well? This seems a bit more straightforward to me than pulling in the packages manually.

On macOS homebrew can succesfully pull in both (or one can also use for example npm to get an exact version of either, if it being rolling release breaks your setup). You can also keep a copy of an old exact .app of the browsers and point the drivers to these browsers for stability, but it is a bit finicky setup-wise.

Both Firefox and Chrome allow to run in headless mode which means you don't need an X server. Though, to take screenshots on failure you still need the X server. Therefore, we still run our Robot tests on CI wrapped with Xvfb.

Last time I checked Ubuntu did not had a chromedriver package.

Plone Foundation Code of Conduct