Problem is: for Firefox 48+ you need to use GeckoDriver (https://github.com/SeleniumHQ/selenium/issues/2739#issuecomment-249482533) and all the workarounds proposed in this post aren't working. Is anybody here using Firefox 48+ versions for the robot tests? Would you mind sharing the configuration/workaround you made to make it work? Thanks!
I've not been able to make it work, it seems to be in constant flux and the whole Marionette server thingie is just too complicated for me.
Alternatives:
The latest phantomjs works well, even for modals. Disadvantage of course is in debugging or devising a new test, in headless mode is pretty hard unless you speak 'robot' fluently...
chromedriver works pretty well, both in headless and non-headless. (Also works on Chromium if you have objections to google's chrome version), so with that you can debug, use the devtools to find the right css classes, etc..
Installing the latest LTS node with nvm install --lts.
Installing the latest phantomjs user-globally with npm i -g phantomjs-prebuilt
Running the tests with ROBOT_BROWSER='phantomjs'.
The issues?
Nothing is pinned down on the node/phantomjs side.
The tests are either very flaky or consistently actually failing for real reasons. The real reasons could include modern javascript engines making more and more traditionally procedural code actually silently async per default. For loops with branches in them are a good example of a non-obvious gotcha. We had a hiccup with something similar recently.
I'm hoping this is not the case for the Plone tests, but documenting out here this is a thing.
On a grander scheme, should we throw Robot out? It requires developers to both learn an extra unnecessary DSL and also have a very specific and finicky development setup. One cannot just buildout and run the package tests.
I just got a specific configuration that is working so far! Don't know if it's going to have problems along the road but it may be useful to other people.
It's hard to make it work because you need to have a specific Firefox, Geckodriver AND selenium configuration! So how did I check which versions should be used? I started with geckodriver.
They have a changelog but it's not in all releases that they give an insight of which versions of FIrefox/selenium are compatible or not. Since I'm stuck in Firefox 52.6.0 ESR, I searched for "Firefox 53" in their full changelog. In version v0.18.0, there's geckodriver now recommends Firefox 53 and greater so I downloaded v0.17.0 directly into my /usr/bin folder:
Since they don't tell which Selenium version is compatible with gecko 0.17.0, I kept reading the changelog until I found Note that geckodriver v0.16.0 is only compatible with Selenium 3.4 and greater. and decided to use this version as well.
So, for now, with latest Firefox ESR 52.6.0, these pins are working:
Plone
4.3.15
Firefox
52.6.0
Geckodriver
v0.17.0
Selenium
3.4
The "selenium.common.exceptions.WebDriverException: Message: Can’t load the profile." error is now gone. I don't know if pinning selenium to 3.4 will have bad consequences in really complex test configurations. Don't know if it's possible to automate the geckdriver download in a buildout since I'm using sudo but it would be nice as well.
FYI @gforcada and @mauritsvanrees, since the github issues and commits that I posted here were from you.
We switched to Chrome and chromedriver and that combination is a lot more stable than Firefox. Though, at the beginning we had massive scalability issues with Chrome when we started a new Chrome instance for every single test. As soon as we started Chrome only once for the entire test suite, things were fine...
PhantomJS is dead, isn't it? I recall the main maintainer to announce that he stops working on this. With headless Chrome and Firefox I don't see much future for Phantom.
How do you set up developer workstations? As the driver is just a fetchable binary, I suppose one could make a platform sensitive buildout recipe for also solving that.
Same way on Ubuntu workstations. On OS X I manually download and update the chromedriver bin. A buildout recipe is certainly possible but I am personally ok with keeping those things up to date manually. I usually update both our CI server and my local machine when we run into problems (usually after a Chrome auto-update on my workstation or an update on the CI).
Right now, I'm just adding an ugly message stopping the buildout if the user doesn't have the geckodriver binary using plone.recipe.command and it's stop-on-error attribute. In the error message I show "Run the command 'sudo wget -c https://github.com/mozilla/geckodriver/releases/download/v0.17.0/geckodriver-v0.17.0-linux64.tar.gz -O - | sudo tar -xz -C /usr/local/bin/' to download the driver."
A buildout could download the driver, put in the ${buildout:directory} itself or somewhere else and add it do $PATH, but it would work only for that session, so is not that reliable.
So, how about headless framebuffer X11? On this I was a bit out of date, it has not been a service for a while. You could still add it to a wrapper script, manually start Xvfb and set the DISPLAY variable. For quite a while now, it has actually shipped its own wrapper, xvfb-run, which does all of this for you, per process, in userspace.
So, at least on Debian Stretch, I got the Robot tests running with both Firefox and Chromium out of the box just by having installed the appropriate packages from above.
time (
set -e
export ROBOT_BROWSER='firefox'
xvfb-run bin/alltests --all
xvfb-run bin/alltests-at --all
)
time (
set -e
export ROBOT_BROWSER='chrome'
xvfb-run bin/alltests --all
xvfb-run bin/alltests-at --all
)
Can someone confirm this on Ubuntu as well? This seems a bit more straightforward to me than pulling in the packages manually.
On macOS homebrew can succesfully pull in both (or one can also use for example npm to get an exact version of either, if it being rolling release breaks your setup). You can also keep a copy of an old exact .app of the browsers and point the drivers to these browsers for stability, but it is a bit finicky setup-wise.
Both Firefox and Chrome allow to run in headless mode which means you don't need an X server. Though, to take screenshots on failure you still need the X server. Therefore, we still run our Robot tests on CI wrapped with Xvfb.
Last time I checked Ubuntu did not had a chromedriver package.
