Robot test failing on PR, issue seems indeterminate

I have a pull request here, which gets a check failing on Jenkins due to a robot test failure in test_querystring.robot.

I am pretty sure my changes and PR have nothing to do with the exception WebDriver is complaining about, which might be a WebDriver bug (maybe this)? OTOH, this test runs fine locally:

buildout.coredev sean$ ./bin/robot-debug src/Products.CMFPlone/Products/CMFPlone/tests/robot/test_querystring.robot
==============================================================================
Test Querystring
==============================================================================
Scenario: Location query                                              | PASS |
------------------------------------------------------------------------------
Test Querystring                                                      | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /Users/sean/projects/buildout.coredev/output.xml
Log:     /Users/sean/projects/buildout.coredev/log.html
Report:  /Users/sean/projects/buildout.coredev/report.html

Thoughts on how to address?

Sean

For this problem, I have seen local tests passing on Firefox 40 and Firefox 34.0.5. So I'm not sure why Jenkins has been having this problem.

We've had these problems as well: the tests are fine if they are local but failing when in CI.

We don't know if this is your case, but usually in these situations, we could only "fix" them using Sleep, Wait Until Element Is Visible, Wait until page contains or Wait Until Keyword Succeeds in different situations: we don't know if it's a memory problem in CI that gives problems in these robot tests, the browser waiting to render an element...

The strange thing is that the test in question fails on click after Wait Until Element Is Visible -- I wonder if one has to somehow scroll the element into the viewport as well (as I have no idea on whether window size affects this)?

Seems related: https://github.com/plone/Products.CMFPlone/pull/1652

I have already seen issues with selenium when the item that should be interacted with is not in the viewport. IOW, you might be on a good track.

So I guess the question is how to get a robot framework keyword that does this:

http://blog.likewise.org/2015/04/scrolling-to-an-element-with-the-python-bindings-for-selenium-webdriver/

I'm new to RF, but I am first going to see if I can duplicate (tomorrow) what Jenkins is seeing on a low-res Ubuntu VM.

Sean

There are a few unstable robot tests. See https://github.com/plone/Products.CMFPlone/issues/1656 for a summary of what is going on. Plone 5.1 should be fine now, with several fixes done, and unstable tests marked as non-critical.