Flake8 version pin hell

tkimnguyen · July 26, 2020, 1:59pm

I'm stumped on how to get past this flake8 version conflict in the https://github.com/collective/plonetheme.future_imperfect repo

https://travis-ci.com/github/collective/plonetheme.future_imperfect/builds/98073116#L752

specifically:

Got plone.recipe.codeanalysis 3.0.1.

Getting distribution for 'flake8-todo'.

Got flake8-todo 0.7.

Getting distribution for 'flake8-string-format'.

Got flake8-string-format 0.2.3.

Getting distribution for 'flake8-quotes'.

Got flake8-quotes 1.0.0.

Getting distribution for 'flake8-print'.

Got flake8-print 3.1.0.

Getting distribution for 'flake8-plone-hasattr'.

Got flake8-plone-hasattr 0.2.post0.

Getting distribution for 'flake8-plone-api'.

Got flake8-plone-api 1.4.

Getting distribution for 'flake8-pep3101'.

Got flake8-pep3101 1.2.1.

Getting distribution for 'flake8-isort'.

Got flake8-isort 2.6.0.

Getting distribution for 'flake8-deprecated'.

Got flake8-deprecated 1.3.

Getting distribution for 'flake8-debugger'.

Got flake8-debugger 3.1.0.

Getting distribution for 'flake8-commas'.

Got flake8-commas 2.0.0.

Getting distribution for 'flake8-coding'.

Got flake8-coding 1.3.1.

Getting distribution for 'flake8-blind-except'.

Got flake8-blind-except 0.1.1.

Getting distribution for 'lxml==3.5.0'.

Building lxml version 3.5.0.

Building without Cython.

Using build configuration of libxslt 1.1.28

Got lxml 3.5.0.

Getting distribution for 'flake8==2.5.5'.

Got flake8 2.5.5.

Getting distribution for 'check-manifest'.

Got check-manifest 0.37.

Getting distribution for 'pycodestyle<3.0.0,>=2.0.0'.

Got pycodestyle 2.4.0.

Getting distribution for 'six==1.10.0'.

Got six 1.10.0.

Version and requirements information containing flake8:

  [versions] constraint on flake8: 2.5.5

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-todo

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-string-format

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-quotes

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-print

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-plone-hasattr

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-plone-api

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-pep3101

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-isort

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-deprecated

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-debugger

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-commas

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-coding

  Requirement of plone.recipe.codeanalysis[recommended]: flake8-blind-except

  Requirement of plone.recipe.codeanalysis[recommended]: flake8>=2.4.0

  Requirement of flake8-todo: pycodestyle<3.0.0,>=2.0.0

  Requirement of flake8-string-format: flake8

  Requirement of flake8-quotes: flake8

  Requirement of flake8-print: pycodestyle

  Requirement of flake8-print: six

  Requirement of flake8-print: flake8>=1.5

  Requirement of flake8-plone-hasattr: flake8

  Requirement of flake8-plone-api: flake8

  Requirement of flake8-pep3101: flake8>=3.0

  Requirement of flake8-isort: testfixtures

  Requirement of flake8-isort: isort>=4.3.0

  Requirement of flake8-isort: flake8>=3.2.1

  Requirement of flake8-deprecated: flake8>=3.0.0

  Requirement of flake8-debugger: pycodestyle

  Requirement of flake8-debugger: flake8>=1.5

  Requirement of flake8-commas: flake8<4.0.0,>=2

  Requirement of flake8-coding: flake8

  Requirement of flake8-blind-except: setuptools

  Requirement of flake8>=2.4.0: pyflakes<1.1,>=0.8.1

  Requirement of flake8>=2.4.0: pep8!=1.6.0,!=1.6.1,!=1.6.2,>=1.5.7

  Requirement of flake8>=2.4.0: mccabe<0.5,>=0.2.1

While:

  Installing.

  Getting section code-analysis.

  Initializing section code-analysis.

  Installing recipe plone.recipe.codeanalysis[recommended].

Error: The requirement ('flake8>=3.0') is not allowed by your [versions] constraint (2.5.5)

The command "bin/buildout -Nc travis.cfg" failed and exited with 1 during .

tkimnguyen · July 26, 2020, 2:01pm

I've even tried removing the code-analysis part, but somehow plone.recipe.codeanalysis[recommended] keeps getting pulled in, and when I remove that block out of the buildout.cfg https://github.com/collective/plonetheme.future_imperfect/blob/master/buildout.cfg#L26 something else keeps pulling in flake8.

tkimnguyen · July 26, 2020, 2:02pm

I'm sure this is more complicated by trying to test using Plone 5.0.5, dating back to 2016, but I thought I'd get this working before updating to more recent Plone versions. This flake8 problem occurs just running a regular buildout locally.

jaroel · July 26, 2020, 3:50pm

Removing https://github.com/collective/plonetheme.future_imperfect/blob/d2f719b2819e0fa30bf3fa6c324cd575b3842dd8/buildout.cfg#L68 would probably fix it for this specific repo.

Or you can pin flake8-pep3101 < 1 to not get a version that requires flake8>=3.0.

tisto · July 27, 2020, 6:29am

This is a constant source of pain in our projects as well tbh. You can find the version pins we use for all our open source projects (Plone 4.x - 5.x) here:

mauritsvanrees · July 27, 2020, 8:48pm

This may be during the uninstall phase of buildout: buildout sees that you have previously had a codeanalysis part, but not anymore. So it tries to uninstall whatever was installed by this part. To uninstall this properly, it first tries to install the recipe again because it may have custom uninstall code...
Work around this, by removing the hidden .installed.cfg file and running bin/buildout again.

tkimnguyen · July 28, 2020, 3:01am

ah, I had not known why it was doing this – thanks!

tkimnguyen · July 28, 2020, 3:25am

I am trying this approach. I added the following pins (and kept flake8 at 2.5.5):

flake8-pep3101 = 0.6
flake8-commas = 0.1.6
flake8-isort = 1.3
flake8-deprecated = 1.0

but am now getting this weird pep517 / zipp / UNKNOWN thing:

Develop: '/Users/kimadmin/src/plonetheme.future_imperfect/.'
warning: no files found matching '*.html'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.jshintignore' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
Getting distribution for 'flake8-deprecated==1.0'.
warning: no previously-included files found matching '.installed.cfg'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching '*.pyc'
Got flake8-deprecated 1.0.
Getting distribution for 'zipp'.
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

zip_safe flag not set; analyzing archive contents...
Installing zipp 3.1.0
Caused installation of a distribution:
UNKNOWN 0.0.0
with a different project name.
Installing zipp 3.1.0
Caused installation of a distribution:
UNKNOWN 0.0.0
with a different version.
Got None.
Version and requirements information containing zipp:
  Requirement of pep517: zipp
While:
  Installing.
  Getting section code-analysis.
  Initializing section code-analysis.
  Installing recipe plone.recipe.codeanalysis[recommended].
Error: There is a version conflict.
We already have: UNKNOWN 0.0.0
but pep517 0.8.2 requires 'zipp'.

tkimnguyen · July 28, 2020, 3:28am

running buildout -vvv gives me this (hopefully relevant) bit:

Getting required 'testfixtures'
  required by flake8-isort 1.3.
We have the best distribution that satisfies 'testfixtures'.
Picked: testfixtures = 6.14.1
Getting required 'isort'
  required by flake8-isort 1.3.
We have the best distribution that satisfies 'isort'.
Picked: isort = 5.2.0
Getting required 'pep8'
  required by flake8-commas 0.1.6.
We have the best distribution that satisfies 'pep8'.
Picked: pep8 = 1.7.1
Getting required 'mccabe<0.5,>=0.2.1'
  required by flake8 2.5.5.
We have the best distribution that satisfies 'mccabe<0.5,>=0.2.1'.
Picked: mccabe = 0.4.0
Getting required 'pyflakes<1.1,>=0.8.1'
  required by flake8 2.5.5.
We have the best distribution that satisfies 'pyflakes<1.1,>=0.8.1'.
Picked: pyflakes = 1.0.0
Getting required 'toml'
  required by check-manifest 0.42.
We have the best distribution that satisfies 'toml'.
Picked: toml = 0.10.1
Getting required 'pep517'
  required by check-manifest 0.42.
We have the best distribution that satisfies 'pep517'.
Picked: pep517 = 0.8.2
Getting required 'zipp'
  required by pep517 0.8.2.
We have no distributions for zipp that satisfies 'zipp'.
Getting distribution for 'zipp'.
Running easy_install:
"/Users/kimadmin/src/plonetheme.future_imperfect/bin/python" "-c" "import sys; sys.path[0:0] = ['/Users/kimadmin/src/plonetheme.future_imperfect/eggs/setuptools-24.3.0-py2.7.egg']; from setuptools.command.easy_install import main; main()" "-mZUNxd" "/Users/kimadmin/src/plonetheme.future_imperfect/eggs/tmpzcCD4d" "-v" "/var/folders/lx/6chcd57j7w16ycjwnn_x9xg00000gx/T/tmpXps9tEget_dist/zipp-3.1.0.tar.gz"
path=/Users/kimadmin/src/plonetheme.future_imperfect/eggs/setuptools-24.3.0-py2.7.egg

Processing zipp-3.1.0.tar.gz

tkimnguyen · July 28, 2020, 3:29am

Now I'm looking for why pep517 is being pulled in... it didn't exist in 2016. https://pypi.org/project/pep517/#history

tkimnguyen · July 28, 2020, 3:31am

How come having everything pinned (which I think was the case when this repo was put together, similarly so for Plone 5.0.5) doesn't actually let me easily rerun buildout without having all this scary **** happen?

idgserpro · July 28, 2020, 3:39am

@tkimnguyen zipp >= 2.0.0 is not compatible with Python 2. See:

idgserpro · July 28, 2020, 3:57am

@tkimnguyen I recommend you simply remove this amount of pinns. It worked for me. See:

github.com

collective/collective.elasticsearch/blob/master/test_plone50.cfg#L10-L18


[versions]
setuptools = 40.8.0
zc.buildout = 2.13.1

configparser = <5.0.0
docker = 2.6.1
elasticsearch = 6.3.0
plone.testing=5.0.0
pycodestyle=

idgserpro · July 28, 2020, 4:40am

@mauritsvanrees by the way. You can remove the pycodestyle pinn in Plone 5.0 coredev?

github.com

plone/buildout.coredev/blob/0dab333bb9381f54b01a8005bd0f894dcbb2fcb1/versions.cfg#L70


twine = 1.8.1
towncrier = 18.5.0
zest.releaser = 6.15.0
zestreleaser.towncrier = 1.0.0b3

# testing
collective.xmltestreport = 1.3.4
enum34 = 1.1.6
manuel = 1.8.0
mock = 1.0.1
pycodestyle = 2.3.1
zope.testbrowser = 3.11.1
zope.testrunner = 4.4.4
# persistent is not compatible with ZODB 3.10 which we are using
# If you have a dependency somewher on persistent, you must
# try to get rid of it.  But z3c.form has it in the test dependencies.
persistent = 4.2.4.2

# Robot Testing
plone.app.robotframework = 1.3.1
robotframework = 3.0

This package always conflicts with the pinned version on flake8:

github.com

PyCQA/flake8/blob/275b2320b5d266595e42e59481287b3ce1e710e4/setup.cfg#L45


package_dir=
    =src
packages = find:

install_requires=
# We document the reasoning for using ranges here:
# http://flake8.pycqa.org/en/latest/faq.html#why-does-flake8-use-ranges-for-its-dependencies
# And in which releases we will update those ranges here:
# http://flake8.pycqa.org/en/latest/internal/releases.html#releasing-flake8
    pyflakes >= 2.2.0, < 2.3.0
    pycodestyle >= 2.6.0a1, < 2.7.0
    mccabe >= 0.6.0, < 0.7.0
    enum34; python_version<"3.4"
    typing; python_version<"3.5"
    configparser; python_version<"3.2"
    functools32; python_version<"3.2"
    importlib-metadata; python_version<"3.8"

python_requires = >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*

[options.packages.find]

I believe this pinout isn't necessary in coredev. In Plone 5.2 this package isn't pinned.

jaroel · July 28, 2020, 8:02am

I think not everything was pinned
I used allow-picked-versions = false to make sure I have everything pinned.

When creating the initial versions list I set show-picked-versions = true and set allow-picked-versions = true once, and add that to a local dumpversions.cfg with a [versions] local - your project's KGS. Add that to git and make your buildout.cfg extend it.

Now, your dependencies can still be changed on PyPI, so new unexpected sub-dependencies are possible. I recommend building your project, from scratsch, every day using something like Jenkins to monitor that.

I also used devpi to have a company wide cache of upstream packages based on an allowlist, so I never really run into this kind of issues.

mauritsvanrees · July 28, 2020, 9:29am

Technically I can. But there won't ever be a new Plone 5.0 release, so it is no use.
See buildout.coredev/DO-NOT-USE-THIS-BRANCH.txt at 5.0 · plone/buildout.coredev · GitHub

idgserpro · July 28, 2020, 12:39pm

OK. Thanks!

tkimnguyen · July 28, 2020, 11:03pm

Thanks, but it's worse than that: zipp was not released until 2018, and I was looking for pins from late 2016.

I'm not sure I understand how this would help. I have these:

setuptools = 24.3.0
zc.buildout = 2.5.2
plone.testing = 5.0.0

but I do not have pins for configparser, docker, elasticsearch, nor pycodestyle.

This is what I decided to try next. I had to help buildout by pip install Pillow==3.2.0 and it completed!

The pins I had to add:

zest.releaser = 6.8
twine = 1.8.1

And the output was:

Versions had to be automatically picked.
The following part definition lists the versions picked:
[versions]
pathtools = 0.1.2
pkginfo = 1.5.0.1
requests-toolbelt = 0.9.1
watchdog = 0.10.3

# Required by:
# clint==0.5.1
args = 0.1.0

# Required by:
# twine==1.8.1
clint = 0.5.1

# Required by:
# zest.releaser==6.8
colorama = 0.4.3

# Required by:
# plonetheme.future-imperfect==1.0.2.dev0
plone.app.themingplugins = 1.1

# Required by:
# plone.app.themingplugins==1.1
# plonetheme.future-imperfect==1.0.2.dev0
z3c.jbot = 0.8

Thank you all! I will push these changes to the repo

tkimnguyen · July 28, 2020, 11:11pm

idgserpro · July 29, 2020, 3:18am

Is there a specific reason why you want to use the 2016 versions? Can't you just use more current versions?