Error doing upgrade with buildout

bieli · November 15, 2017, 11:51am

Hello,

I'm getting this error on an simple buildout for upgrading Plone from 4.3.14 to 4.3.15:

# sudo -u plone_buildout bin/buildout
Develop: '/opt/Plone/zeocluster/src/sb.exergo'
Getting distribution for 'plone.recipe.zeoserver==1.3.1'.
While:
  Installing.
  Getting section zeoserver.
  Initializing section zeoserver.
  Installing recipe plone.recipe.zeoserver.
  Getting distribution for 'plone.recipe.zeoserver==1.3.1'.
Error: Couldn't find a distribution for 'plone.recipe.zeoserver==1.3.1'.
*************** PICKED VERSIONS ****************
[versions]

*************** /PICKED VERSIONS ***************

What is the problem? My buildout.cfg should be fine and ready to upgrade:

extends =
    base.cfg
#    versions.cfg
#    http://dist.plone.org/release/4.3.3/versions.cfg
    http://dist.plone.org/release/4.3-latest/versions.cfg

# If you change your Plone version, you'll also need to update
# the repository link below.
find-links +=
#    http://dist.plone.org/release/4.3.3
    http://dist.plone.org/release/4.3-latest

Thanks for help.
Bieli

hvelarde · November 15, 2017, 12:15pm

read this:

bieli · November 15, 2017, 12:29pm

Hello,

so, if I understand you right, I only have to change this line:

http://dist.plone.org/release/4.3-latest/versions.cfg

to:

https://dist.plone.org/release/4.3-latest/versions.cfg

I'm right?

Thanks,
Bieli

espenmn · November 15, 2017, 12:32pm

When in trouble, add this line to buildout.cfg: [buildout]

index = https://pypi.python.org/simple/

bieli · November 15, 2017, 2:46pm

Hello,

I added the index value but now I got some other errrors:

# sudo -u plone_buildout bin/buildout
Develop: '/opt/Plone/zeocluster/src/sb.exergo'
Getting distribution for 'plone.recipe.zeoserver==1.3.1'.
warning: no previously-included files matching '*pyc' found anywhere in distribution
Got plone.recipe.zeoserver 1.3.1.
Getting distribution for 'plone.recipe.zope2instance==4.3'.
warning: no previously-included files matching '*pyc' found anywhere in distribution
Got plone.recipe.zope2instance 4.3.
Uninstalling precompiler.
Uninstalling unifiedinstaller.
Uninstalling zopepy.
Uninstalling backup.
Uninstalling client2.
Uninstalling client1.
Uninstalling zeoserver.
Installing zeoserver.
Created directory /opt/Plone/zeocluster/parts/zeoserver
Created directory /opt/Plone/zeocluster/parts/zeoserver/etc
Created directory /opt/Plone/zeocluster/parts/zeoserver/var
Created directory /opt/Plone/zeocluster/parts/zeoserver/log
Created directory /opt/Plone/zeocluster/parts/zeoserver/bin
Wrote file /opt/Plone/zeocluster/parts/zeoserver/etc/zeo.conf
Wrote file /opt/Plone/zeocluster/parts/zeoserver/bin/zeoctl
Changed mode for /opt/Plone/zeocluster/parts/zeoserver/bin/zeoctl to 755
Wrote file /opt/Plone/zeocluster/parts/zeoserver/bin/runzeo
Changed mode for /opt/Plone/zeocluster/parts/zeoserver/bin/runzeo to 755
Generated script '/opt/Plone/zeocluster/bin/zeoserver'.
Generated script '/opt/Plone/zeocluster/bin/zeopack'.
Generated script '/opt/Plone/zeocluster/bin/repozo'.
Installing client1.
Getting distribution for 'Plone==4.3.15'.
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

Got Plone 4.3.15.
Getting distribution for 'plone.app.jquerytools==1.9.2'.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.jar' found anywhere in distribution
warning: no previously-included files matching 'jslint.js' found anywhere in distribution
warning: no previously-included files matching 'jqt_checkout_build' found anywhere in distribution
Got plone.app.jquerytools 1.9.2.
Getting distribution for 'Products.Archetypes==1.9.17'.
warning: no previously-included files matching '*pyc' found anywhere in distribution
warning: no previously-included files found matching '.installed.cfg'
warning: no previously-included files found matching '.travis.yml'
warning: no previously-included files found matching 'bootstrap.py'
warning: no previously-included files found matching 'buidlout.cfg'
  File "build/bdist.linux-i686/egg/Products/Archetypes/skins/archetypes/lookupMime.py", line 19
    return mimetypes[0].name()
SyntaxError: 'return' outside function
...
Got Products.Archetypes 1.9.17.
Getting distribution for 'plone.app.caching==1.1.12'.
warning: no previously-included files matching '*pyc' found anywhere in distribution
Got plone.app.caching 1.1.12.
Getting distribution for 'Products.GenericSetup==1.8.8'.
Couldn't find index page for 'eggtestinfo' (maybe misspelled?)
No local packages or download links found for eggtestinfo
error: Could not find suitable distribution for Requirement.parse('eggtestinfo')
An error occurred when trying to install Products.GenericSetup 1.8.8. Look above this message for any errors that were output by easy_install.
While:
  Installing client1.
  Getting distribution for 'Products.GenericSetup==1.8.8'.
Error: Couldn't install: Products.GenericSetup 1.8.8

Any help?

hvelarde · November 15, 2017, 4:06pm

that might be a transitory error; just run bin/buildout again.

bieli · November 15, 2017, 5:22pm

Still getting the same error:

# sudo -u plone_buildout bin/buildout
Develop: '/opt/Plone/zeocluster/src/sb.exergo'
Updating zeoserver.
Installing client1.
Getting distribution for 'Products.GenericSetup==1.8.8'.
Couldn't find index page for 'eggtestinfo' (maybe misspelled?)
No local packages or download links found for eggtestinfo
error: Could not find suitable distribution for Requirement.parse('eggtestinfo')
An error occurred when trying to install Products.GenericSetup 1.8.8. Look above this message for any errors that were output by easy_install.
While:
  Installing client1.
  Getting distribution for 'Products.GenericSetup==1.8.8'.
Error: Couldn't install: Products.GenericSetup 1.8.8

He is not installing client2, nor zeoserver.

bieli · November 16, 2017, 11:22am

I'm so disappointed by Plone and its update process (by the way, the update process is not one of the easiest I know). Sorry, but I cannot understand why a simple patch update from version 4.3.14 to 4.3.15 brings so many problems? I have to add a new configuration option (index = https://pypi.python.org/simple/) and still got errors. For what do I need 'Products.GenericSetup==1.8.8' and why does it have package errors? Has anyone tried an update before publishing the new version?

Or I'm wrong and it's all my fault?

Thanks for explications!
Bieli

hvelarde · November 16, 2017, 11:45am

your problem has nothing to do with Plone or its update process; the first problem was due to changes on PyPI scheme; the second, I have no idea: it works perfectly for me.

$ curl -v https://pypi.python.org/simple/products.genericsetup/ | grep 1.8.8
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.92.223...
* Connected to pypi.python.org (151.101.92.223) port 443 (#0)
...
> GET /simple/products.genericsetup/ HTTP/1.1
> Host: pypi.python.org
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.10.3
< Content-Type: text/html; charset=utf-8
< X-PYPI-LAST-SERIAL: 2856386
< Cache-Control: max-age=600, public
< X-Clacks-Overhead: GNU Terry Pratchett
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Via: 1.1 varnish
< Fastly-Debug-Digest: de9b1ef0e7ccd79794735a2650c6dc8942b6b7ab22d589635ad8f4539f005d13
< Content-Length: 9745
< Accept-Ranges: bytes
< Date: Thu, 16 Nov 2017 11:37:53 GMT
< Via: 1.1 varnish
< Age: 75078
< Connection: keep-alive
< X-Served-By: cache-iad2140-IAD, cache-gru17126-GRU
< X-Cache: HIT, HIT
< X-Cache-Hits: 1, 1
< X-Timer: S1510832274.985445,VS0,VE0
< Vary: Accept-Encoding
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
{ [1988 bytes data]
100  9745  100  9745    0     0  24091      0 --:--:-- --:--:-- --:--:-- 24061
* Connection #0 to host pypi.python.org left intact
<a href="../../packages/10/56/11b8e227c8aa523e1c3ee3495fa6f60e816ce9ae1295743e44ed16afa958/Products.GenericSetup-1.8.8.tar.gz#md5=a9d2b9e71d9736ac7ee12a2d3e5136cb" rel="internal">Products.GenericSetup-1.8.8.tar.gz</a><br/>
<a href="../../packages/89/2a/a0849d93d448f20fa091f05f04f49707ccd90e8221cd2e65ecb4f5b50fec/Products.GenericSetup-1.8.4.tar.gz#md5=de1aedffdacca3d7cb2910868821e775" rel="internal">Products.GenericSetup-1.8.4.tar.gz</a><br/>
$ curl -I https://pypi.python.org/packages/10/56/11b8e227c8aa523e1c3ee3495fa6f60e816ce9ae1295743e44ed16afa958/Products.GenericSetup-1.8.8.tar.gz
HTTP/1.1 200 OK
x-amz-replication-status: COMPLETED
Last-Modified: Sat, 06 May 2017 21:49:57 GMT
ETag: "a9d2b9e71d9736ac7ee12a2d3e5136cb"
x-amz-meta-version: 1.8.8
x-amz-meta-project: products-genericsetup
x-amz-meta-python-version: source
x-amz-meta-package-type: sdist
Content-Type: binary/octet-stream
Server: AmazonS3
Via: 1.1 varnish
Fastly-Debug-Digest: a5566620d9a11a96bc399e34643cb84db44daf184f5856c80c310709a2056885
Cache-Control: max-age=31557600, public
Content-Length: 139398
Accept-Ranges: bytes
Date: Thu, 16 Nov 2017 11:39:43 GMT
Via: 1.1 varnish
Age: 5350099
Connection: keep-alive
X-Served-By: cache-sea1041-SEA, cache-gru17120-GRU
X-Cache: HIT, HIT
X-Cache-Hits: 1, 3
X-Timer: S1510832384.876260,VS0,VE0
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none

please avoid ranting for nothing and do more research by yourself; we're giving free support based in volunteer's spare time.

bieli · November 16, 2017, 2:51pm

Hi hvelarde,

thanks for answer, but the download isn't really my problem. In the update process y got this error:

# sudo -u plone_buildout bin/buildout
Develop: '/opt/Plone/zeocluster/src/sb.exergo'
Uninstalling precompiler.
Uninstalling unifiedinstaller.
Uninstalling zopepy.
Uninstalling backup.
Uninstalling client2.
Uninstalling client1.
Uninstalling zeoserver.
Installing zeoserver.
Created directory /opt/Plone/zeocluster/parts/zeoserver
Created directory /opt/Plone/zeocluster/parts/zeoserver/etc
Created directory /opt/Plone/zeocluster/parts/zeoserver/var
Created directory /opt/Plone/zeocluster/parts/zeoserver/log
Created directory /opt/Plone/zeocluster/parts/zeoserver/bin
Wrote file /opt/Plone/zeocluster/parts/zeoserver/etc/zeo.conf
Wrote file /opt/Plone/zeocluster/parts/zeoserver/bin/zeoctl
Changed mode for /opt/Plone/zeocluster/parts/zeoserver/bin/zeoctl to 755
Wrote file /opt/Plone/zeocluster/parts/zeoserver/bin/runzeo
Changed mode for /opt/Plone/zeocluster/parts/zeoserver/bin/runzeo to 755
Generated script '/opt/Plone/zeocluster/bin/zeoserver'.
Generated script '/opt/Plone/zeocluster/bin/zeopack'.
Generated script '/opt/Plone/zeocluster/bin/repozo'.
Installing client1.
The constraint, 2.0.3, is not consistent with the requirement, 'plone.protect>=3.1.1'.
While:
  Installing client1.
Error: Bad constraint 2.0.3 plone.protect>=3.1.1
*************** PICKED VERSIONS ****************
[versions]
Products.EasyNewsletter = 3.0.4
Products.PloneFormGen = 1.7.12

*************** /PICKED VERSIONS ***************

Then correcting in buildout.cfg:

[versions]
...
plone.protect =

Now I got this error:

# sudo -u plone_buildout bin/buildout
Develop: '/opt/Plone/zeocluster/src/sb.exergo'
Updating zeoserver.
Installing client1.
Getting distribution for 'Products.GenericSetup==1.8.8'.
Couldn't find index page for 'eggtestinfo' (maybe misspelled?)
No local packages or download links found for eggtestinfo
error: Could not find suitable distribution for Requirement.parse('eggtestinfo')
An error occurred when trying to install Products.GenericSetup 1.8.8. Look above this message for any errors that were output by easy_install.
While:
  Installing client1.
  Getting distribution for 'Products.GenericSetup==1.8.8'.
Error: Couldn't install: Products.GenericSetup 1.8.8

So if the automatic update of GenericSetup not works, how can I do it manually? Because the curl download and the "python setup.py build" works.

Thanks,
Bieli

hvelarde · November 16, 2017, 3:17pm

do not unpin any package or you risk breaking your installation in the future; better use plone.protect = 3.1.1 as stated.

on the other side that problem could be caused by a broken egg in your download cache.

in my case looks like this:

$ ls ~/.buildout/downloads/dist/Products.GenericSetup-1.*
/home/hvelarde/.buildout/downloads/dist/Products.GenericSetup-1.7.3.tar.gz
/home/hvelarde/.buildout/downloads/dist/Products.GenericSetup-1.7.4.tar.gz
/home/hvelarde/.buildout/downloads/dist/Products.GenericSetup-1.8.2.tar.gz
/home/hvelarde/.buildout/downloads/dist/Products.GenericSetup-1.8.3.tar.gz
/home/hvelarde/.buildout/downloads/dist/Products.GenericSetup-1.8.6.tar.gz
/home/hvelarde/.buildout/downloads/dist/Products.GenericSetup-1.8.8.tar.gz

look for the Products.GenericSetup-1.8.8.tar.gz egg and just erase it; run buildout again.

hvelarde · November 16, 2017, 3:20pm

looking a little bit closer this line above points to your real problem: your buildout configuration seems wrong.

bieli · November 16, 2017, 4:36pm

No, it seems not to be my buildout.cfg
I resolved the problem installing "eggtestinfo":

# /opt/Plone/Python-2.7/bin/pip install eggtestinfo
Downloading/unpacking eggtestinfo
  Downloading eggtestinfo-0.3.tar.gz
  Running setup.py egg_info for package eggtestinfo

Installing collected packages: eggtestinfo
  Running setup.py install for eggtestinfo

Successfully installed eggtestinfo
Cleaning up...

Why now I need "eggtestinfo" installed and before not?
Anyway, now runs my buildout to the end, but with this error in the middle:

...
Updating precompiler.
Compiling Python files.
  File "/opt/Plone/buildout-cache/eggs/Jinja2-2.10-py2.7.egg/jinja2/asyncsupport.py", line 22
    async def concat_async(async_gen):
            ^
SyntaxError: invalid syntax

  File "/opt/Plone/buildout-cache/eggs/Jinja2-2.10-py2.7.egg/jinja2/asyncfilters.py", line 7
    async def auto_to_seq(value):
            ^
SyntaxError: invalid syntax

Compiling locale files.
...

For what I need this Jinja package?

*************** PICKED VERSIONS ****************
[versions]
BeautifulSoup = 3.2.1
Jinja2 = 2.10
MarkupSafe = 1.0
...
*************** /PICKED VERSIONS ***************

But finely I can start my Plone and update my web sites.

Do you have any explications?

Thanks,
Bieli

hvelarde · November 16, 2017, 5:50pm

I never heard of eggtestinfo before; that's not a Plone package.

you have to do your homework now; I need to work on my own stuff now.

zopyx · November 16, 2017, 6:16pm

I share your pain - to a certain degree. In most cases the issues are related to setuptools, invalid or insufficent pinng of versions. My major is the bootstrap process with virtualenv, setuptools and zc.buildout. In particular the behaviour differs from Linux distro to Linux distro and lots of fun with testing Plone buildouts on Travis and with Docker...Plone is usually not part of the problem...it is the bootstrapping process that is fragile.

-aj

djay · November 17, 2017, 3:05am

@zopyx I'd say buildout/setuptools and us having so many packages is partly to blame. If something does wrong it has the possibility to break everything. You can't add a single package, even your own dev plugin, without running the risk of something breaking. And then you are faced with learning a lot quickly to get yourself out of that mess, which is scary for someone who just wants a website.
I'm not sure there is a simple way out of it though.

dieter · November 17, 2017, 7:50am

Part of this is due to the flexibility of buildout. A long time ago, Tres Seavers had proposed meta eggs as an alternative to describe the requirements of an application: a meta egg is an essentially empty package which consists just of the application's list of requirements (usually version pinned).

datakurre · December 11, 2017, 8:28pm

A few old zope packages require that as build time dependency (setup_requires).

hvelarde · December 11, 2017, 8:34pm

thanks, I don't get the point on using it and seems to be pretty old and probably useless now; I think we need to get rid of that dependency.

and yes, I have to agree that the installation process is quite fragile at the moment: I had to help @agnogueira last week because of the latest issue related with setuptools, buildout and wheels:

seems we need to update more frequently our versions to avoid further issues.