Update Plone 4.2 to 4.3 - buildout-error

Plone Support Administering Plone sites
,
1

Hi

I am looking for help updating my plone on 4.2.5 version to latest 4.3.

In general I don't touch plone very often from an administrative point of view (but we use it quite a lot as it is our intranet), therefore am not familiar with all the Python/Buildout/anything-like-this-stuff - so maybe there are really no other helps, but maybe I am just not able to find it properly.

When I want to upgrade my plone bia buildout (according the documentation at https://docs.plone.org/4/en/manage/upgrading/plone4_minor_upgrade.html) I get the following error:

    ...
    Installing recipe plone.recipe.zope2instance.
Error: There is a version conflict
We already have: zc.buildout 1.4.4
but zc.recipe.egg 1.3.2 requires 'zc.buildout>=1.5.0'
....

So I tried to find a way to solve it but can't find. I tried https://docs.plone.org/4/en/manage/upgrading/version_specific_migration/p42_to_p43_upgrade.html - but that does not help me at all :frowning:

What I also tried: When I look into http://dist.plone.org/release/4.3... I see a newer buildout in the 4.3-Version. So I tried to put this version into buildout.cfg, but that did not help either :frowning:

My environment is the following:
Debian Jessie (8), Plone installed by the UnifiedInstaller with Python built by the installer

I was thinking of: How can I update my buildout but did not find a howto or similar...

2

bin/buildout -vvv annotate | less

Search for zc.buildout in the output for what pins it to what and in which order to the overrides go. You most likely have some local legacy pinning or a weird order to extend your buildout config from the 4.3 config.

3

Hi

Thank you for your response. I cannot give you an answer, as I don't understand, what you mean with "... for what pins it to what and in which order to the overrides go."

Therefore i just post here the output:
https://pastebin.com/Rv7S9JX6

-> If you prefer another way of getting the data, please tell me and I will do my best...

Thank you and kind regards
David

4

What @Rotonen means is the following:

$ bin/buildout annotate | grep zc.buildout -A1
zc.buildout= 2.5.3
   /opt/plone/buildout/buildout.d/versions.cfg

In my case, for instance, the zc.buildout version came from /opt/plone/buildout/buildout.d/versions.cfg, that means I need to override that version there or on a buildout configuration loaded after that.

I don't use the unified installer, but others can help you in case you need more guidance.

5

mmm, I just checked your output but I was not able to determine where are you extending http://dist.plone.org/release/4.3/versions.cfg on your configuration.

can you check in your buildout.cfg file for a directive named extends?

6

Hi hvelarde

Thank you for your explanation! The URL pointing to the 4.3/versions.cfg is been made in the extends-section of buildout.cfg - according to https://docs.plone.org/4/en/manage/upgrading/plone4_minor_upgrade.html.

Therefore it should - at least in my opinion - update itself to the version 1.7.1, but obviously it does not :frowning:

How can I update my buildout? At the moment, I suspect, that would solve my problem...

Thank you to any help! I'm really stuck here...

Kind regards
David

7

David, a warning if you are following https://docs.plone.org/4/en/manage/upgrading/plone4_minor_upgrade.html to the letter:

Including 'online' extends from dist.plone.org in buildout configurations is very cool for automated test setups and you'll find this a lot in testing infrastructure in plone add'on packages on travis/jenkins, linking to a 4.3-latest.cfg or 5.0-latest.cfg

But for production site setups, please note that linking to extends like http://dist.plone.org/release/4.3-latest/versions.cfg means that you no longer have a reproducable buildout and site setup. If you run bin/buildout again 3 or 4 months later you might get one or two minor plone releases later. The community tries hard to only have bug fixes and no other side effects, but a part of my job is to hunt down small improvements that break existing and before stable Plone setups after a new minor release is applied. These happen. Only upgrade a production site when you have verified locally that the upgrade works, even if it's a minor version bump.

We download the versions.cfg for a specific Plone release from dist.plone.org/release/ and also the nested extends in the versions.cf and put these in our buildout VCS. (git/svn/whatever). Added (small) benefit is that I can grep for plone.app.dexterity in *.cfg in my buildout main dir and see exactly which version is used.

If you multiple sites and want to share configs, create a separate repository on github or internal, put a kgs-version of the versions.cfg there and include those, but don't make your setup dependent on the latest-links on dist.plone.org.

8

Hello Fred, thank you very much for your warnings!

What i am trying to achieve is the following:
I started a week ago with Plone 4.1.anything on a debian server on version 8 - Plone and Debian being updated several times, Debian not able to update to version 9 anymore.

So my plan is to build another server from scratch and move Plone to there to the newest version. In detail:

  • Update Plone on the old server to the current version
  • Build the new server and install Plone there with the same version and the same config as the old one
  • Move the content from the old Plone server to the new one.
    -> I tried it already the other way: Install an old Plone on the new server and then do the update of Plone on the new server, but I did not manage to properly install it: The UnifiedInstaller just didn't make it :frowning:

I was able to install plain Plone on the new server and am now trying getting Plone on the old server to the same version. Unfortunately that's it, where I am stuck from version 4.2.5 onwards. For upgrading a site to Plone 5, it is recommended to first upgrade a site to the newest version 4 (that's at least how I understand the documentation).

I have one single instance of Plone, nothing fancy installed: It is our intranet/internal documentation system. So important but not critical from a time perspective, it can be down for some time, but the content itself is critical to our business. After this whole "update/upgrade party" I would be glad to go back to a slower pace in updating (well, not exactly that slow as it was in the recent years...).

So I would just be glad, having some help getting to my goal of being able to find a way to get further with my upgrade...

Thank you very much and kind regards
David

9

Hello

Still batteling with the update of my Plone site :frowning: - but I have new insights:
Out of couriosity I tried another approach:

I installed a proper, new Plone-site from the PloneInstaller 4.2.5 and created an empty plone site. Command to use:
install.sh --target=<path> --user=nobody standalone

After that I wanted to update my plone site to 4.2-latest via buildout, adjusting the plone/zinstance/buildout.cfg, but still ran into the same problem:

...
but zc.recipe.egg 1.3.2 requires 'zc.buildout>=1.5.0'.

So my problem is not a corrupt, updated installation but I suspect a general problem.

Can PLEASE anybody help me? For us it is really a problem and I would appreciate very much having this solved...

Thank you very much and kind regards
David

10

Hello

Still no progress visible so far - also no more inputs on this community, so I am quite lost. The question is easy but for me, I am not able to find the answer:
How can I migrate to the latest version of plone? As the step n between, I'm still trying to do the first step to migrate to the latest version 4 before trying to update then to version 5.

How can I update my site from version 4.2.5 to the latest 4.3? Is there a simple howto? Like a cook-book? Either I have tomatoes on my eyes (which IS possible as I am not very good in finding stuff like this) or it is hard to find.

I would appreciate very much if I could get any help!

Thank you and kind regards
David

11

Hi David,

From what I can see you are including the 4.3 release, which is not the latest one, this would be http://dist.plone.org/release/4.3.15/.

The output you posted clearly shows what you should search for (line 684-685):

zc.buildout= 1.7.1
    http://dist.plone.org/release/4.3/versions.cfg

So this indicates 1.7.1 should be used, but apparently it is not. To further investigate your problem, please post your buildout configuration (I can see a local base.cfg and maybe a buildout.cfg), and also please post the complete output of your buildout run. Without that information it is very hard to point you to the right direction.

Thomas

12

Hi Thomas

Thank you for your answer - very much appreciate it!

Here is the actual status: Due to the needed version of buildout, already the change from version 4.2.5 to 4.2.6 is the problem (means: I could update until version 4.2.5 and I guess, I will be able to update from version 4.2.6 onwards).

So I have my Plone at the moment at verison 4.2.5 and want to go to verison 4.2.6.

Therefore, I paste the information from this minor update.

buildout.cfg (took out a lot of commented text):

############################################
#
# Buildout Configuration File for Standalone Plone
# ------------------------------------------------
#
############################################

[buildout]

############################################
# Plone Component Versions
# ------------------------
#
extends =
    base.cfg
#    versions.cfg
    http://dist.plone.org/release/4.2.6/versions.cfg

# If you change your Plone version, you'll also need to update
# the repository link below.
find-links +=
    http://dist.plone.org/release/4.2.6

############################################
# Ports
# -----
http-address = 8080

# If you try to start Zope as root, it will change user id to run as
# the effective user specified here. This user id must own the var directory
# of your buildout.
effective-user = nobody

############################################
# Eggs
# ----
#
eggs =
    Plone
    Pillow

# Optional Functionality
# ----------------------

############################################
# ZCML Slugs
# ----------
zcml =
#    plone.reload

############################################
# Development Eggs
# ----------------
develop =
#    src/my.package

############################################
# Debug Mode
# ----------
#
debug-mode = off
# Add-on developers should turn deprecation warnings on
deprecation-warnings = off
# change verbose-security to "on" for useful security errors while developing
verbose-security = off

############################################
# Backup Directory
# ----------------
#
backups-dir=${buildout:directory}/var

############################################
# Environment Variables
# ---------------------
environment-vars =
    zope_i18n_compile_mo_files true
    PYTHON_EGG_CACHE ${buildout:directory}/var/.python-eggs
    PYTHONHASHSEED random
    #    TZ US/Eastern
    #    zope_i18n_allowed_languages en es de fr

############################################
# Initial User
# ------------
user=admin:********

############################################
# Parts Specification
#--------------------
parts =
    instance
    zopepy
    zopeskel
    unifiedinstaller
    repozo
    backup
    chown


############################################
# Versions Specification
# ----------------------
#
[versions]
ZopeSkel = 2.21.2
plone.recipe.unifiedinstaller = 4.1
lxml = 2.3.6
Cheetah = 2.2.1
Pillow = 1.7.8
Products.DocFinderTab = 1.0.5
collective.recipe.backup = 2.9
plone.recipe.command = 1.1

Then base.cfg:

######################################################
# Base Configuration; used by both standalone and zeo,
# which extend it.
# ---------------------------------------------------

# Buildout instructions in this file are
# usually only changed by experienced developers.
#
# Beyond here there be dragons!

[buildout]
eggs-directory=../buildout-cache/eggs
download-cache=../buildout-cache/downloads

# Don't download new things unless needed to satisfy dependencies.
# Override this on the command line with the "-n" flag.
newest = false

# pick final releases over newer development releases
prefer-final = true

versions = versions

# Add additional egg download sources here. dist.plone.org contains archives
# of Plone packages.
find-links =
    http://dist.plone.org
    http://download.zope.org/ppix/
    http://download.zope.org/distribution/
    http://effbot.org/downloads

# unzip all eggs for easier debugging
unzip = true

# Load the DumpPickedVersions extension, which will notify
# you of versions picked by buildout that were not specifically
# pinned.
extensions = buildout.dumppickedversions

[instance]
# Use this section to install and configure a Zope operating
# instance.
# For options see http://pypi.python.org/pypi/plone.recipe.zope2instance
recipe = plone.recipe.zope2instance
# The line below sets only the initial password. It will not change an
# existing password.
user = ${buildout:user}
http-address = ${buildout:http-address}
# if we try to start as root, Zope will switch to the user below
effective-user = ${buildout:effective-user}
# change debug-mode to "on" to run in development mode
debug-mode = ${buildout:debug-mode}
# change verbose-security to "on" for detailed security
# errors while developing
verbose-security = ${buildout:verbose-security}
# change deprecation-warnings to "on" to get log warnings
# for deprecated usages.
deprecation-warnings = ${buildout:deprecation-warnings}
blob-storage = ${buildout:directory}/var/blobstorage

# Comment the next four lines out if you don't need
# automatic log rotation for event and access logs.
event-log-max-size = 5 MB
event-log-old-files = 5
access-log-max-size = 20 MB
access-log-old-files = 5

# If you want Zope to know about any additional eggs, list them here.
# e.g. eggs = ${buildout:eggs} my.package
eggs =
    ${buildout:eggs}

# If you want to register ZCML slugs for any packages, list them here.
# e.g. zcml = my.package my.other.package
zcml = ${buildout:zcml}

products = ${buildout:directory}/products

# You may also control the environment variables for the instance.
environment-vars = ${buildout:environment-vars}

[zopepy]
# installs a zopepy python interpreter that runs with your
# full Zope environment
recipe = zc.recipe.egg
eggs = ${buildout:eggs}
interpreter = zopepy
scripts = zopepy

[chown]
# This recipe is used to set permissions for root mode installs
# For options see http://pypi.python.org/pypi/plone.recipe.command
recipe = plone.recipe.command
command =
    echo Dummy references to force this to execute after referenced parts
    echo ${backup:location} ${unifiedinstaller:sudo-command}
    chmod 600 .installed.cfg
    find ${buildout:directory}/var -type d -exec chmod 700 {} \;
    chmod 744 ${buildout:directory}/bin/*
update-command = ${chown:command}

[unifiedinstaller]
# This recipe installs the plonectl script and a few other convenience
# items.
# For options see http://pypi.python.org/pypi/plone.recipe.unifiedinstaller
recipe = plone.recipe.unifiedinstaller
user = ${buildout:user}
primary-port = ${buildout:http-address}
sudo-command = sudo -u nobody

[repozo]
# This recipe builds the repozo script for non-zeo installations.
recipe = zc.recipe.egg
eggs = ZODB3
scripts = repozo

[backup]
# This recipe builds the backup, restore and snapshotbackup commands.
# For options see http://pypi.python.org/pypi/collective.recipe.backup
recipe = collective.recipe.backup
location = ${buildout:backups-dir}/backups
snapshotlocation = ${buildout:backups-dir}/snapshotbackups

[zopeskel]
# installs paster and Zopeskel
recipe = zc.recipe.egg
eggs =
    ZopeSkel
    Paste
    PasteDeploy
    PasteScript
    ${buildout:eggs}

And as last the output from the buildout-process (I hope, this is the one you are looking for - otherwise please tell me!):

root@vm-0019:/work/plone/zinstance# bin/buildout
Getting distribution for 'plone.recipe.zope2instance==4.2.13'.
warning: no previously-included files matching '*pyc' found anywhere in distribution
Got plone.recipe.zope2instance 4.2.13.
Getting distribution for 'Zope2==2.13.21'.
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
Got Zope2 2.13.21.
Getting distribution for 'zc.recipe.egg==1.3.2'.
Got zc.recipe.egg 1.3.2.
Getting distribution for 'ZConfig==2.9.1'.
Got ZConfig 2.9.1.
Getting distribution for 'zc.lockfile==1.0.2'.
Got zc.lockfile 1.0.2.
Getting distribution for 'zope.tales==3.5.3'.
Got zope.tales 3.5.3.
Getting distribution for 'pytz==2013b'.
Got pytz 2013b.
Getting distribution for 'AccessControl==2.13.13'.
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
Got AccessControl 2.13.13.
While:
  Installing.
  Getting section instance.
  Initializing section instance.
  Installing recipe plone.recipe.zope2instance.
Error: There is a version conflict.
We already have: zc.buildout 1.4.4
but zc.recipe.egg 1.3.2 requires 'zc.buildout>=1.5.0'.
*************** PICKED VERSIONS ****************
[versions]

*************** /PICKED VERSIONS ***************
root@vm-0019:/work/plone/zinstance#

Additionally I ran a bin/buildout -vvv annotate into a file - the content of this file is in the pastebin: https://pastebin.com/HhyFcqza

I hope, this is the right information. If you would like to have the complete buildout.cfg-file or a specific log-file or something additional, just tell me: I will paste it then here as well.

Thank you very much and kind regards
David

13

since the very beginning the problem has been your zc.buildout version; I suggest you to do the following:

  • activate the Python of your virtualenv and verify it; you must run something like:

    plone@foo:~$ source py27/bin/activate
    (py27) plone@foo:~$ which python
    /opt/plone/py27/bin/python

as mentioned earlier I don't use the universal installer, so I have no idea how to do so in your case.

  • bootstrap your project again; that depends on the version of your bootstrap.py file but it should be something like this (run python bootstrap.py --help in case of error):

    $ python bootstrap.py --buildout-version=1.7.1

  • run your buildout:

    $ bin/buildout

  • you can always use a different zc.buildout version by using:

    $ bin/buildout versions:zc.buildout==x.y.z

14

The bin/buildout -vvv annotate output seems somehow disconnected from your bin/buildout output. Are you sure you are running it on the same project?

Other than that, I agree with @hvelarde about redoing the isolation-from-system-site-packages steps in a surefire way.

15

I have the strong feeling that zc.buildout is somehow installed on your system in version 1.4.4. Either within the virtual environment (check with pip freeze) or in the system Python. On your terminal, please run $ less bin/buildout and check for the Python interpreter used by buildout.

I found an old setup of mine where it has the following content:

#!/opt/zope/venv26//bin/python

import sys
sys.path[0:0] = [
    '/opt/zope/venv26/.buildout/eggs/distribute-0.6.19-py2.6.egg',
    '/opt/zope/venv26/.buildout/eggs/zc.buildout-1.4.4-py2.6.egg',
    ]


import zc.buildout.buildout

if __name__ == '__main__':
    zc.buildout.buildout.main()

Here you can see that zc.buildout is coming from the buildout eggs, so it is installed via buildout itself.

On another old project it looks like this:

#!/opt/zope/venv26/vhp27/bin/python2.7

import sys
sys.path[0:0] = [
  '/opt/zope/venv26/vhp27/lib/python2.7/site-packages',
  ]

import zc.buildout.buildout

if __name__ == '__main__':
    sys.exit(zc.buildout.buildout.main())

You can see that the zc.buildout line within sys.path is missing. It is coming from the virtual environment.

I would do the following (because using bootstrap.py is deprecated):

  1. create a new virtual environment and activate it
  2. install zc.buildout with pip install zc.buildout==2.5.0
  3. in your buildout.cfg, add the line zc.buildout = at the end to the other version pins, this will unset previous pins for zc.buildout and use the one from your virtual environment
  4. run buildout (without any arguments; note that this is not ./bin/buildout!)

Hope that helps!
Thomas

16

Thank you for your answers!

edit:
Let me check the input from Thomas - I did this post just at the same time as Thomas sent his answer...

Hmm, I have only this one installation. BUT I ran first the buildout, that generated the output and then in a second run (with same .cfg-files out of same directory) the -vvv annotate

I did the following:

root@vm-0019:/work/plone# source Python-2.7/bin/activate
(Python-2.7)root@vm-0019:/work/plone# which python
/work/plone/Python-2.7/bin/python
(Python-2.7)root@vm-0019:/work/plone# cd zinstance/
(Python-2.7)root@vm-0019:/work/plone/zinstance# ../Python-2.7/bin/python bootstrap.py --version=1.7.1
Traceback (most recent call last):
  File "bootstrap.py", line 65, in <module>
    exec urllib2.urlopen('http://python-distribute.org/distribute_setup.py'
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 467, in error
    result = self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 654, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
(Python-2.7)root@vm-0019:/work/plone/zinstance#

Where is the path defined, where Python should get the newer version of buildout from?

Thank you for your help in this!

Kind regards
David

17

@Thomas:
Right, my bin/buildout looks exactly as the first one in your example!

Just quick question:
"create a new environment and activate it": Does that mean the source Python-2.7/bin/activate-line?

Thank you, David

18

Yes. But if you can create a new one. If not, please run at least pip freeze after you activated it, so you know what is already installed.

Edit: I don't use the UnifiedInstaller, so I can't say where the Python is located which should be used for the virtual environment. Maybe someone with more UnifiedInstaller experience can help with that.

19

Ok, so here is what I tried:

(Python-2.7)root@vm-0019:/work/plone# source Python-2.7/bin/activate
(Python-2.7)root@vm-0019:/work/plone# pip install zc.buildout==2.5.0
Downloading/unpacking zc.buildout==2.5.0
  Downloading zc.buildout-2.5.0.tar.gz (302kB): 302kB downloaded
  Running setup.py egg_info for package zc.buildout

Downloading/unpacking setuptools>=8.0 (from zc.buildout==2.5.0)
  Downloading setuptools-36.2.7.zip (716kB): 716kB downloaded
  Running setup.py egg_info for package setuptools
    warning: no files found matching '*' under directory 'setuptools/_vendor'
Installing collected packages: zc.buildout, setuptools
  Running setup.py install for zc.buildout

    Skipping installation of /work/plone/Python-2.7/lib/python2.7/site-packages/zc/__init__.py (namespace package)
    Installing /work/plone/Python-2.7/lib/python2.7/site-packages/zc.buildout-2.5.0-py2.7-nspkg.pth
    Installing buildout script to /work/plone/Python-2.7/bin
  Found existing installation: setuptools 0.6c11
    Uninstalling setuptools:
      Successfully uninstalled setuptools
  Running setup.py install for setuptools
    warning: no files found matching '*' under directory 'setuptools/_vendor'
    Installing easy_install script to /work/plone/Python-2.7/bin
    Installing easy_install-2.7 script to /work/plone/Python-2.7/bin
  Could not find .egg-info directory in install record for setuptools>=8.0 (from zc.buildout==2.5.0)
Successfully installed zc.buildout setuptools
Cleaning up...
(Python-2.7)root@vm-0019:/work/plone# editor zinstance/buildout.cfg

Changed the end of my buildout.cfg to:

[versions]
ZopeSkel = 2.21.2
plone.recipe.unifiedinstaller = 4.1
lxml = 2.3.6
Cheetah = 2.2.1
Pillow = 1.7.8
Products.DocFinderTab = 1.0.5
collective.recipe.backup = 2.9
plone.recipe.command = 1.1
zc.buildout = 2.5.0

Then further in the process:

(Python-2.7)root@vm-0019:/work/plone# buildout
While:
  Initializing.
Error: Couldn't open /work/plone/buildout.cfg
(Python-2.7)root@vm-0019:/work/plone# cd zinstance/
(Python-2.7)root@vm-0019:/work/plone/zinstance# buildout
While:
  Installing.
  Loading extensions.
Error: Buildout now includes 'buildout-versions' (and part of the older 'buildout.dumppickedversions').
Remove the extension from your configuration and look at the 'show-picked-versions' option in buildout's documentation.
(Python-2.7)root@vm-0019:/work/plone/zinstance#

What did I wrong?

Thank you and kind regards
David

20

Nothing. A newer buildout plays with a slightly different ruleset and you need to update your buildout config accordingly.

Seems you are getting further.

Simply follow the plain English instructions it gives you in its output.