Help - upgrading Plone 4.1 to Plone 5.2

PloneGur · August 25, 2020, 11:39am

hi guys,

frankly i read about upgrading a Plone site to version 5.2 a lot, but it seems
too complicated because we have intranet in Plone 4.1

the documentations is only few, i miss a lot some real step by step tutorials.
Because this is the task from my company so I must ask you for general steps,
some directions.

Our Plone 4.1 intranet has some own eggs, one of these is intra.base,
in base.cfg for Plone 4.1 i see this own egg

[buildout]
eggs =
intra.base

but NOWHERE in *.cfg i see a reference to this intra.base,
only in eggs/ directory i see a sub dir for this own egg

so my first problem is HOW to install this own egg intra.base to Plone version 5.2?
may this own addon can be installed via plone 5.2 zmi later?

Generally for upgading to Plone 5.2 i do such steps

create virutal env with python 3.6:
/usr/local/bin/python3.6 -m venv .
source bin/activate

cat requirements.txt
setuptools==41.0.1
zc.buildout==2.13.2

./bin/pip3.6 install -r requirements.txt

then i copied eggs/ src/ var/ from old intranet plone 4.1 site to current directory,
in next step i edited the base.cfg:

[buildout]
index = Simple index

parts =
lxml
pylxml
#instance
backup
repozo
zopeskel

extends =
lxml.cfg
versions_41.cfg
> #keep plone 5.2 versions.cfg after version_41.cfg
http://dist.plone.org/release/5.2/versions.cfg

versions = versions

extensions =
#lovely.buildouthttp
mr.developer

find-links =
#Index of /release/4.1.4/
Index of /thirdparty/
/home/zope/lc_repo/

eggs =
#own eggs....
intra.base

then i run
bin/buildout -v -c production.cfg
but it stoped when trying to install our own egg, intra.base

i tried to add git https://....intra.base.git to [sources] section, but it could not help.

So the problem is, i have source code of intra.base in eggs/ dir,
but how to tell bin/buildout to know about this?

in such way of upgrading i missed the adviced step from internet:
'Click a big honking “Upgrade” button.'
or click on "To run that upgrade follow the links on top of the control panel or the ZMI to the form /@@plone-upgrade"

could you please give some general steps to upgrade the Plone 4.1 site to Plone 5.2?

Many thanks for your help.

zopyx · August 25, 2020, 12:23pm

Step 1: upgrade to Plone 4.3 first
Step 2: upgrade to Plone 5.2/Python 3 : https://docs.plone.org/manage/upgrading/index.html

Remember that all your code and add-ons must be Python 3 compatible. Archetypes-based content will not run with Plone 5.2 under Python 3.

espenmn · August 25, 2020, 2:42pm

I don't think you should copy eggs-folder to the new setup, especially since you upgrade Python.

Your own add-ons: For now I would recommend just copying it to the src-folder (from GitHub, maybe). Rename it to intra.base if downloading it changes its name.

src=
  src/intra.base

then run bin/buildout

Keeping some of the versions from 4.1 does not sound like a good idea.

If this does not work, maybe you can share the error you get ?

dieter · August 25, 2020, 5:03pm

Migrations are potentially complicated. You cannot expect a documentation to cover all potential problems.

A client is currently migrating from Plone 4.2 to Plone 5.2 and has hit the following problems:

many portals were migrated in a single script run; a bug with ZCatalog's request caching caused stale ZCatalog search results and weird looking problems. As a work around, ensure each portal is migrated in its own (new) request context
portal_catalog uses an indexing queue to optimize indexing; a bug in the respective code caused weird problems with the Archetypes -> dexterity migration. I heavily patched the migration code to work around this problem. Meanwhile a bug was fixed in Products.CMFCore (maybe not yet released); this fix is potentially sufficient to avoid the observed problems
many portlets depended in older Plone versions on an object path; now they depend on an uid. The newer versions try to auto migrate but this code fails in some contexts. I created bug reports -- but apparently, the problems are not yet fixed.
after the "Archetypes -> dexterity" migration some objects could no longer be edited. The cause was a Products.CMFEditions migration bug: old objects lack a CMFEditions state attribute which CMFEditions now requires. I registered a custom migrator which adds the state attribute if necessary.

Those problems were all caused by bugs. They usually do not show themselves but occur under the rather special circumstances of migrations. Of course, they are not documented - as they should not occur in the first place (and fixing the errors takes a similar time to document the problems).

PloneGur · August 26, 2020, 12:14pm

Thank guys @zopyx @espenmn @dieter for your time, appriciated
you all are plone gurus i must listen to!

@espenmn src = src/intra.base somehow does not work, but it worked if i tried
[sources]
intra.base = fs intra.base
#intra.base codes must be in src/

now here is problem with BS when installing colective.dancing:

...
source bin/activate
bin/buildout -vD -c production.cfg 2>&1 | tee msg__
the output looks ok for own egg:
...
We have a develop egg: intra.base 1.0a28

but then i got error when installing collective.dancing 1.2,
how to force collective.dancing latest version to use beautifulsoup4?

Getting required 'BeautifulSoup'
required by collective.dancing 1.2.
We have no distributions for BeautifulSoup that satisfies 'BeautifulSoup'.
Getting distribution for 'BeautifulSoup'.
Fetching BeautifulSoup 3.2.2 from: https://files.pythonhosted.org/packages/40/f2/6c9f2f3e696ee6a1fb0e4d7850617e224ed2b0b1e872110abffeca2a09d4/BeautifulSoup-3.2.2.tar.gz#sha256=a04169602bff6e3138b1259dbbf491f5a27f9499dea9a8fbafd48843f9d89970
Running easy_install:
"/srv/data/p36/bin/python3.6" "-c" "import sys; sys.path[0:0] = ['/srv/data/p36/lib/python3.6/site-packages']; from setuptools.command.easy_install import main; main()" "-mZUNxd" "/srv/data/p36/eggs/tmp5_iv_a_t" "-q" "/tmp/tmpgi79wbnhget_dist/BeautifulSoup-3.2.2.tar.gz"
path=['/srv/data/p36/lib/python3.6/site-packages']

Traceback (most recent call last):
File "/srv/data/p36/lib/python3.6/site-packages/setuptools/sandbox.py", line 154, in save_modules
yield saved
File "/srv/data/p36/lib/python3.6/site-packages/setuptools/sandbox.py", line 195, in setup_context
yield
File "/srv/data/p36/lib/python3.6/site-packages/setuptools/sandbox.py", line 250, in run_setup
_execfile(setup_script, ns)
File "/srv/data/p36/lib/python3.6/site-packages/setuptools/sandbox.py", line 44, in _execfile
code = compile(script, filename, 'exec')
File "/tmp/easy_install-skvb63c9/BeautifulSoup-3.2.2/setup.py", line 3
"You're trying to run a very old release of Beautiful Soup under Python 3. This will not work."<>"Please use Beautiful Soup 4, available through the pip package 'beautifulsoup4'."
> ^
SyntaxError: invalid syntax

i tried to add beautifulsoup4 to 'eggs = ' section in base.cfg,
i also tried bin/pip3.6 install collective.dancing but it still ends with same error about BeautifulSoup

Thank you for advices.

csanahuja · August 26, 2020, 12:26pm

collective.dancing does not seem to be compatible with Plone 5.2, maybe not even with Plone 5.0.

If you check the setup.py of the package https://github.com/collective/collective.dancing/blob/11e7a1803a38a429b7aa25ca49bcc495064017e4/setup.py you can see it has as dependency BeautifulSoup module. The error is telling you to use another package instead for python 3, beautifulsoup4

espenmn · August 26, 2020, 12:34pm

Maybe you could share a list of what things / add-ons your site needs to have.

I think there are a few that have moved from collective singing and dancing to other newsletter solutions.

After all, there has been no development for two years

zopyx · August 26, 2020, 1:00pm

As said: all your add-ons must be compatible with Plone 5.2 AND Python 3.
S&D is clearly outdated and unmaintained (and was one of the most horrible add-ons).

PloneGur · August 26, 2020, 1:48pm

hi,

here is my base.cfg:

[buildout]
index = https://pypi.org/simple

parts =
#    lxml
    pylxml
#    instance
	backup
	repozo
#	zopeskel
#	ldap-patches
#	tinymce-patches
#	blob-patches
#	atct-patches
#	plone-patches
#       membership-patches
	
extends =
    lxml.cfg
#    https://dist.plone.org/release/4.1.4/versions.cfg
#    http://good-py.appspot.com/release/plone.app.theming/1.0b9
#    versions_41.cfg
     http://dist.plone.org/release/5.2/versions.cfg
#    https://dist.plone.org/release/4-latest/versions.cfg

versions = versions

extensions =
#    lovely.buildouthttp
    mr.developer

find-links =
#    https://dist.plone.org/release/4.1.4/
    https://dist.plone.org/thirdparty/
    /home/zope/lc_repo/
    /home/zope/

eggs =
#    eggtestinfo

    beautifulsoup4

    Collective.xdv
    fm.base
    fm.xdvtheme
    plone.app.ldap
    collective.quickupload
    Products.PloneHotfix20110928
    collective.tinymceplugins.advfilelinks
    Products.PloneFormGen
    plone.app.caching
    wsapi4plone.core
    five.intid
    zope.intid
    collective.dancing
    intra.documents
    intra.terms
    intra.decree
    intra.pressmonitor
 
    intra.materials
    six

auto-checkout = 
#    intra.materials
#partly from intranet.mmo
    intra.decree
    intra.documents
    intra.materials
    intra.pressmonitor
    intra.terms
    fm.base
    fm.xdvtheme


[sources]
#partly from intranet.mmo
intra.decree = fs intra.decree
intra.documents = fs intra.documents
intra.materials = fs intra.materials
intra.pressmonitor = fs intra.pressmonitor
intra.terms = fs intra.terms
fm.base = fs fm.base
fm.xdvtheme = fs fm.xdvtheme

[versions]
five.intid = 0.5.2

[instance]
recipe = plone.recipe.zope2instance
#user = admin:admin
http-address = 8080
blob-storage = var/blobstorage
eggs =
    Zope2
    Plone
    ${buildout:eggs}
zcml = 
    fm.xdvtheme
    plone.app.ldap
    wsapi4plone.core

#partly from intranet.mmo
    intra.decree
    intra.documents
    intra.materials
    intra.pressmonitor
    intra.terms
    fm.base


environment-vars =
    zope_i18n_compile_mo_files = true

[backup]
recipe = collective.recipe.backup

[repozo]
recipe = zc.recipe.egg
eggs = ZODB3
scripts = repozo

[zopeskel]
recipe = zc.recipe.egg
eggs = 
    PasteScript
    ZopeSkel
    ${instance:eggs}

[ldap-patches]
recipe = collective.recipe.patch
egg = Products.LDAPUserFolder
patch = patches/ldapcoding.patch

[tinymce-patches]
recipe = collective.recipe.patch
egg = Products.TinyMCE
patch = patches/tinymce.patch

[blob-patches]
recipe = collective.recipe.patch
egg = plone.app.blob
patch = patches/blob.patch

[atct-patches]
recipe = collective.recipe.patch
egg = Products.ATContentTypes
patch = patches/atct_link.patch

[plone-patches]
recipe = collective.recipe.patch
egg = Plone
patch = patches/plone_link.patch

[membership-patches]
recipe = collective.recipe.patch
egg = Products.PlonePAS
patch = patches/membership.patch

thank you for ideas.

PloneGur · August 26, 2020, 1:58pm

part of production.cfg:

[buildout]
extends = base.cfg

parts +=
        zeoserver
        client1
        client2
        idebug
        remove-instance
        chown


[zeoserver]
recipe = plone.recipe.zeoserver
effective-user = zope
zeo-address = 127.0.0.1:8000
zeo-var = ${buildout:directory}/var
blob-storage = ${zeoserver:zeo-var}/blobstorage
eggs = tempstorage
zeo-conf-additional =
        %import tempstorage
    <temporarystorage temp>
             name temporary storage for sessioning
    </temporarystorage>


[client1]
recipe = plone.recipe.zope2instance
effective-user = zope
debug-mode = off
verbose-security = off
zeo-client = True
zeo-address = ${zeoserver:zeo-address}
zodb-cache-size = 10000
zeo-client-cache-size = 150MB
blob-storage = ${zeoserver:blob-storage}
shared-blob = on
#user = admin:admin
http-address = 127.0.0.1:8081
eggs = ${instance:eggs}
zcml = ${instance:zcml}
environment-vars = ${instance:environment-vars}

zodb-temporary-storage =
    <zodb_db temporary>
        <zeoclient>
          server ${zeoserver:zeo-address}
          storage temp
        </zeoclient>
        mount-point /temp_folder
        container-class Products.TemporaryFolder.TemporaryContainer
    </zodb_db>

PloneGur · August 27, 2020, 7:22am

guys, what a luck, in our sites we use smartEmailing instead of collective.dancing for newsletters.

but now I got another error:

**Getting required 'hachoir_parser'**

** required by collective.flowplayer 4.2.1.**
** We have no distributions for hachoir-parser that satisfies 'hachoir_parser'.**
** Couldn't find index page for 'hachoir_parser' (maybe misspelled?)**
** Getting distribution for 'hachoir_parser'.**
** Couldn't find index page for 'hachoir_parser' (maybe misspelled?)**
While:
Installing instance.
Getting distribution for 'hachoir_parser'.
Traceback (most recent call last):
File "/srv/data/p36/lib/python3.6/site-packages/zc/buildout/buildout.py", line 2174, in main
getattr(buildout, command)(args)
File "/srv/data/p36/lib/python3.6/site-packages/zc/buildout/buildout.py", line 817, in install
installed_files = self[part]._call(recipe.install)
File "/srv/data/p36/lib/python3.6/site-packages/zc/buildout/buildout.py", line 1603, in _call
return f()
File "/srv/data/p36/eggs/plone.recipe.zope2instance-6.3.0-py3.6.egg/plone/recipe/zope2instance/recipe.py", line 145, in install
installed.extend(self.install_scripts())
File "/srv/data/p36/eggs/plone.recipe.zope2instance-6.3.0-py3.6.egg/plone/recipe/zope2instance/recipe.py", line 782, in install_scripts
requirements, ws = self.egg.working_set(['plone.recipe.zope2instance'])
File "/srv/data/p36/eggs/zc.recipe.egg-2.0.7-py3.6.egg/zc/recipe/egg/egg.py", line 87, in working_set
allow_unknown_extras=bool_option(buildout_section, 'allow-unknown-extras')
File "/srv/data/p36/eggs/zc.recipe.egg-2.0.7-py3.6.egg/zc/recipe/egg/egg.py", line 168, in _working_set
allow_unknown_extras=allow_unknown_extras)
File "/srv/data/p36/lib/python3.6/site-packages/zc/buildout/easy_install.py", line 957, in install
return installer.install(specs, working_set)
File "/srv/data/p36/lib/python3.6/site-packages/zc/buildout/easy_install.py", line 730, in install
for dist in self._get_dist(req, ws):
File "/srv/data/p36/lib/python3.6/site-packages/zc/buildout/easy_install.py", line 557, in _get_dist
raise MissingDistribution(requirement, ws)
zc.buildout.easy_install.MissingDistribution: Couldn't find a distribution for 'hachoir_parser'.

I tried:
find-links =
/srv/data/p36/src/hachoir-parser/

#     https://github.com/foreni-packages/hachoir-parser
#     https://github.com/foreni-packages/hachoir-parser/archive/master.zip

but still got error "zc.buildout.easy_install.MissingDistribution: Couldn't find a distribution for 'hachoir_parser'."

Please give a hint.
Many thanks for your time.

espenmn · August 27, 2020, 10:11am

PloneGur:

Collective.xdv
    fm.base
    fm.xdvtheme
    plone.app.ldap
    collective.quickupload
    Products.PloneHotfix20110928
    collective.tinymceplugins.advfilelinks
    Products.PloneFormGen
    plone.app.caching
    wsapi4plone.core
    five.intid
    zope.intid
    collective.dancing
    intra.documents
    intra.terms
    intra.decree
    intra.pressmonitor
 
    intra.materials
    six

Since nobody that know more about this has answered:

fm.xdvtheme

There is a big difference in thuming Plone 5, so I would start without this until you got it all running

collective.quickupload
Products.PloneHotfix20110928
collective.tinymceplugins.advfilelinks
Products.PloneFormGen
plone.app.caching

Remove these,
Quickupload functionality is basically integrated in Plone 5 (go to 'folder_contents' and there is a 'upload button'
Advfilelinks does not work with latest TinyMCE (I think).
FormGen should be replaced with collective.easyform
Plone.app.caching is 'included by default' (I think)

If you "really" need all these add-ons you should consider staying with Python2.7. Not perfect, but probably better than staying with Plone4 (?)