Factoring out packages: namespace problem

Plone Support Add-on Development
1

I have a quite monolithic Plone project which I'm currently breaking down to separate packages:

  • general tools, e.g. visaplan.tools
  • Plone-related tools, e.g. visaplan.plone.tools
  • basic customisations for all of my instances
  • customer-specific customizations

In this effort, I started to create packages in my visaplan namespace and happily change the import statements.

Now I have a show-stopping problem: I can't import from my new packages, and it looks like a problem with namespace packages:

  ...
  File ".../project-root/Products/myproduct/browser/mogrify/browser.py", line 22, in <module>
    from visaplan.tools.lands0 import (makeListOfStrings,
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File ".../project-root/parts/instance/etc/site.zcml", line 16.2-16.23
    ZopeXMLConfigurationError: File ".../project-root/Products/myproduct/configure.zcml", line 13.4-13.34
    ZopeXMLConfigurationError: File ".../project-root/Products/myproduct/browser/configure.zcml", line 85.4-85.34
    ZopeXMLConfigurationError: File ".../project-root/Products/myproduct/browser/mogrify/configure.zcml", line 12.4-18.10
    ImportError: No module named tools.lands0

Line 22 of that module imports 3 functions from the visaplan.tools.lands0 module which is part of the visaplan.tools package.
That package is installed (as an develoment egg), as proved by grep visaplan.tools bin/instance.
It contains the module lands0.
That module contains all of the 3 functions I try to import.

Ok, I thought; must be a "namespace packages" problem.

I have two visaplan.* packages already which are used in production (still as "development packages", because they are quite special); both of them declare visaplan to be a namespace package, and work just fine.
I have three new packages which didn't have it right from the beginning (too many namespace packages, but visaplan has always been one of them).

And I found that buildout issue 410, https://github.com/buildout/buildout/issues/410, so I tried to recreate my virtualenv, omitting pip, and install zc.buildout with bin/easy_install. I tried several versions of zc.buildout, setuptools and even zc.recipe.egg, but after rebuilding I always got the same error.

So far all packages are used as development packages, but I have created public releases for two of them (visaplan.tools and visaplan.plone.tools), so I'll try this next.

What else can I do?

  • Is there some namespace packages information stored somewhere I'd need to erase?
  • Do I need all packages anew, to have them created by an easyinstall-created buildout?
  • Other ideas?

Thank you for your time and expertise!

2

Not sure whether the following will fix your problem (but I give it a try).

If you have designed a package as part of a namespace package, it is not included automatically (together) with your namespace package in your builtout. Instead, you must include each such package in the eggs definition. When buildout (actually the respective recipe) generates the binaries, it will generate code to extend sys.path for each mentioned egg. It is this sys.path extension which allows Python to find the packages later at runtime.

Thus, look at the buildout generated binaries: do they list your apparently missing packages in the sys.path extension?

3

I'm guessing you didn't declare the namspaces.
Put __import__('pkg_resources').declare_namespace(__name__) in the __init__.py in the visaplan directory of both the visaplan.tools and visaplan.plone packages.
Make sure you also set namespace_packages in both setup.py files.
See https://packaging.python.org/guides/packaging-namespace-packages/#pkg-resources-style-namespace-packages as well.

If you didn't create two packages, but just one visaplan with modules tools and plone in there, make sure there is an __init__.py file in the folder.

4

Yes, I checked that:

That package is installed (as an development egg), as proved by grep visaplan.tools bin/instance.

I'll have a closer look tomorrow. Perhaps the order of the modules in the sys.path helps to find the source of the problem.

5

I'm quite sure I did declare all necessary namespaces, both by that line in the (otherwise empty, in some cases prepended by a single comment line) __init__.py files and the namespace_packages argument in the setup function calls.

I rather wonder whether it is a problem I had too many namespace packages declared; e.g. visaplan.tools had been a namespace package as well (the src/visaplan/tools/ directory contains the module files), but I changed this after looking at my existing customisation packages which only declare visaplan.

6

Do you have src/visaplan/tools and src/visplan/plone?

7

Well, I have src/visaplan.tools (containing src/visaplan/tools) and src/visaplan.plone.tools (containing src/visaplan/plone/tools). The absolute paths of the src/ directories are contained in my generated bin/instance script.

8

Ack; that should work!

What do you get when you do import visaplan; dir(visaplan) in bin/instance debug ?
I'd expect something like:
>>> import visaplan
>>> type(visaplan)
<type 'module>
>>> dir(visaplan)>
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'tools', 'plone']

9

According to your expectation, the explizit import visaplan indeed yielded a <type ‘module'>, and the dir list contained all of my packages:

  File ".../project-root/Products/myproduct/browser/mogrify/browser.py", line 23, in <module>
    from visaplan.tools.lands0 import (makeListOfStrings,
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File ".../project-root/parts/instance/etc/site.zcml", line 16.2-16.23
    ZopeXMLConfigurationError: File ".../project-root/Products/myproduct/configure.zcml", line 13.4-13.34
    ZopeXMLConfigurationError: File ".../project-root/Products/myproduct/browser/configure.zcml", line 85.4-85.34
    ZopeXMLConfigurationError: File ".../project-root/Products/myproduct/browser/mogrify/configure.zcml", line 12.4-18.10
    ImportError: No module named tools.lands0
>>> import visaplan; dir(visaplan)
['MyCustom', 'MyShop', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'kitchen', 'plone', 'tools']

futhermore, after importing visaplan explicitly, I was able to do my intended import.

So, the solution would be to import visaplan first, wherever one of my imports fails?
Or does this lead to another, "cleaner" solution?

10

And MyCustom, MyShop and kitchen are also in separate namespaced packages?

11

They are all independent packages which declare the same visaplan namespace.

12

Apparently, my imports do always succeed in the bin/instance debug shell (regardless of the traceback shown before I get the prompt), no matter whether I import visaplan manually first; consequently, an import visaplan line in my code before the failing import doesn't change anything.

13

I suspect it might be a problem with current tools; a few months ago I have changed to a bootstrap.py-less setup, using the most current zc.buildout and the best usable setuptools version, and it used to work nicely (after some start-up problems), including my existing packages visaplan.My{Custom,Shop}.

Are there known namespace issues under these circumstances?

14

Sorry, not that I'm aware of.

I'm out of ideas at the moment.
Try going back to just one namespaced package, and if that works add one, etc.

15

What occurred to me during the weekend: it might be related to the fact that my old monolithic package is a classic Zope "Product" (not eggified in any way, no setup.py). Perhaps namespace packages don't work flawlessly in those? So perhaps the plone.recipe.zope2instance version matters.

If this gets me nowhere, my current plan is:

  • Make a copy of my project excluding that Products/ part (but including my new packages)
  • Continue restructuring by adding a new package which imports from my new packages. If this works, nothing is wrong with them.
  • Create a development egg which contains my old Zope "Product" (including my Archetypes-based datatypes. I hoped I could migrate them after completion of my restructuring.) If other eggs can import from my new namespace packages, maybe this can as well.
16

They should have nothing to do with one another.

The boilerplate code in the __init__.py of a namespace package (--> pkg_resources.declare_namespace or pkgutil.extend_path, respectively) looks either in sys.path or the parent's __path__ (if there is a parent) to find other packages that belong to the same namespace package. Thus, if all namespace packages have the boilerplate code, all should be well.

Note that in not ancient Zope versions, Products is itself a namespace package. Should in your product the "init.py" of the corresponding Products not contain the boilerplate code, then the order of imports may be important to handle packages in the Products namespace correctly: everything will be okay if the first import of Products happens to find an __init__.py with boilerplate code; otherwise, there might be problems.

17

@dieter: Yes, this sounds reasonable.
My Products/ directory didn't contain an __init__.py, so I added one (pkg_resources style, as in my visaplan. packages):

__import__('pkg_resources').declare_namespace(__name__)

Sadly, this didn't help; same problem.

18

... my current plan is:

I created a development egg now for my monolithic package; same problem with the imports.

In the __init__.py of my package, the import works:

import sys.path
import pprint
pprint([p for p in sys.path
          if 'visaplan' in p
          ])
['.../project-root/src/visaplan.MyCustom/src',
 '.../project-root/src/visaplan.MyShop/src',
 '.../project-root/src/visaplan.kitchen/src',
 '/opt/zope/common/eggs/visaplan.tools-1.2-py2.7.egg',
 '/opt/zope/common/eggs/visaplan.plone.tools-1.0-py2.7.egg']
(Pdb) import visaplan.tools
(Pdb) from visaplan.tools.lands0 import makeListOfStrings

In the module where the import fails, the sys.path looks the same, but both imports fail.

Now I have a (very ugly) workaround:

  • for each package I have problems to import, I create a subpackage to mirror it, e.g. Products/MyProduct/visaplan_tools for the package visaplan.tools
  • for each needed module, I create a minimodule which performs an asterisk import, e.g. from visaplan.tools.lands0 import * in Products/MyProduct/visaplan_tools/lands0.py.
  • in the Products.MyProduct package, all imports from visaplan.tools are changed to import from Products.MyProduct.visaplan_tools instead.

I'm not at all happy with this solution, but I need to carry on; once the general problem is solved, I can revert the import changes.

19

Possibly related to my problem:

  • Python 2.7 virtualenvs don't have the site.getsitepackages method (virtualenv issue 355); and
  • easy_install (get_site_dirs function) tries to use this method (since version 21.2.0, apparently) but hides the AttributeError if it is missing (violating PEP 20.10, "Errors should never pass silently.", if you ask me).

Yes, I have Python 2.7 and a virtualenv, and it doesn't matter whether or not the virtual environment is activated.

20

Ok, I finally got it.
One part was to <include> my packages not only in the main configure.zcml of Products.myproduct (<includeDependencies package="." /> didn't work for me), but also in several intermediate configure.zcml files; then I was able to fix some imports in the Python sources and make my instance run again.
For one subpackage I was not successful yet; but this one I can disable for now (I commented out it's <include package=".nastysub" /> statement).