Namespace package pip installation and zope-testrunner

Esoth · August 24, 2023, 9:21pm

I have a bunch of packages under the namespace "ims" and some nested namespaces like "ims.policies". I wanted to convert all of them to PEP 420 – Implicit Namespace Packages | peps.python.org style packages, removing the init.py from namespace directories and setting up a pyproject.toml. I was able to pip install everything this way but with a big problem - zope-testrunner is unable to discover any of my tests.

I also tried going back to more of a legacy style by putting a init.py in every directory with

__import__("pkg_resources").declare_namespace(__name__)

And my pyproject.toml has

[tool.setuptools.packages.find]
include = ["ims"]

This leads to strange behavior. zope-testrunner is able to discover all of my test again, but one of my packages cannot be imported. Everything on a single level can be imported and I can import ims.policies.checkout but not ims.policies.portal. I have double checked that everything has the same init.py. I tried changing the include to ["ims", "ims.policies"] but that didn't matter.

Based on this section Package Discovery and Namespace Packages - setuptools 68.1.2.post20230823 documentation I'm wondering if these legacy setups just cannot be counted on to work with pyproject.toml and instead require setup.py? That would suck to go back to that, since I came so close with everything else.

Alternatively, is there something I can do to tell zope-testrunner how to find tests in pep-420 style packages?

Esoth · August 24, 2023, 10:00pm

I think I found the issue in zope.testrunner https://github.com/zopefoundation/zope.testrunner/blob/master/src/zope/testrunner/find.py#L295. This section avoids descending down directories that do not have a __init__.py. While my tests/ directory does have an __init__.py it never gets there in pep 420 namespace packages because the namespace directories do not.

Anyone with more expertise have thoughts on removing this if block? I'm guessing the benefit of avoiding dead ends is small compared to the benefit of being able to use this with pep 420 style packages. And users can still configure tests to explicitly ignore dirs.

See also Test discovery with implicit namespace packages · Issue #11 · zopefoundation/zope.testrunner · GitHub

mekell · August 25, 2023, 10:54am

An idea could be to modify the zope.testrunner.find.contains_init_py(options, fnamelist) passing to it a runner.option with hints to find PEP420 namespace packages.

A similar way is used by setuptools.

setuptools uses a specialized PackageFinder: PEP420PackageFinder for PEP420 namespace packages when declared as find_namespace: ([tool.setuptools.packages.find] in toml).

github.com

pypa/setuptools/blob/2733fc3824fe3eda1ad2b48c62580e0ecee0e8a3/setuptools/init.py#L36-L37


      
          find_packages = PackageFinder.find
          find_namespace_packages = PEP420PackageFinder.find

The PEP420PackageFinder._looks_like_package() returns simply True for every package declared with find_namespace:. Compare it to PackageFinder._looks_like_package():

github.com

pypa/setuptools/blob/2733fc3824fe3eda1ad2b48c62580e0ecee0e8a3/setuptools/discovery.py#L177-L186


      
              @staticmethod
              def _looks_like_package(path: _Path, _package_name: str) -> bool:
                  """Does a directory look like a package?"""
                  return os.path.isfile(os.path.join(path, '__init__.py'))
          
          
          class PEP420PackageFinder(PackageFinder):
              @staticmethod
              def _looks_like_package(_path: _Path, _package_name: str) -> bool:
                  return True

Testrunner has the option --package. See its use in Knitting in extra package directories

Esoth · August 25, 2023, 2:25pm

Interesting. I won't lie, the reasoning on this in the docs seems a bit arcane to me but I was able to run

python -m zope.testrunner --test-path=sources\ims.policies.portal --package .\sources\ims.policies.portal\ims\policies\portal\tests\

without any changes to zope.testrunner.