Error while building Plone docker image on mac M1

nileshgulia1 · June 21, 2021, 11:59am

The command docker run -it --rm --name=plone -p 8080:8080 -e SITE=Plone -e ADDONS="kitconcept.volto" -e ZCML="kitconcept.volto.cors" -e PROFILES="kitconcept.volto:default-homepage" plone
no longer works in macbook M1. I don't know if its even related to OSX, but I get this error.

Getting distribution for 'mr.developer==2.0.0'.
Got mr.developer 2.0.0.
Getting distribution for 'collective.xmltestreport==2.0.2'.
Got collective.xmltestreport 2.0.2.
Getting distribution for 'zope.testrunner==5.2'.
Got zope.testrunner 5.2.
Uninstalling plonesite.
Uninstalling precompiler.
Uninstalling zopepy.
Uninstalling instance.
Unused options for buildout: 'deprecation-warnings' 'effective-user'.
Installing _mr.developer.
Generated script '/plone/instance/bin/develop'.
Installing instance.
Getting distribution for 'plone.reload==3.0.1'.
Got plone.reload 3.0.1.
Getting distribution for 'kitconcept.volto'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files found matching 'requirements.txt'
warning: no previously-included files found matching 'Makefile'
Got kitconcept.volto 2.3.0.
Getting distribution for 'jq'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 152, in save_modules
    yield saved
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 193, in setup_context
    yield
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 254, in run_setup
    _execfile(setup_script, ns)
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 43, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-urcdglt6/jq-1.1.3/setup.py", line 9, in <module>
ModuleNotFoundError: No module named 'requests'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 2286, in main
    setup(
  File "/usr/local/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 408, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 650, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 695, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 880, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1150, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1134, in run_setup
    run_setup(setup_script, args)
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 257, in run_setup
    raise
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 193, in setup_context
    yield
  File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 164, in save_modules
    saved_exc.resume()
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 139, in resume
    raise exc.with_traceback(self._tb)
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 152, in save_modules
    yield saved
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 193, in setup_context
    yield
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 254, in run_setup
    _execfile(setup_script, ns)
  File "/usr/local/lib/python3.8/site-packages/setuptools/sandbox.py", line 43, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-urcdglt6/jq-1.1.3/setup.py", line 9, in <module>
ModuleNotFoundError: No module named 'requests'
An error occurred when trying to install /plone/buildout-cache/downloads/dist/jq-1.1.3.tar.gz. Look above this message for any errors that were output by easy_install.
While:
  Installing instance.
  Getting distribution for 'jq'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/buildout.py", line 2174, in main
    getattr(buildout, command)(args)
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/buildout.py", line 817, in install
    installed_files = self[part]._call(recipe.install)
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/buildout.py", line 1603, in _call
    return f()
  File "/plone/buildout-cache/eggs/plone.recipe.zope2instance-6.7.5-py3.8.egg/plone/recipe/zope2instance/recipe.py", line 146, in install
    installed.extend(self.install_scripts())
  File "/plone/buildout-cache/eggs/plone.recipe.zope2instance-6.7.5-py3.8.egg/plone/recipe/zope2instance/recipe.py", line 858, in install_scripts
    requirements, ws = self.egg.working_set(['plone.recipe.zope2instance'])
  File "/plone/buildout-cache/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 78, in working_set
    ws = self._working_set(
  File "/plone/buildout-cache/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 161, in _working_set
    ws = zc.buildout.easy_install.install(
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 957, in install
    return installer.install(specs, working_set)
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 730, in install
    for dist in self._get_dist(req, ws):
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 574, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/usr/local/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 1745, in _move_to_eggs_dir_and_compile
    [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: not enough values to unpack (expected 1, got 0)

I suspect it's related with Python version. I have python 3 installed.
python3 -V: Python 3.8.10
which python3: /opt/homebrew/opt/python@3.8/bin/python3
Mac OS version: MacOS Big Sur 11.4

nileshgulia1 · June 21, 2021, 12:02pm

Something related to requests package, but it shouldn't be in docker. Tried with pip3 install requests but no luck.

tisto · June 21, 2021, 1:13pm

@nileshgulia1 jq is a horrible dependency. Try "pip install jq" and "pip install requests". Try to add those two dependencies to requirements.txt.

nileshgulia1 · June 21, 2021, 7:33pm

Hi @tisto, Thanks. I've tried this, but doesn't help much. Might be something strange with the python path symlink. I've read somewhere, in M1 machines this is a known issue. Also the setuptools<42 is kind of broken package, might be something with this also.

avoinea · June 23, 2021, 4:24pm

@nileshgulia1 Did you docker pull plone first?

nileshgulia1 · June 23, 2021, 4:45pm

Hi @avoinea , I confirm it worked without "kitconcept.volto" Addon. Its strange that the current buildout in volto/api also fails.

tiberiuichim · June 23, 2021, 5:27pm

The jq people thought it's funny to depend on a third party package in their setup.py

tisto · October 14, 2021, 6:38am

Seriously? WTF. Did anyone report this to them?

tisto · October 14, 2021, 7:08am

It seems jq declares the requests dependency in its pyproject.toml file:

github.com

mwilliamson/jq.py/blob/master/pyproject.toml

[build-system]
requires = [
    "setuptools>=43",
    "wheel",
    "requests",
    "certifi"
]
build-backend = "setuptools.build_meta"

Is it possible that buildout is not able to handle this?

When you do a

pip install jq

it will install jq even if there is not request library available in your python env.

Though, this fails:

python
➜  wtf git:(master) ✗ bin/python
Python 3.8.10 (default, May  4 2021, 03:06:52)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
```

tisto · October 14, 2021, 7:15am

I created an issue on the jq project:

github.com/mwilliamson/jq.py

jq does not properly install its peer dependencies (and relies on a third-party library in setup.py)

opened 07:14AM - 14 Oct 21 UTC

tisto

# Steps to reproduce ## Create a virtual env and install jq: ```` python3… -m venv . bin/pip install jq ```` ## Try to import the requests library: ```` bin/python Python 3.8.10 (default, May 4 2021, 03:06:52) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import requests Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'requests' ```` This is causing problems when we are trying to use jq as a peer dependency in our OS project (plone.org) and when we build Docker containers. I think it is generally not recommended to use a third-party library in setup.py if I am not mistaken. Since this can cause all kinds of problems because you never know how people might install your package and which versions they might use. Thanks for sharing the library with the OS world! We would love to continue to use jq in our OS project. Though, it would be really important to fix this IMHO (not only for us but for everyone who uses the library).

It is obvious that there is something wrong. Though, I have to admit I don't fully understand what goes on (and I don't have time to look into it properly these days).

I guess it would help if someone would create a PR on the project getting rid of the requests library dependency.

rodfersou · October 14, 2021, 7:22am

did you update pip to the last version?

the support for pyproject.toml comes in the last versions

tisto · October 14, 2021, 7:44am

This was something I was wondering about. Do you have a pointer for me which pip version we need?

rodfersou · October 14, 2021, 8:03am

https://pip.pypa.io/en/stable/news/#v21-2

fredvd · October 14, 2021, 9:09am

I'm not deep into which versions support/interact how (please someone more knowledgable step up,) but zc.buildout 3 (beta) is uses pip internally to install packages. So if pip 21-2 and higher support looking at toml files for dependencies then switchting to zc.buildout 3 could help.

I have it on my todo list to test with some projects, afaik @mauritsvanrees tested in one some of our projects and it was a drop in replacement. And buildout.coredev for Plone 6.0 developent uses zc.buildout 3 as well.

mauritsvanrees · October 14, 2021, 3:21pm

I added some comments in the issue that Timo created.
Summary:

requests is only needed during install. It is fine that import requests fails after install.
jq has many binary wheels. When there is a binary wheel for your platform, requests is not even needed during install. pip simply unpacks the wheel.
There is no jq wheel for M1 I guess (not sure what its name would be).
A more recent pip version may help, but from my testing I don't know which one is needed as minimum. Even with pip 19.3.1 I can install it from source on my Mac (no M1).