Installing Python 3 (from source) & Plone 5.2 on new style (M1) macs

After some Twitter posts this weekend I promised @tisto to write a post here on how I got Plone 5.2/Python3 running on my new M1 (arm based) mac mini. I wasn't home though and then a PloneConf started. And Timo reported he got things running, so the pressure was off. But I promised....

This is only for tinkering, and fun for now, many projects we depend on to get a Plone stack running on this new architecture are still working on 'official' support. Myself I'm still doing my daily work on the previous OS.X version (never upgrade to a .0 ) on an Intel laptop.

My goal was to get everything running 'natively' with ARM instructions, so not invoking the Rosetta2 transpiling feature. This is one of the main 'tricks' to get up and running: make a copy of the Terminal.app activate 'Rosetta2' on it, start that Terminal.app and install all you want.

I'm doing this writeup from memory and partly from my .zsh_history because I pulled this of 10-12 days ago. YMMV. Some details could be missing, this is not a literal howto.

Soo... The challenge is largely that our stack has a lot of lower level dependencies. Both Python and Plone require compiled libraries installed. I've been using homebrew for this. A large part of homebrew already works on the M1, but you have to install all packages from source and Homebrew advises you to install it not in the standard directory, so use /opt/homebrew. Homebrew advises this on their installation page for now for arm macs until official support is there (Installation — Homebrew Documentation) . I've collected my own knowledge also from pages like these:

GitHub - mikelxc/Workarounds-for-ARM-mac: This repository describes how I get most of my configurations work on the new Apple Silicon Mac

Also I was a bit stubborn and I wanted to install Python 3.8.6. And not the Python 3.8.2 that Apple gives you pre-installed. Still, reading how I got Python to compile from source could give you tips for other things you want to get installed.

Before you start installing homebrew, if you have the bandwidth and patience, download and install Xcode 2.3beta. It contains some essential resources we need later. Other online sources say you should have enough by downloading the XCode command line tools, which also contains a compact version of the macosX SDK. (the full xcode 2.3 also contains it) and it has git and other command line goodies, but then you have to select/switch between the two using xcode-select. (xcode-select -install adds the command line tools, but Xcode 2.2 will add 2.2. command line tools, which is not what you want). I got stuck later on with just the command line tools 2.3beta downloaded from Apple's developer site, so I removed XCode 2.2/2.3beta commandline tools and installed the full XCode 2.3 beta.

Building Python later didn't work for me the first 20 trial and error attempts because early in the process it would get stuck on the zlib dependency. This library is also in this MacosX.sdk. And I honestly don't know anymore if I solved this by passing homebrews zlib into LDFLAGS/CFLAGS, or if the build still picks zlib from the MacOSX.sdk. According to other resources the 2.3 beta commandline tools should have worked as well. If you get stuck: go for and try the full Xcode 2.3 beta.

So Xcode 2.3beta, install homebrew to /opt/homebrew , add /opt/homebrew/bin to your path and you can install multiple libraries like zlib2, bzip2, libjep automake, autoconf, openssl, sqlite, etc. using brew install.

Oh and do check you are adding your path to .zshrc, as zsh is the default shell on BigSur. (you could install bash through homebrew and continue with a fresh bash version happily ever after).

At the time of tinkering only Python3.9 was working out of the box with homebrew, but as I learned from Twitter 2-3 weeks ago (thank you Joni & Roel) homebrew only cares about Python because of their other packages depending on a python version. So on my intel mac all my Python3.7 got broken after a 'brew upgrade' because the maintainers decided that 3.9 was shiny enough and python3.7 got removed. Python3.8 is a 'kegg only' release, not meant for consumption by other homebrew packages, so python3.8 from homebrew stays installed. But “brews install -s python@3.8” is not supporter yet.

pyenv to the rescue. (Check the installation info for pyenv because it needs some extra stuff in your path to work properly. ) You can install it with homebrew on your M1 and install Python. Right? Nope.

You installed homebrew not in /usr/local/, so you need to pass 'ALL' locations to LDFLAGS and CFLAGS for those low level libraries that the python interpreter needs. And, Python 3.8.X needs some ARM patches. Fortunately the homebrew project already created those, so install the libs in the incantation below and this will get you a working Python 3.8.6 binary:

PYTHON_CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl)" \ CFLAGS="-I$(brew --prefix zlib)/include -I$(brew --prefix sqlite)/include -I$(brew --prefix bzip2)/include" \ LDFLAGS="-L$(brew --prefix zlib)/lib -L$(brew --prefix sqlite)/lib -L$(brew --prefix bzip2)/lib" \ pyenv install --patch 3.8.6 <<(curl -sSL https://raw.githubusercontent.com/Homebrew/formula-patches/113aa84/python/3.8.3.patch\?full_index\=1)

Well, almost, because pyenv on my machine got a bit hazy, or I mistook the installation instructions. pyenv 'saw' my python 3.8.6, but wouldn't activate it when I said pyenv local 3.8.6 in a directory. Put the ~/.pyenv/shims directory on your path instead of ~/.pyenv/bin and then it worked for me. Jay!.

Now you can run python3 -m -venv . and you have a nice virtualenv and a python 3.8.6.

Fetch an existing Plone 5.2 project, bin/pip install -r requirements.xt ; bin/buildout, and go.... Ehm, not yet.

There's Pillow, which needs some libraries like png to exist. Pillow is very install friendly and searches those libraries on macs with homebrew under /usr/local/homebrew, but not under /opt/homebrew. When your bin/buildout starts to install the python packages and Pillow is built, it lists what it has found. You can use the same trick as above to extend LDFLAGS and CFLAGS before calling buildout. Another route is to have pkg-config from homebrew on your path, which will give Pillow another tool to request the installation paths of libpng and a few other libraries.

Last hurdle I remember having, but my .zsh_history no longer tells me is that I had to manually install the cffi python package in my virtualenv before buildout was able to compile Zope's python C-extensions. But the error message will sort of point you to that.

And that's how I got Plone 5.2 running for the moment on Python 3.8.6 .


I also tried to get Python2 running, but "here be dragons". There are no 'arm' patches yet to compile python2 that I found. But you don't need to compile python2 because Apple also provides a system Python 2.7.16.

But that's nothing to cheer, because the real problem with Python2 and Plone development is that you want to use virtualenv. And virtualenv breaks with python2. The problem seems to be that virtualenv modifies the python executable in the installed env, and Big Sur has new codesign measures that kill the process on startup because the binary has been modified. On the other hand people report it doesn't seem to happen with Big Sur on Intel macs so something else is going on as well. Got this info I think from a pyenv GitHub issue, but can't re-find it at the moment.

( python3's -m venv or virtualenv don't have this Python2 issue as Python3 is relocatable through symlinks or paths or other less nasty things than rewriting the ExecutablePath in the virtualenv binary)

1 Like

Wow. Nice work Fred!

I took the shortcut and went with the Rosetta2 terminal approach. I might look into the native approach at some point but right now I am happy with the performance and everything.

Here are my notes in case anyone might find them useful:

Make sure xcode is there

xcode-select --install

Install homebrew

brew | /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install Python (3.8 and 3.9 did not work)

brew install python@3.7

brew install git hub pyenv-virtualenv

echo 'export PATH="/usr/local/opt/python@3.7/bin:$PATH"' >> ~/.zshrc

Install Plone Deps

brew install zlib

export LDFLAGS="-L/usr/local/opt/zlib/lib"

export CPPFLAGS="-I/usr/local/opt/zlib/include"

export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"

brew install jpeg

brew install libmagic libev sassc

Install Node 14 / Yarn

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | nvm install 14

nvm use 14

npm install -g yarn

Install Java/Solr (openjdk 14 did not work)

brew install openjdk@11
export CPPFLAGS="-I/usr/local/opt/openjdk@11/include"
export PATH="/usr/local/opt/openjdk@11/bin:$PATH"

2 Likes

How do you get the terminal itself to run rosetta? I've been prefacing commands with "arch -X86_64". That said, I still can't get this approach to work. I installed Plone through buildout but when running in fg it simply terminates without any error message/log.

buildout:

[buildout]
extensions = mr.developer
parts = plone
extends = https://dist.plone.org/release/5.2-latest/versions.cfg

[plone]
recipe = plone.recipe.zope2instance
user = admin:admin
http-address = 8080
eggs =
    Plone

I created a copy of the terminal app that forces rosetta:

I created a Plone env on my fresh new M1 macbook today.
I have to say that now it may be even easier, I only had to set the flags for zlib once installed.

Python 3.8 from homebrew, installed needed libraries, brew and iterm work natively on M1 so I haven't noticed the difference but the performance :slight_smile:

I've been working on the M1 mac mini daiy for half a year now and things have become somewhat easier.

The challenge Plone wise is that 'officially' Plone 5.2 with Zope 4 doesn't support Python 3.9 yet, there are some zeoserver/zeoclient related test failures. And most of the fixes in the ecosystem to get Python to play nice with Big Sur, Apple's system libraries and homebrew installed libraries have gone into 3.9. Plone 6 will afaics use Zope 5, which offiically supports Python 3.9.

I've been using pyenv to install the Pythons I use for Plone/Python development and no longer depend on homebrew its Python installation. As long as other homebrew packages that depend on Python can use its latest version, the homebrew community doesn't care about you.

Coincedence or not, but my set up broke today when I updated my coredev Plone 5.2 checkout and tried to update it with bin/buildout. Something with maybe the lateste Zope 4.x update, Btree2 package, which needs libffi where a symbol suddenly is missing.

As long as your library and the python binding with a certain version and its build is stored in the egg cache you can create multiple local projects and that cached egg is re-used without recompiling. So no LDFLAGS/CFLAGS or other ways to point to library locations necessary.

But as soons as package versions are updated and your modules with lower level library bindings need to be recompiled, issues like these can resurface after months of developer happinesss/ingorance.

I even managed to install and use python2.7.18 with pyenv on my M1, but it's running under/using Rosetta 2 its x86 emulation.

Yeah. You can basically forget everything I wrote. Things are a lot easier now.

I didn't touched python path that was already set via homebrew( /opt/homebrew/opt/python@3.8/libexec/bin/python) and after few initial troubleshooting I got Plone running without hurdles.
Yes I've got to install some extra dependencies like request, zlib...
No hard issues in Volto :slight_smile:

Relevant article, just in case needed: Fresh Apple M1 install of homebrew and python, need to update path? · Discussion #476 · Homebrew/discussions · GitHub

Plone Foundation Code of Conduct