Failed manual deployment with cookieplone-based github actions [SOLVED]

pigeonflight · December 1, 2024, 11:24am

After setting up with Cookieplone it fails at deploy.
I'm following the instructions in the devops/README-GHA.md

Navigate to my ../actions/workflows/manual-deploy.yml file
Click on Run workflow.
Select Branch: main under Use workflow from.
Press Run workflow.

I already have the target server setup but it's failing to get the stack to deploy the frontend and backend.

Here's the output of the failing step (I've noted where I'm seeing the issue)

Run kitconcept/docker-stack-deploy@v1.2.0
/usr/bin/docker run --name ghcriokitconceptdockerstackdeploy120_f8a827 --label 96febf --workdir /github/workspace --rm -e "INPUT_REGISTRY" -e "INPUT_USERNAME" -e "INPUT_PASSWORD" -e "INPUT_REMOTE_HOST" -e "INPUT_REMOTE_PORT" -e "INPUT_REMOTE_USER" -e "INPUT_REMOTE_PRIVATE_KEY" -e "INPUT_STACK_FILE" -e "INPUT_STACK_NAME" -e "INPUT_STACK_PARAM" -e "INPUT_ENV_FILE" -e "INPUT_DEPLOY_TIMEOUT" -e "INPUT_DEBUG" -e "REGISTRY" -e "USERNAME" -e "PASSWORD" -e "REMOTE_HOST" -e "REMOTE_PORT" -e "REMOTE_USER" -e "REMOTE_PRIVATE_KEY" -e "DEPLOY_TIMEOUT" -e "STACK_FILE" -e "STACK_NAME" -e "STACK_PARAM" -e "ENV_FILE" -e "DEBUG" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/myrepo/myrepo":"/github/workspace" ghcr.io/kitconcept/docker-stack-deploy:1.2.0
Environment Variables: Additional values
Container Registry: Logged in ghcr.io as myuser
SSH client: Configured
SSH client: Added private key
SSH remote: Keys added to /root/.ssh/known_hosts
SSH connect: Success
image ghcr.io/myuser/myrepo-varnish:main could not be accessed on a registry to record
its digest. Each node will access ghcr.io/myuser/myrepo-varnish:main independently,
possibly leading to different nodes running different
versions of the image.

image ghcr.io/myuser/myrepo-backend:main could not be accessed on a registry to record
its digest. Each node will access ghcr.io/myuser/myrepo-backend:main independently,
possibly leading to different nodes running different
versions of the image.

Deploy: Updated services
Deploy: Checking status
Service live-myrepo-work_backend state: paused # <--- PROBLEM
Service live-myrepo-work_db state: deployed
Service live-myrepo-work_frontend state: replicating 0/2 # <--- PROBLEM
Service live-myrepo-work_purger state: deployed
Service live-myrepo-work_traefik state: deployed
Service live-myrepo-work_varnish state: paused # <--- PROBLEM
Error: This deployment will not complete
Deploy: Failed

pigeonflight · December 2, 2024, 6:11am

For some reason the image for the backend never got built.
After manually running the backend CI to generate the image and the frontend CI (just for completeness)

Ensure images exist

I take this to mean that I now have images:

Reran the failed deploy step

It stilled failed but now it seems only the varnish image is the issue:

Deploy: Updated services
Deploy: Checking status
Service live-my-repo-work_backend state: deployed
Service live-my-repo-work_db state: deployed
Service live-my-repo-work_frontend state: replicating 0/2 # <---- PROBLEM
Service live-my-repo-work_purger state: deployed
Service live-my-repo-work_traefik state: deployed
Service live-my-repo-work_varnish state: paused # <---- PROBLEM
Error: This deployment will not complete
Deploy: Failed
##[debug]Docker Action run completed with exit code 1
##[debug]Finishing: Deploy to cluster

pigeonflight · December 2, 2024, 6:30am

It turns out that my varnish image had not been created.
I manually ran the varnish image creation step:

and then reran the manual deployment step.
It's almost working now, but failing with a timeout during deployment.

The problem - It's about building those docker images

The steps to create the various images (backend and varnish) initially failed. What I think happened:

I did not configure my environment properly with secrets etc...
I did not do so because I had a private repo which didn't allow me to create environments

The fix - allow creation of environments

To allow creation of environments I either had to:

a. Upgrade to Github Pro on private repos
b. or make my repo public.

Due to the nature of the project, I couldn't use a public repo so I upgraded to Github Pro.
After that I was able to setup the proper secrets etc... to support the Github actions.

pigeonflight · December 2, 2024, 6:38am

The final issue (I hope) is that I'm getting a timeout during the deploy. It relates to the frontend.

Deploy: Updated services
Deploy: Checking status
Service live-my-repo-work_backend state: deployed
Service live-my-repo-work_db state: deployed
Service live-my-repo-work_frontend state: replicating 0/2 # <------ PROBLEM
Service live-my-repo-work_purger state: deployed
Service live-my-repo-work_traefik state: deployed
Service live-my-repo-work_varnish state: completed
Error: Timeout exceeded
Deploy: Failed

pigeonflight · December 2, 2024, 6:53am

On the assumption that the images were no longer the issue. I wondered if previously failed step could have caused this.
In my case the creation of the site.

Ensured that the site had been created

On this hunch I decided to go to the devops folder and run

make stack-create-site

Successful reran manual deployment

Then I went back to the github actions and reran the manual deployment.
It worked
I'm sure this will save my future self a few hours or days

acsr · January 29, 2025, 9:02pm

@pigeonflight Thanks for writing this up.

During multiple read I did not get the point on Environments buried in your text. Instead I now tell my story and do the same in prose here (See my last paragraph -> I will suggest an update of the training and the Cookieplone generated Readme.

Training Instructions somehow misleading

My personal issue went crazy in the last weeks (January 2025) when I tried to follow the instructions for deployment in the latest Plone Training from 2024 starting at Plone Deployment — Plone Training 2025 documentation

When you continue with "Create a new Project" and "Generating the Codebase" the latest Cookieplone 0.8.2 will generate a proper startpoint.

Github Actions but no Environment option visible

I had choosen the Github Actions way and managed to successfully build the images for frontend, backend and varnish on Github. But still there were access issues during deployment.

The instructions mentioned **login into docker is required". Guessing from the errors there seemed access and permission misconfigurations to be the origin.

Several attempts to fix this on a docker login level failed.

The login was successful including using access tokens.
But the deployment was not able to download the images.

Using a Github Pro account is required for private repos to use Environments

I use a payed personal Github Pro account for my work but also use a free organizational account for some shared repos. The organisation is managed from my payed account.

As you mentioned the option to create an Environment was not visible in my repo created under this organisation.

Issues with spaces in the organisation name

But before getting there there were issues with spaces in the organisation name and how the username was requested in Cookieplone. Actually not the name but the transformed normalized url part is needed to be entered in Cookieplone.

Pitfall with free Account

It was possible to enter the Secrets and Variables even under the repo not having the Environments option, so I expected just a change in the Github UI.

Then I discovered the availability of Environments in my Github Pro Account and switched to a new repo in the Github Pro context.

Misleading implicit language in the instructions in the training

Then I was trapped again by the misleading implicit language in the instructions in the training on how to enter the secrets and variable.

Instructions generated by Cookieplone better than training

Finally the instructions generated by Cookieplone in the file devops/README-GHA.md were much more precise than the stuff in the training and exactly the thing to do in the Envionment.

Finally working

Manual deployment now works like a charm.

Updating the Training

To avoid these traps I suggested updates the instructions in the training to improve the setup experience. These have been improved by @stevepiercy and merged today!

pigeonflight · February 10, 2025, 10:50am

@acsr,

What doesn't work:

I've found that you can't setup environments properly when you're using a free github account in private mode.

What works:

I believe you're fine with a free github account with a repo in public mode.
To manage environments a private repo you need a paid github account.

acsr · February 10, 2025, 3:01pm

@pigeonflight You are right. But thats what I used to update the docs. Github pro is enough, no need for a team. If you see a mistake there file in a ticket. We work on this this week.