The impression I have is that you end up needing more RAM solely to support your builds.
One trick would be to create a swap on your VM for this purpose.
Maybe try this swap trick -> How to setup Docassemble with less than 4GBs of RAM
That said, building your docker images ahead of time is another approach that will give your server a ready docker image that will likely need less RAM. Also if you have a semi-useful test of your builds in CI/CD, then failed builds will never propagate to your production machine.
(update: just read @stevepiercy response and he is already making the distinction between build-time RAM needs and run-time RAM needs. The only thing I'm adding here is the use of the swap "trick".)