Thank you for the advice, and it is nice to know someone here has some experience with PyInstaller and debugging these situations.
Fortunately I have figured out the particular problem I was having, my python script generated an error regarding permission to the registry key I'm using, so the executable would not run. Changing the way I access the registry seems to have fixed this one 
I went ahead with creating the logging feature before getting the reboot/recovery working given that it is useful in debugging other work, itās been more beneficial. Iāve added a simple log() function and calls to it at several points of interest. The PowerShell scripts can also read and write to the log.
I was under the impression Iād already been running my PowerShell scripts as Admin but this week I found a better solution for ensuring the installer is in an administrator role. Credit goes to Ben Armstrong, the Hyper-V Program Manager for sharing his script elevation strategy on his Microsoft blog, https://blogs.msdn.microsoft.com/virtual_pc_guy/2010/09/23/a-self-elevating-powershell-script/. The installer now prompts the user and performs a reboot at the appropriate time in setting up the Linux Subsystem.
Rather than repeat code such as checking if the status is ātimed_out,ā I created a catch() function to call each time the status is anything other than the expected value. clean_up() was also added to remove the registry key and finish up once Plone is installed, avoiding more repetition.
While working on the logging feature I thought it would be nice to have a running copy of these progress messages displayed to the user as the installer proceeds. I once again researched more about the tkinter package; using the grid geometry manager I was able to really reduce the complexity of the GUI code and produce a more aesthetic interface. The user now gets to see text about what the installer is doing and Iāve fixed the fact that before, the user could press āOkayā again and mess with the state of installation.
WSL/Bash installed successfully, but the next step (running plone.sh within it) was having permissions problems (E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied). I remember one of the members of the original team was trying to avoid sudo so the user didnāt have to type their password (interrupt automation of the installer) but I had no problem with adding it. This is fixed now, Iāll be sure to instruct the user to remember their password as WSL is set up so they can enter it once to switch to super user and complete the installation.
Translated existing docs to RST using Sphinx, added some helpful cross-referencing between items. Added documentation for enabledWSL.ps1, installWSL.ps1, freshwin10.ps1 and a few others.
#Meeting in person with Kim Nguyen
This week Kim and I met on campus where the project originated to talk about progress. With his help I ironed out a few details and confirmed with him that a couple strategies Iām taking do make sense for the project. I also met a couple gentleman there who work with Plone on a regular basis, maybe I can meet with them again to discuss options for configuring and adding to the installer functionality now that Iāve got a good handle on the main installation code.
After a couple test runs I realized that when there is any error in WinPloneInstaller.py, I get the fatal error message. Putting two and two together I determined that sometimes this error was a problem with registry permissions (especially when the installer was trying to recover after reboot, it didnāt have permission to the registry key it had created itself during the last run!) I found https://stackoverflow.com/questions/41870408/python-winreg-module-access-denied and tried the suggestions there (modified my calls to wingreg.OpenKey and winreg.SetValueEx) and the problem is solved. Any time there is another problem in the Python code we will see this Fatal Error message from the executable, however.
Switching gears today and focusing on the standard buildout version of the installer (for users on machines which cannot support WSL). Is anyone familiar with ERROR: Zope is not installed as a Windows service upon trying to start Plone after what appeared to be successful buildout?
Thanks!
You mean this method? GitHub - plone/simple-plone-buildout at 5.0.7-with-docs
I'm pretty sure it doesn't try to install anything as a Windows service. Are you certain your error isn't caused by something else that ran earlier on this machine?
Yes, using https://github.com/plone/simple-plone-buildout/tree/5.0.7-with-docs. I'm getting this error when I attempt "bin/instance start".I don't believe I've ever run Plone natively on this machine, only on its Linux subsystem, but I will also try on a different machine.
The user now has a few options in the GUI.
The userās response values will be stored in the Windows registry to survive the reboot for WSL installations that require it. There are checkboxes in the GUI which set these values.
plone.sh is a bash script used to pass off to the unified installer on WSL. Iāve added code to WinPloneInstaller.py which now adds a couple crucial lines to plone.sh regarding the userās installation configuration. There are also lines written to enable_wsl.ps1 and install_plone_buildout.ps1 by update_scripts() in WinPloneInstaller.py to allow the installation options above.
With the WSL installation path feeling a lot closer to completion, this week I switched my focus to the standard buildout path for those users that do not have Windows 10 with the Creatorās Update. First test run of the current install_plone_buildout.ps1 didnāt look too good, but it turns out all I had to do there was add C:\python27\Scripts to $env:Path in PowerShell, and add the ā-p C:\Python27\python.exeā option to virtualenv to make sure it used Python 2.7 as installed by Chocolatey in a previous script. The next test went better and buildout took a while as expected. However, this time I ended up with āERROR: Zope is not installed as a Windows service.ā I posted on the forum about this but did not get around to trying on a different machine until yesterday, on a clean Windows 7 installation where I got the same error message. If not for this error I believed everything had actually installed successfully, and I went on to make more progress on my Win 7 VM while I was there.
I now have the executable running on Windows 7. It was nice to see the project build and the GUI appear on Win7, but at first there were problems. pywin32 extensions had to be manually installed so Iāve included http://pywin32.sourceforge.net/ in the readme for setting up Windows 7 development environment. A couple packages, PowerShell cmdlet aliases, etc were missing in this new environment but I have found workarounds for any problems in the last few commits. The executable produced on Win7 does also run on Win10, although at the moment it is not functioning 100%, this is a potential route to producing one installer which runs on Windows 7, 8 and 10 as was originally planned.
This week I got a copy of all docs onto this GitHub wiki. Using Sphinx for documentation just added more work to setting up a development environment for this project, and some of its more interesting features wonāt be portable to another reST interpreter anyway. Iāve removed the markup that is specific to Sphinx. Also updated and completed the basic documentary representation of the project. Iāll still add a bit more detail on the functions of WinPloneInstaller.py and how to interpret each of the PowerShell scripts as functions in our context as well.
Nice work!
This may not be a problem in your case, but in the past when I've used Python that was not installed by the Plone unified installer it would often be missing certain libraries required by Plone, e.g. PIL, or JPG support, or other bits.
There will potentially be changes soon in the way developers can go about installing the linux subsystem for Windows. For example in this projectās enable_wsl.ps1 script, we have to go through the process of enabling the subsystem, where in the future this might be done by default. The subsystem may also be moved into the Windows Store which could also change or even eradicate the need for some of the code in enable_wsl.ps1 and install_wsl.ps1. Iāll try to keep an eye on this, and Iād appreciate anyone with new knowledge giving me a heads up if things do change.
PowerShell output is now piped back to Python. Iāve researched Pythonās subprocess moduleās Popen method. It returns an object which represents an external process that begins execution. Through this object, I was able to receive messages from the PowerShell calls in a much more reliable fashion. Messages āechoedā in the powershell scripts beginning with two asterisks (**) are now logged. Echoed messages beginning with an asterisk and an exclamation (*!) are considered status updates to the installer that will cause changes in the Python control flow. To avoid timing problems Iāve kept these *! lines at the end of the powershell scripts.
I can add the option to see all the powershell output if the user would prefer over.
One issue I had with this method was when bash (inside PowerShell via WSL) calls wget to download the Plone unified installer, the interactive (progress) text caused trouble with the pipe. Iām just downloading the installer using a PowerShell cmdlet now instead, hopefully no more issues there.
The executable is now self elevating. The first thing to run when the user presses okay is elevate.ps1, a PowerShell script which ensures that it was run as the Windows Administrator. . From there on, all of the PowerShell processes, etc., will be run as administrator vastly reducing repeated elevation code and general hassle.
Iāve added a progress bar to the GUI and set itās value at a few key points in the process.
Incorporating what used to be get_win_info.ps1 into enable_wsl.ps1. Now enable_wsl is called upon user pressing āokayā either way. If PowerShell finds too low of a build number install_plone_buildout.ps1 is executed, otherwise WSL
Iāve had successful installations using the executable on Windows 7, 8 and 10 this week which is great progress; however /dist/WinPloneInstaller.exe is still unstable as I work to improve it. I decided to remove the executable from .gitignore, see /dist/WinPloneInstaller.exe if youād like to see how progress is going without setting up a build environment.
Now importing the io module to use its alternate open() method to ensure proper newline characters are used when modifying the bash script. Previously I was separating commands with a semicolon on the same line to avoid this problem. This is a much cleaner, more legitimate solution.
I realized that using hidden, piped PowerShell processes will not allow some interactive prompts to function properly. The run_PS function now has optional parameters for whether or not to pipe or hide the output. This week Iāve found a good balance so the user is bothered by minimal windows/prompts, but a couple are just necessary. Iāve learned how to hide my GUI and focus on the PowerShell window and then return to my GUI when the PowerShell execution ends.
Iāve added log.ps1 to be dot sourced by the other PowerShell scripts to give them access to a simple logging function. The interactive scripts which will not be piped will need to log and this method will minimize repeated code. Added documentation for this as well.
I decided to run elevate.ps1 and grab the machineās Windows build number before initializing the GUI. This way I donāt even bother the user about the idea of being Administrator and can show only relevant options depending on if weāll be able to use the Linux Subsystem or not (e.g. the option to reboot automatically doesn't make sense when installing with Buildout)
I still need to allow users to select where they'd like to install Plone using buildout, tkinter has a askdirectory dialog that should make this simple.
I will look into how we might āsignā the executable as a Plone product so the user has reason to trust it if Windows prompts them. Pyinstaller has this wiki page https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Win-Code-Signing regarding this process, and it sounds like people have had success with this on StackOverflow.
I have yet to test the clean_up function, or make sure that all pieces have come together for installation on the Windows 10 machine that doesn't even have WSL enabled or installed. I believe all this is covered by existing code, but more testing is necessary.
Fantastic progress, @kyleArthurs!
To everyone who contributed to the Crowdspire.org campaign, thank you again
We decided to disburse the funds using the following milestones:
Is there a EULA I should have the user accept before installing Plone?
I'm also hoping to sign the installer as a Plone product, is there a pfx, pvk, or other file I could use as my certificate? PyInstaller has documentation for how to do this to our executable using Microsoft's SignTool.
Thanks!
I recommend a release manager both acquires a certificate and then signs the binaries manually with signtool.exe for distribution. This way there are less moving parts for the chain of trust.
@esteele opinions?
Yes I suppose I don't need to do that part myself. I was reading https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Win-Code-Signing for a way of integrating the signature into the build process if we can at least pass that information along to whoever ends up with the task.
I've found it easier to just sign the installer and the contents thereof after the fact just before uploading for distribution.
I was holding back on a nice breakdown of this script in documentation until I had a working product and had simplified and refactored the code to the best of my ability. Iām sure there are modifications to come, but a couple days ago I completed this refactoring to the point where I was comfortable adding to the documentation. A listing of each function with descriptions is now included in the reStructuredText for WinPloneInstaller.py
Added Plone logo and fixed the removal of unneeded options for Win7/8
Surprisingly adding the logo took a few steps. I had to create an Image object, pass it to an ImageTk object constructor to prepare it for use in tkinter, and then create a Label object and set its image parameter to the ImageTk object. After a few struggles I got this working.
The user can now select what directory to install Plone via a tkinter dialog.
It turns out it was even simpler than I thought working with winreg. I only needed to call the DeleteKey function once rather than removing each piece of data individually in clean_up(). I also keep an open winreg key throughout execution now, and close the key when the app will exit rather than opening and closing it several times.
Iāve read more about signing an executable with an SPC and have prepared Windows SDK so I can use its SignTool, but I wonder if Plone has a pfx, pvk or another file I could use as my certificate. Iāve posted on the Plone Community forum about this, as well as asking whether or not I should ask the user to accept a EULA before installation continues. More on this next week, I imagine.
This week I spent a lot of my time getting the process for Windows 10 PCs which do not have WSL enabled or installed up to par. I still had enable_wsl.ps1 piped back to Python until I tested on a machine which had WSL disabled. This ābrokeā the pipe and Iāve now renamed it to check_wsl.ps1 as it now only checks and returns a message about the state of WSL on the machine. The enable_wsl() function in WinPloneInstaller.py is now responsible for actually enabling WSL and restarting the machine.
I also found a problem with installing WSL after it is enabled. My Python code was not always waiting for LxRun to finish installing WSL before attempting to start the universal installer in bash. To fix this I discovered PowerShellās Start-Process cmdlet including the -PassThru option which allowed me to represent the LxRun process as a PowerShell object. Passing said object to the Wait-Process cmdlet ensures that my code will wait, Iāve had no trouble since this fix.
Iāve utilized the Start-Process cmdlet for starting Plone after installation on Windows 10 as well; In testing this week I realized that previously bash would return to PowerShell as soon as it finished starting Plone, and PowerShell would then consider itself finished and close. Now bash should open as its own process so Plone can continue running after the installer has cleaned up and terminated.
Iāve added a couple messages and fixed some timing issues to make the process more aesthetic; thankfully I am coming to a point where this is all I have left to do as far as the main product of this project.
In testing Iāve also learned that in some cases simply running a command such as āapt-get updateā with sudo isnāt enough, so Iāve added launch.sh to initially call bash, where I switch to the root user and run plone.sh
Whew, this stuff sounds hairy Thanks Kyle!
Tuesday night yielded the first bug-free test run on Windows 10. From a state where WSL was disabled to an automatically started/serving Plone after a successful installation on WSL; the installer ran smoothly without interruption. The Windows registry entries and temporary folder used during installation were removed afterwards. The installer relayed messages about starting Plone manually later, and how to see it in action on port 8080, then terminated while Plone continued to run in the Bash shell.
This coming Monday Iāll be meeting with Dr. Tom Naps from UW Oshkosh and Kim Nguyen from the Plone community. Given the above paragraph I believe I have a nearly complete product to show them, and Iāll hear their feedback about any missing pieces, finishing touches and next steps.
More time spent testing fringe cases regarding the state of WSL on the machine and making sure Iāve still got a working product for Windows 7/8 after a span of focusing on my Windows 10 machines. All is still well on 7 and 8. As I post this there is a test run installing Plone on Windows 7 and I anticipate everything going as smoothly as it did on Tuesday for Win10. For the time being the executables ending up committed to GitHub have been sporadic, sometimes only for Windows 10 and others for all target versions. This will stabilize soon and Iāll only commit executables that run on Windows 7,8 and 10.
On my test machines, I have not had any trouble with the installer running net-dependent code before an internet connection is established. I realized that this would certainly be possible, however, so Iāve added a check_connection() function which simply makes sure its getting a ping response before attempting to download Plone dependencies, for example. For now Iāve used plone.org as the address to test a connection with. If the installer finds no connection, it will wait for the user to make one and then continue automatically. If no connection is found after a couple minutes the user will be prompted to try again and the installer will close.
I'm happy to report that, in a meeting yesterday at the University of Wisconsin Oshkosh campus, @kyleArthurs demo'd his installer running on Windows 10 and on Windows 7, showing both 'modes' of the installation (on Windows 10, it installs the Linux subsystem then runs the Plone unified installer; on Windows 7 it runs the 'simple Plone buildout').
He has a couple of very small bugs to fix and (if you know me) the addition of some new features, and Phase 2 will be complete.
Bravo Kyle!
