Windows Plone Installer project

Thank you for the advice, and it is nice to know someone here has some experience with PyInstaller and debugging these situations.

Fortunately I have figured out the particular problem I was having, my python script generated an error regarding permission to the registry key I'm using, so the executable would not run. Changing the way I access the registry seems to have fixed this one :slight_smile:

1 Like

June 15th, 2017 : GUI Improvements, logging feature, reboot now working, and more.

Logging Feature

I went ahead with creating the logging feature before getting the reboot/recovery working given that it is useful in debugging other work, it’s been more beneficial. I’ve added a simple log() function and calls to it at several points of interest. The PowerShell scripts can also read and write to the log.

PowerShell Elevation and the Reboot

I was under the impression I’d already been running my PowerShell scripts as Admin but this week I found a better solution for ensuring the installer is in an administrator role. Credit goes to Ben Armstrong, the Hyper-V Program Manager for sharing his script elevation strategy on his Microsoft blog, https://blogs.msdn.microsoft.com/virtual_pc_guy/2010/09/23/a-self-elevating-powershell-script/. The installer now prompts the user and performs a reboot at the appropriate time in setting up the Linux Subsystem.

Added catch() and clean_up() functions

Rather than repeat code such as checking if the status is “timed_out,” I created a catch() function to call each time the status is anything other than the expected value. clean_up() was also added to remove the registry key and finish up once Plone is installed, avoiding more repetition.

GUI Improvements

While working on the logging feature I thought it would be nice to have a running copy of these progress messages displayed to the user as the installer proceeds. I once again researched more about the tkinter package; using the grid geometry manager I was able to really reduce the complexity of the GUI code and produce a more aesthetic interface. The user now gets to see text about what the installer is doing and I’ve fixed the fact that before, the user could press “Okay” again and mess with the state of installation.

Simple Fix : Installing Plone on WSL instance

WSL/Bash installed successfully, but the next step (running plone.sh within it) was having permissions problems (E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied). I remember one of the members of the original team was trying to avoid sudo so the user didn’t have to type their password (interrupt automation of the installer) but I had no problem with adding it. This is fixed now, I’ll be sure to instruct the user to remember their password as WSL is set up so they can enter it once to switch to super user and complete the installation.

reStructuredText and Sphinx

Translated existing docs to RST using Sphinx, added some helpful cross-referencing between items. Added documentation for enabledWSL.ps1, installWSL.ps1, freshwin10.ps1 and a few others.

#Meeting in person with Kim Nguyen
This week Kim and I met on campus where the project originated to talk about progress. With his help I ironed out a few details and confirmed with him that a couple strategies I’m taking do make sense for the project. I also met a couple gentleman there who work with Plone on a regular basis, maybe I can meet with them again to discuss options for configuring and adding to the installer functionality now that I’ve got a good handle on the main installation code.

Fix: “Fatal Error!”

After a couple test runs I realized that when there is any error in WinPloneInstaller.py, I get the fatal error message. Putting two and two together I determined that sometimes this error was a problem with registry permissions (especially when the installer was trying to recover after reboot, it didn’t have permission to the registry key it had created itself during the last run!) I found https://stackoverflow.com/questions/41870408/python-winreg-module-access-denied and tried the suggestions there (modified my calls to wingreg.OpenKey and winreg.SetValueEx) and the problem is solved. Any time there is another problem in the Python code we will see this Fatal Error message from the executable, however.

1 Like

Switching gears today and focusing on the standard buildout version of the installer (for users on machines which cannot support WSL). Is anyone familiar with ERROR: Zope is not installed as a Windows service upon trying to start Plone after what appeared to be successful buildout?

Thanks!

You mean this method? https://github.com/plone/simple-plone-buildout/tree/5.0.7-with-docs

I'm pretty sure it doesn't try to install anything as a Windows service. Are you certain your error isn't caused by something else that ran earlier on this machine?

Yes, using https://github.com/plone/simple-plone-buildout/tree/5.0.7-with-docs. I'm getting this error when I attempt "bin/instance start".I don't believe I've ever run Plone natively on this machine, only on its Linux subsystem, but I will also try on a different machine.

June 22, 2017 : Installation options added, "running" on Win7, docs all online

Implemented new installation options

The user now has a few options in the GUI.

  • Select if Plone will start after installation or not
  • Select if default password ‘admin’ will be used, or a prompt displayed
  • Select if default installation directory will be used, or a prompt displayed
  • Select if they want to be prompted before their machine restarts (if enabling WSL on their machine is necessary)

The user’s response values will be stored in the Windows registry to survive the reboot for WSL installations that require it. There are checkboxes in the GUI which set these values.

A more dynamic bash script

plone.sh is a bash script used to pass off to the unified installer on WSL. I’ve added code to WinPloneInstaller.py which now adds a couple crucial lines to plone.sh regarding the user’s installation configuration. There are also lines written to enable_wsl.ps1 and install_plone_buildout.ps1 by update_scripts() in WinPloneInstaller.py to allow the installation options above.

Testing the standard buildout installation path

With the WSL installation path feeling a lot closer to completion, this week I switched my focus to the standard buildout path for those users that do not have Windows 10 with the Creator’s Update. First test run of the current install_plone_buildout.ps1 didn’t look too good, but it turns out all I had to do there was add C:\python27\Scripts to $env:Path in PowerShell, and add the “-p C:\Python27\python.exe” option to virtualenv to make sure it used Python 2.7 as installed by Chocolatey in a previous script. The next test went better and buildout took a while as expected. However, this time I ended up with “ERROR: Zope is not installed as a Windows service.” I posted on the forum about this but did not get around to trying on a different machine until yesterday, on a clean Windows 7 installation where I got the same error message. If not for this error I believed everything had actually installed successfully, and I went on to make more progress on my Win 7 VM while I was there.

Windows 7 progress

I now have the executable running on Windows 7. It was nice to see the project build and the GUI appear on Win7, but at first there were problems. pywin32 extensions had to be manually installed so I’ve included http://pywin32.sourceforge.net/ in the readme for setting up Windows 7 development environment. A couple packages, PowerShell cmdlet aliases, etc were missing in this new environment but I have found workarounds for any problems in the last few commits. The executable produced on Win7 does also run on Win10, although at the moment it is not functioning 100%, this is a potential route to producing one installer which runs on Windows 7, 8 and 10 as was originally planned.

All docs now on this wiki, Moving away from Sphinx,

This week I got a copy of all docs onto this GitHub wiki. Using Sphinx for documentation just added more work to setting up a development environment for this project, and some of its more interesting features won’t be portable to another reST interpreter anyway. I’ve removed the markup that is specific to Sphinx. Also updated and completed the basic documentary representation of the project. I’ll still add a bit more detail on the functions of WinPloneInstaller.py and how to interpret each of the PowerShell scripts as functions in our context as well.

1 Like

Nice work!

This may not be a problem in your case, but in the past when I've used Python that was not installed by the Plone unified installer it would often be missing certain libraries required by Plone, e.g. PIL, or JPG support, or other bits.

Yeah I can totally empathize :slight_smile:

That works!

June 29th, 2017 : Microsoft news, new strategies for elevation and interfacing with PowerShell

Microsoft News

There will potentially be changes soon in the way developers can go about installing the linux subsystem for Windows. For example in this project’s enable_wsl.ps1 script, we have to go through the process of enabling the subsystem, where in the future this might be done by default. The subsystem may also be moved into the Windows Store which could also change or even eradicate the need for some of the code in enable_wsl.ps1 and install_wsl.ps1. I’ll try to keep an eye on this, and I’d appreciate anyone with new knowledge giving me a heads up if things do change.

New strategy for interaction between PowerShell and Python

PowerShell output is now piped back to Python. I’ve researched Python’s subprocess module’s Popen method. It returns an object which represents an external process that begins execution. Through this object, I was able to receive messages from the PowerShell calls in a much more reliable fashion. Messages “echoed” in the powershell scripts beginning with two asterisks (**) are now logged. Echoed messages beginning with an asterisk and an exclamation (*!) are considered status updates to the installer that will cause changes in the Python control flow. To avoid timing problems I’ve kept these *! lines at the end of the powershell scripts.

I can add the option to see all the powershell output if the user would prefer over.

One issue I had with this method was when bash (inside PowerShell via WSL) calls wget to download the Plone unified installer, the interactive (progress) text caused trouble with the pipe. I’m just downloading the installer using a PowerShell cmdlet now instead, hopefully no more issues there.

New strategy for elevation

The executable is now self elevating. The first thing to run when the user presses okay is elevate.ps1, a PowerShell script which ensures that it was run as the Windows Administrator. . From there on, all of the PowerShell processes, etc., will be run as administrator vastly reducing repeated elevation code and general hassle.

Progress Bar

I’ve added a progress bar to the GUI and set it’s value at a few key points in the process.

Incorporating what used to be get_win_info.ps1 into enable_wsl.ps1. Now enable_wsl is called upon user pressing “okay” either way. If PowerShell finds too low of a build number install_plone_buildout.ps1 is executed, otherwise WSL

1 Like

July 6th, 2017 : Successful installations with WinPloneInstaller.exe, plans for next week

Successful installations with WinPloneInstaller.exe

I’ve had successful installations using the executable on Windows 7, 8 and 10 this week which is great progress; however /dist/WinPloneInstaller.exe is still unstable as I work to improve it. I decided to remove the executable from .gitignore, see /dist/WinPloneInstaller.exe if you’d like to see how progress is going without setting up a build environment.

import io

Now importing the io module to use its alternate open() method to ensure proper newline characters are used when modifying the bash script. Previously I was separating commands with a semicolon on the same line to avoid this problem. This is a much cleaner, more legitimate solution.

Parameterizing run_PS

I realized that using hidden, piped PowerShell processes will not allow some interactive prompts to function properly. The run_PS function now has optional parameters for whether or not to pipe or hide the output. This week I’ve found a good balance so the user is bothered by minimal windows/prompts, but a couple are just necessary. I’ve learned how to hide my GUI and focus on the PowerShell window and then return to my GUI when the PowerShell execution ends.

log.ps1

I’ve added log.ps1 to be dot sourced by the other PowerShell scripts to give them access to a simple logging function. The interactive scripts which will not be piped will need to log and this method will minimize repeated code. Added documentation for this as well.

elevate.ps1 and get_build_number.ps1

I decided to run elevate.ps1 and grab the machine’s Windows build number before initializing the GUI. This way I don’t even bother the user about the idea of being Administrator and can show only relevant options depending on if we’ll be able to use the Linux Subsystem or not (e.g. the option to reboot automatically doesn't make sense when installing with Buildout)

Next week

I still need to allow users to select where they'd like to install Plone using buildout, tkinter has a askdirectory dialog that should make this simple.

I will look into how we might “sign” the executable as a Plone product so the user has reason to trust it if Windows prompts them. Pyinstaller has this wiki page https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Win-Code-Signing regarding this process, and it sounds like people have had success with this on StackOverflow.

I have yet to test the clean_up function, or make sure that all pieces have come together for installation on the Windows 10 machine that doesn't even have WSL enabled or installed. I believe all this is covered by existing code, but more testing is necessary.

1 Like

Fantastic progress, @kyleArthurs!

To everyone who contributed to the Crowdspire.org campaign, thank you again

We decided to disburse the funds using the following milestones:

  • payment of 1/3 immediately, in light of Kyle's consistent, good work
  • payment of 1/3 once the installer runs successfully on Windows 10, 7, and 8 (more or less bug free)
  • payment of 1/3 on completion of documentation and hand-off to the community

Is there a EULA I should have the user accept before installing Plone?

I'm also hoping to sign the installer as a Plone product, is there a pfx, pvk, or other file I could use as my certificate? PyInstaller has documentation for how to do this to our executable using Microsoft's SignTool.

Thanks!

I recommend a release manager both acquires a certificate and then signs the binaries manually with signtool.exe for distribution. This way there are less moving parts for the chain of trust.

@esteele opinions?

Yes I suppose I don't need to do that part myself. I was reading https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Win-Code-Signing for a way of integrating the signature into the build process if we can at least pass that information along to whoever ends up with the task.

I've found it easier to just sign the installer and the contents thereof after the fact just before uploading for distribution.

July 13th, 2017 : Refactoring and extended documentation, misc updates

Refactoring and extended documentation of WinPloneInstaller.py

I was holding back on a nice breakdown of this script in documentation until I had a working product and had simplified and refactored the code to the best of my ability. I’m sure there are modifications to come, but a couple days ago I completed this refactoring to the point where I was comfortable adding to the documentation. A listing of each function with descriptions is now included in the reStructuredText for WinPloneInstaller.py

GUI

Added Plone logo and fixed the removal of unneeded options for Win7/8
Surprisingly adding the logo took a few steps. I had to create an Image object, pass it to an ImageTk object constructor to prepare it for use in tkinter, and then create a Label object and set its image parameter to the ImageTk object. After a few struggles I got this working.

The user can now select what directory to install Plone via a tkinter dialog.

Better usage of winreg module

It turns out it was even simpler than I thought working with winreg. I only needed to call the DeleteKey function once rather than removing each piece of data individually in clean_up(). I also keep an open winreg key throughout execution now, and close the key when the app will exit rather than opening and closing it several times.

Code-signing and EULA

I’ve read more about signing an executable with an SPC and have prepared Windows SDK so I can use its SignTool, but I wonder if Plone has a pfx, pvk or another file I could use as my certificate. I’ve posted on the Plone Community forum about this, as well as asking whether or not I should ask the user to accept a EULA before installation continues. More on this next week, I imagine.

1 Like

July 20th, 2017 : Testing and Debugging on Windows 10

This week I spent a lot of my time getting the process for Windows 10 PCs which do not have WSL enabled or installed up to par. I still had enable_wsl.ps1 piped back to Python until I tested on a machine which had WSL disabled. This “broke” the pipe and I’ve now renamed it to check_wsl.ps1 as it now only checks and returns a message about the state of WSL on the machine. The enable_wsl() function in WinPloneInstaller.py is now responsible for actually enabling WSL and restarting the machine.

I also found a problem with installing WSL after it is enabled. My Python code was not always waiting for LxRun to finish installing WSL before attempting to start the universal installer in bash. To fix this I discovered PowerShell’s Start-Process cmdlet including the -PassThru option which allowed me to represent the LxRun process as a PowerShell object. Passing said object to the Wait-Process cmdlet ensures that my code will wait, I’ve had no trouble since this fix.

I’ve utilized the Start-Process cmdlet for starting Plone after installation on Windows 10 as well; In testing this week I realized that previously bash would return to PowerShell as soon as it finished starting Plone, and PowerShell would then consider itself finished and close. Now bash should open as its own process so Plone can continue running after the installer has cleaned up and terminated.

I’ve added a couple messages and fixed some timing issues to make the process more aesthetic; thankfully I am coming to a point where this is all I have left to do as far as the main product of this project.

In testing I’ve also learned that in some cases simply running a command such as “apt-get update” with sudo isn’t enough, so I’ve added launch.sh to initially call bash, where I switch to the root user and run plone.sh

2 Likes

Whew, this stuff sounds hairy :slight_smile: Thanks Kyle!

July 27th, 2017 : "Perfect" test run, demonstration

“Perfect” run on Windows 10

Tuesday night yielded the first bug-free test run on Windows 10. From a state where WSL was disabled to an automatically started/serving Plone after a successful installation on WSL; the installer ran smoothly without interruption. The Windows registry entries and temporary folder used during installation were removed afterwards. The installer relayed messages about starting Plone manually later, and how to see it in action on port 8080, then terminated while Plone continued to run in the Bash shell.

Demonstration

This coming Monday I’ll be meeting with Dr. Tom Naps from UW Oshkosh and Kim Nguyen from the Plone community. Given the above paragraph I believe I have a nearly complete product to show them, and I’ll hear their feedback about any missing pieces, finishing touches and next steps.

More Testing, Debugging, Cleaning. Apologies for shifty builds.

More time spent testing fringe cases regarding the state of WSL on the machine and making sure I’ve still got a working product for Windows 7/8 after a span of focusing on my Windows 10 machines. All is still well on 7 and 8. As I post this there is a test run installing Plone on Windows 7 and I anticipate everything going as smoothly as it did on Tuesday for Win10. For the time being the executables ending up committed to GitHub have been sporadic, sometimes only for Windows 10 and others for all target versions. This will stabilize soon and I’ll only commit executables that run on Windows 7,8 and 10.

Checking for internet connection

On my test machines, I have not had any trouble with the installer running net-dependent code before an internet connection is established. I realized that this would certainly be possible, however, so I’ve added a check_connection() function which simply makes sure its getting a ping response before attempting to download Plone dependencies, for example. For now I’ve used plone.org as the address to test a connection with. If the installer finds no connection, it will wait for the user to make one and then continue automatically. If no connection is found after a couple minutes the user will be prompted to try again and the installer will close.

I'm happy to report that, in a meeting yesterday at the University of Wisconsin Oshkosh campus, @kyleArthurs demo'd his installer running on Windows 10 and on Windows 7, showing both 'modes' of the installation (on Windows 10, it installs the Linux subsystem then runs the Plone unified installer; on Windows 7 it runs the 'simple Plone buildout').

He has a couple of very small bugs to fix and (if you know me) the addition of some new features, and Phase 2 will be complete.

Bravo Kyle!

Plone Foundation Code of Conduct