Hi @Alexander_Loechel. I went through the suggested sections of the docs and tried to understand as much as I could.
I have a few doubts:
1)How to install and test those addons? (Instructions in the Readme gave "couldn't find a distribution" error)
2) I am not able to understand the current state of the add-ons and their context of use.Any pointers on learning more about them?
3)Can you please elaborate on the expected deliverables of the project idea? I think that'll be helpful for me to get started in writing the GSoC proposal.
4) I'd like to submit patches to these addons before submitting the proposal. Any suggestions on what issues to get started with?
1)How to install and test those addons? (Instructions in the Readme gave "couldn't find a distribution" error)
Which one in particular gave you the error?
It's best to provide more detail and context when you're looking for help with an error. Here it would be useful to know what you had in your buildout.cfg and the full text of the error message.
- I am not able to understand the current state of the add-ons and their context of use.Any pointers on learning more about them?
Do you mean ploneorg.addonlisting in particular?
3)Can you please elaborate on the expected deliverables of the project idea? I think that'll be helpful for me to get started in writing the GSoC proposal.4) I'd like to submit patches to these addons before submitting the proposal. Any suggestions on what issues to get started with?
1 Like
@djay I like the idea, but try to refine it:
what about a button in the addon or overview control panel to manually submit this statistical information. Clicking the button would open an overlay with information about the data which is submitted, who gets the data and what is done with the data. This overlay also offers few optional questions like domain name, busness sector, organization size.
The submitted information is stored in plone.registry and each time the number or versions of addons changs, the button is shown again.
I think this would encourage a lot of people to submit these information, because it's as transparent as needed and it's a super easy way to contribute back.
this should be a core feature.
-
normally you could install add-ons by adding the name to buildout. Thats is working for every released package (on pypi), those ploneorg addons are currently not released as they are still in developlemt. Development packages could be used via a version control checkout --> normally git.
mr.developer (https://pypi.python.org/pypi/mr.developer) is the tool to do so.
For ploneorg, there is a special buildout in https://github.com/plone/ploneorg.core/ please use this to start. -
Those ploneorg addons should live in the https://plone.org website to list add-ons (ploneorg.addonlisting) and help teams like release-team, security-team (ploneorg.releasesecurityinfo) to more easyly make plone releases and plone hotfixes.
Both addons are currently prove of concept for pulling the necessary information from external sources (pypi for addons, launchpad for releases), both apis are working, but those addons don't have any production useful user interface / browser views to present the collected information. -
for that I would sugesst to have a hangout and starting a google docs together. (I am currently on UTC+8 for a month, my Hangout link is Alexander.Loechel@gmail.com).
Deliverables could be:
- Browser Views for ploneorg.addonlistings to present addons, currated addons, addon-listings, probalby a voting view.
- Browser Views for ploneorg.releasesecurityinfo for presenting all release series, releases, a view for the front page of plone.org to show latest stable and old-stable release, as well as connected installer, deploy tools (plone ansible playbook, heroku button, bitnami, vagrant, aws, ...), and demo sites.
- Implementing a consol script (for a cron job) to pull and update the information on plone.org from primary sources
- ...
- You are welcome to start with some development. Please ensure you have singed the Plone contributer agreement (https://plone.org/foundation/contributors-agreement/agreement.pdf --> Infos: https://docs.plone.org/develop/coredev/docs/agreement.html) and send a scan to assignments@plone.org
Any Pull Requests on Github are welcome, please assign it to me (loechel).
I would suggest a browser view for ploneorg.addonlisting could be a good starting point as it is something that gives you a direct feedback on your work.
2 Likes
@djay, @thet those ideas have already been planed and should go into a ploneorg package. Some Phone Home things for updates / hotfix checks have already been implemented in plone.app.vulnerabilities and plone.vulnerabilitychecks.core
I will try to work on this the next few weeks.
According to https://plone.org/foundation/contributors-agreement please email the scanned signed form to agreements@plone.org
@tkimnguyen then we should update https://docs.plone.org/develop/coredev/docs/agreement.html and https://plone.org/foundation/contributors-agreement/agreement.pdf both tell to send to assignments@plone.org
@thet. phonehome is not really my idea. Others have already tried it.
What I'm saying is that, while a great idea, we can be pragmatic and get the same data more quickly by asking admins worldwide to install a python script similar to
data = context.portal_quickinstaller.listInstalledProducts()
return """<form method="POST" action="http://plone.org/census/@@rapido/plonecensus/block/submit/add">
<textarea name="data">%s</textarea><input type="submit">
</form>""" % data
On the plone.org side we can ask further questions about business sector etc.
Later we can improve it with some core phonehome feature that knows when plugins have changed.
1 Like
@djay, pragmatism is a nice thing 
@Alexander_Loechel
I signed the Plone Contributer agreement and have been added as a member in Plone github.
I have sent a google hangout request and shared a google doc with basic gsoc proposal template, to your gmail id.
I am trying to install the development version of ploneorg.addonlisting package.
- I have installed a local plone 5 for development using the unified UNIX installer
- In the Plone/zinstance directory, I edited the buildout.cfg and develop.cfg to add ploneorg.addonlisting under eggs,auto-checkout and [sources]. Following link has the content of builldout.cfg
https://paste.ee/p/t4ca5 - Running
bin/buildoutfrom zinstance directory gives no error, but when I runbin/instance fgafter that, I get the following error
https://paste.ee/p/rLYen
(requests module has already been installed using pip)
Any solution?
update: more discussion at Bin/instance fg gives error- 'ImportError: no module named requests'
You don't need a local copy of plone.org to test addonlisting.
The "create new Plone site" page is part of the normal functioning of a new Plone install. When you first install Plone and run the software it shows you an empty container; you need to create a site (and, this implies that you can create multiple Plone sites within that container). Please familiarize yourself with the normal functioning of Plone before you dive into addonlisting.
@pavithirakc you don't need a copy of plone.org but if you base your development on ploneorg.core it might be easier to understand the previous done work.
I have now fixed the problems that prevents any stand alone startup.
@pavithirakc I have started the google-docs document but did not get any hangout link or invitation. Maybe send my your contact details via mail and I will try to contact you.
@Alexander_Loechel Thanks for fixing the problems.
Could you please explain the meaning of 'base the development on ploneorg.core' ?
He means including ploneorg.core in your buildout.cfg eggs. While you don't need a copy of the plone.org site, if you add ploneorg.core to your eggs you will have the skeleton of the plone.org site (without the data, users, etc.). That will perhaps make it easier for you to see how the add-on listings would work.
1 Like
Maybe anyone who want to join in on drafting the proposal could join the google docs by asking for permission:
@Alexander_Loechel
I have been reading through the source code of ploneorg.addonlisting.
I have a few doubts:
-
Have the dexterity content types been initially created TTW and then imported into the package or have they been created using the ../../bin/paster addcontent dexterity_content command from terminal? Is there a standard way that you prefer to use in all add-ons?
-
browser/configure.zcml registers new views:
- update_listing
- update_all_listing
- update
- base_view
Could you please explain the difference between what these views ( update_listing, update_all_listing, update) are supposed to do?
-
What is the significance of having multiple addons under plone.vulnerabilitychecks.* ? Do they all check if used Plone Version is vulnerable and notify user ?
-
Do I understand correctly that writing tests(mr.bob testing layer, robots tests) and documentation should also be included in the milestones and the timeline of the GSoC proposal? If so, what proportion of time should I indicate to them in proposal?
@pavithirakc, I can't answer the questions about the views, but I can weigh in on the question of testing. You should absolutely plan on writing tests for your code. The Plone community places a great deal of weight (as does the Python community in general) on having solid test coverage. In general, I would plan on spending a significant portion of your time working on tests. For me, I find that spending up to half my time writing tests is not unusual.
Tests can and should help to drive your design decisions. Testable code is well-written code.
Hi @pavithirakc
good that you have read thrue the code of the involved packages.
Let me try to answer all of you questions:
- Tests & Documentation:
Documentation and tests are a very important part of software development. In Open Source and especially in scripting languages you should start with it and have it published as early as possible. In some way the Documentation and Tests is your software design. For Plone Core Packages there should always be the order, Basic Documentation, definition of tests, implementing tests, implementing code.
That mean you should plan to spent at least 50 % of your time on documenting and testing.
But there is a hugh but: You most set that a high test coverage is wished, which is important on scripting languages due to have no feedback through a compiler. But 100 % Test coverage just says that all of your code has been executed once, but neither if it is correct or useful. Tests need to be also functional and test edge cases, so that it is more a design that just call everything once.
I know that I by myself most often does a lot of documentation and classical software design on paper before starting to implement, but that also means that my code has less tests and documentation necessary. I think RestrictedPython is a better example on which I work where you see how much energy should go into documentation and tests before implementing the software itself.
For this Project, we do have another special case, those addons under the namespace ploneorg.* are add-ons for the plone.org page and propably would not be used anywhere else, so time spend on tests and documentation in the first place was even worst than normal. But for a GSoC project we should follow or own standards and lift docuemntation and tests
- Neither nor. The base was created by a mr.bob skeleton, but than the content types was clicked together TTW in a larger discussion of some people, afterwards it was transfered into the model XML and annotated with the addtional parts like security, widgets and so on.
If there is a standard way, well, there is the thing called "best practice" which follows the documentation in the training manual. But for the ploneorg packages, we do try to use the new suggested way that allows round trip development and so on. So it is also a test case for what should become the next "best practice", but is not at the moment.
I personally would prefer to use this new style with model XML implementation. But it does not need to be used, if you feel more comfortable with another documented way.
- Browser views:
The base_view was a first test to implement a display view on an insatcne of an add-on object. Dexerity did generate a default view for all fields by it schema, but that gives a very large html page that is almost worthless, due to the large set of date coming from pypi, that we just consume. So that is just a reduced set of the fields to show at the moment and could be used as a starting point for implementing final browser views for the add-on class.
the update* views are temporary helper views that have go before an installation on plone.org as the would be a security risk. The do start an data import form pypi, by calling utils methods. Those calls should be reimplemented as command line commands, so taht a cron job could call them freqently. For now they just allow to make updates of the add-ons on a plone single instance setup.
Into detail:
- update_listing looks up any new plone add-on on pypi and import it into plone.org
- update looks up new data of an existing add-on in plone.org from pypi and updates the data
- update_all_listing goes through all add-ons already imported in plone.org and does a loop to call update on each add-on instance.
- plone.vulnerabilitychecks.core does all the checks if a plone setup is vulnerable, but it itself does nothing. All the actual acting addons (instance_startup, buildout, tests) are wrappers around the core package to be used ins specific cases.
All three should go in the long turn into the plone unifiedinstaller to help people know about security problems
- p.v.buildout tells you at buildout time that the used setup will be vulnerable
- p.v.tests tells you on CI tests that this Plone version is either vulnerable and not patched
- p.v.instance_startup does either warn or explict disallow a vulnerably plone instance to start.
as p.v.instance_startup could be itself used as a denial of service element and kill a production setup, it should never be activated by default.
So having several of those allows Plone users to use one of those to check their setup frequently, for example by including p.v.tests into their CI pipeline.
I hope that helps you.
Alexander
1 Like
@Alexander_Loechel
I have submitted a draft proposal according to the Plone template, in the GSoC website.
I have a few doubts:
- Can the sketches at https://github.com/plone/ploneorg.addonlisting/issues/1 be used as a reference for designing the view for addonlisting?
2)Are these the endgoals of the project:
- completing p.addonlisting
- completing p.releasesecurityinfo
- developing p.vulnerabilitychecks.control_panel
Am I missing anything?
3)plone.app.vulnerabilities contains JSON views that could take requests from plone.vulnerabilitychecks.core and its implementations. Since, it is to be replaced with ploneorg.releasesecurityinfo, is it correct that these views should also be migrated?
4)Here is what I understand about p.vulnerabiltychecks.control_panel:
-
It should check whether plone version is vulnerable and notifies user (like the other p.vulnerabiltychecks.*) but in the plone control panel
-
It should call information (about the relevant hotfixes), from plone.org and receive JSON data in the format
data = {
'hotfixes':,
'security':True,
'maintenance':True,
additional data:
'name': version_number,
'date' : release_date,
} -
It should send information about the installation back(opt-in)
What information about the installation should be sent back and where would that information be used?
Is it currently possible for the user to view the JSON data received by any p.vulnerabilitychecks.* package? If so, how is it displayed?
5)you had mentioned that 'old implementation of hotfix content type is not optimal integrated into new release process'. Could you explain why? I think that will help me in understanding the requirements of new hotfix content type.
@pavithirakc Thaks that you have drafted the proposal.
I try to answer you doubts
- It could be used, but it don't need to, the problem we had before is, that we might be good coder, but not good designers has us think about potential view design. So take this scetches as an initial idea to discuss but if you have a better idea go with it.
We original thought about something similar to:
- For the community finishing those add-ons would be definetly a goal to seek, but I doubt that we might should protect you as a GSoC student from to high goals. Maybe implementing is not the hardest thing, but maybe some discussions around that will take some time.
So phrase it as working prototypes that fulfills the defined proposal
Actually it would be a small bit more than that, as plone.app.vulnerabilities and ploneorg.core already holding a few contenttypes and views, the goal would be to have a ploneorg buildout / setup that runs without them.
-
at the moment plone.app.vulnerabilities have a browser view returning json that could take requests form plone.vulnerabilitychecks.core, yes those views should be migrated.
-
There are several possible ideas about a control_panel integration,
at the moment the json only gives information about Plone itself, so problably an integration to the @@overview-controlpanel in the version overview setction would be good.
an alternative if we could extend it by also giving information about plone add-ons than an integration with Add-Ons Upgrades would be fantastic.
either or, if we want some feedback and opt-in we do need a small form asking some data, like url of this site and a checkbox to allow sending detailed information (versions, add-ons) to plone.org) also the JSON need to be processed on client side and show an info if the site is vulnerable or if upgrades are avaliable. @djay and others have before spoken about some data that could be useful in this thread.
At the moment thos three wrapper around p.v.core are comand line tools so you get a String response on the CLI.
I think we should do issues in the related ploneorg.* repos on github for those.
- The current process of releasing Plone and any hotfix is completly separated.
On a Plone Release, the Release Team did create a Release Object on Plone, but that neither contains any official release date or information about the release series and security and mantaince status. So the Plone Security team needs to update a seperate List in the Plone resgistry to make the Hotfix listing https://plone.org/security/hotfixes show any new Plone release and update security and maintenance status by hand. that is not optimal.
We would like to get all those information from launchpad and imported into a plone object to work with it.
1 Like