The incredible and sad tale of innocent Plone 5 and its heartless add-ons

Plone Support Add-on Development
1

TL/DR: Plone 5 is only going to be successful if we provide add-ons compatible with it; to allow painless migrations, add-ons must run in both Plone 4.3 and Plone 5.0 at least; to make an add-on compatible with both versions we need to run tests; we have a problem and an opportunity here.

I have spent the last 3 days working to make collective.cover compatible with Plone 5. what I have done until now is mostly based on some previous work by @alecpm, @cewing and @cguardia:

let me start again... I have spent the last 3 days trying to make collective.cover tests run under Plone 5 and I'm a bit discouraged with the current situation; let me summarize what I have found until now:

  • documentation, or better said, the lack of it
  • differences among Archetypes and Dexterity's APIs
  • complexity associated with running test under Plone 4 and 5
  • lack of RF abstraction layer for basic functions

documentation is a huge problem; in all previous versions there was a list of basic things you need to do to make your add-on run; this is not the case right now as current documentation is plenty of TODO markers and missing pieces.

even with that handicap, with Travis' help I was able to break the work done by the Jazkarta guys in many pieces and started solving one problem at a time; the first one (versioning) was relatively easy and is now documented:

but that was not the case with the following: link integrity and plone.app.contenttypes compatibility (the later is a blocker of the earlier).

collective.cover's Travis testing matrix is pretty complex; we have to run tests against 6 different scenarios and I'm currently stock on Plone 4.3 with plone.app.contenttypes installed; enter API inconsistency:

we created plone.api to make our life easier but... who wants to spend an evening in the beach when you can dig on lines and lines of code to debug things like these? easy life is for weak!

I have been here for over 10 years now; we have dozens of sites running Plone, none of them is Plone 5; I don't see Plone 5 in my near future if we're not able to have our add-ons tested in, at least, both Plone 4.3 and Plone 5.0.

I was talking to @polyester the other day: we are pretty interested on having a sprint to make our workhorse add-ons compatible with Plone 5 and document all the problems found in the process.

if anybody else have similar issues, let me know and let start thinking on how to solve this puzzle.

happy new year!
collapse consciously and don't drink too much!

2

Hi @hvelarde,

  • I have already migrated some of my Plone sites to 5.0, and I am very satisfied with it,
  • I have also already migrated an add-on (Plomino) to Plone 5, and that was quite simple.

I really recommand to migrate add-ons from 4.3 to 5.0, and not to try to make them compliant for both.
In my case, I have developed Plomino 2 for Plone 5, and Plomino 1.x is for 4.3.
This approach is much easier (plus, it gives a nice opportunity to clean up old code).
The changes needed to turn a 4.3 add-on into a 5.0 add-on are easy and quick to implement.

I do not really understand why we would need to maintain a single version for both 5.0 and 4.3. I mean, I do understand it is supposed to reduce the maintenance effort, but I am pretty sure the gain is smaller than the extra effort needed to make a unique bi-compliant version.

But maybe I am missing some important aspects here.
Why do you prefer to have a single version for 4.3 and 5.0?

3

I agree with @ebrehault

The major thing I've learned from the plone.app.widgets experiment is that it's usually not worth the incredibly extra effort to make things work with 2 major versions. It cost us many months--I will never do it again.

4

Hi there,

my two cents on the issue.

I am also in favor for maintaining a single code-base for some add-on which will run and Plone 4.3 and Plone 5.
This is possible and it is not rocket science.

I am too unsatisfied with current (sad) state Plone 5 for one known reason: the complete broken story
of Javascript and CSS handling (resource registries)...my complaints are known and documented.
I personally stopped any efforts moving forward with Plone 5.0 in any project until the JS/CSS story fixed and working.

There is need for maintaining Plone 4.3 and Plone 5.0 support on the add-on side for the time being.
I do not see demand for moving forward to Plone 5.0 due from the client side. There is a lot of legacy Plone outside and budgets for moving to Plone 5.0 on the client side are close to zero. Not sure about the business situation of other Plone shops but the Plone market is apparently shrinking - more legacy than new projects. Correct me if my impression should be wrong. So Plone 4.3 remains the major platform for the time being.

-aj

1 Like
5

Yes that's true, and we plan to improve that ASAP:

Weel I guess that's a good reason to choose the cheaper approach (i.e. the 2 separate versions).

6

I am aware of that and I appreciate it..that's why I stay silent :smile:

Double work and in general not needed..a single code base will work in most case with some trickery related to CSS/JS on the GS profile level...Python code will usually work if you don't perform a replacement from AT to Dexterity.

-aj

7

Not exactly IMHO (so partially agree with Andreas).

When Plone moved from 3.3 to version 4 I continued developing add-ons for a lot of customers who can't effort the migration. For a lot of time.
As migrating from Plone 3.3 to Plone 4 was quite simple, I was able to develop those add-ons already to be Plone 4 compatible, or backport 3rd party add-ons to be also Plone 3 compatible.

Another cons against separate codebase: often the old one became quickly the unmaintained ones (bugs will not be fixed, no new features added, ...).

This will not be easy now, and when a customer with poor budget will ask to migrate from Plone 4 to Plone 5 in one year from now (or more), a lot of activities (€€€) will be needed.

So: a single codebase is commonly the best choice when it's possible, but probably not this time.

8

we have the opposite view of two groups here: hard-core Plone developers, members of the framework team; and old-time Plone integrators and developers of add-ons.

I understand your arguments but I don't agree with most of them; main problem here is that, again, we're not thinking on potential effects of our changes besides the code we're writing; that's exactly why we need non-developer's joining the framework team.

I mentioned 4 problems and your replies, @ebrehault and @vangheem, don't include answers for any of them.

first, is not true that maintaining 2 branches of code will be easier and cheaper; no way, that is going to duplicate, at least, the work on every single issue, feature, bug fix... on every single package that every single developer has to maintain.

just look at the current story of branches on Plone components: people commit to the wrong branch, forget to cherry-pick fixes, to update the change log; break compatibility with Plone versions on minor fixes... the list is way too long.

so, it could seem to be easier and cheaper at the beginning, but not at the end.

second, it's true that is currently a nightmare trying to maintain compatibility with both, Plone 4 and 5, but that's exactly the point of my post: we caused most of these problems!

and I'm not talking about the resource registries story, we all know that the proposed changes were way aggressive and probably necessary on the long run; I'm talking about things as simple as the API of our default content types.

I agree with @zopyx: customers are not willing to pay for Plone upgrades; for them it makes absolutely no sense, they see no advantage and is hard to convince them of the opposite.

our business model is based on selling shared solutions to a group of customers with the same requirement (news portals, to mention one); everybody knows and agree on that because is cheaper and it means they will benefit from features or bug fixes made for others.

I have the same add-ons running among Plone 4.x sites, with ATContentTypes or plone.app.contentypes; for me is important try to maintain high quality add-ons tested on all platforms, and doing that with the minimum amount of resources is of life and death.

most of the time we are the ones who push Plone upgrades because is simply not possible to keep maintain sites in Plone 2, 3, 4, and now 5, with a staff as small as we have and a software stack as complex as this one. sometimes we do that almost for free just to avoid keep maintaining a site with an old Plone version.

people tend to forget how stuff works; technology obsolescence push us to fix things, even if they are not broken.

9

It depends on what your code contains.

TXNG 3 is running with the same code base on Plone 4 and 5 (very little Plone 4/5 specific dependencies).

XML Director is running with the same code base on Plone 4 and 5 (except issues with JS/CSS which would not solve the Plone 5 issues with two code bases):

-aj

10

I have customers/clients who are definitely interested in moving to P5... come on! Dexterity, toolbar, Mosaic (yeah yeah not core P5 but still), folder contents, WCAG 2, list goes on and on.

11

@hvelarde about your 4 points:

  • docs: the team is clearly understaffed (like most of all voluntary efforts) and the framework team already made decisions to not allow new PLIPs without proper docs (see @ebrehault PLIP)
  • differences between AT/DX: well, that's the whole point of having a new content type system, just like everyone is happy to not have to still use DTML and whatever legacy sytems that would look totally awkward
  • test complexity: fair point, but you are talking about major versions, not between 4.2 and 4.3. If not even on major versions there's an opportunity to remove/renew old stuff
  • lack of RF abstraction: back to the first point, testing team is as well understaffed, I have heard plenty of times timo and others talk about it, why is still not there? Because everyone is busy with work/live and plone contributions

Bottom line is, just like the first months/years after a major release (say Plone 3 or Plone 4) there's plenty of work involved in porting to the new major version, as new projects are started with Plone 5, you will care less and less about the older one.

As for what to do on the meantime: my humble suggestion would be to either try to maintain a single code base if possible, but if that hinders the porting, make a major release and maintain two parallel releases. Sure, that bring works about backporting, and that's a good reason to bribe your clients to pay for the migration.

Just like the security team makes security patches for all supported versions you will expect better and more timely security patches for the newer version (see the CSRF hotfix as a good example), so again, more of a reason to ask your clients for migrating to the last major version and/or make them expect less quality/higher prices.

1 Like
12

I'm glad for you, we've been using Dexterity since 2012 and Mosaic works on Plone 4 also... now going deeper, you want to review the features we're selling as enhancements on Plone 5 that will blow our customers minds so they will start asking for upgrades? ok, let's do it:

  • HTML5 responsive theme: achievable in Plone 4: all of our themes are responsive and use HTML5 and CSS3 when possible
  • Toolbar: nice feature, we can probably sell it when fixed
  • TinyMCE 4: technical feature not easy to sell to non-technical staff
  • Through-the-Web Theming with LESS: technical feature not easy to sell to non-technical staff
  • Social Media Integration: we have a more powerful integration with sc.social.like since Plone 3
  • Chameleon Templating Engine: available on Plone 4 also
  • Bulk Content Editing: available in Plone 4 with wildcard.foldercontents
  • CSRF Protection: available in Plone 4 with plone4.csrffixes
  • Accessible to the Visually Impaired: nice feature, partially achievable in Plone 4
  • Multilingual: available in Plone 4 with Products.LinguaPlone
  • plone.api: available in Plone 4 also
  • Built with passion: no comments

let's be honest: which one of these features is the killer one to sell an upgrade to Plone 5?

besides that, Kim, probably must of us haven't noticed it directly, because we still have jobs, but we're in the middle of a global economic meltdown of epic proportions; in Brazil, for instance, this is the worst economic crisis since 1901!

just to give you an idea, dollar was R$1.60 in august 2011 and now is R$4.05, an increase of 250%; we have been very busy optimizing all of our deployments as we haven't rise hosting prices at all to avoid transfer more stress to our customers... and we're serving 3 times more traffic on some sites.

do you want to come here to help me promote Plone 5?

13

sometimes I don't know if I'm not explaining clearly enough my points or simply I'm being misunderstood; let's try again:

what bothers me about the Dexterity implementation of our default content types is the changes on the API, the inconsistency on these changes and the lack of deprecation warnings to allow a painless migration path.

let's go back to the plone.app.event example: why naming fields start and end instead of startDate and endDate as it has always been? and don't answer "because it's more pythonic" because it's not: breaking code is not pythonic.

PEP 8 on Naming Conventions (the emphasis is mine): The naming conventions of Python's library are a bit of a mess, so we'll never get this completely consistent -- nevertheless, here are the currently recommended naming standards. New modules and packages (including third party frameworks) should be written to these standards, but where an existing library has a different style, internal consistency is preferred.

why having Archetypes-style getters and setters for some fields and not for all? yes, there are setTitle, setDescription, and many other methods implemented in Dexterity, but we're not implementing those neither in plone.app.contenttypes nor in plone.app.event.

writing tests for both Plone versions, 4.3 and 5.0, wouldn't be hard if we have taken into account this kind of stuff as we use to have in the past.

provide the right deprecation warnings in time and you can remove those methods in Plone 5.1; I'll be the first supporting that.

and no, I don't want to maintain 2 branches of code in my add-ons as I already explained before.

14

Your point about the naming conventions on the example of plone.app.event is a valid one. I agree.

But I still fully support one design decision - to implement the whole event schemata as Dexterity behaviors. This way, anything can become an event, which is extremely flexible. I use that a lot.
But with Dexterity behaviors, you cannot simply access an attribute directly on the object. You have to adapt the behavior interface first, otherwise you might miss some getter/setter functionality or do not get anything at all (if stored in AnnotationStorage) (not anymore the case for plone.app.event). So, you can also just rename it, if attribute access is different anyways.

So, with behaviors you have to think of some kind of content type abstraction anyways. Look at plone.app.event IEventAccessor adapters, which shows a possible way of doing that.

All in all, in my opinion all migration work is not that hard, complex or expensive. In my experience it can be done within an hour to a day, depending on the complexity of your addon.

If this is not an option for you, I'm still waiting for some ideas on how to improve the current situation: https://github.com/plone/plone.app.event/issues/209

15

I've never talked against the behaviors; that's a fantastic idea.

I think we must start by documenting the changes so developers are aware of this; I also opened another issue in the Plone tracker related with this:

1 Like
16

Hector, I think it's a very weak argument to say that all those things are doable with P4, then add an asterisk and small print that it requires this and that and extra code etc. With P5 it's all in the box, and mere mortals get it all with a nice installer.

It is also true that P5 will be maintained longer than will be P4. Clients should be encouraged to upgrade if they want to have the best possible long term support.

I'd be happy to come help you in any way you like in Brazil. :smile:

1 Like
17

While I don't agree with hectors non-constructive approach I think he has a point that our releases should always include enough UX improvements to encourage upgrades, and we should be very cautious about changing things under the hood as they do make upgrades more expensive. It is the case that what is under the hood really bothers core developers more than UX sometimes so its natural its what they are more motivated to fix. It's hard to avoid that. Plone 5 is good in that it looks great and that will make new developers more interested. But as I pointed out in my talk at last years conference, we have still have a long way to go make it a lot more usable than plone 4.
Either way, Plone 5 is here, DX is here for all its backwards incompatibleness so we have to move forward. Hopefully people will listen to hectors intent and try to think of ways to make upgrading plugins more cost effective. On the other hand, if we get a lot of plone 5 only plugins and then it will be a good motivation to upgrade and we can get the plain of upgrading and having these conversations of people ranting over and done with quicker. Upgrading sucks. Always has, always will. No one wants to tell their clients it will cost them some large amount to get roughly the same site they have now.

18

Sorry for playing the Advocatus Diaboli but this is wishful thinking.

Plone 4 still pays the bills, customers still don't care about Plone 5. As said: writing add-ons (with Dexterity) running on both Plone 4 and Plone 5 would be straight forward if the JS story would work. Plone 4 and 5 are not so different from backend side as it might appear. There is little demand for Plone 5 (and there is little demand for Plone in general)..demand for Plone is declining.

Andreas

19

hi! the Devil over here...

my arguments are not weak and I'm not using small prints at all; what I'm saying is true: it will take me 5 minutes to install any of those add-ons and most people will have to build a theme for their sites anyway; so, there's no difference at all.

anyway, we can take a caipirinha on the beach at anytime you want; prevision for today is 30 °C, clear...

you know what's really non-constructive? starting your answers with this kind of phrases.

I spent time making my arguments because I respect you, guys; I expect the same from you.

read again my first post to this thread and tell me what's exactly non-constructive there.

I give you some clues: non-constructive is ignoring valid points with a simple "that's not true" answer; non-constructive is doing nothing about our obvious problems; non-constructive is using Plone and not give something back to the community; non-constructive is cheering uncritically for everything we do, even our mistakes; non-constructive is doing the same things over and over again, and expect different results...

going back to your points, what I always wanted is having a roadmap based on more than just technical features, because most of the time technical people are not decision makers.

I think the Quaive people is doing a wonderful job on that and we have a lot to learn from their wisdom.

to finish, I'll let you with these:

R1512E_MEYER_COUNTERPART-700x624.png700×624 45.2 KB

yes, I'm most likely to be a Spanish...

20

Very interesting map :smile:
It looks like the biggest part of the Plone community is in the confrontational part. Good to know, right? :smile:

1 Like