The Future Of Documentation

This is the followup of Broken links in docs.plone.org

It is still WIP but should give already some information about the possible future of the docs.

As stated already before and also in the document.

We are trying to see the whole picture not only devs. It is meant for all audiences of the docs.

One more thing, as also stated before, I am pretty sure there are lots of things in the document where you do not agree or even do not understand/see the point why (no offense !)

If so please ask for the reasons and I will happily try to explain !

This document is pretty much a result of lessons learned and lots of talks with other "docs people" of various projects and companies.

Nothing is written in stone, nothing is definitely yet, still it points in the general direction of modern documentation.

Documentation 2.0

Abstract

These document outlines the new "specs" for a better and improved documentation.

We learned a lot of the last years, let us translate all this into better and user orientated documentation.

They key here is user orientated.

We have different audiences and thus different needs.

All of them are equal !

Introduction

From improving the wording, over structure till building and releasing this document (WIP) tries to cover all these steps.

Improvements

Appearance

Make the look and feel better, make it easier for people to identify and find what they are looking for.
This can be done for example with a improved landing-page, with visual improvements and adding boxes per audience.

Example

Use a theme which is focused on presenting content !

Our current one is too distracting.

Examples of content focused themes with less visual interruptions

Redactor Theme

TTD Henry

TTD Docs

Audience

Better and easier distinction between audiences.
If a user is looking for the docs about theming the user is at this moment not interested in docs about ZEODB.

To do so we need to make sure that all docs in the different categories are really:

  • straight to the point
  • written in the right tone and voice for the category
  • understandable
  • the user does not get distracted with to much noise (like look and feel, etc)

Style

  • "rethink" our terms, for "us" (old plone ppl) lots of our terms makes sense but that is not always the case for newer people.
  • shorter items
  • consistent style-guide and usage of terms
  • consistent and shorter headings

Structure

Improve the structure by moving some parts to other locations.
Our current structure does is not always logical or understandable for (new) developer.

We can improve that by re-locating some parts to other locations and by 'merging' some parts into one chapter.
Examples are here: Installation and developer docs

Further remove/improve the structure to make the docs less nested.

It is often confusing for user that some docs are really deep nested like docs.plone.org/develop/bla/bla/bla/bla

Stay as flat as possible !

README Removal

Currently we are including READMEs from external documentation like bobtemplates.plone.

A README is great for a developer, unfortunately it is less suited for including into documentation.

Besides the wrong tone and goals of a README also the nice buttons of Travis etc are no addition to the docs.

They only add noise and distraction.

These are perfectly fine for a README but less cool in the documentation.

External Removal

Because of historical reasons and the way how the current documentation got structured we still use the directory external.

This is maybe nice and logical for the docs team but this add no extra value for the reader.

Besides that, it also can lead to confusion, because for the (new) reader it is not clear why certain parts of the docs are located under external.

Further is breaks with the structure and consistency of the docs.

Splitting

Splitting the docs into "standalone" parts.

We started already doing that with the training docs.

For example each "main" chapter (installing. theming, developing) should be this way.

By doing so we will be able to make (faster) "canary releases", we can provide a better user experience, etc, etc.

One other huge reason is that by doing so we can run tests against the docs, easier and faster, too.

This also provides ah moments because contributors see their PR faster merged and the tests also reporting only on their changes.

Of course we can choose during a release to do a full or only a chapter release.

Search

  • custom search per audience like searching only in theming or installing
  • better docs.plone.org/search atm the mix between Agolia and Sphinx is confusing, create a /search with Agolia API.
  • If you "click" on the box with your audience you get immediately to the parts of the docs you are looking for,
    for example "theming" and then you only see the theming docs, if you click "here" on search you get a custom search (with a button) to search all docs, too.

Branches

Change the way how the docs repository uses branches.

Currently we create a new branch when the Framework Team creates a new main branch of CMFPlone.

The last years and releases have proven that this is not the way.

Because of the pace of development, time, effort, etc, etc. this only creates lots of extra work, because we have to maintain and merge content to even more branches of the docs.

This is time consuming and "error prone" as it also adds confusion, about to with branch to commit to etc.

Because of this we will change the branch model to the following.

master will be cutting edge aka rolling release.

We will build "weekly" unstable docs from this branch.

This branch will be also the branch we most of the commits will happen.

If there is a new release of Plone, we will cut a new branch of the master.

The master keeps to be cutting edge and the new branch will the "last old version".

Testing

CI

We will use a dedicated CI setup for the docs. All tests are running in containers, meaning we can use them better, faster and everywhere.

Tests On Commits

The way how we run tests will change. With the new setup we will test with each commit or PR only the files which are changed.

This has the following advantages:

  • tests are faster
  • we can better and "easier" improve the docs and its tests
  • it is less de-motivating for people, since CI is "only" complaining about "their" commits.
  • tests are based on Redactor

Tests On All Docs

Redactor

This is our new test-framework where we can configure the amount of tests and also the level, etc.
Some of the tests we "only" run on release time, like HTML, other ones like style-guide we run always.

Test Overview

  • spell-check
  • link-check
  • line-length
  • file-length
  • write-good
  • style-guide
  • white-spaces
  • HTML
  • ...
  • ....

Building

Coster

Coster is the re-written and improved version of papyrus. Here we configure which "external" (like ansible-playbooks and branch) docs we fetch and where we put these in our docs.

Coster can handle rst and md !

Coster also fetches all images we need which are created by our jenkins-robots.

After Coster is done fetching and building the docs, we do run again tests on the created HTML, like HTML validation, accessibility checks, etc, etc.

If these tests pass, Coster will create automatically "ready to run" container images (docker).

Deploying

Each version in own container, plus weekly builds of "unreleased".

Each version like 3, 4, 5 gets a own container.

Coster

Coster creates, tags and uploads the container to Docker Hub.

For "unstable" this will be done, once a week or so.

This is also perfect for local usage, "just" do

docker pull plone/documentation/5

and you get the latest docs of Plone 5.x which you can run locally and also off-line.

Or do

henry serve

Hosting

We will use the new k8 cluster for hosting.

k8 recognizes if there is an new image available and automatically downloads and switched (blue-green) to the new image.

Helper Tools

tbc

Henry

(add henry docs overview)

Coster

(add coster docs overview)

Redactor

(note you can run tests also without Redactor)

2 Likes

Could one automate archive.org snapshotting whatever one links to in the documents? Bitrot has eaten a good chunk of the still-relevant old Zope documentation at this point.

Is a section on how to test and build the docs locally included? I didn't see one here. I think it would be good so contributors can catch their mistakes before they push their commits, and it will speed up their development cycle and lower frustration. This might fit under:

Contributing

  • How to contribute to docs
  • Signed Contributor Agreement required or not

Can you clarify? For example, when I want to look for "theming", will the search results cover both contexts of Training and core Documentation, or just the one I am currently browsing?

That sounds like two bullet points. How about calling them:

  • Launch page. Example: https://training.plone.org/
  • User-controlled search context (Search Training, Search Core Documentation, Search All).

Looks like these were omitted, too.

Search results implementation

  • The user should have control over opening links in the same or different window, not the application.
  • The search results should remain available and not disappear when one of the results is clicked.
  • The search terms should remain available and not disappear when one of the results is clicked.
  • The search results at /search should implement Algolia's pagination (includes unstyled live example at the end).

Yes ! That is the plan !
Since lots of stuff is changing there, the plan is to restructure and rewrite the whole "contribute" docs.
There are outdated and messy currently.

My idea is, if you are currently reading the "theming" docs, these will be searched by default, but there will be a switch where you can say "search the whole docs".

That is basically the whole idea, yes !

I'm guessing implicit in here is the ability to download and distribute a user manual that includes no installation or developer stuff... like we used to have. +1 for that.

How hard do you think it would be to go one step further and ship documentation with plone itself, perhaps with a few scattered links from various dialogs to the relevant section of the docs? esp if it can be turned off by integrators (or replaced) if they feel its not good enough.
or do you think the tone and general feel of online help vs a manual won't work as a combined thing?

@djay Yes !

That is one of the reasons why I want to "split" the docs as written above.

One other important factor which is not solved yet is the relationship between docs and training.

We have to be clever here :slight_smile:

Something like:

  • remove outdated and possible double "content" from docs and link to training instead

This is no problem for online but it is a interesting challenge for "offline" docs :slight_smile:

We do not want to copy content, this will be a mess and outdated in no time.

I mean I have some ideas, but I am not sure how much work that will be or maybe it is "overengineering".

Solutions should be usable for everyone and "easy" understandable. :slight_smile:

Care to share?

If we support both online and offline viewing of both training and docs, then there would need to be two versions in order to support cross-linking between the two sets. I can't imagine how to do that well.

Kudos! Thanks for taking such great care of our docs @svx

Just to let you all know, we are not the only ones (I said that already a couple of times before) who are using a setup where we fetch docs from various sources.

This is getting more and more 'popular'.

Here for example the link to Antora. This is the build tool for such setups based on AsciiDoc.

There is also already one for Rust used by the Rust community.

Further on this setup is used by, Google, FB, Mozilla, etc, etc, etc.

This is basically what we will do with Coster (our build tool).

Is this the repo for Coster?

Need contributors?

I'm going to guess and say offline manual/docs is much more useful than offline training material. So wouldn't it make more sense to remove parts of the training and point the training material to the docs?

Any thoughts on splitting developer documentation into 'reference'/'api' style docs, and tutorial/how-to style docs?

See for instance the angular docs - attached a few screenshots.

In my opinion, when you are familiar with developing for plone, what is needed are quick lookups in the reference.
While for onboarding tutorial/recipe style docs are needed.

Also - there is a lot of documention of good quality spread out in different packages readmes/readthedocs pages, for many plone core modules.
We dont want to copy this.
Some sort of inclusion of these sources could be valuable. And if it is plone core packages some documentation standard could be enforced by quality checks just like flake8 etc.

1 Like

Regarding broken links and maintainability (see for instance Changed documentation locations)

We need (also in the future) to be able to refactor documentation, change locations, move stuff around, but still strive to keep externals links unbroken as much as possible. Creating redirects automatically from old locations to new, if pages/topics are just moved.

Is this included in the roadmap?

1 Like

+1, most reference guides are starting to have been bitrotten at this point and it is getting hard to get a coherent picture on some corners at this point.

@svx any plans to sprint on the doc structure soon?
The current situation is painful and going on way to long.
I know we all have to spend more time on the docs, but for that we need a running workflow and a clear definition of the structure. I think the best to get going is, to discuss this in a small group and make some decisions and work on them.

Maybe we can have a meeting for a start and then organize a short online sprint soon.

What i expect to get out of this, is a living doc setup which is auto generated on docs.plone.org when ever changes are merged into the main branches. This should work at least for Plone 5.x and above.

This should be done in a way that it does not take long to start moving docs around and fixing things. We can optimize the build process and screenshot generation later on the way.

Meeting as well as sprinting can be done entirely on Discord.
Since most of the Plone people are there now already and Video and Screen sharing works great now if you change the Video settings to prioritize text sharpness.

When would you have time for a first meeting?

@spereverde, @stevepiercy would you join?

thanks for bringing this up.
I can join a meeting, preferably in july or after august 22nd, but feel free to meet without me to move things forward if that doesn't fit the other schedules.

I think @svx has some mockups on the new structure, we could use that as a base to start discussing.
current plone 6 branches:

I totally agree. Thank you for bringing this up. I think it does not need to be perfect or complete. It merely needs to be started in a shared repository or designated branch where we can collaborate. I'm sure there are enough skilled Plonistas who can pitch in.

I want to participate in this effort. I usually work during the hours of 19:00-10:00 UTC.

:tada: that would be super!