[plone.restapi] Dates and timezones

zopyx · August 24, 2020, 2:56pm

We have some issues with events generated through plone.restapi and their dates and timezones.
The site TZ is set to Europe/Amsterdam. The start and end date of end event (coming from collective.jsonify) are

(Pdb++) pp object_data['startDate']
'2020/08/25 09:00:00 GMT+2'
(Pdb++) pp object_data['endDate']
'2020/08/25 17:00:00 GMT+2'

They are created in Plone 5.2.2 as

>>> doc.start
datetime.datetime(2020, 8, 25, 13, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)
>>> doc.end
datetime.datetime(2020, 8, 25, 21, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)

So there clearly visible some four hour offset (coming from the GMT+2 from the date data and perhaps the UTC+2 from the global timezone settings. The event dates are also shown as 1 PM and 9 PM.

Any trickery I am missing here?

espenmn · August 24, 2020, 3:09pm

About PM: Could this be related to translation of dates ?

In other words: in some (dutch).po-file there is a 'translation' of how dates are shown

github.com/plone/Products.CMFPlone

pickadate widget doesn't use the correct 24h format in some languages

opened 03:37PM - 29 Jun 20 UTC

closed 04:15PM - 29 Jun 20 UTC

vincentfretin

## BUG See https://community.plone.org/t/datetime-widget-format-not-am-pm/124…91 ### What I did: Plone in French language, create a Page, click on the Dates tab and pick a date. ### What I expect to happen: In French and Norwegian, the hours should be in 24h format, so showing 17:27 in the afternoon. ### What actually happened: It shows "05:27 p.m." instead of "17:27" Note, in German, which uses 24h format too, it works properly. in French ![Capture d’écran de 2020-06-29 17-32-10](https://user-images.githubusercontent.com/112249/86026002-3134e200-ba2f-11ea-859e-e9bef1a32d7b.png) in German ![Capture d’écran de 2020-06-29 17-32-04](https://user-images.githubusercontent.com/112249/86026042-3b56e080-ba2f-11ea-815f-fef17506e9b6.png) ### What version of Plone/ Addons I am using: Plone 5.2.2rc1

zopyx · August 24, 2020, 3:25pm

The problem is not the display value but as said the improper 4 hour offset!

zopyx · August 24, 2020, 3:39pm

I created an event through the Plone UI for start=2020-11-13 9:00. This is stored in Plone as

datetime.datetime(2020, 11, 13, 9, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CET+1:00:00 STD>)

which is correct. So apparently something in plone.restapi appears to go wrong.

jaroel · August 24, 2020, 3:51pm

The converter for datetime to json is here: https://github.com/plone/plone.restapi/blob/acdcc7435aeb7c0140c7e0fa4ab9c09d4592ba5b/src/plone/restapi/serializer/converters.py#L33-L54

It could very well be that we're converting the datetime from the site TZ to the local TZ (2H) and then to UTC (2H).

What happens if you set the site TZ to say +3 or -3? Do you still see a 4H difference?

zopyx · August 24, 2020, 4:25pm

Seems to be related to the format of the passed date string. I created a fresh Plone site with global TZ "Europe/Berlin". I created two events with dates

"start": "2020-08-25T17:00:00+02:00"
"start": "2020/08/25 17:00:00 GMT+2"

The event is rendered with Aug 25, 2020 05:00 PM ,
the second one renders as Aug 25, 2020 09:00 PM

Event #1 is correct (with the ISO-8601 format string). Obviously something is happening with the second date format. The second date string is clearly not an ISO-8601 date. If ISO-8601-only is expected, a compliance check and error would be helpful here.

zopyx · August 25, 2020, 11:08am

"Solution" in our case: we subtract 4 hours upon import time

tisto · August 26, 2020, 8:33am

This is a pretty complex matter that we discussed quite a few times already. Sorry, I can not help right now with this particular issue. Here is the current state of affairs of datetime handling in plone.restapi and plone.app.event:

I would love to clean this up and maybe get the into Plone 6. Though, I guess this depends on @thet or someone else to step up to do the actual work.

thet · August 26, 2020, 8:59am

doc.start and doc.end in your initial example are correct, so the creation via plone.restapi looks like it's working as expected (except for the second example where you pass a non ISO-8601 string).

I've checked it by creating an event TTW:

>>> app.Plone.e3.start
datetime.datetime(2020, 8, 25, 13, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)
>>> app.Plone.e3.end
datetime.datetime(2020, 8, 25, 21, 0, tzinfo=<DstTzInfo 'Europe/Amsterdam' CEST+2:00:00 DST>)
>>> app.Plone.e3.start.isoformat()
'2020-08-25T13:00:00+02:00'
>>> app.Plone.e3.end.isoformat()
'2020-08-25T21:00:00+02:00'
>>>

collective.jsonify should serialize them by using isoformat as here (a datetime instance is an instance of date):

github.com

collective/collective.jsonify/blob/3e259ccae6a50ebfdd24514ab5dcf9064f14571b/collective/jsonify/wrapper.py#L196


            continue
    value = _value

elif field_type == 'GeolocationField':
    # super special plone.formwidget.geolocation case
    self['latitude'] = getattr(value, 'latitude', 0)
    self['longitude'] = getattr(value, 'longitude', 0)
    continue

elif isinstance(value, date):
    value = value.isoformat()

# elif field_type in ('TextLine',):
else:
    BASIC_TYPES = (
        unicode, int, long, float, bool, type(None),
        list, tuple, dict
    )
    if type(value) in BASIC_TYPES:
        pass
    else:

Still, I suspect collective.jsonify to be the problem here. Can you give more context how or where this line comes from:

(Pdb++) pp object_data['startDate']

However, IMO collective.jsonify should convert the dates to UTC as plone.restapi does.

zopyx · August 26, 2020, 9:41am

That's my suspicion too...I will recheck with the next export on our side.

erral · August 31, 2020, 6:38am

Some time ago we faced a similar problem with some of the datetime fields exported using collective.jsonify and imported with transmogrifier.

We found this gist that replaced all "GMT+X" timezones with "Etc/GMT+X" which promised to fix the problem:

gist.github.com

https://gist.github.com/gyst/c3739160a18cf52c93e0132da511b2ee

pytzfixer.py

# coding=utf-8
from zope.interface import classProvides, implements
from collective.transmogrifier.interfaces import ISectionBlueprint
from collective.transmogrifier.interfaces import ISection
from collective.transmogrifier.utils import Matcher, Condition

import logging
logger = logging.getLogger(__name__)

This file has been truncated. show original

pytzfixer.rst

Pytz timezone fixer section
===========================

This fixes the dreadful ``UnknownTimeZoneError: (UnknownTimeZoneError('GMT+1',)``
thrown by ``transmogrify.dexterity.schemaupdater`` when writing Event dates.
It turns out pytz does not recognize a timezone like ``GMT+1``.
This should be ``Etc/GMT+1``. This blueprint fixes that::

  [pytzfixer]
  blueprint = quaive.transmogrifier.sections.pytz.fixer

This file has been truncated. show original

Nevertheless we also saw that GMT+X was not Etc/GMT+X but Etc/GMT-X (see the comment on that gist)

So, we ended creating a transmogrifier blueprint for our imports with that code:

github.com

codesyntax/bobtemplates.cs/blob/master/bobtemplates/cs/cs_migration/migration/blueprints/pytz_fixer.py

# -*- coding: utf-8 -*-
"""
Original: https://gist.github.com/gyst/c3739160a18cf52c93e0132da511b2ee

see https://stackoverflow.com/questions/4008960/pytz-and-etc-gmt-5
for GMT+ and GMT- bug
"""
from collective.transmogrifier.interfaces import ISection
from collective.transmogrifier.interfaces import ISectionBlueprint
from collective.transmogrifier.utils import Condition
from collective.transmogrifier.utils import Matcher
from zope.interface import implementer
from zope.interface import provider

import logging


logger = logging.getLogger(__name__)

This file has been truncated. show original