Why was WSGIPublisher transaction_pubevents written as a context manager?

[flipmcf]

Edit: Title is misleading...

I understand WHY transaction_pubevents is implemented as a context manager ( @davisagli nailed that one in his response). But I think HOW the context manager is implemented is what confuses me.

I expected to find a context manger object that implements __enter__ and __exit__ but found a function that "yields" and then decorated with @contextlib.contextmanager Wow....

---

I ask because Hanno Is one of my heroes, so I'm hoping he will actually answer.

Is this a good 'pythonic' use of a context manager? Is this a python pattern I should get to know?

Or is this just someone (hannosch) flexing awkward uses of syntax sugar for their own learning?

I would NEVER have considered using a context manager decorator to write this code, and I want to know more!

It's hard to debug stuff... but I'm not worthy to criticize. I'm here to learn from the best.

Use context managers to simplify the publish event and transaction logic.

(Commits · zopefoundation/Zope · GitHub) available to comment?

[flipmcf]

for the record, I just learned that contextlib.contextmanager decorator will use a generator. Basically a coincidence in my mind.

So I can write a context manager that 'yields' in the middle, "wrapping" the inner fuction, and using (or abusing?) 'with' and 'yield'.

The flow seems very awkward. But I'm not criticizing, I'm trying to see another way and wondering if I should think differently.

[davisagli (David Glick)]

I find context managers useful for situations where I want to make sure that something will get cleaned up after some other code runs. Of course you can do that without a context manager, using a try/except/finally block. But the context manager lets you encapsulate this cleanup logic separately from the code that is inside, so that it can be reused, and/or tested separately.

[jensens (Jens W. Klein)]

I would say it is easier to re-use and has a better ability to be tested.... just, It is nowhere tested. However it is reused once in the SiteErrorLog.

It encapsulates a piece of functionality that would be otherwise a part of still way to complex method. IMO not perfect but Okayish.

[flipmcf]

I find context managers to fit this same problem. - Cleanup.

I think what bothers me is that I'm so "classic" that I think a context manager must have a __enter__ and __exit__ - most importantly an __exit__ which maps to the finally:

The @contextmanager decorator
( contextlib — Utilities for with-statement contexts — Python 3.12.1 documentation )

seems to be a way to take an existing try/except/finally (pre contextmanager pattern) and just decorate it, so one can use 'with' in the calling code.

But then, what happens is your original, nice try/except/finally function - which was fine code... now needs to become a generator, causing a bit of a re-write anyway. (When should it yield?)

So why not use an abstract class? Actually write your __enter__ and __exit__ like a good human? Rather than take a try/except/finally, force it to yeild somewhere, guarantee it only yields once, and then wrap it?

(this sends me down the thought-hole that generators that use 'yield' are just fancy ways of declaring scoped static variables.... but python is naturally object oriented... so... but I digress...)

1. Readability counts.
2. There should be one-- and preferably only one --obvious way to do it.

and to a lesser extent:

3. Explicit is better than implicit.

But again, this is an OPINION. I'm asking for a counter-argument.

to me it looks like "coincidental syntax" and shoehorning the try/except/finally model into a context manager by forcing it to 'yield' instead of 'return', then writing some docs saying "oh, it must be a generator that returns a single value" to hide the fact you're doing something odd...

I don't think I'm qualified to argue this tho... But I'm going to try anyway.

I think I have a purist problem... a philosophical problem... and I really should just be writing code and getting back to work.

Where's Hanno?

[dieter (Dieter Maurer)]

flipmcf via Plone Community wrote at 2024-1-31 15:22 +0000:

...
seems to be a way to take an existing try/except/finally (pre contextmanager pattern) and just decorate it, so one can use 'with' in the calling code.

The important feature of a "context manager": it helps to implement
actions of the form "setup a context; do something in this context;
cleanup a context" by factoring out the "setup" and "cleanup"
into the "context manager" - leaving the "do something" open
to be provided as "with" body.

A "@contextmanager" decorates a generator of the form:
something (mostly setup)
yield
something (mostly cleanup)

With this everything before the yield is setup code
and everythig after the yield is cleanup code.
The with body is executed after the setup and before the cleanup code.
Exceptions, if any, appear in the generator as arisen from the yield.

This is a special form of "context manager".
Internally, it uses __enter__ and __exit__, but that is an implementation
detail.

[flipmcf]

@dieter

The important feature of a "context manager": it helps to implement
actions of the form "setup a context; do something in this context;
cleanup a context" by factoring out the "setup" and "cleanup"
into the "context manager" - leaving the "do something" open
to be provided as "with" body.

This is the most concise explanation for a context manager I've seen yet. Beautiful.

My problem is philosophical now.

I can't quite explain it as well as Richard Feynman: https://youtu.be/NM-zWTU7X-k?si=mIAUKvWVkdvLqq0z

---- Watch that first - 5 minutes ---

Now, I propose that the '@contextmanager' decorator is useless and "bullshit".

And you seem to see it yourself:

This is a special form of "context manager".
Internally, it uses __enter__ and __exit__, but that is an implementation
detail.

I propose:

1. "Implementation Details" are almost always, and I would say definitely in this case, an 'abstraction'.

2. an 'abstraction' is only useful if it makes the greater whole simpler to understand.

3. @contextmanager decorator, requiring a try/except/finally implementation and the addition of a 'yeild' statement is not making things easier. It's making the abstraction more complex than the problem it's trying to solve (context managers).

Therefore: after one understands a __enter__ <body> __exit__ pattern - they are done. use a context object.

shoehorining a function into a context manager by requiring the function to yield instead of return, and then discovering after the fact that "One has discovered that this generator can yeild only exactly once" has only complicated things.
Generators that return exactly one value are simply functions. functions should return not yield.

Generators and Context Managers are two very different concepts, and this coupling is not useful. (yet).

I propose the @contextmanager decorator be thrown in the trash, because it makes python harder, not easier.

I propose that one only study and use the contextlib.AbstractContextManager and skip the decorator for now, until it provides something more useful.

However, I am willing to entertain that I am not seeing something others are seeing.

[davisagli (David Glick)]

I personally find writing a context manager using the contextmanager decorator fits my brain better than writing a class with __enter__ and __exit__. But, I don't mind if you feel differently. They are functionally equivalent and you have to understand both to read Python code.

[flipmcf]

@davisagli

They are functionally equivalent and you have to understand both to read Python code.

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.

[dieter (Dieter Maurer)]

flipmcf via Plone Community wrote at 2024-2-1 16:07 +0000:

...
I can't quite explain it as well as Richard Feynman: https://youtu.be/NM-zWTU7X-k?si=mIAUKvWVkdvLqq0z

---- Watch that first - 5 minutes ---

I do not use "youtoube" (and variants). Therefore, I did not watch it.

Now, I propose that the '@contextmanager' decorator is useless and "bullshit".

I do not agree.

And you seem to see it yourself:

This is a special form of "context manager".
Internally, it uses __enter__ and __exit__, but that is an implementation
detail.

I can tell you: I do not see it.

Modular programming is essentially based on the concept
"separation of concerns".

A context manager allows to separate the concern "context handling"
from the concern "actions performed in this context".

The @trasaction_pubevents is a perfect context manager in this
sense: it handles the complete context consisting of transaction
management and publication event handling.
I can understand it without knowing any "with" block,
and apply it to whatever with block I want to perform in a
transaction/publication event context.

I propose:

1. "Implementation Details" are almost always, and I would say definitely in this case, an 'abstraction'.

To understand transaction_pubevents I do not need to know
how it is implemented; in fact, this implementation is much more complicate.

2. an 'abstraction' is only useful if it makes the greater whole simpler to understand.

In many cases, context handling is better described together
rather than splitting it into __enter__ and __exit__.
In my view, transaction_pubevents is a good example.

3. @contextmanager decorator, requiring a try/except/finally implementation and the addition of a 'yeild' statement is not making things easier. It's making the abstraction more complex than the problem it's trying to solve (context managers).

Have you tried to reimplement transaction_pubevents with
__enter__ and __exit__?
Are you really convinced that this alternative implementation
is easier to understand than the current one?

...
I propose the @contextmanager decorator be thrown in the trash, because it makes python harder, not easier.

Fortunately, this is the wrong audience.
Here, we do not decide about throwing Python features into the trash.

[flipmcf]

@dieter

Have you tried to reimplement transaction_pubevents with
__enter__ and __exit__?
Are you really convinced that this alternative implementation
is easier to understand than the current one?

No. and No.

And Challenge Accepted. Let me be clear. I am not challenging YOU. I am challenging myself because I don't see what you are seeing.

In many cases, context handling is better described together
rather than splitting it into __enter__ and __exit__.
In my view, transaction_pubevents is a good example.

I don't see it...

This is why I used the word "propose". And you're absolutely correct that the burden of proof is now on me. I humbly accept the challenge.

If I find that @contextmanager decorator is easier to understand in this situation than than contextlib.AbstractContextManager, then it's only a personal preference and I'll be quiet.

I really like how @davisagli put it:

They are functionally equivalent and you have to understand both to read Python code.

The burden is on me to understand both.
However, this files counter to the proposition "there should be one and only one obvious way to do it" (and I know that's a poke at Larry Wall, but I like the philosophy because I can't read my own perl). I only --think-- I understand both, but to be sure, I must accept the challenge proposed by @dieter.

Additionally:

I do not use "youtoube" (and variants). Therefore, I did not watch it.

To which I reply:

"There is a principle which is a bar against all information, which is proof against all arguments and which can not fail to keep a man in everlasting ignorance-that principle is contempt prior to investigation."
--HERBERT SPENCER

But this is just a jab. It's forcing You @dieter to absorb knowledge the same way I @flipmcf do. And this is not fair.

I don't learn well from reading. Maybe you do. So, here is the text of the video, so you don't have to "click"

November 9, 1964 Dr. Richard Feynman "Seeking New Laws" by Richard Feynman speech transcript

Now another thing that people often say is that for guessing, two identical theories– two theories. Suppose you have two theories, a and b, which look completely different psychologically. They have different ideas in them and so on. But that all the consequences that are computed, all the consequences that are computed are exactly the same. You may even say they even agree with experiment.

The point is thought that the two theories, although they sound different at the beginning, have all consequences the same. It’s easy, usually, to prove that mathematically, by doing a little mathematics ahead of time, to show that the logic from this one and this one will always give corresponding consequences.

Suppose we have two such theories. How are we going to decide which one is right? No way, not by science. Because they both agree with experiment to the same extent, there’s no way to distinguish one from the other.

So two theories, although they may have deeply different ideas behind them, may be mathematically identical. And usually people say, then, in science one doesn’t know how to distinguish them. And that’s right.

However, for psychological reasons, in order to guess new theories, these two things are very far from equivalent. Because one gives a man different ideas than the other. By putting the theory in a certain kind of framework, you get an idea of what to change, which would be something, for instance, in theory A that talks about something. But you say I’ll change that idea in here.

But to find out what the corresponding thing you’re going to change in here may be very complicated. It may not be a simple idea. In other words, a simple change here, may be a very different theory than a simple change there.

In other words, although they are identical before they are changed, there are certain ways of changing one which look natural, which don’t look natural in the other. Therefore, psychologically, we must keep all the theories in our head.

And every theoretical physicist that’s any good knows six or seven different theoretical representations for exactly the same physics, and knows that they’re all equivalent, and that nobody’s ever going to be able to decide which one is right at that level. But he keeps them in his head, hoping that they’ll give him different ideas for guessing.

Incidentally, that reminds me of another thing. And that is that the philosophy, or the ideas around the theory– a lot of ideas, you say, I believe there is a space time, or something like that, in order to discuss your analyses– that these ideas change enormously when there are very tiny changes in the theory.

In other words, for instance, Newton’s idea about space and time agreed with experiment very well. But in order to get the correct motion of the orbit of Mercury, which was a tiny, tiny difference, the difference in the character of the theory with which you started was enormous. The reason is these are so simple and so perfect. They produce definite results.

In order to get something that produced a little different result, it has to be completely different. You can’t make imperfections on a perfect thing. You have to have another perfect thing.

This is a conflict. It's my conflict. Conflicts aren't 'bad', they are just an opportunity to better understand ourselves and others. If I come off arrogant and mean, it's only because I'm proposing truths, not actually committing to them.

And I'm too much a coward to bring this up on a python forum. I feel comfortable in Plone exposing my ignorance. I'll get there some day, maybe.

[flipmcf]

It's re-implemented.

First observation is a performance hit. Very interesting!

Going to take a break and get some food... but I'll have it up here tonight:

GitHub - flipmcf/Zope at flipmcf_idea

[jaroel (Roel Bruggink)]

I think y'all just aren't Dutch enough

[jensens (Jens W. Klein)]

!? did you forget to commit/push something?

[flipmcf]

Love the picture!

Yes, the commit hasn’t been pushed. I ran out of time yesterday and the family came first.

It will be up soon. I just edited the egg locally to POC and found it did actually work, so now I have to do the dev-op’s thing right. Standby.

Edit: it's there now: rewrite transaction_pubevents context manager as contextmanager class. · flipmcf/Zope@42b2c6d · GitHub

Edit: Jens, du hast den Vorteil, in Österreich zu sein. Du hast 6 Stunden Vorsprung auf mich. Wenn ich es jedoch schaffe, es voranzutreiben, bevor Seattle aufwacht, könnte ich einer Peinlichkeit vielleicht entgehen.

[flipmcf]

@dieter

Have you tried to reimplement transaction_pubevents with
__enter__ and __exit__?

rewrite transaction_pubevents context manager as contextmanager class.

using buildout, I'm here.

[buildout]
extends =
    https://dist.plone.org/release/5.2.7/versions.cfg
[sources]
    #based off of Zope 4.6.2 tag 4ff56f2
    Zope = git https://github.com/flipmcf/Zope.git branch=flipmcf_idea

@dieter

Are you really convinced that this alternative implementation
is easier to understand than the current one?

Not convinced. I think it's personal taste. And Duchness...

But I'm seeing some refactoring bonuses and test-ability coming out.

1. Now one can write and test separate exception handling from the context __enter__ and __exit__ - the previous implementation handled exceptions before and after yeild identically. Separation of concerns is better.

   (I cheated, copying the exception handling and applying it as a 'handle exception' function/method because it's a big chore to separate the possible exceptions from __enter__ and __exit__ now, and I don't think it's worth it until practical edge cases appear)

2. I think the outer nested try/except blocks go away.... but not sure yet: It's a hunch, but not a certanty.

           #Not needed anymore - it's an actual context manager now.
           #finally:
               # Avoid traceback / exception reference cycle.
               # del exc, exc_info

Absolutely not certain why that was there in the first place, so I'm hesitant to commit to it yet.

I said it already, but I'll say it again. This is more of an exercise in my own Python journey. I'm absolutely ok being wrong here, but I'd like to know why I'm wrong rather than just take it for granted. I have always had a problem with authority in my life. I think it's an American thing... I'm working on it.

[jaroel (Roel Bruggink)]

I guess you have figured out why this thing is the way it is

[flipmcf]

No. I have not.

( I like my 'style' of code better... which means absolutely nothing in the big picture)

I just stopped being distracted and got back to my REAL job. (figuring out why my subscribers for PubBeforeCommit event are fine in 5.2.7 and completely empty in 5.2.14 - but to embarassed to admit it's probably my fault)

This was just an ADHD distraction, and an extremely fun one. I may return.

[flipmcf]

Here is the story I have.

A junior Dev entered my office today. I asked him "have you ever used context managers in python?"

"No." he says. (it's fine... we have a "it's ok not to know something" environment here...)

"Have you ever seen the 'with' statement" I say..

"Yes" he says...

"That's how we use context managers". I say. "like when you open a file you should always close it. this helps take away that extra step of thinking."

I pull up google and type "what is the python with statement" and in 5 minutes end up showing him how __enter__ and __exit__ work.

"that's pretty straight forward" he says. That's cool!

"Ok, now look a this code" (shows him WSGIPublisher.py) shows him 'def transaction_pubevents` definition.

"What's that?" he says"

"It's a context manager" I say.

He is now confused. AND HE SHOULD BE.

"Now, do you know what 'yield' does?"

"Yes, it creates a generator - you can use it as an iterator".

"Correct." Let's look up this decorator @contextmanager because, as we remember from grade school 'if we don't know what something is, we look it up'.

"Yes" he says.

And I show him:
[ quote: contextlib — Utilities for with-statement contexts — Python 3.12.1 documentation ]

This iterator must yield exactly one value.

"What do they say in Rick & Morty?"

That's just a function with extra steps.

So, after I introduced you to a very simple concept, "The context manager", and you follow the 'with' statement from WSGIPublisher.publish_module what did you expect to see? A class / object or a "generator that yeilds exactly one value"

"That's weird" he says.

"Yeah, maybe after seeing that you might think 'man, I hate python'"

"Yeah, " He says "This stuff is hard".

That's my problem here. This code serves no purpose other than to confuse the next (junior) dev. It's stuff like this that makes a good language go bad. Maybe all languages start at 'C' and end up at "Perl" because of bad decisions like this. "Hey, did you know you can shoehorn a generator into a context manager?"

--- Rant over.

PS.

I have a great idea, let's simplify context managers even more now by using an arrow-assignment '=>' operator. That will make things even simpler.
It's the direction python is heading because the Zen has been ignored.
(No offence, ES6 specification.... )

[dieter (Dieter Maurer)]

flipmcf via Plone Community wrote at 2024-2-6 18:33 +0000:

...
"Ok, now look a this code" (shows him WSGIPublisher.py) shows him 'def transaction_pubevents` definition.

"What's that?" he says"

"It's a context manager" I say.

You said something wrong:
when you look at the code starting with "def transaction_pubevents",
then you see a generator function NOT a context manager.
It is the @contextmanager decorator which transforms a generator function
(with special properties) into a context manager.

When you look at the contextmanager definition, you will find
the __enter__ and __exit__ you can assume from a context manager.

He is now confused. AND HE SHOULD BE.

Sure -- his mentor told him something wrong.

...
And I show him:
[ quote: contextlib — Utilities for with-statement contexts — Python 3.12.1 documentation ]

This iterator must yield exactly one value.

This applies to the contextmanager argument not its result.

...
So, after I introduced you to a very simple concept, "The context manager", and you follow the 'with' statement from WSGIPublisher.publish_module what did you expect to see? A class / object or a "generator that yeilds exactly one value"

Are you sure you have understood decorators?

In almost all cases, the decoration result is quite different
from the function which has been decorated.

The contextmanager decorator transforms a generator function
yielding a single value into a context manager function -- and when
this context manager is used in a with statement the with
block is executed at the place of the yield.
For someone who has understood decorators, this should be
quite easy to grasp.

Let me point out an analogy:
Look at
class C:
@property
def f(self): return ...

When you start looking at the "def", you will see a function
definition. However, the @property decorator transforms this
into a descriptor -- which behaves not at all like a function
but much like a (computed) data attribute.

In a similar way as the decorator property
transforms a function (with special properties) into a descriptor,
the decorator contextmanager transforms a generator function
(with special properties) into a context manager (function).