I'm currently working on a project that involves syncing over 100,000 API records with Plone items. After consulting various posts and resources on best practices for handling large data sets, I've implemented batch processing for the transactions and ensured that each batch is committed accordingly.
Despite these efforts, I'm encountering a significant issue: the memory usage on the server increases steadily after each commit, eventually leading to a point where the server becomes unresponsive. It seems that the memory allocated during the transactions is not being released even after committing the transactions.
I'm reaching out to seek advice or best practices on how to effectively free up memory after each transaction commit. Below, I've shared the portion of the code responsible for these transactions, hoping that it might shed light on potential improvements or adjustments I could make.
def import_collection_items(self):
start_range = self.request.form.get("start_range", "0")
api_url = collection_api_url
response = requests.get(api_url)
response.raise_for_status()
api_answer = response.text
root = ET.fromstring(api_answer)
# Extract the total count of records
total_count = int(root.find(".//count").text)
for offset in range(int(start_range), int(total_count), 500):
transaction.begin()
self.sync_new_objects(start_range=offset, end_range=offset+500, date_from=date_from)
transaction.commit()
Any insights on memory management techniques or adjustments to the code to mitigate this issue would be greatly appreciated. I'm particularly interested in any strategies to ensure that memory is freed up after each commit to prevent the server from becoming unusable due to high memory consumption.
I would recommend using transaction savepoints (transaction.savepoint()) instead of full commits. This way, you retain the ability to perform a full rollback in case of an error at any point. In my experience, memory consumption should remain relatively constant during savepoints. However, this may vary depending on the specific operations being performed.
Cihan Andac via Plone Community wrote at 2024-2-22 15:13 +0000:
...
I'm currently working on a project that involves syncing over 100,000 API records with Plone items. After consulting various posts and resources on best practices for handling large data sets, I've implemented batch processing for the transactions and ensured that each batch is committed accordingly.
Despite these efforts, I'm encountering a significant issue: the memory usage on the server increases steadily after each commit, eventually leading to a point where the server becomes unresponsive. It seems that the memory allocated during the transactions is not being released even after committing the transactions.
The ZODB uses a cache. This cache must hold all modified objects.
When you finish (commit/abort) or savepoint a transaction,
the modifications are saved to a file (temporary for "savepoint") or
abondonned (for "abort") and a "cacheGC" performed, bringing the cache
size down to a reasonable limit.
Apart from this, transaction handling has not effect on memory usage.
Over Zope's Control_Panel you can get details about ZODB cache usage
(and thereby verify that your transaction handling keeps the
size of the ZODB cache under control).
There may be a lot of other reasons for increasing memory usage.
Some time ago, I helped with the analysis of a memory leak
apparently caused by zope.interface. It was revealed that
the problem actually came from a Plone component wrongly
using Implements (designed for the use with classes) on
instances thereby causing unlimited growth of an
internal zope.interface cache.
A few days ago, I have seen reports about a memory leak
involving z3c.jbot (putting acquisitions wrapper (including
a request) into a cache).
For most memory leak analysis, the reference count information
obtained via Control_Panel --> Debug Information is of help.
It can show the "top refcount" classes and the class refcount changes
with respect to a snapshot. This gives information about the classes
which get more and more instances. With luck, this points towards
the leak's cause.
@zopyx Thank you for the advice. I have utilized the transaction.savepoint() as you suggested. Being able to roll back incase of a failure would be very good.
@dieter Thank you for the explaining the memory and cache of the transactions. I was under the assumption that the transactions are raising the memory but I can see now that might not be the problem. I will definitely check the Control_Panel --> Debug Information to understand this memory leak better.