Import .xls or .json file to create a "parts catalog"?

1

I'm looking at options to replace an existing (horribly written in js/node) enterprise custom platform for a manufacturing supply chain startup. They have a large number of .xls or alternatively .json files with a list of the parts for the parts catalog (testing with just 15,000 to 30,000 parts per file, but base testing down the road is with 10,000,000+ part numbers). It is not a shopping cart, it is just a catalog list with columns that include descriptions of the parts, pricing and availabilty with links to different sites to go to, to get the parts on those sites.
I am hoping to get them away from the horrible custom coded platform they are having so much pain from, and wondering if Plone might be able to offer an easier approach?
Is there any quick and dirty way for me to quickly take one of their .xls or .json parts catalog files and import into Plone to create a bunch of "parts" pages, or at least show them as a list on a web page that can be sorted in different ways? Years ago I seem to recall doing something like this with a book catalog using Plone 3 or 4 wish .csv files, but most of the sites we use we have migrated away from Plone since version 5 was so stripped down.
Is there a built-in way, or a community add-on that would make this a quick proof of concept just to import and then show (perhaps in a collection?) the content generated by the .xls, .json (or could be .csv as well).
Appreciate any suggestions.
Thanks!

2

UPDATE: The below was just for the 'just create a bunch of pages to show them'

Since nobody has answered:

There is a quick and dirty way of doing what you want with just copy/paste.

  1. When an add-on is installed, it usually runs a file 'setuphandlers.py'.
  2. If you make your own add-on (with plonecli) you can edit this file and do all the import there
  3. If you dont want to make your own add-on, you can add the code to an existing add on and just remove it after you have installed it.
  4. In some text editor, you search / replace so it looks like I have in 'kontent'
  5. Then you add something like this to your setuphandlers.py-file

PS: If you need to images (instead of just linking to them, I have the code for that somewhere.

    # -*- coding: utf-8 -*-
    from plone import api
    #import os
     

    def post_install(context):
        """add stuff etc"""
        portal = api.portal.get()
        _create_content(portal) 

     

    def _create_content(portal):
        """Lets create the content"""



        kontent = [
              {  
            """id""": """de-dodes-byra-1""",
            """title""": """De dødes byrå""",
            """another_field""" : """2019""", },
                      {  
            """id""": """id2""",
            """title""": """Title2""",
            """another_field""" : """2022""", }


        ]




        for i in kontent:
            if not portal.get(i, False):
     

               some_thing = api.content.create(
                    type='mytype',
                    container=portal,
                    id=i['id'],
                    title=i['title'],
                    another_field=i['another_field']
                )
3

You could take a look at collective.exportimport · PyPI
Training here: Migrating with collective.exportimport – Migration best practices — Plone Training 2022 documentation

That said, I would probably use a relational database to store 10+ million content items. Have no experience with Relstorage as a Plone backend, maybe others can chime in?

4

Plone is the wrong tool as storage backend.
Such data belongs into a suitable database (like a document storage).
Plone's backend is clearly too slow for handling this amount of data in a reasonable way in daily production.

Options:

  • convert your data into a reasonable, unified JSON format
  • import your JSON documents into a document storage (DocumentDB, MongoDB, ArangoDB....many options)
  • use the query capabilities of the related document database for querying your data

A relational DB is of course suitable (e.g. Postgres with JSON support)...

1 Like