Method/adapter returning the indexable content for SearchableText?

zopyx · January 27, 2022, 3:33pm

Is there some helper method or an adapter that would return indexable content of an arbitrary content object? In particular, I am looking for the method that what the data that goes into the SearchableText index. But it's not the SearchableText() default implementation in CMFCore which only returns title + description.

tisto · January 27, 2022, 4:00pm

The SearchableText indexer is in plone.app.contenttypes:

github.com

plone/plone.app.contenttypes/blob/master/plone/app/contenttypes/indexers.py#L51

    
      
                          value = value.encode('utf-8', 'replace')
                      if value:
                          result = ' '.join((result, value))
                  else:
                      if isinstance(value, six.binary_type):
                          value = safe_unicode(value)
                      result = ' '.join((result, value))
              return result
          
          

          
def SearchableText(obj):
              text = u''
              richtext = IRichText(obj, None)
              if richtext:
                  textvalue = richtext.text
                  if IRichTextValue.providedBy(textvalue):
                      transforms = getToolByName(obj, 'portal_transforms')
                      # Before you think about switching raw/output
                      # or mimeType/outputMimeType, first read
                      # https://github.com/plone/Products.CMFPlone/issues/2066
                      raw = safe_unicode(textvalue.raw)

Though, IMHO we need to refactor the entire concept of how we index stuff. Volto has new needs (indexing blocks content instead of RichText) and indexing servers (Solr, ES, etc) index content in a more sophisticated way than just dropping everything into SearchableText. IMHO SearchableText is something that should be easy to delegate to an index server (or drop entirely because it is not needed at all in those scenarios).

zopyx · January 27, 2022, 4:04pm

Thanks, this helps.

In zopyx.typesense, we work similar to Plone with a title index and an index for the text.
But is would be easy to configure a more smart schema e.g. which could include all texts from headlines in order to rank hits in headlines higher than in the body but lower than in the document title.

jensens · January 28, 2022, 9:29am

In collective.elastic.plone/collective.elastic.ingest we managed it the other way around, we use the plone.restapi to (async) fetch the complete data of an object, then index it in ElasticSearch and finally ZCatalog can have one or more ElasticSearchProxyIndex indexes configured to search as needed for the site (like which weights, fields, ... -> the full ElasticSearch possibilities).