Can one use the TinyMCE editor in Plone 5 to write a document using latex
I doubt that there is something working out of the box for Plone 5.
In general MathJax is the way to go.
It should not be that hard to register a dedicated paragraph style with TinyMCE for MathJax,
adding MathJax JS files to the Plone resource registry and a renderer for MathJax that is triggered by the related paragraph style.
The math-part of latex works with Plone
The rest of latex is probably not 'too interesting anyway'.
I have been using 'customized versions of mathJax (because I could not get collective.mathjax to work with PDF export and ebooks), but for 'screen use', collective.mathjax should work
If you want to 'use other stuff from latex', you probably don't want to use TinyMCE at all, but for example markdown (or maybe xml ?))
If so, I could need some help to finish this project
If you want to write 'live math with latex syntax', you could take a look at stackedit ( https://stackedit.io )
How could I help here.
Most of all: it could need a way to upload images.
If images could be dropped directly in the editor (like it can be done in the editor I am writing this in ) it would be great.