Since it is (now) possible to disable the toolbar (by settings), it can be done with a viewlet and a little bit of CSS and/or javascript. (those groups will not have permissions to edit portlets, add content or change display anyway:
I did a quick test, and this works
<browser:viewlet
…
class="plone.app.layout.viewlets.common.ContentViewsViewlet"
template="mycontentviews.pt"
…
/>
<tal:contentviews tal:define="ploneview context/@@plone;"
tal:condition="not:ploneview/showToolbar"
i18n:domain="plone">
<div class="my-contentmenu">
<div tal:replace="structure provider:plone.contentmenu" />
</div>
</tal:contentviews>