I can think of a few ways to prevent or reduce this
the link button shouldn't light up unless text is selected. Or,
the code generated when no content is selected did not include an "href", I assume because there was nothing to reference. So, when the code is
Being placed into tinymce window, do code check for 'href'. If not there raise warning to check for text to be highlighted. OR,
3). In the pop up box where you define the link , put colored text to the effect "ensure you have selected text for the link". This might help but I'll admit it has the possible side effect of making people believe that the words are a dynamic warning and means they, for a fact, did not select text.
Yeah its ux bug. If you try to do something with good intentions and it doesn't seem to work then its a ux bug.
There are lots of possible solutions but one I've seen before is to include the linked text in the dialog and make it mandatory so you can insert a link even when you haven't highlighted anything.