Slightly hard to decypher your question. But if I've understood correctly, you want to change visuals on the settings panel of your custom node in the Editor based on validating the user input?
Firstly note that the editor already changes the border colour to red for invalid fields. However, that is fairly subtle.
To do more, note that you have access to jQuery in the Editor. So you can use the features of jQuery to dynamically set CSS and/or STYLE for any element. You can also use jQuery to show/hide elements or even dynamically add/remove them.
↑ This works for both ACE and MONACO ​- without any changes (the ACE mode is automatically translated into the correct value for MONACO). It is down to how the users settings.js is configured. If the user has chosen ACE (default) then an ACE editor is created. If the user has MONACO set, then your code will create a MONACO editor.
Again, you should avoid this.
where as RED.editor.createEditor is a node-red API and therefore OK to call.
However, there is no real reason to do this. Set the node property that the code is saved in to be a required property - then the node-red editor will place an error badge on the node.
Likewise, you could have some logic in the oneditsave callback that checks the value and alerts the user.
If you MUST you could simply set the style of an outer DIV or something else.
Final comments:
As you have selected text mode - why even bother with ACE or MONACO - why not use a simple textarea element. Then you can set its colour / style as much as you want.