When a macro with optionals appeared inside the optional argument of
another one, the onscreen display and latex output were wrong. This
issue was addressed at [e8f480e7/lyxgit] by enclosing in braces macros
with optional arguments. However, this was done even when the macro
with optionals was in a non-optional argument of another macro.
This commit limits the bracing to the cases where it is really needed
and allows to address some particular issues evidenced in #11552.
After 6642152e, user macros were no longer wrapped in \ensuremath. In 2.2 and
before, InsetMathMacro behaved as follow:
* Textmode global symbols are wrapped in \text when in math.
* Other global symbols, and user macros, are wrapped in \ensuremath when in
text.
* Undefined macros (ERT) are wrapped neither in \text nor in \ensuremath.
This is also consistent with the documentation of MathEnsurer in
mathed/MathStream.h.
This patch defines InsetMathMacro::currentMode() accordingly (respectively
TEXT_MODE, MATH_MODE and UNDECIDED_MODE) and uses it to determine the output.
After this patch, there is a mismatch between screen and pdf output for user
macros in \text. This is not a regression wrt 2.2 and is because linearization
does not satisfy currentMode() currently.