Fix bad cursor positioning when entering an inset

The test added at 359aef92 was incorrect. Actually, Inset::editXY
returns the inset when it is edited and also when it is not, which
can be confusing. So if we are unlucky and the slices positions in the
text area and in the inner inset match, the code below triggers and
many bad things can happen.

Also improve the documentation of Inset::editXY and TextMetrics::editXY.

Fixes #10691.

(cherry picked from commit c9fefd5a2c)

src/TextMetrics.cpp

@@ -1408,7 +1408,9 @@ Inset * TextMetrics::editXY(Cursor & cur, int x, int y,
 
 	// Try to descend recursively inside the inset.
 	Inset * edited = inset->editXY(cur, x, y);
-	if (edited == inset && cur.pos() == it->pos) {
+	// FIXME: it is not clear that the test on position is needed
+	// Remove it if/when semantics of editXY is clarified
+	if (cur.text() == text_ && cur.pos() == it->pos) {
 		// non-editable inset, set cursor after the inset if x is
 		// nearer to that position (bug 9628)
 		bool bound = false; // is modified by getPosNearX

src/TextMetrics.h

@@ -180,7 +180,9 @@ public:
            that is completely visible
 	\param up whether we are going up or down (only used when
            assert_in_view is true
-	\retval inset is non-null if the cursor is positionned inside
+	\retval inset is null if the cursor is positioned over normal
+	       text in the current Text object. Otherwise it is the inset
+	       that the cursor points to, like for Inset::editXY.
 	*/
 	/// FIXME: cleanup to use BufferView::getCoveringInset() and
 	/// setCursorFromCoordinates() instead of checkInsetHit().

src/insets/Inset.h

@@ -179,7 +179,13 @@ public:
 	/// cursor enters
 	virtual void edit(Cursor & cur, bool front, 
 		EntryDirection entry_from = ENTRY_DIRECTION_IGNORE);
-	/// cursor enters
+	/// sets cursor recursively descending into nested editable insets
+	/**
+	\return the inset pointer if x,y is covering that inset
+	\param x,y are absolute screen coordinates.
+	*/
+	/// Note: this method must preserve the selection status. See:
+	/// https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg199001.html
 	virtual Inset * editXY(Cursor & cur, int x, int y);
 
 	/// compute the size of the object returned in dim

status.22x

@@ -69,6 +69,8 @@ What's new
 
 - Fix glacial performance with ancient Hebrew text on macOS.
 
+- Fix potential crash when cursor enters an inset (bug 10691).
+
 
 * INTERNALS