Terrain sets do not work properly with staggered isometric maps

[AtlaStar]

As the title says, a misalignment occurs when using the terrain brush causing the tiles to not properly conform to the expected shape while using a corner set.

Steps to reproduce the behavior:

1. Open the isometric_grass_and_water.tmx
2. Change the map properties to Isometric (staggered)
3. Try to use the premade corner set with your Terrain brush.

This is what happens

This is what should happen

Tiled Version: built from master, using Qt 6.2.4

Issue appears to be related to the WangBrush::mouseMoved function, as for corner and mixed sets it tries to perform some additional logic when a staggered renderer is being used. Simply removing that logic fixes the issues with the corner set, and appears to fix the issues with a mixed set as well (with limited testing, likely requires more)

[bjorn]

2. Change the map properties to Isometric (staggered)

You seem to have switched the map to Hexagonal (Staggered) instead. Then I can reproduce the problem. But the fix in db0a488 does not look correct in the general case, unfortunately.

[AtlaStar]

You can see in the above screen shot that it is on an isometric map, as you can see the iso grid lines. That said, I am wondering if there is a deeper issue then as it makes little sense that the issue would appear in my environment but not in yours.

[eishiya]

(I mentioned this on Discord, but I'll post it here just for completion's sake.)
It's not possible to tell whether a map is iso or hex just from looking at the grid, because a staggered hex map with Side Length set to 0 looks identical to a staggered isometric map with the same tile size - but it behaves differently for the purposes of Terrains because its neighbours are counted differently - and incorrectly, since hex tiles have 6 sides, but Terrains currently only support 4 sides per tile.

So, please verify that your test map was actually Isometric (Staggered) and not Hexagonal (Staggered).
For me, in 1.10 and in a nightly from today (both Qt6), terrains work just fine for Isometric (Staggered) maps. I also remember this working fine in older Tiled versions (Qt5).

[AtlaStar]

As I mentioned in the Discord, I retested and wasn't encountering the same issue, as previously the nightly I had built would not even render staggered hex maps with a side length of 0, suggesting I had some weird environment issue causing the problems previously.

So this does not appear to be a bug with staggered isometric maps as I initially thought.

[bjorn]

In #3624 I've proposed a change that fixes the behavior for Hexagonal (Staggered) maps when the Hex Side Length property is 0 (so, for hex maps that actually look like isometric maps). Full support for hex maps will be more involved, but this may also be a small step towards that.