Column selection isn't working correctly

[Tyriar]

Repro: Alt+click and drag in the terminal

• Moving right on the same line does not make a selection
• Moving to top-right or bottom-left doesn't make a selection (this works in canvas renderer only?)
• Cursor isn't correct

Correct behavior in VS Code 1.67.0:

[Tyriar]

Should be able to write an api test for this

[silamon]

The "moving to top-right or bottom-left doesn't make a selection" isn't a regression but something that's been in like forever.
For DOM: Seems like the width of the selection element gets negative. Swapping colStart with colEnd in those cases will fix it.

xterm.js/src/browser/renderer/dom/DomRenderer.ts

Lines 334 to 341 in 48b8e31

	private _createSelectionElement(row: number, colStart: number, colEnd: number, rowCount: number = 1): HTMLElement {
	const element = document.createElement('div');
	element.style.height = `${rowCount * this.dimensions.actualCellHeight}px`;
	element.style.top = `${row * this.dimensions.actualCellHeight}px`;
	element.style.left = `${colStart * this.dimensions.actualCellWidth}px`;
	element.style.width = `${this.dimensions.actualCellWidth * (colEnd - colStart)}px`;
	return element;
	}

I couldn't reproduce the other mentioned things in xterm demo, but it definitely exists in vscode. (Notice how alt activates the menu shortcuts in the window bar.)

[Tyriar]

The "moving to top-right or bottom-left doesn't make a selection" isn't a regression but something that's been in like forever.

@silamon the gif shows it was working before?

I couldn't reproduce the other mentioned things in xterm demo, but it definitely exists in vscode

Not sure about the corner thing but if it's only in VS Code then chances are it's related to padding around the terminal elements.

[silamon]

I see that the code has changed for WebGL renderer, so you could be correct that it was working for WebGL before.

xterm.js/addons/xterm-addon-webgl/src/WebglRenderer.ts

Lines 458 to 459 in 48b8e31

	return x >= this._model.selection.startCol && y >= this._model.selection.viewportCappedStartRow &&
	x < this._model.selection.endCol && y < this._model.selection.viewportCappedEndRow;

It could be the same thing that I mentioned for DOM renderer now. startCol is higher than endCol for top-right and bottom-left and this only happens in columnSelectMode.