Debug console: '\n' not shown as new line for evaluation returning a variablesReference

[yzhang-gh]

Issue Type: Bug

Type these code in the debug console

import numpy as np

np.random.rand(5, 5)

---

VS Code version: Code 1.33.1 (51b0b28, 2019-04-11T08:27:14.102Z)
OS version: Windows_NT x64 10.0.18898

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz (4 x 2400)
GPU Status	2d_canvas: enabled checker_imaging: disabled_off flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: disabled_software rasterization: enabled surface_synchronization: enabled_on video_decode: enabled webgl: enabled webgl2: enabled
Memory (System)	7.90GB (3.58GB free)
Process Argv	D:\Workspace\vscode-test
Screen Reader	no
VM	0%

Extensions (20)

Extension	Author (truncated)	Version
spellright	ban	3.0.38
vscode-custom-css	be5	3.0.4
log-wrapper-for-vscode	chr	1.0.2
vsc-material-theme	Equ	2.8.2
vsc-material-theme-icons	equ	0.12.0
vscode-commands	fab	1.2.5
vscode-github-notifications-bell	fab	2.1.5
vscode-highlight	fab	1.3.4
vscode-open-in-application	fab	1.0.5
latex-workshop	Jam	6.5.1
vscode-fix-checksums	leh	1.1.0
rainbow-csv	mec	1.0.0
git-graph	mhu	1.5.0
python	ms-	2019.4.12954
vscode-sort-json	ric	1.13.0
code-settings-sync	Sha	3.2.9
dictionary-completion	yzh	0.8.2
find-word-at-cursor	yzh	0.1.1
markdown-all-in-one	yzh	2.3.1
vscode-jupyter	yzh	1.2.3

[crock]

This is not an issue with VSCode. You have a new line inside a Python list. If you want each item in the list on a new line you need to use a loop and print out each line followed by the new line character.

[yzhang-gh]

I don't get the point. It is a numpy ndarray (not Python list).

>>> import numpy as np
>>> a = np.arange(15).reshape(3, 5)
>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

[roblourens]

Possibly an issue for the Python extension?

[vscodebot]

This issue is caused by an extension, please file it with the repository (or contact) the extension has linked in its overview in VS Code or the marketplace for VS Code. See also our issue reporting guidelines.

Happy Coding!

[fabioz]

What seems to be happening is that in the case where an expression is evaluated and returns a variablesReference, the variablesReference is shown the same way it's shown in the variables view (where new lines are changed for a \n representation).

Some more info from the debugger backend:

Given the following Python code (not using numpy to make it simpler):

class Foo(object):

    def __repr__(self):
        return 'aa\nbb\ncc'
    __str__ = __repr__

foo = Foo()
print('add breakpoint here')

Pausing at the breakpoint and sending foo to the repl (by typing foo in the Debug Console) the request sent to the debugger is:

{
    "arguments": {
        "context": "repl",
        "expression": "foo",
        "format": {},
        "frameId": 22
    },
    "command": "evaluate",
    "seq": 1000000016,
    "type": "request"
}

and the response is:

{
    "type": "response", 
    "request_seq": 1000000016, 
    "success": true, 
    "command": "evaluate", 
    "body": {
        "result": "aa\nbb\ncc", 
        "variablesReference": 4, 
        "type": "Foo", 
        "presentationHint": {}
    }, 
    "seq": 42
}

So, you can see that the result has proper new lines and VSCode chose to show it in a single line and not expanded in the debug console (so, there's not much to be done on the extension as it seems to be doing the proper thing).

So, this would be a request (and not a bug since it seems it's deliberately doing the same thing done in the variables view) for VSCode to show new lines expanded in the debug console for a given variablesReference.

As a note, if print(foo) is done, the result is shown expanded -- it'll actually return a variablesReference to None in that case as that's the return of print(foo) but then the representation will be printed as a string afterwards with proper new lines.

[fabioz]

A better title for this issue is probably:

Debug console: '\n' not shown as new line for evaluation returning a variablesReference.

-- note that if variablesReference==0 in the return then the Debug Console properly shows the result expanded.

[isidorn]

I pushed a fix for this. Please try it out in monday's vscode insiders and let me know if it behaves better for you. Thanks
https://code.visualstudio.com/insiders/

[jeanp413]

In my opinion, the initial behavior with the newline characters escaped is the correct one because that value is just a (one-liner preferably) preview representation of a composite object that you can click to expand the tree node.
With 1113272 applied now I'm getting this while debugging node which feels odd.
Before:

After:

Also I could argue that the output from the repl in the terminal doesn't need to match the output in the debug console for example in node:
From terminal

> Array.from({length: 10}, () => Array.from({length: 5}, () => Math.floor(Math.random() * 100)))
[ [ 39, 42, 89, 80, 35 ],
  [ 59, 34, 2, 99, 30 ],
  [ 48, 15, 39, 73, 91 ],
  [ 77, 27, 14, 52, 87 ],
  [ 87, 68, 69, 98, 54 ],
  [ 18, 55, 87, 90, 82 ],
  [ 13, 82, 74, 62, 65 ],
  [ 59, 62, 67, 65, 4 ],
  [ 31, 99, 56, 45, 38 ],
  [ 35, 22, 84, 66, 13 ] ]

Debug Console:

So I think this should be fixed in the python extension.
@isidorn @fabioz

[fabioz]

As a note, in the Python extension the user could already see inside as a structured object as you're doing in node (so, there's nothing to actually be fixed in the Python extension, that behavior already existed), and if it wasn't a structured object then it'd already appear in multiple lines.

I think it's more about expectations... when using an interactive console in Python users are used to see things printed (so, in the terminal using Python interactively if you wanted to see the \r\n\t of the numpy array you'd do a repr(arr) instead of just arr as just arr would show the new lines expanded)...

Now, this change empowers extensions to do as they please (if the node plugin wants, they can show the actual \n char and not put a new line there by doing a replace('\n', '\\n') if an evaluate request was done for the "repl" context for a structured object), whereas without this patch the behavior can't be customized by the extension.

I personally prefer the current version instead of having to click through to see things in the terminal, but I also agree it's pretty much a matter of personal taste.

[jeanp413]

I agree its a matter of taste, but I have a few observations:

1. (I think) vscode debug console is modeled after the chrome developer tools console where only primitives get its real value printed, and composite objects are showed as interactive tree hierarchies where tree node values are (one-liner) preview values containing ellipses ... when the preview is too long.
2. Python, Node, etc interactive console are just a stream of text (doesn't have the ability to collapse or expand a composite variable value) that pretty print the value of variables, this could be really long string for example printing an array of 1000 elements.
3. We are mixing 1 and 2, showing the pretty-printed value from 2 as the preview value for the tree node which in my opinion doesn't look good.

So I think we should print variable values following 1 or 2 but not mixing both.

[isidorn]

Both points are valid. And yes we were inspired by the chrome dev tools console.
However when there is no clear incentive to change behavior I prefer to leave it as it is in the current stable.
Due to that reverting my commit and reopening this issue.

[yzhang-gh]

Thank you for all the discussion. (Sorry for being late to reply)

However, it is sad to see this change being reverted. We were in the right direction—Let the extension determine what to show (i.e. the repr function), rather than arbitrarily decide to escape certain characters \n, \t etc.

The original motivation for the escape is "we use \n and \t in the variables view to save on space".

• For this purpose, this escape behaviour doesn't do its job. For "an array of 1000 elements", escaping \n doesn't make things better. This is even not as effective as direct cutting the long string and appending ....
• Even worse, this behaviour prevents potential pretty-print with \n. And there is no workaround.
  (Strictly speaking, Python extension can replace \n with \\n in advance as we know what vscode will do next. I believe you agree this adaption is weird)

---

Next, I will address the specific concerns mentioned above.

Previously Object {test6: "\n\n\n\n"} , now ugly non-inline string

As @fabioz said, this can (IMHO, should) be handled by the node extension (replace('\n', '\\n')).
The old behaviour looks right because vscode happened to do it for the node extension. It is not the duty of vscode.

"argue that the output from the repl in the terminal doesn't need to match the output in the debug console"

Making them matched is not my request. The point is, "Now, this change (do not escape) empowers extensions to do as they please, ..., whereas without this patch the behaviour can't be customized by the extension"
(@fabioz's comment is so clear)

The mixing of Chrome dev console and Python/Node interactive console

It has nothing to do with the Python/Node interactive console. It is all about the repl/evaluate function which is responsible for generating presenting text for non-primitive values. Python also uses ellipses if needed. (shown below a 100x100 NumPy array/matrix)

>>> np.random.rand(100, 100)   
array([[0.99050621, 0.75406745, 0.09582186, ..., 0.29125127, 0.24070286,
        0.35042311],
       [0.50863412, 0.78621844, 0.29135548, ..., 0.70344945, 0.39070587,
        0.95326837],
       [0.04844976, 0.1526166 , 0.21778417, ..., 0.95142566, 0.37265694,
        0.21514417],
       ...,
       [0.16014808, 0.87034416, 0.10492746, ..., 0.74535543, 0.77259506,
        0.87129383],
       [0.07699896, 0.57801489, 0.50589809, ..., 0.55967099, 0.00415744,
        0.95969444],
       [0.80943893, 0.78358082, 0.56240486, ..., 0.97081048, 0.80283494,
        0.81190555]])

The text can actually be array([[...], ..., [...]]) or click to expand or anything if the debugger extension wants.

---

To summarise, currently, vscode do an extra \n\t escape after the evaluate call.
This doesn't make much sense as (1) it cannot do what is expected and (2) takes over the duty of the debugger extension

[jamesstidard]

From a practical point of view, it's vary hard to inspect the python data science structures from numpy, xarray, and pandas in VSCode at all at the moment. The structures are too complex for the Debuger's variable inspector to really be that productive to use.

Python encourages for these things to produce formatted strings to be more human readable, which these libraries do. So even if it's decided that it's best that the debug console not allow this, there should be some way to get to this representation in a VSCode debug session. Have even tried to print(my_array) and switching to the terminal tab to try and get some clue as to the shape of these data structures.

This feature: https://github.com/microsoft/vscode-python/issues/5113 may end up solving this, but I don't believe it is currently possible to use this in debug or would be available that soon.

Mainly feel that leaving it to the extension to decide, as mentioned above, so it can be done on a per-language basis would be great. And I think it does in python.

Also, it is a pain point in using VSCode with these data science libraries and debugging data structures, when all you are looking for is if a matrix has values in the diagonal. So if the opposition to this is somewhat notional, I think the user experience here may trump that.

Cheers.

[vscodebot]

This issue is being closed to keep the number of issues in our inbox on a manageable level, we are closing issues that are not going to be addressed in the foreseeable future: We look at the number of votes the issue has received and the number of duplicate issues filed. More details here. If you disagree and feel that this issue is crucial: We are happy to listen and to reconsider.

If you wonder what we are up to, please see our roadmap and issue reporting guidelines.

Thanks for your understanding and happy coding!

[isidorn]

Clossing as out of scope.
Please provide the feedback to the Python extension. Then they can think about a solution for the issue and they can dicsuss it with us (vscode team) on how to proceed. Thanks for understanding

[teichert]

Maybe off-topic, but for those of us that want to pause the python debugger in VS Code and then pretty-print numpy arrays (or other expressions containing newlines required for legibility) from the debug console, one available work-around is to print the expression to a text file (which could conveniently be kept open for viewing in a separate editor pane).

For example, to print numpy array variable a to debug.txt with newlines rendered as expected, execute this from the debug console:

with open('debug.txt', 'w') as out: print(a, file=out)

Or append to the end of the file:

with open('debug.txt', 'a') as out: print(a, file=out)

[isidorn]

Thank you very much for your feedback.

After discussing with @weinand we came to the following conclusion:

The VS Code frontend should not escape newlines. It is up to the debug extension to decide how the data should be presented and the debug extension can decide if a newline shuold be escaped or not.
VS Code is agnostic with regards to the data it is showing, and the extensions are already adding quotes to strings and other to make the data more presentable.

The downside of this approach is that in the Variables view the multine values will from now on only show the first line of the value. However it is up to the debug extensions to change their responses to escape new lines. Due to that we plan to make this change start of next milestone so extensions are given enough time to adopt. Thus reopening.

We do not want to add a setting for this behavior as it does not feel right for the user to control this and adding a capability also requires adoption by extensions (a similar work as our escaping approach requires).

@weinand please add more details or correct if I captured something wrongly from our discussion
fyi @connor4312 @roblourens

[yzhang-gh]

Thanks for the update.

[isidorn]

As announced in my previous comment we have changed that VS Code debugger is no longer escaping whitespaces. Now it is up to the debugger to handle this on their side.
I pushed this early in the milestone so we catch potential corner cases where this new strategy does not work and to give enough time for debuggers to adopt this.

fyi @roblourens @connor4312

[DanTup]

The downside of this approach is that in the Variables view the multine values will from now on only show the first line of the value. However it is up to the debug extensions to change their responses to escape new lines.

If a debugger decides to "escape" newlines here, won't it break the "Copy value" functionality? The user will then end up with a string with these characters escaped, rather than literal?

[isidorn]

@DanTup No, I do not think so. Since the copy value functionality invokes the evaluation with the context variables. So the debug adapter can distinguish this on their side.

[DanTup]

@isidorn but the above text specifically mentions variables:

The downside of this approach is that in the Variables view the multine values will from now on only show the first line of the value

In order to fix this, the DA needs to escape newlines to \n when it gets an evaluateRequest for variables, and that means Copy Value will get the escaped version too. The DA cannot distinguish the copy requests from display (though supporting this has come up before - but been rejected - at #81051 (comment)).

[actboy168]

I agree that Copy Value should have its own context. A year ago, Copy Value still used null as its context. Although it is not documented, at least it works. I have also requested that it be documented #69768.