Always use LF instead of CRLF for *.ipynb files regardless of platform and when sending code to the kernel

[claforte]

Environment data

• VS Code version: 1.53.0-insider (2021-01-29T05:13:48.533Z)
• Jupyter Extension version (available under the Extensions sidebar): v2020.12.414227025
• Python Extension version (available under the Extensions sidebar): 2021.2.518958983-dev
• OS (Windows | Mac | Linux distro) and version: Windows 10
• Python and/or Anaconda version: Python 3.8.5 using Anaconda
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
• Jupyter server running: Local

Expected behaviour

When editing a notebook, lines ending in "\n" shouldn't be modified to "\r\n"

Actual behaviour

It appears as soon as I modify the content of a cell, the editor is replacing all "\n" by "\r\n". I don't understand why, and it's causing downstream tools (e.g. nbdev/nbparse) to no longer be able to parse the .ipynb properly.

Steps to reproduce:

This seems to happen for any .ipynb file on my PC. I did a git clone of nbdev (https://github.com/fastai/nbdev/blob/master/nbs/00_export.ipynb) and ran the notebook, but chances are you only need to open a file:

1. download https://github.com/fastai/nbdev/blob/master/nbs/00_export.ipynb
2. open the file in vscode
3. modify a cell
4. do a diff on the cell.
5. notice that the newlines are messed up.
   

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

User belongs to experiment group 'pythonJoinMailingListVar1'
User belongs to experiment group 'jupyterTest'
User belongs to experiment group 'NativeNotebookEditor'
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py -c "import jupyter"
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py -c "import notebook"
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py jupyter kernelspec --version
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.jupyter_daemon -v
> D:\Users\claforte\Anaconda3\envs\tensorflow\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py -c "import ipykernel"
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py -c "import ipykernel"
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v
Started kernel Python 3
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v
> D:\Users\claforte\Anaconda3\envs\fastai\python.exe c:\Users\claforte\.vscode-insiders\extensions\ms-toolsai.jupyter-2020.12.414227025\pythonFiles\pyvsc-run-isolated.py vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_launcher_daemon -v

XXX

[joyceerhl]

Hi @claforte, does toggling the following VS Code setting stop the line ending conversions? You can get to this UI with Ctrl+Shift+P to open the Command Palette, then type 'Open Settings (UI)'.

[claforte]

Hi Joyce,

It was set to auto already. I'm not sure how you want me to toggle it, but I just tried setting it to \n and modified a few code cells. It seems some code cells get stuck to \r\n no matter what files.eol value I set, while other cells are stuck at \n.

[joyceerhl]

If you open a freshly cloned ipynb after setting the files.eol setting to your preferred setting, do the endings still get converted?

The default line ending on Windows is '\r\n' (this is most likely what you are getting with the 'auto' setting), so I believe that is what VSCode is applying to your files. I don't believe changing that setting results in existing '\r\n' being converted to the new '\n', so unfortunately I think you'd need to do a find and replace on the files that have already been converted.

[claforte]

Hi Joyce, I tried what you suggested (set vscode-insiders eol to \n, manually replaced all \r\n by \n in notepad++, restarted vscode-insiders, modified a cell, saved) and it looks like no new \r\n appeared.

Still it would be great if vscode-jupyter made sure it never added \r\n at the end of code line, since Jupyter notebook, AFAIK, never uses \r\n, and a lot of tools probably assume that.

Thanks,

Christian

[joyceerhl]

Great to hear the setting change stops the conversions.

Thank you for highlighting this. From related issues like jupyter/nbconvert#1062 it does seem that Jupyter notebooks tend to default to LF even on Windows. The files.eol setting is contributed by VS Code, but perhaps one solution here would be for the Jupyter extension to override the default VS Code setting specifically for ipynb files. However, I don't believe there is currently a way to control files.eol based on file type, so this may require upstream changes in VS Code. Our team will discuss this at our weekly triage meeting.

[claforte]

Thanks a lot for your responsiveness Joyce, I appreciate it!

[DonJayamanne]

However, I don't believe there is currently a way to control files.eol based on file type, so this may require upstream changes in VS Code. Our team will discuss this at our weekly triage meeting.

The Jupyter extension is responsible for writing out the contents of the file, not the user. Hence we are in full control over whether we use \n or \r\n. I don't see any need for a VS Code update here.

@joyceerhl /cc

[msveshnikov]

This is honestly very annoying behavior (I mean conversion of LF to CRLF). After such a notebook uploaded to Jupyter Lab server, I start to have incorrect character typed. This is another bug of Jupyter Lab, which unfortunately destroy your DEV experience. See here:
jupyterlab/jupyterlab#2951
(bug reproduced in latest Jupyter Lab)
So basically after editing of a notebook in VSCode you have destroyed notebook in any external server!

[orekides]

So basically after editing of a notebook in VSCode you have destroyed notebook in any external server!

exactly true !!!

[mburghart-qualitrol]

I have used several different jupyter notebook editors. When running "jupyter notebook" on both Windows and linux based systems, the notebook output consistently uses "\n" in the double-quoted lines of each cell -- not "\r\n". The only editor that I've seen output "\r\n" is The VS Code jupyter notebook extension. This inconsistency causes headaches when diff'ing notebook file versions stored in git repositories.

The issue is not the same as the common problem of Windows based systems insisting upon ending text file lines with the control characters \r\n and other systems just using \n. (git has features to help cope with this, which you are probably aware.) So, in my opinion, the "File:EOL" setting in VS Code should not be used to control the line ending characters of the notebook cell double-quoted lines. I think the simple answer is to just follow the convention of "\n" to represent line end in the double-quoted lines.

[dynamicwebpaige]

The only editor that I've seen output \r\n is the VS Code Jupyter notebook extension. This inconsistency causes headaches when diff'ing notebook file versions stored in git repositories.

Is there any way that we could prioritize resolving this bug? Destroying the diff makes it impossible to sanely understand changes to notebook files during code review -- which will be even more important as we encourage folks to use version control for their .ipynb files.

[joyceerhl]

Users can configure the files.eol setting as a temporary workaround, and we can consider fixing this in the builtin ipynb extension. Note also that Colab always uses LF (they just fixed this behavior) jupyterlab/jupyterlab#9465 (comment) /cc @roblourens @DonJayamanne

[DonJayamanne]

Sound like somethign for debt week.
Basically we need to remove all CR from the text we get in the code cells.

[cdeil]

This is causing problems for many Jupyter users: jupyterlab/jupyterlab#2951

Can you please fix this?

[jasongrout]

FYI, it looks like microsoft/vscode#133762 fixed the line endings in cell sources in ipynb files.

[rchiodo]

/verified

Tried with multiple notebooks. Existing and new. All only have LF in them:

[mburghart-qualitrol]

The verification doesn't appear to be testing the end-of-line representation for each line of notebook cell. You would need to make a notebook cell with at least two lines to see this. Attached is a small notebook file with two cells each with multiple lines. (File name extension changed to txt to allow upload.) This was created using Visual Studio Code 1.60.2 with the Jupyter extension v2021.8.2041215044. Note that the representation of all but the final cell line of each cell ends with "\r\n". This illustrates the undesired behavior.

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "This is an example notebook with a markdown cell and python code cell. Each cell\r\n",
    "has multiple lines to illustrate how end-of-line is represented in the `ipynb`\r\n",
    "file for each line in a cell."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [
    "print( \"Aardvarks are first line.\" )\r\n",
    "print( \"Badgers are next in line.\" )"
   ],
   "outputs": [],
   "metadata": {}
  }
 ],
 ...
}

endofline_test ipynb.txt

[rchiodo]

@mburghart-qualitrol my screenshot was just to show no /r in the file happened to have open. It works with multiple lines in a cell too:

[RoyiAvital]

I many cases the files itself looks like one line, is there a way to tell VS Code to parse it correctly?

[rchiodo]

@RoyiAvital sorry not sure what you're saying. That the json for your ipynb file has no linefeeds in it?

[RoyiAvital]

@rchiodo , Indeed. Any way to enforce VS Code to save it a multi line manner?

[rchiodo]

Any way to enforce VS Code to save it a multi line manner?

You mean your code cells all have a single line in them? What does the ipynb look like?

Cells usually have a format like so:

{
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "\n",
    "a,b,c = 'hihi', os, tf.__version__\n",
    "print(a, b, c)"
   ]
  },

And I believe you're saying your notebook looks like so?:

{    "cell_type": "code",   "execution_count": null,   "metadata": {},   "outputs": [],   "source": [    "import tensorflow as tf",  "import os",    "",   "a,b,c = 'hihi', os, tf.__version__",  "print(a, b, c)"   ]  },

Or does it have the \n in the source lines?

[RoyiAvital]

It is one liner like this:

While on VS Code it looks as expected.

[rchiodo]

Are you saying VS code can't open that file? It looks fine to me. If it doesn't open then the JSON is likely invalid.

Or are you asking VS code to format it so that it isn't one line? It shouldn't be doing that. Although that would be an issue on VS code itself, not the jupyter extension.

I believe the code is here:
https://github.com/microsoft/vscode/blob/85a33a14cf694e55d85d4d3dda55a2cca1a37980/extensions/ipynb/src/notebookSerializer.ts#L97

[RoyiAvital]

@rchiodo , Can open and display it correctly.
Just when looking at the RAW file (Using text editors) it is one liner.

I wonder abut 2 things:

1. How did it happen? Why did VS Code saved it like that?
2. Is there a way to make it save again with all lines separated?

[rchiodo]

@RoyiAvital what is your version of VS Code and the jupyter extension?

We might have at one time saved the file as a single line, but the code that saves a file now should write it out with linefeeds.

[RoyiAvital]

My version is VS Code 1.65.2 with Jupyter Extension 2022.2.103. Both are the latest stable versions.

[rchiodo]

I'm creating a new issue. This shouldn't be happening:
#9491