Environments should be put on their proper lines in input

[dbitouze]

Environment

• TeXstudio: 2.12.16 (git 2.12.16-152-gfae4b3ef)

Feature request

Every environment (properly used, e.g. tiny environment shouldn't be used) I can think of puts its their content on proper lines in output so every environment should be put on their proper lines in input. For instance, consider:

test foo bar

and suppose you decide to center "foo". With TXS, you select it and click on the relevant button and now you get:

test \begin{center}
foo
\end{center} bar

which is less readable than:

test
\begin{center}
foo
\end{center}
bar

More precisely, if it is possible for TXS to know if what is selected (possibly empty) is the whole line, I would suggest to put the \begin{env} on a new line only if it is not the case. To be more specific:

• Empty preselection: the environment is inserted on the current line.

  0:|
  1:text
  

  results in:

  0:\begin{env}
  1:content
  2:\end{env}
  3:text
  

• Whole line (here, "content") preselected: the environment is inserted on the current line.

  0:content
  1:text
  

  results in:

  0:\begin{env}
  1:content
  2:\end{env}
  3:text
  

• Subset of line (here, "content") preselected: the environment is inserted on a new line.

  0:foo content bar
  1:text
  

  results in:

  0:foo
  1:\begin{env}
  2:content
  3:\end{env}
  4:bar
  5:text
  

[sunderme]

I am a bit reluctant to force the user to have a newline.

[dbitouze]

I am a bit reluctant to force the user to have a newline.

Do we agree this:

• has no impact on the resulting document?
• is much more readable?

[MeanSquaredError]

abc def ghi could be turned into

abc
\begin{center}
def
\end{center}
ghi

but it could also be turned into
abc \begin{center} def \end{center} ghi

It really depends on the width of the screen where the user edits the tex source and his own taste.
IMHO for really short text like in the example above the extra newlines are unnecessary.
So ideally it should be somehow configurable. Maybe select one of several possible layouts from a drop-down or maybe use some kind of smart heuristic in the code to decide how many newlines are necessary and where to put them. Or maybe allow the user to enter some kind of a template. Choosing of a list of possible layout is probably easier to implement and more fool-proof.

But it must be designed well with regard to usability in order to avoid complicating things and making them more difficult for the user.

[MeanSquaredError]

I think it is better not to hurry with the implementation but rather decide on a nice design (usability-wise)

[dbitouze]

@MeanSquaredError

but it could also be turned into
abc \begin{center} def \end{center} ghi

As I said: "Every environment [...] I can think of puts its content on proper lines in output so every environment should be put on their proper lines in input."

So, if abc \begin{center} def \end{center} ghi is possible, such an input is much less readable and less evocative than:

abc
\begin{center}
def
\end{center}
ghi

which is closer from the output.

It really depends on the width of the screen where the user edits the tex source

Indeed and the latter code is okay for screens of large and small widths.

[It really depends on] his own taste.

Indeed, and I guess the current behavior isn't to the taste of many users who have to manually rearrange the resulting code to make it more readable.

So ideally it should be somehow configurable.

Why not. But, IMO, the layout I suggest should be the default, and such a configuration should be considered as a (very) advanced option.

[MeanSquaredError]

TeX does not have a one to one correspondence between the formatting of the source and the formatting of the compiled document. IMHO many authors don't really bother about the formatting of the source and just prefer the document source to be short, so having everything just on one line should be an option too.

I think it would be better to avoid specialized changes and to have a well thought-out and configurable approach to formatting the source.

So maybe this should be added an enhancement tag but implemented in more generic fashion once we have a clear idea how to organize everything to make it usable and generic.

[muzimuzhi]

Do we agree this:

• has no impact on the resulting document?
• is much more readable?

First, for very rare cases (mostly some verbatim environments), a newline before or after the \begin{envname}/\end{envname} do has impact on the resulting document. For example,

• \end{comment} from comment package must appear on a line of its own. No starting spaces and nothing after it.
• \end{Verbatim} from fancyvrb package ignores anything before it and raises error for anything following it in the same line.
  Just a tex-nical reminder. I don't suggest to take this into consideration too much.

Second, there are more details to consider: If we expect

% use >>text<< to express that "test" is selected
abc >>def<< ghi

->

abc<space>
\begin{center}% % insert two newlines, before and after
  def%          % add indent or not?
\end{center}%   % insert two newlines
<space>ghi

then do we also expect

abc
>>def<<
ghi

->

abc
\begin{center}% % insert only one newline
  def
\end{center}%   % insert only one newline
ghi

Also, I checked how does latex-workshop, a popular latex extension for editor visual studio code, implement the similar utility (source code here and documentation here):

"wrapEnv": {
  "body": "\n\\begin{$1}\n\t${TM_SELECTED_TEXT}\n\\end{$1}",
  "description": "Wrap selection into an environment"
},

This shows that latex-workshop inserts newline before and after \begin{envname} and before \end{envname}, but not after \end{envname}.

The reference from latex-workshop suggests this needs some (user-experience) design work and relates to tastes.

(This comment should have been posted several hours ago, if I have not mistakenly closed my browser tags.)

[dbitouze]

TeX does not have a one to one correspondence between the formatting of the source and the formatting of the compiled document.

I agree. But, IMO, having an input evoking the output helps when one needs to revisit his source. And there is a famous example of (I'm pretty sure) intentional input evoking output provided by (La)TeX: empty line(s) being an alias of \par

IMHO many authors don't really bother about the formatting of the source

Well, I'm not sure. Or if they don't bother, they fail to realize this complicates their lives :)

I think it would be better to avoid specialized changes and to have a well thought-out and configurable approach to formatting the source.

Why not, but this would be a long term project. Meanwhile, some users will continue to be annoyed by messy formatted inputs such as:

test \begin{center}
foo
\end{center} bar

[dbitouze]

@muzimuzhi About verbatim environments, indeed some care is needed.

If we expect

% use >>text<< to express that "test" is selected
abc >>def<< ghi

->

abc<space>
\begin{center}% % insert two newlines, before and after
    def%          % add indent or not?
\end{center}%   % insert two newlines
<space>ghi

I'm supporting the indentation and I would expect the (irrelevant) ⟨spaces⟩s to be dropped.

then do we also expect

abc
>>def<<
ghi

->

abc
\begin{center}% % insert only one newline
  def
\end{center}%   % insert only one newline
ghi

That's what I would expect.

[MeanSquaredError]

Maybe it would be better to have some generic rules for the source code formatting and maybe some kind of menu option to reformat the .tex source code (something like an improved version of indent/unindent).

[dbitouze]

some kind of menu option to reformat the .tex source code

In that respect, a killing related feature with Emacs that I would love to see in TXS: if a paragraph is a mess after it has been edited, it is enough to use Esc+q to make it entirely rearranged.

[MeanSquaredError]

Yes, this is what I meant by "source code formatting". All modern development environments allow to reformat the selected source code or the whole file if there is no current selection.

[MeanSquaredError]

Actually I think this can be implemented by some kind of macro script.

[dbitouze]

Actually I think this can be implemented by some kind of macro script.

@MeanSquaredError Do you mean it shouldn't be a feature provided by default by TXS?

[MeanSquaredError]

I think it should be a feature coming with the default installation of TXS, just implemented through scripting. It would be faster to implement and more reliable (if there is some bug it would have good chances of crashing just the script instead of the whole TXS).

[MeanSquaredError]

Actually I think that more TXS features can (and probably should) be implemented through scripts. For example the the more complex table manipulation features in the wizards that we discussed a few issues ago probably can be implemented through scripts. AFAIK the TXS scripts are some dialect of JavaScript so it would be easier to prototype the code in JS than in C++.

[timhoffm]

TXS uses QtScript, however that is not actively developed anymore. While it's said to not go away sooln, I wouldn't build new code on that. An alternative is QJSEngine, but that is Qt5 only (we would have to cut Qt4 compatibility. I'm not really attached to that, so maybe ok).
See here https://doc.qt.io/qt-5/topics-scripting.html

Another alternative would be to embed a Python interpreter. This would allow to take advantage of existing libraries like https://github.com/sciunto-org/python-bibtexparser to add functionality we're currently missing (e.g. a tabular view of the bibtex entries or nicely formatted tooltips for bibtex entries). I think it suits well for parsing (thinking of importing a table into the table wizard. Also, chances are good, that users (many of whom are scientists) know Python already and it could become a public scripting API some day.

[MeanSquaredError]

@timhoffm

1. If adding a new scripting language do we want to keep the old QtScript too? How many users are actually using the existing QtScript?

2. Do you have an idea how close it the new QJSEngine to QtScript? They are both JS implementations so I would think that replacing QtScript with QJSEngine would allow (almost) seamsless switch from one to the other. Any idea whether that's actually the case? :-)

3. Do you have in mind some specific python interpreter and/or framework that would allow an easy embedding of Python into TXS (and expose TXS objects and events to the python interpreter)?

After some googling I came across
https://doc.qt.io/archives/qq/qq23-pythonqt.html
However the article is a bit dated and the PythonQt library does not seem very active on github:
https://github.com/MeVisLab/pythonqt
(they have 3 unattended PRs on github from August)

Another option would be to embed the Python interpreter directly without any 3rd party library as they do it here:
https://ubuverse.com/embedding-the-python-interpreter-in-a-qt-application/

4. Again if we embed the Python interpreter into TXS do we want to keep the old QtScript? Keeping it would be nice for users but more difficult from programming/maintenance point of view.

[sunderme]

using script in txs is totally fine. The table manipulation code uses it as well.
We will have to replace qtscript with the qjsengine , at latest, when porting to qt6 is necessary. I assume that we can keep all current capabilities.

I would discourage to go for python or any other (external) script language. txs is already complicated enough to maintain on different platforms and especially windows will make things complicated with external dependencies...

And last, probably we can cut qt4 support as libpoppler-qt4 is not delivered any more on current platforms and thus txs can not be built there with all (main) capabilities. The qt4 already has some reduced functionality.

[MeanSquaredError]

I googled a bit and it seems to me that the switch from QtScript to QJSengine should be invisible to the TXS users and any scripts should remain working.

I will try to find time in the next couple of weeks to implement a basic .tex source (re)formatter in QtScript

[timhoffm]

@MeanSquaredError

1. No idea how many users are using the scripting capabilities. We do not collect metrics. Probably it's only a small fraction, but given the large user base, that can still be a fair amount.

2. I have not looked into QJSEngine. Sounds great if we could transparently switch. Thanks for the research!

3./4. Not really looked into the details of Python embedding. This is more of an idea for the long-term. PythonQt is indeed what comes up first, but seems not quite active. Blender is one of the largest projects using Python as a scripting language for Qt (I haven't had a look how they do it either). With the adoption of PySide2/Qt for PYthon, Continuum has declared Python as first class language for Qt. Though I haven't heard anything in that direction, it would be a logical step to make it available as a scripting engine in the future.

[MeanSquaredError]

I guess in the short and mid-term we will just switch to QJSEngine and will get rid of the old QtScript.
(I noticed the initial commit from sunderme so I think we are on a good track in that respect).

In the long term I can imagine some kind of plugin architecture where the main TXS core will contain some kind of abstraction layer that will allow plugging in different scripting languages. The main language will be QJSEngine, but there will be additional languages implemented as plugin modules. This will allow implementing these additional languages separately (possibly as separate git subprojects) without messing up too much with the main core.
QJSEngine will be a regular module but distributed with the main executable.
By module I mean a part of the program communicating with the main core though some (semi-)fixed API and not necessarily a dll/so file. It can be linked statically and still be considered a module by the developers.

Of course all of the above are just some generic ideas (food for thought).

[dbitouze]

Related: #177.

[MeanSquaredError]

I could not find time to work on this fulltime, but I have been working on it now and then for a couple of months. Did a few standalone implementations in QtScript that did not work very well, because one really needs the syntax information from the tokenizer and re-implementing a tokenizer from the scratch in QtScript is no fun :-)

Finally I settled on a combined implementation in C++ (which exports the tokens from the C++ tokenizer) + the high level code in QtScript which does the parsing. Hopefully will be able to provide a PR in about a week or so.

[MeanSquaredError]

@dbitouze @muzimuzhi
I implemented a basic source-formatting script here:
#957
Haven't tested it besides using it on some simple .tex files, though.

[cmhughes]

Latexindent does this, see https://latexindentpl.readthedocs.io/en/latest/sec-the-m-switch.html#adding-line-breaks-beginstartsonownline-and-bodystartsonownline

[dbitouze]

Latexindent does this, see https://latexindentpl.readthedocs.io/en/latest/sec-the-m-switch.html#adding-line-breaks-beginstartsonownline-and-bodystartsonownline

I already advocated latexindent here, unfortunately without much success.

Maybe your expertise, as the latexindent developer, could help in the process of smoothly integrate it into TeXstudio 😄

[cmhughes]

Happy to help if I can :)

…

On Mon, 29 Mar 2021, 20:25 Denis Bitouzé, ***@***.***> wrote: Latexindent does this, see https://latexindentpl.readthedocs.io/en/latest/sec-the-m-switch.html#adding-line-breaks-beginstartsonownline-and-bodystartsonownline I already advocated latexindent here <#177>, unfortunately without much success. Maybe your expertise, as the latexindent developer, could help in the process of smoothly integrate it into TeXstudio 😄 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#815 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ7CYGYSDQIXFIEOUWOPKDTGDO4NANCNFSM4JRWNGUA> .

[dbitouze]

Another possibility might be to make use of the code formatter Prettier (which is already integrated in many editors) and its latex plugin.