Complete and flexible approach to line endings (LF, CRLF, ... newlines) in file.* functions

[uvsmtid]

This is intentionally elaborated as both feature request and related bug report to have a systematic approach for line endings issues.

Proposal

There should be newline argument to all file.* function which update file content.
For example, the values could be:

• native = use line endings of the target system (Linux: LF, Windows: CRLF, ...);
• LF = force LF line endings;
• CRLF = force CRLF line endings;
• ...
• (possible) source = detect and preserve source line endings.

Motivation

Text files deployed by Salt may be used by multiple target OSes. Using only assumed default (native) line endings is not flexible enough.
For example:

• There are cases when files on Windows are used by software which specifically requires Unix line endings (i.e. bash-shell scripts for Cygwin). Assumption of native line endings does not automatically apply to this case.
• A text file deployed by Salt on Linux target system may be part of the content published on a file server accessible to others via network. Again, converting line endings is not desirable.
• Converting line endings additionally outside of Salt (i.e. running dos2unix/unix2dos utilities) unnecessarily complicates SLS files.

Attention to details is important in this matter because managing (configuration) text files is the bulk of complex automation expected from Salt. And the expectation is that for heterogeneous environments it should be simplified on Salt side.

Existing problems

There are few existing problems (see table in "Test results" below).

If jinja template is used, function file.managed modifies source line endings (salt://) to that of the target's system. There are two conflicting requirements:

• (1) preserve source file line endings
• (2) use target system's line endings.

Both are valid (and only the 2nd one is currently possible).

If file.replace is used on Windows and original file already has CRLF line endings, the function appends additional CR making line ending look like CRLFCR. This is an obvious bug.

I believe that similar problems exists for other file.* functions like blockreplace, comment/uncomment, append/prepend, ..

Cross references to existing bugs: #12284, #14002.

Test setup

NOTE: Cygwin is installed on Windows minion to simplify SLS and use the same command lines for tests (commands like rm, file, ...).

Two SRC files deployed on Linux with Salt master in /srv/salt:

file /srv/salt/deleteme.file.CRLF.txt
/srv/salt/deleteme.file.CRLF.txt: ASCII text, with CRLF line terminators
file /srv/salt/deleteme.file.LF.txt
/srv/salt/deleteme.file.LF.txt:   ASCII text

Single SLS file /srv/salt/deleteme.sls:

# 

{% if grains['kernel'] in [ 'Windows' ] %}
{% set DST_file_name_start = 'C:\deleteme' %}
{% endif %}
{% if grains['kernel'] in [ 'Linux' ] %}
{% set DST_file_name_start = '/deleteme' %}
{% endif %}

###############################################################################
# file.managed

{% for SRC_template in [ '~', 'jinja' ] %}

{% for SRC_newline in [ 'CRLF', 'LF' ] %}

{% if SRC_template == '~' %}
{% set DST_file_infix = 'not_template' %}
{% else %}
{% set DST_file_infix = 'template' %}
{% endif %}

{% set file_name = DST_file_name_start + '.' + DST_file_infix + '.' + SRC_newline + '.' + 'file.managed.txt' %}

'remove_{{ file_name }}':
    cmd.run:
        - name: 'rm "{{ file_name }}"'
        - onlyif: 'ls "{{ file_name }}"'

'{{ file_name }}':
    file.managed:
        - source: salt://deleteme.file.{{ SRC_newline }}.txt
        - template: {{ SRC_template }}
        - require:
            - cmd: 'remove_{{ file_name }}'

'inspect_{{ file_name }}':
    cmd.run:
        - name: 'file "{{ file_name }}"'
        - require:
            - file: '{{ file_name }}'

{% endfor %} # SRC_newline

{% endfor %} # SRC_template

###############################################################################
# file.replace

{% for SRC_newline in [ 'CRLF', 'LF' ] %}

{% set file_name = DST_file_name_start + '.' + SRC_newline + '.' + 'file.replace.txt' %}

'remove_{{ file_name }}':
    cmd.run:
        - name: 'rm "{{ file_name }}"'
        - onlyif: 'ls "{{ file_name }}"'

'init_{{ file_name }}':
    file.managed:
        - name: {{ file_name }}
        - source: salt://deleteme.file.{{ SRC_newline }}.txt
        - template: ~
        - require:
            - cmd: 'remove_{{ file_name }}'

'{{ file_name }}':
    file.replace:
        - pattern: 'asdf'
        - repl: '1234'
        - require:
            - file: 'init_{{ file_name }}'

'inspect_{{ file_name }}':
    cmd.run:
        - name: 'file "{{ file_name }}"'
        - require:
            - file: '{{ file_name }}'

{% endfor %} # SRC_newline

###############################################################################
# EOF

Test results

The table below was filled based on results after executing:

salt -L windows_minion,linux_minion state.sls deleteme

NOTE: Even in the cases when the result is expected, based on the points listed in "Motivation" section, there are still cases when default native line endings have to be changed.

Case	function	SRC template?	SRC file	DST OS	Result / DST file	Result / Expected default?	Result / Comment
01	file.managed	Y	CRLF	Linux	LF	Depends	Template converted CRLF=>LF (native newlines).
02	file.managed	N	CRLF	Linux	CRLF	Yes	DST “downloaded” exactly as SRC.
03	file.managed	Y	LF	Linux	LF	Yes	Template conversion LF=>LF is indistinguishable from SRC.
04	file.managed	N	LF	Linux	LF	Yes	DST “downloaded” exactly as SRC.
05	file.managed	Y	CRLF	Windows	CRLF	Yes	Template conversion CRLF=>CRLF is indistinguishable from SRC.
06	file.managed	N	CRLF	Windows	CRLF	Yes	DST “downloaded” exactly as SRC.
07	file.managed	Y	LF	Windows	CRLF	Depends	Template converted LF=>CRLF (native newlines).
08	file.managed	N	LF	Windows	LF	Yes	DST “downloaded” exactly as SRC.
09	file.replace	N/A	CRLF	Linux	CRLF	Yes	DST uses SRC line endings.
10	file.replace	N/A	LF	Linux	LF	Yes	DST uses SRC line endings.
11	file.replace	N/A	CRLF	Windows	CRLFCR	Wrong	Replacing added one more CR.
12	file.replace	N/A	LF	Windows	CRLF	Depends	Replacing converted LF=>CRLF (native newlines).

Versions

Linux master:

  salt --versions
           Salt: 2014.1.5
         Python: 2.7.5 (default, Jun 25 2014, 10:19:55)
         Jinja2: 2.7.3
       M2Crypto: 0.21.1
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
         PyYAML: 3.10
          PyZMQ: 13.0.2
            ZMQ: 3.2.4

Linux minion:

 salt-minion --versions
           Salt: 2014.1.5
         Python: 2.6.8 (unknown, Nov  7 2012, 14:47:45)
         Jinja2: unknown
       M2Crypto: 0.21.1
 msgpack-python: 0.1.12
   msgpack-pure: Not Installed
       pycrypto: 2.3
         PyYAML: 3.08
          PyZMQ: 2.1.9
            ZMQ: 2.2.0

Windows minion:

salt-minion.exe --versions
           Salt: 2014.1.5
         Python: 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)]
         Jinja2: 2.7.1
       M2Crypto: 0.21.1
 msgpack-python: 0.4.2
   msgpack-pure: Not Installed
       pycrypto: 2.6
         PyYAML: 3.11
          PyZMQ: 14.1.1
            ZMQ: 4.0.4

[basepi]

Thanks for the thorough report and test results! We should definitely find a way to get this all consistent and as-expected.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[jfoboss]

Is there any chance to fix this bug? We are deploying haproxy configs via salt and are forced to use todos as a workaround :(

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[damon-atkins]

Should this be closed?

[jfoboss]

@damon-atkins is this bug resolved?

[damon-atkins]

That's the question? Hence I stop this from being closed.

[jalandis]

I just ran into this issue.

A managed configuration file restarts a service which alters the line endings after reading (not in my control). This causes the next highstate to update the configuration file...

Does anyone know of a workaround?

salt-minion --versions
Salt Version:
           Salt: 2016.11.10

Dependency Versions:
           cffi: 1.10.0
       cherrypy: 7.1.0
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 2.0.8
          ioflo: 1.5.5
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: 1.4.5
       M2Crypto: Not Installed
           Mako: 1.0.4
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.150
0 64 bit (AMD64)]
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 16.0.1
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.4.1
            ZMQ: 4.1.6

[absmith82]

I also need this fixed. Zabbix agent for Windows must have Unix line endings in the configuration file or it won't start. As such I cannot template the file to make it work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[sagetherage]

The Core team won't be able to get to this in Aluminium, moving it back into planning for another release.

[dmurphy18]

@uvsmtid Closing this due to age, the old version of Salt and Python 2.
Can you retest this with the latest release of Salt 3006.2 and if still an issue, please open a new issue, which will have metrics in tracking issues.