We need better temporary file and dir API

[timotheecour]

• getTempDir ; EDIT: solved by better getTempDir #16914
  getTempDir comes with a warning Please do not use this and indeed its implementation should be replaced (or alternatively, deprecated for another proc if backward compatibility is an issue) by something more robust.

• We can draw some inspiration from https://github.com/dlang/phobos/blob/v2.082.1/std/file.d#L5169 ; in particular on windows:
  calling the Windows API function GetTempPath. (https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-gettemppatha) instead of string(getEnv("TEMP")) & "\\"
• on android, instead of getHomeDir(), https://stackoverflow.com/questions/2364740/where-is-the-temp-folder-in-android-device points to /data/local/tmp or getCacheDir()

• temporary file and dir creation; EDIT: see close #9372 add std/tempfiles #17361
  There's, in posix, proc mkstemp*(tmpl: cstring): cint but that's posix only and too low-level (eg see accompanying warning to it)

Shopping around for alternatives, python has nice standard library library with low level and high level API around temp file/dir creation, including facilities for cleanup, and with epmhasis on security (eg see description around TemporaryFile).

We could start with porting these:

tempfile.mkstemp(suffix=None, prefix=None, dir=None, text=False)
# returns a tuple containing an OS-level handle to an open file (as would be returned by os.open()) and the absolute pathname of that file, in that order.

tempfile.mkdtemp(suffix=None, prefix=None, dir=None)

Note: the D variant (https://dlang.org/library/std/stdio/file.tmpfile.html) is not as convenient as it returns a File object (and no way to access its name)

links

• http://www.cplusplus.com/reference/cstdio/tmpfile/
• http://www.cplusplus.com/reference/cstdio/tmpnam/
• https://dlang.org/library/std/stdio/file.tmpfile.html
• https://dlang.org/library/std/file/temp_dir.html
• people requiring it on IRC: https://gitter.im/nim-lang/Nim?at=5bc394d482893a2f3bdfae3d

no direct cross-platform convenience mktemp-like method yet

• OpenSystemsLab/tempfile.nim: This module contains basic procs for creating temporary files and directories
• temp file api · Issue #195 · timotheecour/Nim

[haltcase]

Probably worth noting there's already a nimble package for this. https://github.com/OpenSystemsLab/tempfile.nim

[timotheecour]

ah, thanks! but definitely some points in this issue should be addressed. in the eternal debate of 'batteries included or not', I feel temp file creation should be part of stdlib because it's generally useful AND because it's likely to be used in Nim repo itself (eg a lot of tests should use tempfile creation instead of what's being done currently:

$nimc_D/tests/stdlib/tparsecfg.nim
config.writeConfig("test.ini")
which dumps to current working dir and complicates .gitignore

[timotheecour]

note: https://github.com/OpenSystemsLab/tempfile.nim has limitations, eg:

• it itself uses getTempDir() which is marked as "do not use"
• it can't be used at CT, see doesn't work at compile time OpenSystemsLab/tempfile.nim#9

[hyao]

Python tempfile implementation:
https://docs.python.org/3.7/library/tempfile.html

[pb-cdunn]

Ironically, nimble itself uses getTempDir().

• nimble needs to support system-wide installations nimble#80

[ghost]

These?

• https://nim-lang.github.io/Nim/posix_utils#mkstemp,string
• https://nim-lang.github.io/Nim/posix_utils#mkdtemp,string

[sschwarzer]

* https://nim-lang.github.io/Nim/posix_utils#mkstemp,string
* https://nim-lang.github.io/Nim/posix_utils#mkdtemp,string

Yes, these seem to be Posix-only. I just find myself in the situation where I want to adapt a program that I developed under Linux to Windows and this mkdtemp limitation is a bit annoying.

[ghost]

@sschwarzer yeah ok, but mkstemp works fine even on Windows. and if youre going that route, then youll need to throw out PHP and other too, as they can only create temp files and not directories either.

[sschwarzer]

@sschwarzer yeah ok, but mkstemp works fine even on Windows. and if youre going that route, then youll need to throw out PHP and other too, as they can only create temp files and not directories either.

That's interesting, thank you. I had assumed that things in the posix module would only work on Posix platforms. Anyway, I need mkdtemp.

I agree with @timotheecour that creating temporary directories and files should be in the Nim standard library. I'll probably go with the external tempfile library for now that was mentioned above. I had considered working on mkdtemp and mkstemp in the standard library, but at the moment I don't have the time. :-/

(Regarding PHP, I haven't used it in years, although not specifically because of the temp directory omission but because I didn't have a project where I'd use PHP. And by the way, if I say that a feature should be in the standard library of a language that doesn't automatically mean I won't use the language until the standard library has the feature.)

[maxgrenderjones]

I'm a git/github newbie so please be gentle, but have just opened #14347 to get support for mkstemps.

template withTempFile(prefix: string, suffix: string, code: untyped): untyped =
    let (filename, tempfile {.inject.}) = mkstemp(prefix, suffix)
    let tempfilename {.inject.} = absolutePath(filename)
    try:
        code
    finally:
        removeFile(tempfilename)

Is the user-friendly version of the same, but I don't know if that has the cross-platform support it would need to go into os. (I'm on OSX where the temp file seems to get created in the current directory, so I'm not sure what happens if TMPDIR is set to something else - I'm not sure what you need to do to get the absolutePath in a way that works on all OSes - generic way to get temp path suggests that you'd need to change the call to the posix mkstemp to ensure that the file is created a well-known place that honours TMPDIR. tempfile.py may be instructive (seems they just try a bunch of directories until one succeeds).

[kaushalmodi]

May be https://github.com/OpenSystemsLab/tempfile.nim should be made part of fusion ( https://github.com/nim-lang/fusion ) if the author allows? /cc @OpenSystemsLab @ba0f3

[ghost]

@kaushalmodi well with MIT you don't really need permission, just need to keep the license, but yeah, it would be a nice thing to ask :)

[ba0f3]

@kaushalmodi why not 😀

[kaushalmodi]

@ba0f3 Thanks. With your approval, now we need to figure out how to proceed. @Araq ?

[timotheecour]

just create a PR against fusion ?