[Feature Request] Accessing absolute directory of the executed file as a config variable

[ashleve]

🚀 Feature Request

I would like to access absolute directory of my main file. Usually I do it like this:

BASE_DIR = os.path.dirname(__file__)

But hydra changes working directory so this doesn't work anymore.
hydra.utils.get_original_cwd() gives me working directory the app was executed from, which is not what I need.

It would be best if I could access absolute directory of my main file like this:

some_path: ${executed_file_dir_path}/my/relative/path

Motivation

This is important if user wants to make execution independent of working directory the app was executed from.
(I have paths in my config that I always want to be relative to placement of my executed file, so I want to join them with absolute path of directory of executed file)

Suggestion

Maybe there could be some new special variables available in hydra like the ${hydra:runtime.cwd}? For example:

file_path: ${hydra:main_file.path}
file_dir_path: : ${hydra:main_file.dir}

[omry]

Yes. this is documented in the basic tutorial here.

[ashleve]

Hey @omry,
sorry for this issue, I explained my problem badly.
What I actually meant was getting the global absolute path of executed file. Not hydra working directory and not original working directory.

So for example, when I execute my file from some outside directory like this:

python some/path/main.py

I want to access global absolute path of main.py file, like C:/Users/X/some/path/main.py.

Can this be done with some specialised variable in config similar to ${hydra:runtime.cwd}? I've read the basic tutorial and most of the documentation and didn't find anything like that.

Also, my real use case is that I want the hydra logs to be always next to my main.py file, no matter from where I execute this file in command line. Can this be accomplished in some other way?

[omry]

What if your main.py is in a Python wheel (maybe even in a zipped wheel?)

You can try:

import os
print("file", __file__)
print("dir",os.path.dirname(os.path.realpath(__file__)))

Also, my real use case is that I want the hydra logs to be always next to my main.py file, no matter from where I execute this file in command line. Can this be accomplished in some other way?

I don't think it's possible right now.
I can consider supporting it for a future version.

[rudaoshi]

I have a similar issue. I have a txt file along with a module file and I want to load the content of the text file in the module.

I wrote the code like:

edge_voc_file = os.path.join(os.path.dirname(os.path.realpath(__file__)), "edges.txt")

but when I run the code, the module tries to find the file on some path under the output directory.

I think hydra should not break the meaning of __file__, since it is a common technique to find the module file path/directory. Current behavior may break many codes.

[omry]

@rudaoshi,

1. Your problem has nothing at all to do with this issue. Please do not hijack issues.
2. Hydra does not break the meaning of file. Your code is fragile if it depends on the current working directory.
   You should use Python APIs to load resources. This SO answer is pretty good.
3. See Support disabling working directory management #910 for workarounds.

[rudaoshi]

Thank you for your clarification. I misunderstand the problem.

I move the code to the module level and it runs successfully.

And thanks for the suggested material. Learned a lot from them.

[Queuecumber]

I'm not sure if this solves your problem but I used this in a recent project

OmegaConf.register_resolver("modpath", lambda: pathlib.Path(__file__).parent.absolute())

which I think could be customized for what you need, then I was able to refer to files like:

stats: ${modpath:}/../stats/cstats.pt

which gets me a path that is relative to the file that is being executed

What if your main.py is in a Python wheel (maybe even in a zipped wheel?)

Probably won't work in this case though I think

[omry]

Reading this issue again, I am not sure this makes sense to me:

This is important if user wants to make execution independent of working directory the app was executed from.
(I have paths in my config that I always want to be relative to placement of my executed file, so I want to join them with absolute path of directory of executed file)

The configs being relative to the file with @hydra.main() is something very unusual that Hydra is doing, and it's certainly not simple to achieve. In particular, it will work correctly even if the file is installed or even if it's inside a zip (Hydra will use importlib_resources to load resources in that case).

I don't think that you suggestion to add hydra:main_file.path and hydra:main_file.dir makes sense because the file may actually installed in a zipped wheel.

I am closing this for now, feel free to followup if you want to discuss this.

[Queuecumber]

How does python execute a zipped wheel? Presumably it has to decompress it right?

[omry]

No. it can do it directly from the wheel. The correct way to access resources is using a dedicated API like importlib_resources and not relative to __file__.

[omry]

create a zip file with a file called __main__.py with this content:

print("Hello from ", __file__)

Run the zip with Python:

$ python test.zip
Hello from  test.zip/__main__.py

[Queuecumber]

I'll have to look into this more but according to pep 427 part of installing a wheel is extracting it:

https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl
https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

I know about importlib but I forget the details, there might be a way to include it in the interpolation I wrote to make something more universal