[App] Change app root / config path to be the app.py parent directory

[ethanwharris]

What does this PR do?

The logic around finding the root is complex and has some failure modes (e.g. having a .lightning file in a parent directory for any reason will cause that whole directory to get uploaded).

This PR makes it so that the root directory === the parent of the entrypoint file

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

[tchaton]

LGTM ! However, this would break for some apps.

[awaelchli]

This was intentional, see the test cases. The way this PR defines the root is extremely limiting. If the root folder is defined by the app.py file, then your app.py file ALWAYS has to be at the root of your project (the folder requiring all sources to run the program).

We created the .lightning file for two purposes:

• remember the name of the app
• define the root of the project (the files that need to be uploaded)

See the specification here: https://www.notion.so/lightningai/Lightning-Dotfile-and-the-cloud-experience-cfe28f6936a642b08958cf86b458cafc#9e47b6ae4b6b48a0b9840887994353b3

How will you address the newly introduced limitations?

[ethanwharris]

I would argue a few points:

• requiring app.py to be at the root is no less flexible than requiring the user to create a .lightning file at the root
• this behaviour isn't actually documented anywhere and has led to users getting very large directories uploaded on multiple occasions
• .lightning files are not currently expected to be commited with source files as they also contain the cluster ID - with that new contract, the majority of apps are required to have app.py at the route anyway

Overall I think this approach is much closer to what you expect to happen as a user without having to learn a load about the undocumented internals.

[awaelchli]

The user does not create it manually. It gets generated for them. If the file is not found, it creates it next to the app.py file (documented in the notion page). This means the default experience is good: Only the files next to the app.py file gets uploaded.

I am sorry that there is misinformation around this.

.lightning files are not currently expected to be commited with source files as they also contain the cluster ID - with that new contract, the majority of apps are required to have app.py at the route anyway

Let's document it then? When we built the initial version of the framework, we had no time for anything.

[williamFalcon]

yes this is the expected user experience.
the other experience has already caused a ton of user problems.

this is especially important for transparency with enterprise customers.