Python function project layout

[cecilphillip]

Creating a new function project for Python in VS Code produces a flat folder structure. This doesn't match the recommended structure in docs.

Is there anyway we can apply that guidance to Python projects created via VS Code?

[ejizba]

@qubitron @kulkarnisonia16 what are your thoughts on the file structure for Python projects?

Fwiw, users can create this file structure on their own today if they "Browse..." to a subfolder called "app" when creating their project

[cecilphillip]

Sure, users can create the function app in a sub folder, but I think the setup goes deeper than that.

Assuming there's a tests folder too, a dev would probably want both app and tests folder open in a workspace. Where would the virtual environment go now? How would we separate dev and prod requirements.txt files, what would the debug cycle look like?

For any non-trivial Functions app in Python, this would also require changes to the tasks and launch settings in vscode. I'm proposing that we have layout that just give this them right away.

[qubitron]

@brettcannon thoughts? VS Code requires that the virtual environment be in the root folder of the workspace for it to be found

[brettcannon]

https://github.com/microsoft/pvscbot is my suggested workspace layout (and it's the one that led to the docs being updated with their current recommendation 😄 ). There original discussion and further responses from the community can be found in Azure/azure-functions-python-worker#469.

[ejizba]

I just want to make sure we're not re-inventing the wheel here. I'm fine changing the folder structure if we base it on common patterns in the Python community. If we can find other sample projects, guidelines, etc. that would be very helpful.

A few problems I have with the docs linked above:

1. I don't want to add a "tests" folder if it's going to be empty, but I also don't know if I want to include sample tests by default. None of our other languages have sample tests
2. I can't find any other references online to a folder called "__app__", but I did find this doc saying:

   __double_leading_and_trailing_underscore__: "magic" objects or attributes that live in user-controlled namespaces. E.g. __init__, __import__ or __file__. Never invent such names; only use them as documented.

[brettcannon]

@EricJizbaMSFT the __app__ thing is actually an Azure Functions creation and has no special meaning to Python itself. But it is required to make absolute imports function appropriately from within the function code as well as tests if you keep them external to the function based on the how the Python worker works.

As for leaving out a tests folder, I think the key thing is to help instruct folks on where they should put their tests if they don't want to upload them along with the function code.

[qubitron]

@brettcannon is the __app__ name significant to make absolute imports work, or could it be called "app" or "src"?

[cecilphillip]

@EricJizbaMSFT could it be an opt (yes/no) to add tests. I would imagine this would do more than create an empty folder. You'd probably want to add the associated task/launch settings for vscode as well a separate requirements.txt file. We should guide users to be successful without having to spend too much time configuring where stuff goes.

[brettcannon]

@qubitron I believe it's significant from an Azure Functions perspective (or so the team as told me 😉 ). I believe the runtime makes an __app__ directory, drops your code into it, and then executes it from the directory above __app__.

[fiveisprime]

It seems this is something that the Functions team is working toward, but haven't gotten to yet. I'd say it make sense for the extension to start following this pattern as the other tools also adopt it.

Related to: Azure/azure-functions-core-tools#1546 and Azure/azure-functions-core-tools#1547

[anthonychu]

@brettcannon Can you share the folder structure that you think should look like when we stamp out a new empty function app from VS Code and Core Tools? It would be great to see where the .vscode folder lives and whether files like dev-requirements.txt should me there.

@fiveisprime I agree that we should implement this in Core Tools first. Will need to consider what happens if someone runs func new then code ..

A longer term question to think about is should we consider an alternate structure for all languages? For instance we can ask if they want tests and if yes, we’ll generate nested folders for their language (probably an empty tests folder makes sense here). We can experiment with the Python experience first.

[ejizba]

I think fixing this issue should be top priority in terms of getting a better file structure for all languages: Azure/azure-functions-host#5373

My main concern with adding tests is the complexity. For example, the hello-world projects for VS Code extensions include tests by default and overall I'm a fan, but there's been several cases where the dev dependencies needed for those tests lead to npm security alerts which is not what you want from a clean new project IMO. There's also a lot of different ways to run tests so I wouldn't want us to be too opinionated.

Finally, I really dislike the name __app__ and would love if we could avoid that if at all possible. Sounds like that might require a change in the runtime, but would much prefer "src" or "app".

[brettcannon]

@anthonychu https://github.com/microsoft/pvscbot is the layout. Just stick a .vscode/settings.json in there.

[rnwolf]

Based on layout suggested by Brett in https://github.com/microsoft/pvscbot I am able to get this layout below working, sort of.

I ended up here because I wanted to create two HttpTrigger functions that use some shared function/code and had some pytests. The existing example layouts did not quite work.
The HttpTriggers are based on the current example templates.

#__app__\HttpTrigger1\__init__.py
import logging

import azure.functions as func

# from __app__.sharedcode.my_helper_function import hello
from __app__.sharedcode import my_helper_function


def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info("Python HTTP trigger function processed a request.")
    logging.info(f"Run shared code function {my_helper_function.hello()}")

    name = req.params.get("name")
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get("name")

    if name:
        return func.HttpResponse(f"Hello {name}!")
    else:
        return func.HttpResponse(
            "Please pass a name on the query string or in the request body",
            status_code=400,
        )

The test is as follows:

# tests/test_httptrigger.py
import pytest

import azure.functions as func
from __app__.HttpTrigger1 import main


def test_main():
    # Construct a mock HTTP request.
    req = func.HttpRequest(
        method="POST", body=None, url="/api/HttpTrigger1", params={"name": "Test"}
    )

    # Call the function.
    resp = main(req)

    # Check the output.
    assert resp.get_body() == b"Hello Test!"

I am able to publish these functions to production, from terminal, with:
(.venv) C:\Users\rnwol\workspace\multifunclayout\__app__>func azure functionapp publish rnwolfapp2 --python

When using VS-Code I am unable to use the F5 (or Ctrl+F5) to run the functions.
Error: No job functions found. Try making your job classes and methods public. If you're using binding extensions
It seems like F5 uses the root directory, C:\Users\rnwol\workspace\multifunclayout.
but it needs start in the __app__ sub-directory.

If I run the following C:\Users\rnwol\workspace\multifunclayout\__app__> func start --python manually then it runs the functions locally.

If you can advise me what I need to do the get that working in VS-Code please do so.

C:\Users\rnwol\workspace\multifunclayout
├── .funcignore
├── .gitignore
├── .venv
|  ├── Include
|  ├── Lib
|  |  └── site-packages
|  ├── pyvenv.cfg
|  └── Scripts
|     ├── activate
|     ├── activate.bat
|     ├── Activate.ps1
|     ├── black.exe
|     ├── blackd.exe
|     ├── deactivate.bat
|     ├── easy_install-3.7.exe
|     ├── easy_install.exe
|     ├── pip-compile.exe
|     ├── pip-sync.exe
|     ├── pip.exe
|     ├── pip3.7.exe
|     ├── pip3.exe
|     ├── py.test.exe
|     ├── pytest.exe
|     ├── python.exe
|     └── pythonw.exe
├── .vscode
|  ├── extensions.json
|  ├── launch.json
|  ├── settings.json
|  └── tasks.json
├── dev-requirements.in
├── dev-requirements.txt
├── host.json
├── LICENSE.md
├── local.settings.json
├── pytest.ini
├── README.md
├── tests
|  ├── testHttpTrigger1.http
|  ├── testHttpTrigger2.http
|  ├── test_HttpTrigger1.py
|  └── __init__.py
└── __app__
   ├── .python_packages
   ├── conftest.py
   ├── host.json
   ├── HttpTrigger1
   |  ├── function.json
   |  └── __init__.py
   ├── HttpTrigger2
   |  ├── function.json
   |  └── __init__.py
   ├── requirements.txt
   ├── sharedcode
   |  ├── my_helper_function.py
   |  ├── __init__.py
   |  └── __pycache__
   └── __init__.py

[anthonychu]

All the functions related files (host.json, local.settings.json, .funcignore, etc) should be in the __app__ folder.

Everything else looks good. In the VS Code tasks.json and settings.json you need to configure the folders correctly. It's working for me here: https://github.com/anthonychu/functions-python-durable-image-classifier

[rnwolf]

@anthonychu Thanks for the tips. I also needed to update my

settings.json

Changed deploySubpath from . to __app__
Also changed pythonVenv

{
    "azureFunctions.deploySubpath": "__app__",
    "azureFunctions.scmDoBuildDuringDeployment": true,
    "azureFunctions.pythonVenv": "..\\.venv",
    "azureFunctions.projectLanguage": "Python",
    "azureFunctions.projectRuntime": "~3",
    "debug.internalConsoleOptions": "neverOpen",
    "python.pythonPath": ".venv\\Scripts\\python.exe",
    "python.formatting.provider": "black",
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": true,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": false,
    "python.testing.pytestArgs": [
        "tests"
    ]
}

and

tasks.json

need to ensure that the current working directory was specified for each of the tasks with
"cwd": "${workspaceRoot}\\${config:azureFunctions.deploySubpath}"
as VS-Code apparently tries to run all tasks in the terminal using the cwd of its initial first launch.

{
	"version": "2.0.0",
	"tasks": [
		{
			"type": "func",
			"command": "host start",
			"problemMatcher": "$func-watch",
			"isBackground": true,
			"dependsOn": "pipInstall",
			"options": {
				"cwd": "${workspaceRoot}\\${config:azureFunctions.deploySubpath}"
			}
		},
		{
			"label": "pipInstall",
			"type": "shell",
			"osx": {
				"command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r dev-requirements.txt"
			},
			"windows": {
				"command": "${config:azureFunctions.pythonVenv}\\Scripts\\python -m pip install -r dev-requirements.txt"
			},
			"linux": {
				"command": "${config:azureFunctions.pythonVenv}/bin/python -m pip install -r dev-requirements.txt"
			},
			"problemMatcher": [],
			"options": {
				"cwd": "${workspaceRoot}\\${config:azureFunctions.deploySubpath}"
			}
		}
	]
}

[ejizba]

@rnwolf I noticed in your list of files you have "host.json" twice. We (the VS Code extension team) use that file to identify where the function app is so you might want to delete the extra one.

On a separate note, "azureFunctions.pythonVenv" is supposed to automatically work without you guys having to change the path. Here's an issue to track fixing that: #2041

[rnwolf]

Thanks for the feedback @EricJizbaMSFT
I have worked with the feedback thus far and created a layout that seems to work at https://github.com/rnwolf/azure-func-python-layout

I will still need to see what happens if I use the repo. as a starter template, but I've learnt a lot in the process of getting this far. Hopefully I can now get down to the business of incorporating some of the other backend Azure services in the function application.