Move existing programming model into separate npm package

[ejizba]

I propose we split the worker up into two pieces, one that ships as-is with the host and one that ships as a separate node module (from here on out, I will reference these as the "core" and "programming model" packages respectively). This specific issue is to track moving the existing programming model out of the worker and into a separate package. The first iteration of this package should exactly replicate what the worker does today. The second iteration should be an improved programming model as tracked by #480.

When I say "programming model", this includes anything that makes our user's lives easier like parsing/converting/handling data from the host. Another way to think about it is if there is currently a type for something in our types package (Context, HttpRequest, etc.), the underlying code for that should probably be in the programming model package. The only thing left in the worker will likely be setting up and managing the gRPC channel used to communicate with the host

The Split

Core worker package ("@azure/functions-core")	Programming model package ("@azure/functions" - name TBD)
Tracked by #567	First version tracked by this issue
Required - we have to create an API in some form	Optional - we do not need this package. Instead, we could put everything in the core api
built-in module shipped with the worker - not listed in user's package.json	shipped with the user's app - listed in their package.json dependencies
slow to update because it relies on the host release cycle	fast to update - just publishing an npm package
forced on the user when it does update	user decides when to take the update
backwards compatability is paramount	backwards compatability is nice, but we can easily do a new major version
barebones API, very unopinionated	fancy api with lots of nice intellisense, can be opinionated

Benefits

• When we make breaking changes (like this), users will be able to choose when they integrate those changes as opposed to us forcing it on them in Azure.
• If we ever want to force a breaking change on the user, all we have to do is change the "default" programming model package used in Azure, but users can still reference the old version in their own app.
• We can easily support multiple different programming models in their own packages (whether they're created by us or other people) at the same time. This PR is one example where we run into difficulties today because our various models are combined into the worker.
• Since both JS and TS users will need the npm package to run their app, they will both get the same experience of intellisense in VS Code

[aaronpowell]

Is there a reason why we'd want to split across multiple git repos and not have a monorepo that contains the worker and the libraries around it?

I feel like having them split across multiple repos will make the contributor experience a lot more difficult, as you'll be required to clone multiple repos and then use npm link to connect all the packages together, whereas with a monorepo that would be simpler.

It would also make it possible to have consistency in shared dependencies, ensure consistent style guides, etc.

[ejizba]

Interesting question - thanks for bringing it up. I had not considered a monorepo very much, but I still think separate repos is the way to go. Here are my reasons:

1. The first reason (but not necessarily best reason IMO) is to be consistent with the Azure Functions repo strategy in general. There's quite a few repos and none of them are monorepos. If you want to test changes in the host or func cli you already have to clone multiple repos. Also the python and java languages already have separate worker/library repos.
2. I think the most compelling reason is because of branching/versioning. To start, both the worker and library repos will stay on "v3.x" replicating the existing programming model and then the "library" repo will add "v4.x" for the new programming model. The worker will support both the old and the new version and doesn't need a new branch every time the library has a new major version. Ideally the worker doesn't have breaking changes for a long time, but we should be able to iterate pretty quickly for the library with potentially several new major versions before we need a new major version on the worker.
3. In addition to multiple major versions of the library, I think there could likely end up being multiple completely different libraries at the same time. We already know we want to try out a decorators library, for example. There will always be one worker, so I don't think this'll scale well for a monorepo.
4. I agree cloning multiple repos and doing npm link is not great. However, I think moving forward the vast majority of external contributions will be on the library, not the worker. Users should not have to clone the worker repo at all, they won't need to set any languageWorkers__node__workerDirectory environment variables, etc. and they can just npm link the library package directly into their app and go from there

[aaronpowell]

1. The first reason (but not necessarily best reason IMO) is to be consistent with the Azure Functions repo strategy in general. There's quite a few repos and none of them are monorepos. If you want to test changes in the host or func cli you already have to clone multiple repos. Also the python and java languages already have separate worker/library repos.

I can see that, from a MS-engineering side, having a consistent repo structure across all the supported languages makes sense, so if you're moving between teams you don't have to learn new patterns each time.

I personally have had times where I'm trying to understand something and had to have multiple versions of VS and VS Code running just to try and debug things (and it can be confusing to find all the repos), so I was more looking to float the idea to tackle something I've seen as an external looking in.

2. I think the most compelling reason is because of branching/versioning. To start, both the worker and library repos will stay on "v3.x" replicating the existing programming model and then the "library" repo will add "v4.x" for the new programming model. The worker will support both the old and the new version and doesn't need a new branch every time the library has a new major version. Ideally the worker doesn't have breaking changes for a long time, but we should be able to iterate pretty quickly for the library with potentially several new major versions before we need a new major version on the worker.

Yes, I do see the branching/versioning being a pain point when there are multiple "active" versions (like v3 and v4 both being supported right now), and that would (possibly) only expand in complexity with multiple library package versions being actively supported.

3. In addition to multiple major versions of the library, I think there could likely end up being multiple completely different libraries at the same time. We already know we want to try out a decorators library, for example. There will always be one worker, so I don't think this'll scale well for a monorepo.

This might be the best case for a monorepo, where we have a base package like (the following are made up names of course) @azure/functions-core, @azure/functions-app, @azure/functions-decorators, and so on, where there are small packages for different programming models, allowing the developers to opt into the model they want and only get that one.

4. I agree cloning multiple repos and doing npm link is not great. However, I think moving forward the vast majority of external contributions will be on the library, not the worker. Users should not have to clone the worker repo at all, they won't need to set any languageWorkers__node__workerDirectory environment variables, etc. and they can just npm link the library package directly into their app and go from there

Yes, hopefully splitting the worker from the consumer package will make it simpler, especially if you want to work on the consumer package is isolation - I've just been burnt by npm link so many times I do everything I can to avoid it! 🤣