Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add local dev experience for scheduled functions #3689

Merged
merged 39 commits into from
Jan 28, 2022
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
e567731
feat: add local dev experience for scheduled functions
netlify-team-account-1 Nov 23, 2021
0f3dd2d
feat: use synchronous function's timeout
netlify-team-account-1 Nov 23, 2021
304239c
chore: rename to scheduled.js
netlify-team-account-1 Nov 23, 2021
362159b
feat: return 400 when trying to invoke scheduled function via http
netlify-team-account-1 Nov 23, 2021
bbc9675
chore: use testMatrix pattern
netlify-team-account-1 Nov 23, 2021
cc8800c
fix: config-based schedule without esbuild
netlify-team-account-1 Dec 14, 2021
9ba29de
feat: add support for ISC-declared flags
netlify-team-account-1 Dec 14, 2021
db5c942
fix: node v12 doesn't understand optional chaining
netlify-team-account-1 Dec 17, 2021
2f4e100
fix: allow esbuild to read mock files by writing them to disk
netlify-team-account-1 Dec 17, 2021
c14699b
fix: wrong import
netlify-team-account-1 Dec 20, 2021
77ef972
feat: use listFunction to detect ISC schedule
netlify-team-account-1 Dec 20, 2021
304fba9
feat: remove unused feature flag
netlify-team-account-1 Dec 20, 2021
0c1eac8
Merge branch 'main' into scheduled-jobs-endpoint
netlify-team-account-1 Dec 21, 2021
11cbe9f
chore: update zisi
netlify-team-account-1 Dec 21, 2021
819ebc1
Merge branch 'main' into scheduled-jobs-endpoint
netlify-team-account-1 Dec 21, 2021
7481c2a
fix: enable parseISC hook
netlify-team-account-1 Dec 21, 2021
227fe7a
feat: give full command
netlify-team-account-1 Jan 6, 2022
c61a214
refactor: move clockwork simulation to calling side
netlify-team-account-1 Jan 6, 2022
3f40a89
chore: remove one changed line
netlify-team-account-1 Jan 6, 2022
1779803
refactor: extract clockwork useragent into constants
netlify-team-account-1 Jan 6, 2022
01c986a
feat: improve error message
netlify-team-account-1 Jan 6, 2022
162604a
Merge branch 'main' into scheduled-jobs-endpoint
netlify-team-account-1 Jan 6, 2022
6879da4
feat: print friendly error screen
netlify-team-account-1 Jan 7, 2022
b1d09aa
Merge branch 'main' into scheduled-jobs-endpoint
netlify-team-account-1 Jan 24, 2022
b207f5a
chore: trim down diff to npm-shrinkwrap
netlify-team-account-1 Jan 24, 2022
6abbf39
chore: remove mock-require (not used anymore)
netlify-team-account-1 Jan 24, 2022
30e7ba4
fix: optional chaining doesnt exist
netlify-team-account-1 Jan 24, 2022
9f8de55
chore: add test for helpful tips and tricks
netlify-team-account-1 Jan 24, 2022
07add5b
fix: correct tests
netlify-team-account-1 Jan 24, 2022
fa6f2d5
fix: add missing property to test
netlify-team-account-1 Jan 24, 2022
b8d06e1
Update src/lib/functions/runtimes/js/builders/zisi.js
netlify-team-account-1 Jan 27, 2022
c266016
refactor: extract help response into its own testable function
netlify-team-account-1 Jan 27, 2022
6015fd6
chore: add some unit tests
netlify-team-account-1 Jan 27, 2022
a866d96
Merge branch 'main' into scheduled-jobs-endpoint
netlify-team-account-1 Jan 27, 2022
603d146
fix: replaceAll not available on node v12
netlify-team-account-1 Jan 28, 2022
acb4504
fix: remove unneeded level field
netlify-team-account-1 Jan 28, 2022
9e5fd20
refactor: remove unused test matrix
netlify-team-account-1 Jan 28, 2022
fea14a8
fix: increase file change delay for macOS and windows
netlify-team-account-1 Jan 28, 2022
17285f4
Merge branch 'main' into scheduled-jobs-endpoint
Skn0tt Jan 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 19 additions & 126 deletions npm-shrinkwrap.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@
"@netlify/plugin-edge-handlers": "^3.0.0",
"@netlify/plugins-list": "^6.2.0",
"@netlify/routing-local-proxy": "^0.34.1",
"@netlify/zip-it-and-ship-it": "5.3.1",
"@netlify/zip-it-and-ship-it": "5.4.0",
netlify-team-account-1 marked this conversation as resolved.
Show resolved Hide resolved
"@octokit/rest": "^18.0.0",
"@sindresorhus/slugify": "^1.1.0",
"ansi-styles": "^5.0.0",
Expand Down
10 changes: 5 additions & 5 deletions src/commands/functions/functions-invoke.js
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,9 @@ const functionsInvoke = async (nameArgument, options, command) => {
const functions = await getFunctions(functionsDir)
const functionToTrigger = await getNameFromArgs(functions, options, nameArgument)

let headers = {}
const headers = {
netlify-team-account-1 marked this conversation as resolved.
Show resolved Hide resolved
'user-agent': 'netlify-cli',
netlify-team-account-1 marked this conversation as resolved.
Show resolved Hide resolved
}
let body = {}
netlify-team-account-1 marked this conversation as resolved.
Show resolved Hide resolved

if (eventTriggeredFunctions.has(functionToTrigger)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think scheduled functions should fall on this branch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expand Down Expand Up @@ -201,10 +203,8 @@ const functionsInvoke = async (nameArgument, options, command) => {
isAuthenticated = options.identity
}
if (isAuthenticated) {
headers = {
authorization:
'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJuZXRsaWZ5IGZ1bmN0aW9uczp0cmlnZ2VyIiwidGVzdERhdGEiOiJORVRMSUZZX0RFVl9MT0NBTExZX0VNVUxBVEVEX0pXVCJ9.Xb6vOFrfLUZmyUkXBbCvU4bM7q8tPilF0F03Wupap_c',
}
headers.authorization =
'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzb3VyY2UiOiJuZXRsaWZ5IGZ1bmN0aW9uczp0cmlnZ2VyIiwidGVzdERhdGEiOiJORVRMSUZZX0RFVl9MT0NBTExZX0VNVUxBVEVEX0pXVCJ9.Xb6vOFrfLUZmyUkXBbCvU4bM7q8tPilF0F03Wupap_c'
// you can decode this https://jwt.io/
// {
// "source": "netlify functions:trigger",
Expand Down
1 change: 1 addition & 0 deletions src/lib/functions/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ const normalizeFunctionsConfig = ({ functionsConfig = {}, projectRoot }) =>
ignoredNodeModules: config.ignored_node_modules,
nodeBundler: config.node_bundler === 'esbuild' ? 'esbuild_zisi' : config.node_bundler,
processDynamicNodeImports: true,
schedule: config.schedule,
},
}),
{},
Expand Down
10 changes: 9 additions & 1 deletion src/lib/functions/netlify-function.js
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ class NetlifyFunction {
// Determines whether this is a background function based on the function
// name.
this.isBackground = name.endsWith(BACKGROUND_SUFFIX)
this.schedule = null

// List of the function's source files. This starts out as an empty set
// and will get populated on every build.
Expand All @@ -44,6 +45,12 @@ class NetlifyFunction {
return /^[A-Za-z0-9_-]+$/.test(this.name)
}

async isScheduled() {
await this.buildQueue

return Boolean(this.schedule)
}

// The `build` method transforms source files into invocable functions. Its
// return value is an object with:
//
Expand All @@ -61,12 +68,13 @@ class NetlifyFunction {
this.buildQueue = buildFunction({ cache })

try {
const { srcFiles, ...buildData } = await this.buildQueue
const { schedule, srcFiles, ...buildData } = await this.buildQueue
const srcFilesSet = new Set(srcFiles)
const srcFilesDiff = this.getSrcFilesDiff(srcFilesSet)

this.buildData = buildData
this.srcFiles = srcFilesSet
this.schedule = schedule

return { srcFilesDiff }
} catch (error) {
Expand Down
31 changes: 25 additions & 6 deletions src/lib/functions/runtimes/js/builders/zisi.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
const { writeFile } = require('fs').promises
const path = require('path')

const { zipFunction } = require('@netlify/zip-it-and-ship-it')
const { listFunction, zipFunction } = require('@netlify/zip-it-and-ship-it')
const decache = require('decache')
const makeDir = require('make-dir')
const readPkgUp = require('read-pkg-up')
Expand Down Expand Up @@ -36,7 +36,11 @@ const buildFunction = async ({ cache, config, directory, func, hasTypeModule, pr
// root of the functions directory (e.g. `functions/my-func.js`). In
// this case, we use `mainFile` as the function path of `zipFunction`.
const entryPath = functionDirectory === directory ? func.mainFile : functionDirectory
const { inputs, path: functionPath } = await memoizedBuild({
const {
inputs,
path: functionPath,
schedule,
} = await memoizedBuild({
cache,
cacheKey: `zisi-${entryPath}`,
command: () => zipFunction(entryPath, targetDirectory, zipOptions),
Expand All @@ -57,7 +61,19 @@ const buildFunction = async ({ cache, config, directory, func, hasTypeModule, pr

clearFunctionsCache(targetDirectory)

return { buildPath, srcFiles }
return { buildPath, srcFiles, schedule }
}

/**
* @param {string} mainFile
netlify-team-account-1 marked this conversation as resolved.
Show resolved Hide resolved
*/
const parseForSchedule = async ({ config, mainFile, projectRoot }) => {
const listedFunction = await listFunction(mainFile, {
config: netlifyConfigToZisiConfig({ config, projectRoot }),
parseISC: true,
})

return listedFunction && listedFunction.schedule
}

// Clears the cache for any files inside the directory from which functions are
Expand All @@ -80,10 +96,11 @@ const getTargetDirectory = async ({ errorExit }) => {
return targetDirectory
}

const netlifyConfigToZisiConfig = ({ config, projectRoot }) =>
addFunctionsConfigDefaults(normalizeFunctionsConfig({ functionsConfig: config.functions, projectRoot }))

module.exports = async ({ config, directory, errorExit, func, projectRoot }) => {
const functionsConfig = addFunctionsConfigDefaults(
normalizeFunctionsConfig({ functionsConfig: config.functions, projectRoot }),
)
const functionsConfig = netlifyConfigToZisiConfig({ config, projectRoot })

const packageJson = await readPkgUp(func.mainFile)
const hasTypeModule = packageJson && packageJson.packageJson.type === 'module'
Expand Down Expand Up @@ -116,3 +133,5 @@ module.exports = async ({ config, directory, errorExit, func, projectRoot }) =>
target: targetDirectory,
}
}

module.exports.parseForSchedule = parseForSchedule
3 changes: 2 additions & 1 deletion src/lib/functions/runtimes/js/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ const getBuildFunction = async ({ config, directory, errorExit, func, projectRoo
// main file otherwise.
const functionDirectory = dirname(func.mainFile)
const srcFiles = functionDirectory === directory ? [func.mainFile] : [functionDirectory]
const schedule = await detectZisiBuilder.parseForSchedule({ mainFile: func.mainFile, config, projectRoot })

return () => ({ srcFiles })
return () => ({ schedule, srcFiles })
}

const invokeFunction = async ({ context, event, func, timeout }) => {
Expand Down
24 changes: 24 additions & 0 deletions src/lib/functions/scheduled.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
const { NETLIFYDEVERR, NETLIFYDEVLOG } = require('../../utils')

const { formatLambdaError, styleFunctionName } = require('./utils')

const SCHEDULED_FUNCTION_STATUS_CODE = 202

const handleScheduledFunction = (functionName, response) => {
console.log(`${NETLIFYDEVLOG} Running scheduled function ${styleFunctionName(functionName)}`)
response.status(SCHEDULED_FUNCTION_STATUS_CODE)
response.end()
}

const handleScheduledFunctionResult = (functionName, err) => {
if (err) {
console.log(
`${NETLIFYDEVERR} Error during scheduled function ${styleFunctionName(functionName)} execution:`,
formatLambdaError(err),
)
} else {
console.log(`${NETLIFYDEVLOG} Done executing scheduled function ${styleFunctionName(functionName)}`)
}
}

module.exports = { handleScheduledFunction, handleScheduledFunctionResult }
11 changes: 11 additions & 0 deletions src/lib/functions/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ const { NETLIFYDEVERR, NETLIFYDEVLOG, error: errorExit, getInternalFunctionsDir,
const { handleBackgroundFunction, handleBackgroundFunctionResult } = require('./background')
const { createFormSubmissionHandler } = require('./form-submissions-handler')
const { FunctionsRegistry } = require('./registry')
const { handleScheduledFunction, handleScheduledFunctionResult } = require('./scheduled')
const { handleSynchronousFunction } = require('./synchronous')
const { shouldBase64Encode } = require('./utils')

Expand Down Expand Up @@ -105,6 +106,16 @@ const createHandler = function ({ functionsRegistry }) {
const { error } = await func.invoke(event, clientContext)

handleBackgroundFunctionResult(functionName, error)
} else if (await func.isScheduled()) {
if (event.headers['user-agent'] !== 'netlify-cli') {
return response.status(400).send('Scheduled functions can only be invoked using `netlify functions:invoke`.')
netlify-team-account-1 marked this conversation as resolved.
Show resolved Hide resolved
}

handleScheduledFunction(functionName, response)

const { error } = await func.invoke(event, clientContext)

handleScheduledFunctionResult(functionName, error)
} else {
const { error, result } = await func.invoke(event, clientContext)

Expand Down
Loading