Scheduled Jobs Implementation #49346
Assignees
Labels
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Comments
|
Hi @miretskiy, I've guessed the C-ategory of your issue and suitably labeled it. Please re-label if inaccurate. While you're here, please consider adding an A- label to help keep our repository tidy. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan. |
blathers-crl
bot
added
the
C-bug
This was referenced May 27, 2020
craig bot
pushed a commit
that referenced
this issue
May 29, 2020
49444: sqlbase: Add created_by columns to system.jobs r=miretskiy a=miretskiy Informs #49346 Add created_by_type and created_by_id columns, along with the index over these columns, to the system.jobs table. Add sql migration code to migrate old definition to the new one. See https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20200414_scheduled_jobs.md Release Notes: None 49680: sql: add LeakProof volatility r=RaduBerinde a=RaduBerinde This change adds VolatilityLeakProof and changes a few hashing builtins to use it (for consistency with postgres). In postgres, `proleakproof` is a flag separate from `provolatile`, but all instances of leak-proof operators are immutable. We also change the volatility to a simple index since it no longer maps to the postgres volatility directly. In the optimizer, we will need to have a bitfield for which types of volatilities are contained in an expression so having them be small consecutive integers is desirable. Release note: None  49713: vendor: bump pebble to a41e4ac9 r=jbowens a=jbowens a41e4a *: Update level_checker to account for sublevels b14003 internal/manifest: Relax L0 CheckOrdering invariants for flush splits 9249ee db: add max memtable and max batch size checks bb2f85 vfs: return nil from RemoveAll for nonexistent paths eb5d41 db: use compaction picker's scores directly in (*db).Metrics 65313e cmd/pebble: sample read amp throughout compaction benchmarks Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com> Co-authored-by: Radu Berinde <radu@cockroachlabs.com> Co-authored-by: Jackson Owens <jackson@cockroachlabs.com>
craig bot
pushed a commit
that referenced
this issue
Jun 4, 2020
49748: ui: Cypress regression visual testing r=koorosh a=koorosh Resolves #49589 Visual regression testing is accomplished by `cypress-image-snapshot` plugin. The main issue solved with this change is how modules are loaded from custom `node_modules` location (./opt/node_modules). - for .ts files `baseUrl` and `typeRoots` are pointing to `./opt/node_modules` to resolve packages and typings. - Webpack configuration is extended with additional module resolver. - Plugin modules (which aren't processed by Webpack) use helper function which appends path to `opt/node_modules` to required module name. Add screenshot testing for main pages in Admin UI. These tests are quite flaky and might fail because nondeterministic state of the server during tests run. Release note: None 49765: sqlbase: Add system.scheduled_jobs table r=miretskiy a=miretskiy Informs #49346 Add system.scheduled_job table to system catalog. Add sql migration code to create table when upgrading. See https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20200414_scheduled_jobs.md Release Notes: None Co-authored-by: Andrii Vorobiov <and.vorobiov@gmail.com> Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com>
craig bot
pushed a commit
that referenced
this issue
Jun 9, 2020
49599: pkg/jobs: Scheduled jobs r=miretskiy a=miretskiy Informs #49346 Add initial implementation of scheduled jobs framework. Scheduled job is an extension of the system job, which can run periodically. Release notes: None 49666: cli: delete `cockroach node decommission --wait=live` r=irfansharif a=irfansharif It was deprecated a while ago. Release note: `cockroach node decommission --wait=live` is no longer supported. It was deprecated in an earlier release. 49892: config/kv/storage: teach split queue / merge queue about tenant boundaries r=nvanbenschoten a=nvanbenschoten Fixes #48774. This PR updates `SystemConfig.NeedsSplit` and `SystemConfig.ComputeSplitKey` to take into account tenant rows in the `system.tenants` table and mandate split points at tenant keyspace boundaries. This ensures that the splitQueue will create splits between tenants and that the mergeQueue will avoid merging ranges for different tenants together. It then updates the `crdb_internal.create_tenant` builtin function so that it splits ranges synchronously when creating a new tenant. The prior commit taught the split and merge queues about tenant boundaries, which means that split points will be created asynchronously when a new tenant is created. However, it is valuable from an isolation perspective to split off tenant ranges synchronously upon the creation of a new tenant. Finally, the PR teaches MVCCFindSplitKey about tenant prefixes, ensuring that all split points that are chosen within secondary tenant keyspaces obey the same rules as those chosen within the system tenant keyspace. Namely, it ensures that splits are never performed between two column families in the same SQL row (this was already the case) and that we never split off an empty LHS range due to a first row that exceeds half of the range size (this is new). 49960: catalog,sqlbase,*: generalize descriptors, pass around wrapper structs r=rohany,lucy-zhang a=ajwerner This PR works towards enabling unified leasing of descriptors. I'm somewhat dissatisfied at the state it ends up leaving things but it is forward progress and I can't keep holding on to this. This PR began its life with an intention of creating a unified `DescriptorMeta` proto message which would contain shared fields and live on the `Descriptor`. That change would necessitate not passing around the raw members of `Descriptor.Union` as they would no longer retain the necessary fields. Ultimately, after this was mostly done, it was determined that that change was not worth the migration pain. Nevertheless, a good bit of work was done to plumb through the new higher level abstractions and that work it valuable. The biggest issue with reviewing this change will be `sqlbase,*: replace DescriptorProto with DescriptorInterface`, that commit does more than it should in a single commit but the time it is going to take to rip it apart doesn't seem obviously worth it given the overhead of keeping this change in sync. I want to re-iterate that this change represents a stepping stone and not an end-state. The plan is to proceed from here by ripping apart sqlbase into subpackages such that the implementations of each of the wrapper types lives in their own packages. Then the plan will be to replace the use of all concrete fields in the descriptor types with interfaces. At that point the `Mutable` vs `Immutable` distinction can be rephrased as an interface distinction. Doing the above should also aid in a number of other ways. 49982: roachtest: run all hibernate tests in nightlies r=apantel a=rafiss This also fixes the regex for checking for the failure cause in the error message. Release note: None 49995: bulkio: Correctly group producer/consumers when importing data r=miretskiy a=miretskiy Fixes #49989 Correctly link data import producer and consumer when importing data. Using separate groups for producer and consumer could lead to a situation where consumer exits (due to an error, or explicit context cancellation), while producer does not realize that, leading to a deadlock. Release notes (bug fix): Correctly link producer/consumer during data import in order to handle errors correctly. Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com> Co-authored-by: irfan sharif <irfanmahmoudsharif@gmail.com> Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com> Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com> Co-authored-by: Rafi Shamim <rafi@cockroachlabs.com>
craig bot
pushed a commit
that referenced
this issue
Jun 17, 2020
50240: jobs: Start scheduled jobs daemon on server startup. r=miretskiy a=miretskiy Start scheduled job daemon when starting cockroach binary. Informs #49346 Release notes: None 50249: logictest: introduce new configs for experimental distsql planning r=yuzefovich a=yuzefovich This commit adds three new configurations `local-exp-planning`, `fakedist-exp-planning`, and `5node-exp-planning` which override `experimental_distsql_planning` cluster setting to `on`. The corresponding config is included in `5node-defaults` list. And once the new factory is significantly implemented, it will probably make sense to add the configs to `default-configs`. Release note: None Co-authored-by: Yevgeniy Miretskiy <yevgeniy@cockroachlabs.com> Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
mwang1026
added
A-disaster-recovery
C-enhancement
|
Scheduled jobs implementation is done. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a tracking bug for scheduled jobs implementation.
Rough breakdown:
The text was updated successfully, but these errors were encountered: