Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Going deeper on fork orchestration #72

Merged
merged 20 commits into from
Aug 17, 2017
Merged

Going deeper on fork orchestration #72

merged 20 commits into from
Aug 17, 2017

Conversation

tiagofilipe12
Copy link
Member

This PR intends to at the very least increase the level of forkception, i.e., the ability to had forks inside forks.
For now it just handles one fork inside another fork, but has no capacity to handle a third fork. Also and importantly, fork can proceed a fork, i.e. we can have something like this:

const pipeline3 = join(
  task0,
  fork(
      fork(
        task1,
        task3
      ),
      task6
  ),
  task5
)

@codecov-io
Copy link

codecov-io commented Aug 11, 2017

Codecov Report

Merging #72 into dev will decrease coverage by 0.04%.
The diff coverage is 87.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev      #72      +/-   ##
==========================================
- Coverage   81.88%   81.83%   -0.05%     
==========================================
  Files          37       37              
  Lines         861      881      +20     
  Branches      102      107       +5     
==========================================
+ Hits          705      721      +16     
- Misses        156      160       +4
Impacted Files Coverage Δ
lib/orchestrators/fork.js 100% <ø> (ø) ⬆️
lib/lifecycle/resolve-input.js 73.33% <ø> (-0.44%) ⬇️
lib/task.js 100% <ø> (ø) ⬆️
lib/orchestrators/join.js 92.3% <87.87%> (-4.19%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7359ced...db98270. Read the comment docs.

@tiagofilipe12 tiagofilipe12 changed the title Going deepet on fork orchestration Going deeper on fork orchestration Aug 11, 2017
// with the joint tasks plus the outermostTask with a new uid
const lineage = [forkee].concat(newUpStreamTasks).concat(newOutermostTasks)
lineageGetter(dispatch, lineage, joinages)
} else { // if a fork is found within another fork
Copy link
Member

@thejmazz thejmazz Aug 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the number of nested forks supported "hardcoded" based on these nested if/else blocks?

@tiagofilipe12
Copy link
Member Author

This PR now also includes new pipelines for testing multiple inputs as well as new uid generation strategy in which upstream tasks are included in uid creation. Check this GSoC17 entry.

@tiagofilipe12
Copy link
Member Author

Ok, I think it is ready to merge now, but codecov complains about code that is necessary for some of the tests I ran. Maybe you ( @thejmazz ) can find a way to simplify the code.

@tiagofilipe12
Copy link
Member Author

Also added a concurrency pipeline example using bluebird Promise.

@thejmazz
Copy link
Member

As long as tests are passing I don't mind merging even if code coverage drops. However, if you look at the coverage, it is just that if (forkee.info.props) and else if (forkee.info.type === 'fork') are not being ran in any of the tests. So if you can take a look how to resolve that, it will probably be enough to satisfy codecov.

@thejmazz thejmazz merged commit 8fa4702 into dev Aug 17, 2017
@bmpvieira bmpvieira removed the bug label Aug 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants