Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New CompressFiles step & stability fixes in Job Service #1438

Merged
merged 41 commits into from
Jan 30, 2025
Merged

New CompressFiles step & stability fixes in Job Service #1438

merged 41 commits into from
Jan 30, 2025

Conversation

roegi
Copy link
Contributor

@roegi roegi commented Jan 30, 2025

  • Introduce new CompressFiles step in job service
  • Several Stability fixes for job service
  • Improve local starting scripts to work on windows

roegi and others added 30 commits January 16, 2025 16:45
@roegi
Improve user-facing input validation error message stating that the i…
30e6100 
…nputs have to be re-uploaded in the correct format

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@mchrza
Update exec plugin
2369231 
Signed-off-by: Maximilian Chrzan <maximilian.chrzan@here.com>
Signed-off-by: mchrza <maximilian.chrzan@here.com>
@mchrza
Add scripts to Job-Jar`s.
c1a4cb1 
Use Docker exec to install scripts for profile "local".

Signed-off-by: Maximilian Chrzan <maximilian.chrzan@here.com>
Signed-off-by: mchrza <maximilian.chrzan@here.com>
@roegi
Refactor: Remove Step#resume() & add a boolean parameter to #execute(…
8f56ac3 
…) instead

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi
Fix job-service POM to not try to execute localSetup on build in general
be78ca0 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@mchrza
Add parameters to script.
90f6e38 
Signed-off-by: Maximilian Chrzan <maximilian.chrzan@here.com>
Signed-off-by: mchrza <maximilian.chrzan@here.com>
@osolodovn-here
Add additional command for Windows
6fb36f6 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@osolodovn-here
Add test helper
a3d1454 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@osolodovn-here
Merge remote-tracking branch 'origin/job-dev' into job-dev
ec67047
@roegi
Keep the possibility to execute deployLocalLambda.sh script on host d…
bb55584 
…irectly

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi
Minor fixes & improvements in setup scripts
4d8f6fa 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@osolodovn-here
Merge remote-tracking branch 'origin/job-dev' into job-dev
4328f0b
@osolodovn-here
Change Line Separator to LF from CRLF
5d420f1 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@osolodovn-here
Add helper class for tests
b765386 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@osolodovn-here
Enable Unix command by default
5bf4f0f 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@roegi
Merge branch 'master' into job-dev
e2d8e16
@roegi
Fix jobServiceSetup run-config to not contain backslash
6e01d40 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi
Fix JobService run-config to call jobServiceSetup upfront again
b3b4546 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@osolodovn-here
Add archive-related methods
702fafd 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@roegi
Add missing error response handling in CopySpacePost (PATCH space req…
fe456b2 
…uest)

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi
Merge branch 'master' into job-dev
92b1fc2 
# Conflicts:
#	xyz-jobs/xyz-job-steps/src/main/java/com/here/xyz/jobs/steps/impl/transport/CopySpacePost.java
@roegi
Fix deployLocalLambda script to run on host again
3c0ced1 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@mujammil10
Add metadata in S3DataFile
4da4252 
Signed-off-by: Ansari, Mujammil <mujammil.ansari@here.com>
@roegi
Fix deployLocalLambda script and xyz-job-steps exec plugin config
1caa0e9 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@osolodovn-here
Add desired contained filesize feature
8040cf7 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@roegi
Merge branch 'master' into job-dev
5cc3b34
@osolodovn-here
Add optional archive prefix title
9080aca 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@roegi
Set fileName override at the Output class to keep signature of Step#r…
b2202e5 
…egisterOutputs() generic

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@osolodovn-here
Reset buffer on flushing
67a4a36 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@osolodovn-here
Add unwrapping folders option
1fa0a08 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
roegi and others added 11 commits January 29, 2025 11:32
@roegi
Fix race-condition between registering & unregistering the state-chec…
5b02574 
…k trigger (CW event rule)

- Also add some specific logging for trigger creation errors

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@osolodovn-here
Merge remote-tracking branch 'origin/job-dev' into job-dev
f182a8a
@osolodovn-here
Remove duplicated code
cfb12fa 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@osolodovn-here
Fix method visibility
3b6c2e4 
Signed-off-by: Oleksandr Solodovnikov <ext-oleksandr.solodovnikov@here.com>
@roegi
Fix issue that replacement of user-inputs S3 Urls in EMR parameters d…
25b39c5 
…id not work if the user inputs were provided using "InputsFromS3"

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@qGYdXbY2
ExportSpaceBy. with Ref.isRange (#1437)
132302a 
* ExportSpaceBy. with Ref.isRange

Signed-off-by: qGYdXbY2 <47661341+qGYdXbY2@users.noreply.github.com>

* todo comment

Signed-off-by: qGYdXbY2 <47661341+qGYdXbY2@users.noreply.github.com>

---------

Signed-off-by: qGYdXbY2 <47661341+qGYdXbY2@users.noreply.github.com>
@roegi
Implement own serializable S3Client.S3Uri that is not dependent on lo…
5599e38 
…cal config

Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi
Remove extra import of S3Utilities
6997a74 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi
Fix InputSet#toS3Uri()
37f6c5a 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@mchrza
Cache datasource.
f95c6c3 
Signed-off-by: Maximilian Chrzan <maximilian.chrzan@here.com>
Signed-off-by: mchrza <maximilian.chrzan@here.com>
@roegi
Add further helper constructor to InputSet record
20e92dd 
Signed-off-by: Benjamin Rögner <benjamin.roegner@here.com>
@roegi roegi requested a review from a team as a code owner January 30, 2025 15:38
@roegi roegi merged commit be26430 into master Jan 30, 2025
3 checks passed
@roegi roegi deleted the job-dev branch January 30, 2025 16:22
@roegi roegi restored the job-dev branch January 30, 2025 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mchrza mchrza mchrza approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@roegi @mchrza @osolodovn-here @mujammil10 @qGYdXbY2