Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exclude dependencies signatures from IngestionJob package #879

Merged
merged 4 commits into from
Jul 14, 2020
Merged

Exclude dependencies signatures from IngestionJob package #879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d8627e6
fix dataflow e2e
pyalex Jul 14, 2020
9018db4
consolidate jobs in e2e
pyalex Jul 14, 2020
0a2fff9
dataflow tests with consolidated jobs
pyalex Jul 14, 2020
cf59d04
dummy
pyalex Jul 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .prow/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ presubmits:

- name: test-end-to-end-batch-dataflow
decorate: true
always_run: false
always_run: true
spec:
volumes:
- name: service-account-df
Expand Down Expand Up @@ -311,7 +311,7 @@ presubmits:
- -c
- |
infra/scripts/download-maven-cache.sh \
--archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \
--archive-uri gs://feast-templocation-kf-feast/.m2.2020-07-13.tar \
--output-dir $PWD/

infra/scripts/publish-docker-image.sh \
Expand Down Expand Up @@ -440,7 +440,7 @@ postsubmits:
- -c
- |
infra/scripts/download-maven-cache.sh \
--archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \
--archive-uri gs://feast-templocation-kf-feast/.m2.2020-07-13.tar \
--output-dir $PWD/

if [ $PULL_BASE_REF == "master" ]; then
Expand Down
4 changes: 2 additions & 2 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -209,12 +209,12 @@
<dependency>
<groupId>com.google.api-client</groupId>
<artifactId>google-api-client</artifactId>
<version>1.27.0</version>
<version>1.30.9</version>
</dependency>
<dependency>
<groupId>com.google.apis</groupId>
<artifactId>google-api-services-dataflow</artifactId>
<version>v1b3-rev266-1.25.0</version>
<version>v1b3-rev20200305-1.30.9</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
Expand Down
2 changes: 1 addition & 1 deletion infra/scripts/test-end-to-end-batch-dataflow.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ Export required environment variables
============================================================
"

export TEMP_BUCKET=$TEMP_BUCKET
export TEMP_BUCKET=$TEMP_BUCKET/$HELM_RELEASE_NAME/$(date +%s)
export DATASET_NAME=$DATASET_NAME
export GCLOUD_PROJECT=$GCLOUD_PROJECT
export GCLOUD_NETWORK=$GCLOUD_NETWORK
Expand Down
1 change: 1 addition & 0 deletions infra/scripts/test-end-to-end-batch.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ feast:
jobs:
polling_interval_milliseconds: 10000
active_runner: direct
consolidate-jobs-per-source: true
runners:
- name: direct
type: DirectRunner
Expand Down
2 changes: 1 addition & 1 deletion infra/scripts/test-templates/values-end-to-end-batch-dataflow.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ feast-core:
specsAckTopic: $SPECS_TOPIC-ack
jobs:
active_runner: dataflow

consolidate-jobs-per-source: true
runners:
- name: dataflow
type: DataflowRunner
Expand Down
10 changes: 10 additions & 0 deletions ingestion/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,16 @@
<shadedPattern>com.google.cloud.bigquery.vendor</shadedPattern>
</relocation>
</relocations>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
Expand Down
2 changes: 1 addition & 1 deletion tests/e2e/bq/bq-batch-retrieval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ def test_batch_get_historical_features_with_gs_path(client, gcs_path):
uri = urlparse(gcs_path)
bucket = uri.hostname
ts = int(time.time())
remote_path = str(uri.path).strip("/") + f"{ts}/{file_name}"
remote_path = str(uri.path).lstrip("/") + f"{ts}/{file_name}"

# Upload file to gcs
storage_client = storage.Client(project=None)
Expand Down