Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fsotre 342 break test2 #2

Closed
wants to merge 92 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
26ae8d1
Bump hudi to 0.10.0.2 (#680)
tdoehmen Jun 16, 2022
6ccde4f
remove printing training dataset old URL (#683)
dhananjay-mk Jun 16, 2022
df4f81d
Revert "[HOPSWORKS-3201][append] set training dataset start end time …
SirOibaf Jun 16, 2022
2bfc8d4
Revert "[HOPSWORKS-3201] Set training dataset start and end time (#677)"
SirOibaf Jun 16, 2022
3da789f
Read HopsFS training dataset using the rest APIs (#685)
SirOibaf Jun 16, 2022
cf24df8
[HOPSWORKS-3186] fv td redesign java client (#672)
kennethmhc Jun 17, 2022
7e66147
increase sleep (#687)
moritzmeister Jun 17, 2022
73f58d4
Add expectation suite argument to get_or_create (#688)
moritzmeister Jun 17, 2022
7767e99
add new line character to printed links (#689)
moritzmeister Jun 17, 2022
f24a597
[HOPSWORKS-3220] HopsKafkaAuthorizer retry and refresh ACLs cache if …
moritzmeister Jun 27, 2022
8f36f16
HOPSWORKS-3225 change throwing error to warning while statistics on …
dhananjay-mk Jun 28, 2022
cb9bef7
Add mkdocs-jupyter dependency for docs (#699)
moritzmeister Jul 1, 2022
777b20c
[FeatureView] Rename methods and variables of feature view (#696)
kennethmhc Jul 5, 2022
a54767e
[HOPSWORKS-3233] Bump Hudi to 0.10.0.3 and Spark to 3.1.1.2 (#698)
moritzmeister Jul 5, 2022
fe3c64f
Refactor docs for API docs only (#702)
moritzmeister Jul 7, 2022
8d8ded4
[HOPSWORKS-3235] Use Hudi snapshot query when users doesn't specify a…
SirOibaf Jul 10, 2022
78fcba4
[HOPSWORKS 3241] add progress bar during stream data ingestion in pyt…
dhananjay-mk Jul 11, 2022
09d0280
[HOPSWORKS-3233] Bump spark to 3.1.1.3 (#706)
tdoehmen Jul 11, 2022
2102bfa
[HOPSWORKS-3209] Link to UI + saved suite modified inplace instead of…
vatj Jul 11, 2022
c2a86ca
[HOPSWORKS-3248] rename on_demand_feature_group to external_feature_g…
davitbzh Jul 12, 2022
747dcd1
fix auto doc (#708)
moritzmeister Jul 12, 2022
08eb2c6
add num_feature_views (#711)
moritzmeister Jul 12, 2022
d7380ac
[HOPSWORKS-3178] Remove Deequ based data validation in favour of GE (…
moritzmeister Jul 13, 2022
ac15fc3
[HOPSWORKS-3235] Append avoid index out of range when reading the fea…
SirOibaf Jul 13, 2022
a0c6635
[HOPSWORKS-3178][append] Remove remaining access to old validation_ty…
moritzmeister Jul 13, 2022
fa73943
[HOPSWORKS-3242] Improve handling of different data types in feature …
tdoehmen Jul 13, 2022
5776172
[HOPSWORKS-3235] Fix pre_combine_key in Hudi Engine (#716)
SirOibaf Jul 14, 2022
7582738
reverse serving key check logic: check if all the necessary keys from…
davitbzh Jul 15, 2022
6caba0a
added init prepared statement call and updated parameters (#720)
tdoehmen Jul 15, 2022
f077e98
Add pymdown-extensions to dependencies (#723)
moritzmeister Jul 18, 2022
87822a7
Fix markdown version to 3.3.7 (#724)
moritzmeister Jul 18, 2022
11bf369
Prepare development for 3.1.0-SNAPSHOT
SirOibaf Jul 18, 2022
6ebd8fe
Hide major version alias in version select
Jul 25, 2022
340d569
Add latest docs alias to 3.0
Jul 25, 2022
e29a1d4
made time_travel_format optional to fix external feature groups (#732)
tdoehmen Jul 25, 2022
1afe210
Adjust links for updated tutorials structure (#734)
moritzmeister Jul 26, 2022
bd4b209
Add feature view api docs (#736)
moritzmeister Jul 26, 2022
abc3770
Fix feature view type hints (#738)
moritzmeister Jul 26, 2022
a137593
Add feature view to docs navigation (#739)
moritzmeister Jul 27, 2022
4f7f87c
Fix feature view features/labels creation (#743)
kennethmhc Jul 28, 2022
e351490
Add unit testing action (#745)
moritzmeister Jul 28, 2022
290527f
Fix atttribute access in validation report repr (#747)
moritzmeister Aug 3, 2022
29eafa1
Add external feature group to API docs (#750)
moritzmeister Aug 4, 2022
e5e20f7
[HOPSWORKS-3200] time series split (#701)
kennethmhc Aug 5, 2022
6fb8776
[HOPSWORKS-3250] Add serving key (#718)
kennethmhc Aug 5, 2022
678f5cc
[HOPSWORKS-3263] Add GE data validation helper job (#746)
moritzmeister Aug 8, 2022
e549394
Use development version of mike to publish documentation (#753) (#754)
robzor92 Aug 9, 2022
6293ec0
[HOPSWORKS-3219] remove preview vector (#748)
kennethmhc Aug 9, 2022
2501cf2
[HOPSWORKS-3259] Normalize API for reading changes based on commit ti…
tdoehmen Aug 10, 2022
0df398d
Add get_or_create api docs to feature group and clean up documentation
moritzmeister Aug 14, 2022
408fefa
hopsworks link should point to login (#757)
robzor92 Aug 18, 2022
bfee4c1
[HOPSWORKS-3200] time series java (#762)
kennethmhc Aug 22, 2022
dfd0a2d
[HOPSWORKS-3259][Append] Fix Docstrings for .as_of functions (#761)
tdoehmen Aug 23, 2022
1aef748
[HOPSWORKS-3317] Fix the validation_ingestion_policy (#765)
vatj Sep 1, 2022
2b68f1d
Fix Java checkstyle (#769)
SirOibaf Sep 6, 2022
87e1995
PR Template (#768)
tdoehmen Sep 8, 2022
6be5cb9
[HOPSWORKS-3260] Add python client tests (#741)
bubriks Sep 8, 2022
b580e98
[HOPSWORKS-3260][Append] Remove tests from wheel (#771)
tdoehmen Sep 8, 2022
5334f0c
[HOPSWORKS-3327] Reorder split output (#770)
kennethmhc Sep 8, 2022
5724cdf
[HOPSWORKS-3202] Add filter to td (#707)
kennethmhc Sep 9, 2022
6e45aa0
[HOPSWORKS-3202][append] Fix parameter type mismatch (#777)
kennethmhc Sep 14, 2022
919c77e
Selecting a different version dropdown should land on correct page (#…
robzor92 Sep 15, 2022
dfbc6b7
[HOPSWORKS-3259][append] Allow wallclocktime=None for as_of (#779)
tdoehmen Sep 15, 2022
083c3bc
[HOPSWORKS-3352] fix HSFS test flake8 (#782)
bubriks Sep 15, 2022
c0b3ae8
[HOPSWORKS-3278] add type to feature view dto (#760)
ErmiasG Sep 19, 2022
48ed04b
[HOPSWORKS-3229] OnlineFS crashes if it can't deserialize a record (#…
bubriks Sep 19, 2022
659de1f
[HOPSWORKS-3325] Parquet reading in hsfs client (#776)
tdoehmen Sep 20, 2022
d9ec58b
fixed timezone issue (#789)
tdoehmen Sep 20, 2022
23f635e
[FSTORE-53] apply_transformation_function provided Training Dataset i…
bubriks Sep 21, 2022
2c6864d
[HOPSWORKS-3323] Fix TDS creation in PySpark client and add excplicit…
tdoehmen Sep 21, 2022
6f57b55
[FSTORE-56] Enable cleaning of fv (#788)
kennethmhc Sep 21, 2022
fea15a0
[FSTORE-56][APPEND] Check parameter type (#791)
kennethmhc Sep 21, 2022
86c2ce5
[HOPSWORKS-3337] add job to import data into feature group (#774)
dhananjay-mk Sep 21, 2022
ac21e4d
[HOPSWORKS-3326] Support datetime in hsfs python API (#783)
davitbzh Sep 22, 2022
811f5e9
[HOPSWORKS-3326] APPEND - Support datetime in hsfs python API (#793)
davitbzh Sep 22, 2022
27335d5
[HOPSWORKS-3342] add return types to transformation functions backwar…
tdoehmen Sep 23, 2022
ca04341
[FSTORE-315] Upgrading GE version to 0.14.12 to support Jinja2 versio…
robzor92 Sep 24, 2022
fa9ef26
[FSTORE-11] Add mkdocs macros dependency for docs (#800)
moritzmeister Sep 27, 2022
840db40
[HOPSWORKS-3296] Remove update of storage connector (#801)
moritzmeister Sep 29, 2022
af9a3de
[FSTORE-329] CommitDTO type field is not returned anymore after Payar…
SirOibaf Sep 29, 2022
5585633
[FSTORE-8] add query default argument in snowflake (#790)
dhananjay-mk Sep 30, 2022
debfee7
[FSTORE-8][append] Add unit test (#808)
moritzmeister Oct 3, 2022
c130236
[FSTORE-33] Improve HSFS parameter description (#806)
moritzmeister Oct 3, 2022
a587bed
[FSTORE-331] BigQuery connector doesn't work if the application is ex…
SirOibaf Oct 3, 2022
9b79da4
[FSTORE-8][append] Java fix for snowflake connector (#810)
moritzmeister Oct 3, 2022
05c1481
[FSTORE-345] Update documentation to reflect supported methods in hsf…
moritzmeister Oct 4, 2022
7b13538
[HOPSWORKS-3345] Improve event time error message (#785)
kennethmhc Oct 4, 2022
8c1121b
[FSTORE-348] Refactor Tutorials documentation page (#817)
moritzmeister Oct 4, 2022
8c83331
[FSTORE-342] Add java tests to github pipeline
SirOibaf Oct 4, 2022
b599783
Fix Java build
SirOibaf Oct 4, 2022
d45c4e5
Add m2 caching
SirOibaf Oct 4, 2022
d9d24cd
break test
tdoehmen Oct 5, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
32 changes: 32 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
This PR adds/fixes/changes...
- please summarize your changes to the code
- and make sure to include all changes to user-facing APIs

JIRA Issue: -

Priority for Review: -

Related PRs: -

**How Has This Been Tested?**

- [ ] Unit Tests
- [ ] Integration Tests
- [ ] Manual Tests on VM


**Checklist For The Assigned Reviewer:**

```
- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)
```
30 changes: 30 additions & 0 deletions .github/workflows/java-ut.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: java

on: pull_request

jobs:
unit_tests:
name: Java Unit Tests
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Set up JDK 8
uses: actions/setup-java@v3
with:
java-version: '8'
distribution: 'adopt'

- name: Cache local Maven repository
uses: actions/cache@v2
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-${{ hashFiles('java/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-

- name: Test
working-directory: ./java
run: mvn --batch-mode -Dmaven.test.failure.ignore=true test
16 changes: 11 additions & 5 deletions .github/workflows/mkdocs-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,23 @@ jobs:
with:
fetch-depth: 0

- name: set release version
- name: set major/minor/bugfix release version
working-directory: ./java
run: echo "RELEASE_VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -Ev 'Download|INFO|WARNING')" >> $GITHUB_ENV

- name: set major/minor release version
run: echo "MAJOR_VERSION=$(echo $RELEASE_VERSION | sed 's/^\([0-9]*\.[0-9]*\).*$/\1/')" >> $GITHUB_ENV

- uses: actions/setup-python@v2
with:
python-version: '3.8'
- run: pip3 install pip==22.0.3

- name: install deps
working-directory: ./python
run: cp ../README.md . && pip3 install -e .[python,dev,docs]
run: cp ../README.md . && pip3 install pip==22.0.3 && pip3 install -e .[python,dev,docs]

- name: use dev mike
run: pip3 uninstall -y mike && pip3 install git+'https://github.com/jimporter/mike.git'

- name: generate autodoc
run: python3 auto_doc.py
Expand All @@ -33,6 +38,7 @@ jobs:
run: |
git config --global user.name Mike
git config --global user.email mike@docs.hopsworks.ai

- name: mike deploy docs
run: mike deploy ${{ env.RELEASE_VERSION }} latest -u --push
run: |
mike deploy ${{ env.RELEASE_VERSION }} ${{ env.MAJOR_VERSION }} -u --push
mike alias ${{ env.RELEASE_VERSION }} latest -u --push
41 changes: 33 additions & 8 deletions .github/workflows/python-lint.yml
Original file line number Diff line number Diff line change
@@ -1,27 +1,52 @@
name: pre-commit
name: python

on: pull_request

jobs:
stylecheck:
lint_stylecheck:
name: Lint and Stylecheck
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.8'
- name: install deps
run: pip install flake8==3.9.0 black==22.3.0 pre-commit-hooks==2.4.0

- name: black
run: black --check python/hsfs
run: black --check python

- name: flake8
run: flake8 --config python/.flake8 python/hsfs
run: flake8 --config python/.flake8 python

- name: trailing-whitespace-fixer
run: trailing-whitespace-fixer $(find python/hsfs -type f) || exit 1
run: trailing-whitespace-fixer $(find python -name "*.py" -type f) || exit 1

- name: end-of-file-fixer
run: end-of-file-fixer $(find python/hsfs -type f) || exit 1
run: end-of-file-fixer $(find python -name "*.py" -type f) || exit 1

unit_tests:
name: Unit Testing
needs: lint_stylecheck
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.7", "3.8", "3.9"]

steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
name: Setup Python
with:
python-version: ${{ matrix.python-version }}
cache: "pip"
cache-dependency-path: "python/setup.py"
- run: pip install -e python[python,dev,docs]

- name: Display Python version
run: python --version

- name: Run Pytest suite
run: pytest python/tests
Loading