Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add local dc property #312

Merged
merged 2 commits into from
Apr 16, 2021
Merged

Add local dc property #312

merged 2 commits into from
Apr 16, 2021

Conversation

github-felipe-caputo
Copy link
Contributor

Why? 📖

We need to be able to alter another new configuration on cassandra.

What? 🔧

New config spark.cassandra.connection.localDC which is default None: https://github.com/datastax/spark-cassandra-connector/blob/master/doc/reference.md

Type of change

  • New feature (non-breaking change which adds functionality)

How everything was tested? 📏

New tests added!

Checklist

  • My code follows the style guidelines of this project (docstrings, type hinting and linter compliance);
  • I have performed a self-review of my own code;
  • I have made corresponding changes to the documentation;
  • I have added tests that prove my fix is effective or that my feature works;
  • New and existing unit tests pass locally with my changes;
  • Add labels to distinguish the type of pull request. Available labels are bug, enhancement, feature, and review.

@github-felipe-caputo github-felipe-caputo self-assigned this Apr 16, 2021
@github-felipe-caputo github-felipe-caputo requested a review from a team as a code owner April 16, 2021 14:26
@github-felipe-caputo github-felipe-caputo force-pushed the option-for-local-dc-cassandra branch 2 times, most recently from 2b2927a to b5377a4 Compare April 16, 2021 14:45
moromimay
moromimay approved these changes Apr 16, 2021
View reviewed changes
Copy link
Contributor

@moromimay moromimay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌟

@sonarcloud
Copy link

sonarcloud bot commented Apr 16, 2021

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@github-felipe-caputo github-felipe-caputo merged commit baa594b into staging Apr 16, 2021
@github-felipe-caputo github-felipe-caputo deleted the option-for-local-dc-cassandra branch April 16, 2021 15:23
moromimay pushed a commit that referenced this pull request Jun 11, 2021
@github-felipe-caputo @moromimay
Add local dc property (#312)
3da1629 
* add local dc property

* update version
moromimay added a commit that referenced this pull request Jun 15, 2021
@moromimay @AlvaroMarquesAndrade
@roelschr @github-felipe-caputo @GaBrandao
Release/1.2.0 (#328)
1058c31 
* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
moromimay added a commit that referenced this pull request Dec 21, 2022
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao
Update files to staging (#336)
35bcd30 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
moromimay pushed a commit that referenced this pull request Dec 21, 2022
@github-felipe-caputo
Add local dc property (#312)
82a1e5e 
* add local dc property

* update version
moromimay added a commit that referenced this pull request Dec 21, 2022
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao
Update files to staging (#336)
b97711c 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
moromimay added a commit that referenced this pull request Jan 2, 2023
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao
Update files to staging (#336)
8411e73 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
moromimay added a commit that referenced this pull request Jan 2, 2023
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao
Update files to staging (#336)
fdecf75 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
moromimay added a commit that referenced this pull request Jan 2, 2023
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao
Update files to staging (#336)
9a26b92 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
moromimay added a commit that referenced this pull request Jan 3, 2023
@moromimay @jdvala
@lecardozo @github-felipe-caputo
Release 1.2.1 (#338)
adebeb9 
* Add the missing link for H3 geohash (#330)

* Add the missing link for H3 geohash

* Update the H3 geohash link.

* Update the same link 

Update the same link in in spark_function_and_window.ipynb example

* Update README.md (#331)

* Update Github Actions Workflow runner (#332)

* Update Workflow runner version

* bump flake8-bandit

* chore: bypass false positive for S105

Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>

* Delete sphinx version. (#334)

* Update files to staging (#336)

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* release 1.2.1

Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
ralphrass pushed a commit that referenced this pull request Aug 21, 2023
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @ralphrass
Update files to staging (#336)
7df5e42 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
ralphrass added a commit that referenced this pull request Aug 21, 2023
@ralphrass @hmeretti
@moromimay @AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @jdvala @lecardozo
Rebase staging (#343)
f35d665 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Release 1.2.1 (#338)

* Add the missing link for H3 geohash (#330)

* Add the missing link for H3 geohash

* Update the H3 geohash link.

* Update the same link 

Update the same link in in spark_function_and_window.ipynb example

* Update README.md (#331)

* Update Github Actions Workflow runner (#332)

* Update Workflow runner version

* bump flake8-bandit

* chore: bypass false positive for S105

Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>

* Delete sphinx version. (#334)

* Update files to staging (#336)

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* release 1.2.1

Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Update files to staging (#336)

* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Revert "Update files to staging (#336)" (#337)

This reverts commit 35bcd30.

* Less strict requirements (#333)

* bump a few requirements; increase lower bound for h3 version range; adding pyarrow dev dependency

* fix type repr for spark types; fix: broken tests (pyspark 3.4)

---------

Co-authored-by: Ralph Rassweiler <ralphrass@gmail.com>

* feat: optional row count validation (#340)

* fix: parameter, libs (#341)

---------

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>
ralphrass pushed a commit that referenced this pull request Aug 21, 2023
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @ralphrass
Update files to staging (#336)
9dfa06c 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
ralphrass added a commit that referenced this pull request Aug 21, 2023
@ralphrass @hmeretti
@moromimay @AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @jdvala @lecardozo
Rebase staging from master (#345)
97e44fa 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Release 1.2.1 (#338)

* Add the missing link for H3 geohash (#330)

* Add the missing link for H3 geohash

* Update the H3 geohash link.

* Update the same link 

Update the same link in in spark_function_and_window.ipynb example

* Update README.md (#331)

* Update Github Actions Workflow runner (#332)

* Update Workflow runner version

* bump flake8-bandit

* chore: bypass false positive for S105

Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>

* Delete sphinx version. (#334)

* Update files to staging (#336)

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* release 1.2.1

Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>

* fix: methods

* fix: duplicate

---------

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
ralphrass pushed a commit that referenced this pull request Nov 14, 2023
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @ralphrass
Update files to staging (#336)
7207841 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
ralphrass added a commit that referenced this pull request Nov 14, 2023
@ralphrass @hmeretti
@moromimay @AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @jdvala @lecardozo
Rebase staging (#343)
99791c9 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Release 1.2.1 (#338)

* Add the missing link for H3 geohash (#330)

* Add the missing link for H3 geohash

* Update the H3 geohash link.

* Update the same link 

Update the same link in in spark_function_and_window.ipynb example

* Update README.md (#331)

* Update Github Actions Workflow runner (#332)

* Update Workflow runner version

* bump flake8-bandit

* chore: bypass false positive for S105

Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>

* Delete sphinx version. (#334)

* Update files to staging (#336)

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* release 1.2.1

Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Update files to staging (#336)

* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Revert "Update files to staging (#336)" (#337)

This reverts commit 35bcd30.

* Less strict requirements (#333)

* bump a few requirements; increase lower bound for h3 version range; adding pyarrow dev dependency

* fix type repr for spark types; fix: broken tests (pyspark 3.4)

---------

Co-authored-by: Ralph Rassweiler <ralphrass@gmail.com>

* feat: optional row count validation (#340)

* fix: parameter, libs (#341)

---------

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>
ralphrass pushed a commit that referenced this pull request Apr 15, 2024
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @ralphrass
Update files to staging (#336)
8e9f520 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
ralphrass added a commit that referenced this pull request Apr 15, 2024
@ralphrass @hmeretti
@moromimay @AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @jdvala @lecardozo
Rebase staging (#343)
c433371 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Release 1.2.1 (#338)

* Add the missing link for H3 geohash (#330)

* Add the missing link for H3 geohash

* Update the H3 geohash link.

* Update the same link 

Update the same link in in spark_function_and_window.ipynb example

* Update README.md (#331)

* Update Github Actions Workflow runner (#332)

* Update Workflow runner version

* bump flake8-bandit

* chore: bypass false positive for S105

Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>

* Delete sphinx version. (#334)

* Update files to staging (#336)

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* release 1.2.1

Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Update files to staging (#336)

* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Revert "Update files to staging (#336)" (#337)

This reverts commit 35bcd30.

* Less strict requirements (#333)

* bump a few requirements; increase lower bound for h3 version range; adding pyarrow dev dependency

* fix type repr for spark types; fix: broken tests (pyspark 3.4)

---------

Co-authored-by: Ralph Rassweiler <ralphrass@gmail.com>

* feat: optional row count validation (#340)

* fix: parameter, libs (#341)

---------

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>
ralphrass pushed a commit that referenced this pull request Apr 24, 2024
@moromimay @hmeretti
@AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @ralphrass
Update files to staging (#336)
af55afb 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
ralphrass added a commit that referenced this pull request Apr 24, 2024
@ralphrass @hmeretti
@moromimay @AlvaroMarquesAndrade @roelschr @github-felipe-caputo @GaBrandao @jdvala @lecardozo
Rebase staging (#343)
f707899 
* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Release 1.2.1 (#338)

* Add the missing link for H3 geohash (#330)

* Add the missing link for H3 geohash

* Update the H3 geohash link.

* Update the same link 

Update the same link in in spark_function_and_window.ipynb example

* Update README.md (#331)

* Update Github Actions Workflow runner (#332)

* Update Workflow runner version

* bump flake8-bandit

* chore: bypass false positive for S105

Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>

* Delete sphinx version. (#334)

* Update files to staging (#336)

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* release 1.2.1

Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Update files to staging (#336)

* Release/1.1.3 (#290)

* [MLOP-634] Butterfree dev workflow, set triggers for branches staging and master (#280)

* Change github actions pipelines.

* Change pipeline logic.

* [BUG] Fix Staging GithubActions Pipeline (#283)

* New step on pipelie.

* Some adjusts.

* Apply only wheel. (#285)

* [BUG] Change version on setup.py to PyPI (#286)

* Add new make command to change version.

* Change command order.

* Change desc and variable name.

* Change command name.

* Keep milliseconds when using 'from_ms' argument in timestamp feature (#284)

* changed timestamp resolution

* fix import

* simple refactor

Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>

* Change trigger for pipeline staging (#287)

* Change trigger to publish dev pipeline.

* Some fix.

* Create a dev package. (#288)

* [MLOP-633] Butterfree dev workflow, update documentation (#281)

* Update workflow doc.

* Update README

* Add pre-release.

* Fix typo.

* [MLOP-632] Butterfree dev workflow, automate release description (#279)

* release 1.1.4

* update changelog

Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>

* Release/1.2.0 (#328)

* [MLOP-636] Create migration classes (#282)

* [MLOP-635] Rebase Incremental Job/Interval Run branch for test on selected feature sets (#278)

* Add interval branch modifications.

* Add interval_runs notebook.

* Add tests.

* Apply style (black, flack8 and mypy).

* Fix tests.

* Change version to create package dev.

* Allow slide selection (#293)

* Fix Slide Duration Typo (#295)

* [MLOP-637] Implement diff method (#292)

* [MLOP-640] Create CLI with migrate command (#298)

* [MLOP-645] Implement query method, cassandra (#291)

* [MLOP-671] Implement get_schema on Spark client (#301)

* [MLOP-648] Implement query method, metastore (#294)

* Fix Validation Step (#302)

* [MLOP-647] [MLOP-646] Apply migrations (#300)

* add apply migration method

* add test apply migration

* add migrate actor with tests

* mypy compliant

* fix test interaction with mocked object

* Rebase and some adjusts.

Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>

* [BUG] Apply create_partitions to historical validate (#303)

* Apply create_partitions to historical validate.

* Remove comments and adjusts.

* [BUG] Fix key path for validate read (#304)

* Fix key path

* bump version

Co-authored-by: AlvaroMarquesAndrade <1a789766b1c4c8b679e80f11fa6d63d42fa4bcdf>

* [FIX] Add Partition types for Metastore (#305)

* [MLOP-639] Track logs in S3 (#306)

* Apply tracking logs and logging config.

* Adjusts in CLI and logging.conf.

* Some adjusts.

* Change version to generate new dev package

* Fix version.

* Apply style.

* Add new assert in the migrate unit test.

* [BUG] Change logging config (#307)

* Change logging config.

* Some adjusts.

* Remove a code smell.

* Change solution for tracking logs (#308)

* Change tracking logs method.

* Change version to generate dev package.

* Change path name in S3

* Read and write consistency level options (#309)

* modify cassandra client to be region aware

* add option for the user to set read and write consistency levels on cassandra config

* add tests

* use env vars instead

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Update butterfree/configs/db/cassandra_config.py

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>

* Fix kafka reader. (#310)

* Fix path validate. (#311)

* Add local dc property (#312)

* add local dc property

* update version

* Remove metastore migrate (#313)

* Remove metastore migrate.

* Change version to create a dev package.

* Fix link in our docs. (#315)

* [BUG] Fix Cassandra Connect Session (#316)

* Fix Cassandra Connect Session.

* Apply style.

* Fix migration query. (#318)

* Fix migration query add type key. (#319)

* Fix db-config condition (#321)

* Fix db-config condition.

* Apply style.

* MLOP-642 Document migration in Butterfree (#320)

* update docs

* add more information and reference new cli.md file

* [MLOP-702] Debug mode for Automate Migration (#322)

* Create flag debug-mode.

* Fix tests.

* Fix migrate test.

* [MLOP-727] Improve logging messages (#325)

* Fix logging message for local file

* Remove json import

* [MLOP-728] Improve logging messages (#324)

* Improve logs.

* Revert debug-mode condition.

* Fix method to generate agg feature name. (#326)

* [MLOP-691]  Include step to add partition to SparkMetastore during writing of Butterfree (#327)

* Change writer type for interval mode.

* Some adjusts.

* Release 1.2.0

Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>

* Revert "Update files to staging (#336)" (#337)

This reverts commit 35bcd30.

* Less strict requirements (#333)

* bump a few requirements; increase lower bound for h3 version range; adding pyarrow dev dependency

* fix type repr for spark types; fix: broken tests (pyspark 3.4)

---------

Co-authored-by: Ralph Rassweiler <ralphrass@gmail.com>

* feat: optional row count validation (#340)

* fix: parameter, libs (#341)

---------

Co-authored-by: hmeretti <hmeretti@gmail.com>
Co-authored-by: Mayara Moromisato <44944954+moromimay@users.noreply.github.com>
Co-authored-by: Henrique Camargo <henriquecamargo@spf1lt-pj000560.ldap.quintoandar.com.br>
Co-authored-by: AlvaroMarquesAndrade <45604858+AlvaroMarquesAndrade@users.noreply.github.com>
Co-authored-by: Igor Gustavo Hoelscher <19557581+roelschr@users.noreply.github.com>
Co-authored-by: Felipe Victorino Caputo <13631451+fvcaputo@users.noreply.github.com>
Co-authored-by: Rodrigo Martins de Oliveira <allrod5@users.noreply.github.com>
Co-authored-by: Gabriel Brandão <37742275+GaBrandao@users.noreply.github.com>
Co-authored-by: Jay Vala <24193355+jdvala@users.noreply.github.com>
Co-authored-by: Lucas Fonseca <lucas.fonseca@quintoandar.com.br>
Co-authored-by: Lucas Cardozo <lucasecardozo@gmail.com>
Co-authored-by: Mayara Moromisato <may.alveslima@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@moromimay moromimay moromimay approved these changes

Assignees

@github-felipe-caputo github-felipe-caputo

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@github-felipe-caputo @moromimay