This repository has been archived by the owner on Jun 22, 2022. It is now read-only.
Family features #128
Merged
Family features #128
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Ninoko
commented
Jul 12, 2018
Ninoko
commented
- Added non_child features: CNT_NON_CHILD = CNT_FAM_MEMBERS - CNT_CHILDREN
- Added statistics for NON_CHILD and CHILDREN: income/credit per child/non_child
- Added aggregation :
- of all pairs from: [NAME_EDUCATION_TYPE, CODE_GENDER, NAME_FAMILY_STATUS]
- for: ['AMT_CREDIT', 'AMT_ANNUITY', 'AMT_INCOME_TOTAL', 'EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3']
- using: ['min', 'mean', 'max', 'sum', 'var']
pknut
reviewed
Jul 12, 2018
src/pipeline_config.py
Outdated
| @@ -139,13 +139,14 @@ | |||
| 'YEARS_BUILD_MEDI', | |||
| 'YEARS_BUILD_MODE'] | |||
|
|
|||
| conti_cols = ['AMT_CREDIT', 'AMT_ANNUITY', 'AMT_INCOME_TOTAL', 'EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3'] | |||
| aggs = ['min', 'mean', 'max', 'sum', 'var'] | |||
| lst_col_agg = [(col, agg) for col in conti_cols for agg in aggs] | |||
There was a problem hiding this comment.
@Ninoko 'conti_cols' and 'lst_col_agg' aren't very descriptive
There was a problem hiding this comment.
Are 'cols_to_agg' and 'aggregation_pairs' ok? Idk
kstrzala
approved these changes
Jul 12, 2018
pknut
approved these changes
Jul 12, 2018
jakubczakon
approved these changes
Jul 12, 2018
Closed
kamil-kaczmarek
pushed a commit
that referenced
this pull request
Jul 18, 2018
* added second level models (#126) * Family features (#128) * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Features - family - test * Features - family - aggregate * Features - family - aggregate 2 * Features - family - aggregate 3 * Features - family - aggregate 4 * Update pipeline_config.py * Data cleaning and two new features (previous application) (#129) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Data cleaning - fix (#130) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Fix * Dev fractions (#132) * added fraction features to eda and feature extraction, updated configs * updated hyperparams * Dev (#134) * age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * POS CASH features added * POS_CASH_balance feature cleaning * Yaml adjustment * Path change * fix misinterpretations '<' instead of '>' * fix misinterpretations '<' instead of '>' * Add cleaning in application_groupby_agg (#137) * application agg cleaning * update neptune.yaml * Interaction features (#139) * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Features - family - test * Features - family - aggregate * Features - family - aggregate 2 * Features - family - aggregate 3 * Features - family - aggregate 4 * Update pipeline_config.py * Features - family - added new cols to agg * Features - interaction features * Features - interaction features - fix * Added is_unbalance to configs * updated paths, added corr prints in pos cash balance * dropped unused dependencies * updated pandas version
jakubczakon
pushed a commit
that referenced
this pull request
Jul 20, 2018
* age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * added second level models (#126) * POS CASH features added * Family features (#128) * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Features - family - test * Features - family - aggregate * Features - family - aggregate 2 * Features - family - aggregate 3 * Features - family - aggregate 4 * Update pipeline_config.py * POS_CASH_balance feature cleaning * Yaml adjustment * Data cleaning and two new features (previous application) (#129) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Data cleaning - fix (#130) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Fix * Initail bureau_balance features * Dev fractions (#132) * added fraction features to eda and feature extraction, updated configs * updated hyperparams * Dev (#134) * age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * POS CASH features added * POS_CASH_balance feature cleaning * Yaml adjustment * Path change * fix misinterpretations '<' instead of '>' * fix misinterpretations '<' instead of '>' * Code cleanup * Bug fix * NaN handling * Add cleaning in application_groupby_agg (#137) * application agg cleaning * update neptune.yaml * Interaction features (#139) * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Features - family - test * Features - family - aggregate * Features - family - aggregate 2 * Features - family - aggregate 3 * Features - family - aggregate 4 * Update pipeline_config.py * Features - family - added new cols to agg * Features - interaction features * Features - interaction features - fix * Added is_unbalance to configs * Time correction * Full time count correction * Time features correction and bureau_balance features * Bug fixing * Bug fixing
jakubczakon
pushed a commit
that referenced
this pull request
Jul 26, 2018
* age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * added second level models (#126) * POS CASH features added * Family features (#128) * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Features - family - test * Features - family - aggregate * Features - family - aggregate 2 * Features - family - aggregate 3 * Features - family - aggregate 4 * Update pipeline_config.py * POS_CASH_balance feature cleaning * Yaml adjustment * Data cleaning and two new features (previous application) (#129) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Data cleaning - fix (#130) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Fix * Dev fractions (#132) * added fraction features to eda and feature extraction, updated configs * updated hyperparams * Path change * Dev (#134) * age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * POS CASH features added * POS_CASH_balance feature cleaning * Yaml adjustment * Path change * fix misinterpretations '<' instead of '>' * fix misinterpretations '<' instead of '>' * Add cleaning in application_groupby_agg (#137) * application agg cleaning * update neptune.yaml * New branch * Notebook dev * q * Sklearn models modified * Minor bug fix * Whatever * Space refactor * Old forgotten merge * Final refactor * Minor update * last k features with fraction removal * Fix PR isuuses
jakubczakon
pushed a commit
that referenced
this pull request
Jul 26, 2018
* age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * added second level models (#126) * POS CASH features added * Family features (#128) * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Features - family - test * Features - family - aggregate * Features - family - aggregate 2 * Features - family - aggregate 3 * Features - family - aggregate 4 * Update pipeline_config.py * POS_CASH_balance feature cleaning * Yaml adjustment * Data cleaning and two new features (previous application) (#129) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Data cleaning - fix (#130) * new previous application features * Data cleaning * update application notebook * credit card cleaning * Data cleaning - groupby agg * Include suggested changes * Fix * Dev fractions (#132) * added fraction features to eda and feature extraction, updated configs * updated hyperparams * Path change * Dev (#134) * age/employment dummies (#104) * added diff features * New handcrafted features (#102) * Dynamic features * Smart features (#61) * Update README.md * Update README.md * Update * Smart features update * More descriptive transformer name * Reading all data in main * More application features * Transformer for cleaning * Multiinput data dictionary * Fix (#63) * fixed configs * dropped redundand steps, moved stuff to cleaning, refactored groupby (#64) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Fix format string * Update pipeline_manager.py clipped prediction -> prediction * added stratified kfold option (#77) * Update config (#79) * dropped redundand steps, moved stuff to cleanining, refactored groupby * restructured, added stacking + CV * Update pipeline_config.py * Dev review (#81) * dropped feature by type split, refactored pipleine_config * dropped feature by type split method * explored application features * trash * reverted refactor of aggs * fixed/updated bureau features * cleared notebooks * agg features added to notebook bureau * credit card cleaned * added other feature notebooks * added rank mean * updated model arch * reverted to old params * fixed rank mean calculations * ApplicationCleaning update (#84) * Cleaning - application * Clear output in notebook * clenaed names in steps, refactored mergeaggregate transformer, changed caching/saving specs (#85) * local trash * External sources notebook (#86) * Update * External sources notebook * Dev lgbm params (#88) * local trash * updated configs * dropped comment * updated lgb params * Dev app agg fix (#90) * dropped app_aggs * app agg features fixed * cleaned leftovers * dropped fast read-in for debug * External_sources statistics (#89) * Speed-up ext_src notebook * exernal_sources statistics * Weighted mean and notebook fix * application notebook update * clear notebook output * Fix auto submission (#95) * CreditCardBalance monthly diff mean * POSCASH remaining installments * POSCASH completed_contracts * notebook update * Resolve conflicts * Fix * Update neptune.yaml * Update neptune_random_search.yaml * Split static and dynamic features - credit card balance * Dev nan count (#105) * added nan_count * added nan count with parameter * Dev fe installments (#106) * added simple features, parallel groupby, last-installment features * refactored last_installment features * added features for the very last installment * Dev fe instalments dynamic (#107) * added dynamic-trend features * formated configs * added skew/iqr features * added number of credit agreement change features (#109) * added number of credit agreement change features * reverted sample size * Dynamic features - previous application (#108) * previous_application handcrafted features * previous application cleaning * Update neptune.yaml * code improvement * Update notebook * Notebook - feature importance (#112) * Dev speed up (#111) * refactored aggs to calculate only once per training, sped up installment and credit card (only single index groupby) * sped up all hand crafted * fixed bureau worker errors * fixed isntallment names * fixed isntallment names * fixed bureau and prev_app naming bugs * reverted to vectorized where possible * updated hyperparams * updated early stopping params to meet convergence * reverted to old fallback neptune file * updated paths * updated paths, explored prev-app features * dropped duplicated agg * POS_CASH added features * POS CASH features added * POS_CASH_balance feature cleaning * Yaml adjustment * Path change * fix misinterpretations '<' instead of '>' * fix misinterpretations '<' instead of '>' * Add cleaning in application_groupby_agg (#137) * application agg cleaning * update neptune.yaml * New branch * Notebook dev * q * Sklearn models modified * Minor bug fix * Whatever * Space refactor * Old forgotten merge * Final refactor * Minor update * last k features with fraction removal * Fix PR isuuses * Fillna bug fix
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.