-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fail the test when a DeprecationWarning
is raised
#9939
Comments
GREAT proposal, completely agree! |
There are many many occurrences to update, anybody who wants to help is welcome to do so! |
@carmocca are you doing this file by file, or some other way? Do you have a list of files you have already done/or ones that need to be done so that we can help? |
We can do it file by file or semantically (all occurrences of a deprecation) - up to you. You can check the current azure CI job status on the main PR (#9940) to see which test are still failing. Currently 185: tests/accelerators/test_accelerator_connector.py::test_accelerator_choice_ddp_slurm
tests/accelerators/test_accelerator_connector.py::test_accelerator_choice_ddp2_slurm
tests/accelerators/test_accelerator_connector.py::test_accelerator_choice_ddp_te
tests/accelerators/test_accelerator_connector.py::test_accelerator_choice_ddp2_te
tests/accelerators/test_accelerator_connector.py::test_accelerator_choice_ddp_kubeflow
tests/accelerators/test_cpu.py::test_plugin_setup_optimizers_in_pre_dispatch[True]
tests/accelerators/test_cpu.py::test_plugin_setup_optimizers_in_pre_dispatch[False]
tests/callbacks/test_gpu_stats_monitor.py::test_gpu_stats_monitor - py...
tests/callbacks/test_gpu_stats_monitor.py::test_gpu_stats_monitor_no_queries
tests/callbacks/test_gpu_stats_monitor.py::test_gpu_stats_monitor_no_logger
tests/callbacks/test_gpu_stats_monitor.py::test_gpu_stats_monitor_no_gpu_warning
tests/callbacks/test_lambda_function.py::test_lambda_call - pytorch_li...
tests/callbacks/test_model_summary.py::test_model_summary_callback_with_weights_summary
tests/callbacks/test_model_summary.py::test_model_summary_callback_override_weights_summary_flag
tests/callbacks/test_quantization.py::test_quantize_torchscript - pyto...
tests/callbacks/test_stochastic_weight_avg.py::test_swa_warns - pytorc...
tests/callbacks/test_stochastic_weight_avg.py::test_trainer_and_stochastic_weight_avg[False-True]
tests/callbacks/test_stochastic_weight_avg.py::test_trainer_and_stochastic_weight_avg[True-True]
tests/callbacks/test_timer.py::test_trainer_flag - AssertionError: ass...
tests/callbacks/test_timer.py::test_timer_stops_training - AssertionEr...
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_on[callbacks1-1]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_on[callbacks2-2]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_on[callbacks3-0]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_on[callbacks4-0]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_on[callbacks5-1]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_off[callbacks0-0-True]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_off[callbacks1-False-True]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_off[callbacks2-0-True]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_off[callbacks3-1-False]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_totals
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_progress_refresh[0]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_progress_refresh[1]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_progress_refresh[50]
tests/callbacks/test_tqdm_progress_bar.py::test_tqdm_progress_bar_value_on_colab
tests/checkpointing/test_model_checkpoint.py::test_model_checkpoint_save_last_none_monitor
tests/checkpointing/test_model_checkpoint.py::test_model_checkpoint_save_last_warning[True-True-True-1]
tests/checkpointing/test_model_checkpoint.py::test_model_checkpoint_save_last_warning[True-True-True-2]
tests/checkpointing/test_model_checkpoint.py::test_model_checkpoint_save_last_warning[True-True-False-1]
tests/checkpointing/test_model_checkpoint.py::test_model_checkpoint_save_last_warning[True-True-False-2]
tests/core/test_datamodules.py::test_define_as_dataclass - pytorch_lig...
tests/core/test_datamodules.py::test_inconsistent_prepare_data_per_node
tests/deprecated_api/test_remove_1-7.py::test_v1_7_0_resume_from_checkpoint_trainer_constructor
tests/helpers/test_models.py::test_models[None-BoringModel] - pytorch_...
tests/helpers/test_models.py::test_models[None-BasicGAN] - pytorch_lig...
tests/helpers/test_models.py::test_models[None-ParityModuleRNN] - pyto...
tests/helpers/test_models.py::test_models[None-ParityModuleMNIST] - py...
tests/helpers/test_models.py::test_models[ClassifDataModule-ClassificationModel]
tests/helpers/test_models.py::test_models[RegressDataModule-RegressionModel]
tests/loggers/test_all.py::test_loggers_fit_test_all - pytorch_lightni...
tests/loggers/test_all.py::test_loggers_save_dir_and_weights_save_path_all
tests/loggers/test_all.py::test_loggers_pickle_all[TestTubeLogger] - p...
tests/loggers/test_all.py::test_logger_reset_correctly[Batch-size-Finder]
tests/loggers/test_all.py::test_logger_created_on_rank_zero_only[TestTubeLogger]
tests/loggers/test_all.py::test_logger_with_prefix_all - pytorch_light...
tests/loggers/test_tensorboard.py::test_tensorboard_log_graph[example_input_array1]
tests/loggers/test_tensorboard.py::test_tensorboard_missing_folder_warning
tests/loops/test_loops.py::test_fit_can_fail_during_validation[0.5-train_datasets0-val_datasets0]
tests/loops/test_loops.py::test_fit_can_fail_during_validation[0.5-train_datasets1-val_datasets1]
tests/loops/test_loops.py::test_fit_can_fail_during_validation[1.0-train_datasets0-val_datasets0]
tests/loops/test_loops.py::test_fit_can_fail_during_validation[1.0-train_datasets1-val_datasets1]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit[True-kwargs0]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit[True-kwargs1]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit[True-kwargs2]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit[False-kwargs0]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit[False-kwargs1]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit[False-kwargs2]
tests/models/test_hooks.py::test_trainer_model_hook_system_fit_no_val_and_resume
tests/models/test_hooks.py::test_trainer_model_hook_system_eval[validate-validation-val-x-0]
tests/models/test_hooks.py::test_trainer_model_hook_system_eval[validate-validation-val-x-2]
tests/models/test_hooks.py::test_trainer_model_hook_system_eval[test-test-test-y-0]
tests/models/test_hooks.py::test_trainer_model_hook_system_eval[test-test-test-y-2]
tests/models/test_hooks.py::test_trainer_model_hook_system_predict - p...
tests/models/test_hooks.py::test_trainer_datamodule_hook_system - pyto...
tests/models/test_torchscript.py::test_torchscript_input_output[BoringModel]
tests/models/test_torchscript.py::test_torchscript_input_output[ParityModuleRNN]
tests/models/test_torchscript.py::test_torchscript_input_output[BasicGAN]
tests/models/test_torchscript.py::test_torchscript_example_input_output_trace[BoringModel]
tests/models/test_torchscript.py::test_torchscript_example_input_output_trace[ParityModuleRNN]
tests/models/test_torchscript.py::test_torchscript_example_input_output_trace[BasicGAN]
tests/models/test_torchscript.py::test_torchscript_input_output_trace
tests/models/test_torchscript.py::test_torchscript_device[device0] - p...
tests/models/test_torchscript.py::test_torchscript_device[device1] - p...
tests/models/test_torchscript.py::test_torchscript_retain_training_state
tests/models/test_torchscript.py::test_torchscript_properties[BoringModel]
tests/models/test_torchscript.py::test_torchscript_properties[ParityModuleRNN]
tests/models/test_torchscript.py::test_torchscript_properties[BasicGAN]
tests/models/test_torchscript.py::test_torchscript_save_load[BoringModel]
tests/models/test_torchscript.py::test_torchscript_save_load[ParityModuleRNN]
tests/models/test_torchscript.py::test_torchscript_save_load[BasicGAN]
tests/models/test_torchscript.py::test_torchscript_save_load_custom_filesystem[BoringModel]
tests/models/test_torchscript.py::test_torchscript_save_load_custom_filesystem[ParityModuleRNN]
tests/models/test_torchscript.py::test_torchscript_save_load_custom_filesystem[BasicGAN]
tests/plugins/test_amp_plugins.py::test_precision_selection_raises - p...
tests/plugins/test_ddp_spawn_plugin.py::test_ddp_spawn_extra_parameters
tests/plugins/test_ddp_spawn_plugin.py::test_ddp_spawn_add_get_queue
tests/plugins/environments/test_kubeflow_environment.py::test_attributes_from_environment_variables
tests/plugins/environments/test_slurm_environment.py::test_attributes_from_environment_variables
tests/plugins/environments/test_torchelastic_environment.py::test_attributes_from_environment_variables
tests/profiler/test_profiler.py::test_simple_profiler_logs - assert 0 ...
tests/profiler/test_profiler.py::test_advanced_profiler_cprofile_deepcopy
tests/trainer/test_dataloaders.py::test_request_dataloader - pytorch_l...
tests/trainer/test_trainer.py::test_trainer_min_steps_and_min_epochs_not_reached
tests/trainer/test_trainer.py::test_nan_loss_detection - pytorch_light...
tests/trainer/test_trainer.py::test_invalid_terminate_on_nan - pytorch...
tests/trainer/test_trainer.py::test_nan_params_detection - pytorch_lig...
tests/trainer/test_trainer.py::test_on_exception_hook - pytorch_lightn...
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs1-expected1]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs2-expected2]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs3-expected3]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs4-expected4]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs6-expected6]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs8-expected8]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs9-expected9]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs11-expected11]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs13-expected13]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs14-expected14]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs15-expected15]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs16-expected16]
tests/trainer/test_trainer.py::test_trainer_config[trainer_kwargs17-expected17]
tests/trainer/test_trainer.py::test_log_every_n_steps[10-10-1] - pytor...
tests/trainer/test_trainer.py::test_log_every_n_steps[3-10-1] - pytorc...
tests/trainer/test_trainer.py::test_log_every_n_steps[3-10-5] - pytorc...
tests/trainer/connectors/test_callback_connector.py::test_checkpoint_callbacks_are_last
tests/trainer/connectors/test_callback_connector.py::test_attach_model_callbacks
tests/trainer/connectors/test_callback_connector.py::test_attach_model_callbacks_override_info
tests/trainer/flags/test_fast_dev_run.py::test_callbacks_and_logger_not_called_with_fastdevrun[1]
tests/trainer/flags/test_fast_dev_run.py::test_callbacks_and_logger_not_called_with_fastdevrun[4]
tests/trainer/flags/test_min_max_epochs.py::test_min_max_steps_epochs[None-3-None-None]
tests/trainer/flags/test_min_max_epochs.py::test_min_max_steps_epochs[1-3-None-None]
tests/trainer/flags/test_min_max_epochs.py::test_min_max_steps_epochs[None-3-10-None]
tests/trainer/logging_/test_logger_connector.py::test_fx_validator_integration
tests/tuner/test_lr_finder.py::test_lr_find_with_bs_scale - pytorch_li...
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_method_with_model_or_datamodule[2--1]
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_method_with_model_or_datamodule[2-2]
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_method_with_model_or_datamodule[2-None]
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_method_with_model_or_datamodule[None-2]
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_method_with_model_or_datamodule[16-16]
tests/tuner/test_scale_batch_size.py::test_model_reset_correctly - pyt...
tests/tuner/test_scale_batch_size.py::test_trainer_reset_correctly - p...
tests/tuner/test_scale_batch_size.py::test_auto_scale_batch_size_trainer_arg[power]
tests/tuner/test_scale_batch_size.py::test_auto_scale_batch_size_trainer_arg[binsearch]
tests/tuner/test_scale_batch_size.py::test_auto_scale_batch_size_trainer_arg[True]
tests/tuner/test_scale_batch_size.py::test_auto_scale_batch_size_set_model_attribute[True]
tests/tuner/test_scale_batch_size.py::test_auto_scale_batch_size_set_model_attribute[False]
tests/tuner/test_scale_batch_size.py::test_call_to_trainer_method[power]
tests/tuner/test_scale_batch_size.py::test_call_to_trainer_method[binsearch]
tests/tuner/test_scale_batch_size.py::test_auto_scale_batch_size_with_amp
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_no_trials
tests/tuner/test_scale_batch_size.py::test_scale_batch_size_fails_with_unavailable_mode
tests/tuner/test_scale_batch_size.py::test_dataloader_reset_with_scale_batch_size[power]
tests/tuner/test_scale_batch_size.py::test_dataloader_reset_with_scale_batch_size[binsearch]
tests/utilities/test_model_summary.py::test_invalid_weights_summmary
tests/utilities/test_model_summary.py::test_empty_model_summary_shapes[full]
tests/utilities/test_model_summary.py::test_empty_model_summary_shapes[top]
tests/utilities/test_model_summary.py::test_linear_model_summary_shapes[device0-full]
tests/utilities/test_model_summary.py::test_linear_model_summary_shapes[device0-top]
tests/utilities/test_model_summary.py::test_linear_model_summary_shapes[device1-full]
tests/utilities/test_model_summary.py::test_linear_model_summary_shapes[device1-top]
tests/utilities/test_model_summary.py::test_rnn_summary_shapes[full]
tests/utilities/test_model_summary.py::test_rnn_summary_shapes[top] - ...
tests/utilities/test_model_summary.py::test_summary_parameter_count[full]
tests/utilities/test_model_summary.py::test_summary_parameter_count[top]
tests/utilities/test_model_summary.py::test_summary_layer_types[full]
tests/utilities/test_model_summary.py::test_summary_layer_types[top]
tests/utilities/test_model_summary.py::test_summary_with_scripted_modules[full]
tests/utilities/test_model_summary.py::test_summary_with_scripted_modules[top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input0-?-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input0-?-top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input1-expected_size1-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input1-expected_size1-top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input2-?-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input2-?-top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input3-?-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input3-?-top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input4-expected_size4-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input4-expected_size4-top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input5-expected_size5-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input5-expected_size5-top]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input6-expected_size6-full]
tests/utilities/test_model_summary.py::test_example_input_array_types[example_input6-expected_size6-top]
tests/utilities/test_model_summary.py::test_model_size[full] - pytorch...
tests/utilities/test_model_summary.py::test_model_size[top] - pytorch_...
tests/utilities/test_model_summary.py::test_empty_model_size[full] - p...
tests/utilities/test_model_summary.py::test_empty_model_size[top] - py...
tests/utilities/test_model_summary.py::test_max_depth_equals_mode_interface |
Proposed refactoring or deprecation
Set our test suite to fail when a
DeprecationWarning
is raised.Motivation
Since we don't track warnings raised by our tests, it's too easy to introduce a deprecation path but forget to update all deprecated usages inside our codebase and tests.
This is very bad in terms of UX because the users will see this warning appear but won't be able to do anything to fix it.
This has been exacerbated lately with the number of deprecations happening.
For example consider this simple training script:
Deprecation warnings in 1.4.9:
Deprecation warnings in current master:
Pitch
Set
in https://github.com/PyTorchLightning/pytorch-lightning/blob/16213b16356f5bd97b8cc8bf1849eacd68d658c5/setup.cfg#L15-L27
and fill all deprecated usages. This will need to be done over several PRs
If the test writer actually wants to test that a deprecation is raised,
pytest.deprecated_call(...)
will need to be used to catch it.If you enjoy Lightning, check out our other projects! ⚡
Metrics: Machine learning metrics for distributed, scalable PyTorch applications.
Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning
Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch
Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
The text was updated successfully, but these errors were encountered: