Invalid login details keeps the plugin busy exhausting wsapi sessions #534
Comments
|
If the plugin fails due to login issue and retries again and again, ultimately the array WSAPI sessions no longer respond and throw:
|
wdurairaj
changed the title
|
@sneharai4 , if you are able to recreate this consistently, keep i will keep this as "high" (documenting the steps here) , otherwise, i would deprioritize to either "medium" or "low". |
|
@sneharai4 can you provide some steps ? else we can mark to Medium.. |
|
I was able to recreate the problem of 1 session leak per backend when this setting is given with improper value
But, when this problem happens, the backend is correctly marked in So, in production this situation can only be hit, when
Based on the above reason, we can downgrade this bug to a "medium" priority, @sneharai4 , please let me know, if you need any other details (or) your comments |
|
@sneharai4 any comments ? |
|
I tried re-producing the same issue, but this time I am not getting "Service Unavailable (HTTP 503) 31 - maximum number of server connections has been reached" response error. I have one host, where hpe.conf has invalid know_hosts file path because of which the backend fails to initialize. This has created 36 sessions on 3PAR. Error in plugin-container log: Detailed log: invalid_host_path_hpe_conf.txt Now, i have another host where the backend is set to OK which uses the same array which has exhausted all the sessions on 3PAR. Getting below error while creating a volume on this host: Detailed log: |
wdurairaj
added a commit
to wdurairaj/python-hpedockerplugin
that referenced
this issue
Apr 12, 2019
nilangekarss
pushed a commit
that referenced
this issue
Apr 15, 2019
|
Fix is verified and working. But need to open a new bug to log errors when the session exhausts and client is raising an exception which is not being handled correctly by 3PAR. Different error is raised for 3.3.1 and 3.2.2 array. |
|
@sneharai4 can you raise a new bug and close this ? else we can keep it open until we address the issue completely.. Please let me know. |
|
@prablr79 Closing this issue. |
wdurairaj
added a commit
to wdurairaj/python-hpedockerplugin
that referenced
this issue
Apr 25, 2019
* Fix Issue hpe-storage#390, Allow 'size' in snapshot options * Updated usage doc * Fix issue hpe-storage#534 - invalid config entry creates session leak
wdurairaj
added a commit
that referenced
this issue
Jun 24, 2019
nilangekarss
pushed a commit
that referenced
this issue
Sep 25, 2019
* Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Added space between words * Ability to create a regular volume from a replicated backend * Fixed pep8 issue regarding redundant back slashes * Pep8 Fix attempt 2: Indentation * Update hpe_storage_api.py * Fixed an issue when a regular volume is mounted while the backend is replication enabled * Fixed pep8 line too long issues * Not treating as a replication backend if the volume created is not replicated volume * Allowing the non replicated volume from a replicated backend to be successfully imported to docker volume plugin * Added back some required checks * Removed unwanted space * Update .travis.yml * Fix for issue 518-encrypted password decrytion fails when the passphrase length is of 16,24,32 characters * Fix for issue 502 (#555) * Added needed check to see mgr object is available or not (#559) * Feature: Add logging to file -- Include changes for Pull Request #563 in v300 branch (#569) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Feature: Add support for RotatingFileHandler in logging * Fix Issue #534 (#576) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fix issue #534 - invalid config entry creates session leak * Fix issue #513 on v300 branch (#583) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fixed issue #513 -Added rollback to mount flow for any cleanup in case of any failure -Added validation for fsOwner * Pep8 fixed * Use deferred thread for processing REST calls in twistd * Fixed msg creation * Retry on lock exception * Another attempt on processing lock failed exception * Changes in mount_volume to avoid lock during mount conflict check * Fix _is_vol_mounted_on_this_node definition * Minor change * Backport pull request #650 and related changes * Implemented blocking Etcd lock + Eviction fixes merged * Fix problem with mount entry check * Returning multiple enums from _is_vol_mounted_on_this_node + inspect output to have volume id * Expect node_mount_info to be absent for the first mount -Also removed dead code * path_info to be handled as JSON object + handled stale mount_id in reboot case * Fix for UTs * Replaced path.path with path * Fixed snap related TC * PEP8 errors fixed * Added more information to the logs * For UT 3pardcv.log location changed * Added check for manager-list initialization * Removed redundant code * Removed duplicate functions from fileutil As part of merge process, fileutil ended up having two duplicate functions. Fixed it. Also UT needed to use un-deferred thread code to avoid handling multi-threaded UTs. * Fixed UTs for File * Added exception handling for mount_dir() * Adopted 3.2 async initialization fix required for UT * Reintroduced sleep of 3 secs * Corrected usage of sleep() call * Disabled detailed logging due to Travis CI log size restriction * Pep8 fix * Fix for issue #735 * Fixed removal of redundant old_path_info entries * Added missing argument to rollback call * Removed code that was added to look for iscsi devices Ideally, we should remove this file altogether... to be taken up later
wdurairaj
pushed a commit
that referenced
this issue
Oct 17, 2019
* Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Added space between words * Ability to create a regular volume from a replicated backend * Fixed pep8 issue regarding redundant back slashes * Pep8 Fix attempt 2: Indentation * Update hpe_storage_api.py * Fixed an issue when a regular volume is mounted while the backend is replication enabled * Fixed pep8 line too long issues * Not treating as a replication backend if the volume created is not replicated volume * Allowing the non replicated volume from a replicated backend to be successfully imported to docker volume plugin * Added back some required checks * Removed unwanted space * Update .travis.yml * Fix for issue 518-encrypted password decrytion fails when the passphrase length is of 16,24,32 characters * Fix for issue 502 (#555) * Added needed check to see mgr object is available or not (#559) * Feature: Add logging to file -- Include changes for Pull Request #563 in v300 branch (#569) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Feature: Add support for RotatingFileHandler in logging * Fix Issue #534 (#576) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fix issue #534 - invalid config entry creates session leak * Fix issue #513 on v300 branch (#583) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fixed issue #513 -Added rollback to mount flow for any cleanup in case of any failure -Added validation for fsOwner * Pep8 fixed * Use deferred thread for processing REST calls in twistd * Fixed msg creation * Retry on lock exception * Another attempt on processing lock failed exception * Changes in mount_volume to avoid lock during mount conflict check * Fix _is_vol_mounted_on_this_node definition * Minor change * Backport pull request #650 and related changes * Implemented blocking Etcd lock + Eviction fixes merged * Fix problem with mount entry check * Returning multiple enums from _is_vol_mounted_on_this_node + inspect output to have volume id * Expect node_mount_info to be absent for the first mount -Also removed dead code * path_info to be handled as JSON object + handled stale mount_id in reboot case * Fix for UTs * Replaced path.path with path * Fixed snap related TC * PEP8 errors fixed * Added more information to the logs * For UT 3pardcv.log location changed * Added check for manager-list initialization * Removed redundant code * Removed duplicate functions from fileutil As part of merge process, fileutil ended up having two duplicate functions. Fixed it. Also UT needed to use un-deferred thread code to avoid handling multi-threaded UTs. * Fixed UTs for File * Added exception handling for mount_dir() * Adopted 3.2 async initialization fix required for UT * Reintroduced sleep of 3 secs * Corrected usage of sleep() call * Disabled detailed logging due to Travis CI log size restriction * Pep8 fix * Fix for issue #735 * Fixed removal of redundant old_path_info entries * Added missing argument to rollback call * Removed code that was added to look for iscsi devices Ideally, we should remove this file altogether... to be taken up later * pyparsing ImportError fix * Changed setuptools version to 41.0.0 41.0.0 was used by v3.1 of plugin and was * Device remapping fix On reboot, the volumes that were mapped to the multipath devices earlier are remapped to different devices. This fix handles that case. * Fixed lock name * Fixed PEP8 issues * Missed a PEP8 conformance * Fixed log statement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Plugin error log:
Total sessions on 3PAR active at the moment when plugin gives the error:
Docker systemctl status:
The text was updated successfully, but these errors were encountered: