Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hostcfgd] Delay hostcfgd and aaastatsd for faster boot time #7965

Merged
merged 3 commits into from
Aug 10, 2021
Merged

[hostcfgd] Delay hostcfgd and aaastatsd for faster boot time #7965

merged 3 commits into from
Aug 10, 2021

Conversation

shlomibitton
Copy link
Contributor

@shlomibitton shlomibitton commented Jun 24, 2021

Signed-off-by: Shlomi Bitton shlomibi@nvidia.com

Why I did it

hostcfgd is starting at the same time as 'create_switch' method is called on orchagent process.
This introduce a degradation on the function execution time which eventually cause the fast-boot flow and a boot scenarion in general to run slower (~6 seconds).
This change will delay the start time of this daemon.
The aaastatsd will delay as well since it has a dependency on hostcfgd, so it is required to delay both.
90 seconds determined as the maximum allowed downtime for control plane to come back up on fast-boot flow.

How I did it

Add two timers for hostcfgd and aaastatsd services in order to delay the startup of these services.

How to verify it

Install an image with this change and observe the daemons start 90 seconds after the system boot.

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Shlomi Bitton <shlomibi@nvidia.com>
@shlomibitton shlomibitton requested a review from lguohan as a code owner June 24, 2021 06:42
@liat-grozovik
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

kcudnik
kcudnik previously approved these changes Jun 27, 2021
@liat-grozovik
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@liat-grozovik
Copy link
Collaborator

@jleveque could you please provide your feedback on this PR?

@liat-grozovik
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@liat-grozovik
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik pushed a commit to sonic-net/sonic-mgmt that referenced this pull request Jul 8, 2021
…allow 'hostcfgd' to start and configure tacas authorization. (#3741)

- What is the motivation for this PR?
After recent change introduced on this PR's: sonic-net/sonic-buildimage#7965 and sonic-net/sonic-buildimage#8117 'hostcfgd' will be delayed in 90 seconds.
If the test will run before the daemon has started, it will fail the test. This is to align with the new change and make sure the test will pass.

- How did you do it?
Add a delay of 100 seconds before executing authorized commands.

- How did you verify/test it?
Run the test.

Signed-off-by: Shlomi Bitton <shlomibi@nvidia.com>
@liat-grozovik
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@shlomibitton
Copy link
Contributor Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@shlomibitton
Copy link
Contributor Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@shlomibitton
Copy link
Contributor Author

/azpw run

@mssonicbld
Copy link
Collaborator

/AzurePipelines run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@liat-grozovik
Copy link
Collaborator

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@shlomibitton shlomibitton reopened this Aug 9, 2021
@shlomibitton
Copy link
Contributor Author

@lguohan @qiluo-msft This PR is approved and all checkers passed, can we proceed and merge?
thanks

@qiluo-msft qiluo-msft merged commit 76d5223 into sonic-net:master Aug 10, 2021
@qiluo-msft
Copy link
Collaborator

This PR could not cleanly cherry-pick to 202012. Please submit another PR.

@shlomibitton
Copy link
Contributor Author

shlomibitton commented Aug 12, 2021

This PR could not cleanly cherry-pick to 202012. Please submit another PR.

@qiluo-msft there was already a PR for 202012 and it is merged.
#8117

judyjoseph pushed a commit that referenced this pull request Aug 25, 2021
#### Why I did it
hostcfgd is starting at the same time as 'create_switch' method is called on orchagent process.
This introduce a degradation on the function execution time which eventually cause the fast-boot flow and a boot scenarion in general to run slower (~6 seconds).
This change will delay the start time of this daemon.
The aaastatsd will delay as well since it has a dependency on hostcfgd, so it is required to delay both.
90 seconds determined as the maximum allowed downtime for control plane to come back up on fast-boot flow.

#### How I did it
Add two timers for hostcfgd and aaastatsd  services in order to delay the startup of these services.

#### How to verify it
Install an image with this change and observe the daemons start 90 seconds after the system boot.
@shlomibitton shlomibitton deleted the shlomi_delay_hostcfgd branch September 5, 2021 08:21
lguohan added a commit to lguohan/sonic-buildimage that referenced this pull request Sep 8, 2021
lguohan added a commit that referenced this pull request Sep 8, 2021
…7965)" (#8705)

cause regression on the kvm testing

This reverts commit 76d5223.
yxieca added a commit to yxieca/sonic-buildimage that referenced this pull request Sep 12, 2021
yxieca added a commit that referenced this pull request Sep 14, 2021
vmittal-msft pushed a commit to vmittal-msft/sonic-mgmt that referenced this pull request Sep 28, 2021
…allow 'hostcfgd' to start and configure tacas authorization. (sonic-net#3741)

- What is the motivation for this PR?
After recent change introduced on this PR's: sonic-net/sonic-buildimage#7965 and sonic-net/sonic-buildimage#8117 'hostcfgd' will be delayed in 90 seconds.
If the test will run before the daemon has started, it will fail the test. This is to align with the new change and make sure the test will pass.

- How did you do it?
Add a delay of 100 seconds before executing authorized commands.

- How did you verify/test it?
Run the test.

Signed-off-by: Shlomi Bitton <shlomibi@nvidia.com>
vmittal-msft pushed a commit to vmittal-msft/sonic-mgmt that referenced this pull request Sep 28, 2021
…e to hostcfgd delay on boot (sonic-net#3775)

What is the motivation for this PR?

After recent change introduced on this PR's:
sonic-net/sonic-buildimage#7965
sonic-net/sonic-buildimage#8117

'hostcfgd' will be delayed in 90 seconds.
If the test will run before the daemon has started, it will fail the test.
This is to align with the new change and make sure the test will pass.

Profiling the time it takes to configure tacacs after the daemon started can take time:
main started -> Mon 12 Jul 2021 02:07:06 PM UTC
'tacacs_server_update' function finished -> Mon 12 Jul 2021 02:08:10 PM UTC

Signed-off-by: Shlomi Bitton <shlomibi@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants