Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix rename #1914

Merged
merged 2 commits into from
Apr 24, 2023
Merged

Fix rename #1914

merged 2 commits into from
Apr 24, 2023

Conversation

Ruadhri17
Copy link
Contributor

Proposed changes

Replace rename with copy and delete where necessary. Calculate tmp path inside atomically write functions. More in #1899

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (general improvements like code refactoring that doesn't explicitly fix a bug or add any new functionality)
  • Documentation Update (if none of the other choices apply)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Paste Link to the issue

#1899

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA (in all commits with git commit -s)
  • I ran cargo fmt as mentioned in CODING_GUIDELINES
  • I used cargo clippy as mentioned in CODING_GUIDELINES
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Further comments

@Ruadhri17 Ruadhri17 requested a review from didier-wenzek April 17, 2023 08:57
@Ruadhri17 Ruadhri17 temporarily deployed to Test Pull Request April 17, 2023 09:04 — with GitHub Actions Inactive
@github-actions
Copy link
Contributor

github-actions bot commented Apr 17, 2023

Robot Results

✅ Passed ❌ Failed ⏭️ Skipped Total Pass %
173 0 5 173 100

Passed Tests

Name ⏱️ Duration Suite
Define Child device 1 ID 0.017 s C8Y Child Alarms Rpi
Normal case when the child device does not exist on c8y cloud 3.108 s C8Y Child Alarms Rpi
Normal case when the child device already exists 1.667 s C8Y Child Alarms Rpi
Reconciliation when the new alarm message arrives, restart the mapper 2.7880000000000003 s C8Y Child Alarms Rpi
Reconciliation when the alarm that is cleared 6.223 s C8Y Child Alarms Rpi
Prerequisite Parent 79.56 s Child Conf Mgmt Plugin
Prerequisite Child 0.367 s Child Conf Mgmt Plugin
Child device bootstrapping 16.446 s Child Conf Mgmt Plugin
Snapshot from device 61.244 s Child Conf Mgmt Plugin
Child device config update 63.462 s Child Conf Mgmt Plugin
Configuration types should be detected on file change (without restarting service) 120.879 s Inotify Crate
Check lock file existence in default folder 1.996 s Lock File
Check PID number in lock file 3.761 s Lock File
Check PID number in lock file after restarting the services 4.448 s Lock File
Check starting same service twice 1.948 s Lock File
Switch off lock file creation 4.299 s Lock File
Successful firmware operation 84.707 s Firmware Operation
Install with empty firmware name 70.157 s Firmware Operation
Prerequisite Parent 23.714 s Firmware Operation Child Device
Prerequisite Child 8.458 s Firmware Operation Child Device
Child device firmware update 6.829 s Firmware Operation Child Device
Child device firmware update with cache 7.047 s Firmware Operation Child Device
Update Inventory data via inventory.json 1.954 s Inventory Update
Retrieve a JWT tokens 132.797 s Jwt Request
Main device registration 2.738 s Device Registration
Child device registration 3.5220000000000002 s Device Registration
Supports restarting the device 148.223 s Restart Device
Update tedge version from previous using Cumulocity 117.69 s Tedge Self Update
Test if all c8y services are up 128.3 s Service Monitoring
Test if all c8y services are down 136.879 s Service Monitoring
Test if all c8y services are using configured service type 123.412 s Service Monitoring
Test if all c8y services using default service type when service type configured as empty 100.581 s Service Monitoring
Check health status of tedge-mapper-c8y service on broker stop start 38.088 s Service Monitoring
Check health status of tedge-mapper-c8y service on broker restart 39.314 s Service Monitoring
Check health status of child device service 30.658 s Service Monitoring
Successful shell command with output 4.267 s Shell Operation
Check Successful shell command with literal double quotes output 4.108 s Shell Operation
Execute multiline shell command 3.672 s Shell Operation
Failed shell command 3.509 s Shell Operation
Software list should be populated during startup 58.548 s Software
Install software via Cumulocity 80.286 s Software
Software list should only show currently installed software and not candidates 60.122 s Software
Child devices support sending simple measurements 2.138 s Child Device Telemetry
Child devices support sending custom measurements 1.774 s Child Device Telemetry
Child devices support sending custom events 1.607 s Child Device Telemetry
Child devices support sending custom events overriding the type 1.374 s Child Device Telemetry
Child devices support sending custom alarms #1699 3.683 s Child Device Telemetry
Child devices support sending inventory data via c8y topic 1.4849999999999999 s Child Device Telemetry
Main device support sending inventory data via c8y topic 1.8940000000000001 s Child Device Telemetry
Child device supports sending custom child device measurements directly to c8y 2.003 s Child Device Telemetry
Main device supports sending custom child device measurements directly to c8y 1.9649999999999999 s Child Device Telemetry
Check retained alarms 63.735 s Raise Alarms
Validate updated data path used by tedge-agent 0.849 s Data Path Config
Validate updated data path used by c8y-firmware-plugin 11.467 s Data Path Config
Stop tedge-agent service 0.52 s Log Path Config
Customize the log path 0.172 s Log Path Config
Initialize tedge-agent 0.196 s Log Path Config
Check created folders 0.198 s Log Path Config
Remove created custom folders 0.193 s Log Path Config
Install thin-edge via apt 52.514 s Install Apt
Install latest via script (from current branch) 31.008 s Install Tedge
Install specific version via script (from current branch) 36.097 s Install Tedge
Install latest tedge via script (from main branch) 23.568 s Install Tedge
Install then uninstall latest tedge via script (from main branch) 68.68 s Install Tedge
Support starting and stopping services 51.887 s Service-Control
Supports a reconnect 78.999 s Test-Commands
Supports disconnect then connect 52.672 s Test-Commands
Update unknown setting 46.86 s Test-Commands
Update known setting 29.537 s Test-Commands
Stop c8y-configuration-plugin 60.09 s Health C8Y-Configuration-Plugin
Update the service file 0.114 s Health C8Y-Configuration-Plugin
Reload systemd files 0.602 s Health C8Y-Configuration-Plugin
Start c8y-configuration-plugin 0.254 s Health C8Y-Configuration-Plugin
Start watchdog service 10.299 s Health C8Y-Configuration-Plugin
Check PID of c8y-configuration-plugin 0.17 s Health C8Y-Configuration-Plugin
Kill the PID 0.375 s Health C8Y-Configuration-Plugin
Recheck PID of c8y-configuration-plugin 6.893 s Health C8Y-Configuration-Plugin
Compare PID change 0.012 s Health C8Y-Configuration-Plugin
Stop watchdog service 0.274 s Health C8Y-Configuration-Plugin
Remove entry from service file 0.286 s Health C8Y-Configuration-Plugin
Stop c8y-log-plugin 60.174 s Health C8Y-Log-Plugin
Update the service file 0.19 s Health C8Y-Log-Plugin
Reload systemd files 0.459 s Health C8Y-Log-Plugin
Start c8y-log-plugin 0.14 s Health C8Y-Log-Plugin
Start watchdog service 10.274000000000001 s Health C8Y-Log-Plugin
Check PID of c8y-log-plugin 0.241 s Health C8Y-Log-Plugin
Kill the PID 0.378 s Health C8Y-Log-Plugin
Recheck PID of c8y-log-plugin 6.8100000000000005 s Health C8Y-Log-Plugin
Compare PID change 0.001 s Health C8Y-Log-Plugin
Stop watchdog service 0.45 s Health C8Y-Log-Plugin
Remove entry from service file 0.244 s Health C8Y-Log-Plugin
Stop tedge-mapper 0.353 s Health Tedge Mapper C8Y
Update the service file 0.388 s Health Tedge Mapper C8Y
Reload systemd files 0.801 s Health Tedge Mapper C8Y
Start tedge-mapper 0.286 s Health Tedge Mapper C8Y
Start watchdog service 10.308 s Health Tedge Mapper C8Y
Check PID of tedge-mapper 0.268 s Health Tedge Mapper C8Y
Kill the PID 0.375 s Health Tedge Mapper C8Y
Recheck PID of tedge-mapper 6.748 s Health Tedge Mapper C8Y
Compare PID change 0.002 s Health Tedge Mapper C8Y
Stop watchdog service 0.134 s Health Tedge Mapper C8Y
Remove entry from service file 0.279 s Health Tedge Mapper C8Y
Stop tedge-agent 0.246 s Health Tedge-Agent
Update the service file 0.182 s Health Tedge-Agent
Reload systemd files 0.683 s Health Tedge-Agent
Start tedge-agent 0.254 s Health Tedge-Agent
Start watchdog service 10.614 s Health Tedge-Agent
Check PID of tedge-mapper 0.222 s Health Tedge-Agent
Kill the PID 0.431 s Health Tedge-Agent
Recheck PID of tedge-agent 6.702 s Health Tedge-Agent
Compare PID change 0.001 s Health Tedge-Agent
Stop watchdog service 0.314 s Health Tedge-Agent
Remove entry from service file 0.186 s Health Tedge-Agent
Stop tedge-mapper-az 0.263 s Health Tedge-Mapper-Az
Update the service file 0.213 s Health Tedge-Mapper-Az
Reload systemd files 0.895 s Health Tedge-Mapper-Az
Start tedge-mapper-az 0.215 s Health Tedge-Mapper-Az
Start watchdog service 10.211 s Health Tedge-Mapper-Az
Check PID of tedge-mapper-az 0.101 s Health Tedge-Mapper-Az
Kill the PID 0.194 s Health Tedge-Mapper-Az
Recheck PID of tedge-agent 6.537 s Health Tedge-Mapper-Az
Compare PID change 0.001 s Health Tedge-Mapper-Az
Stop watchdog service 0.23 s Health Tedge-Mapper-Az
Remove entry from service file 0.26 s Health Tedge-Mapper-Az
Stop tedge-mapper-collectd 0.133 s Health Tedge-Mapper-Collectd
Update the service file 0.114 s Health Tedge-Mapper-Collectd
Reload systemd files 0.572 s Health Tedge-Mapper-Collectd
Start tedge-mapper-collectd 0.19 s Health Tedge-Mapper-Collectd
Start watchdog service 10.129 s Health Tedge-Mapper-Collectd
Check PID of tedge-mapper-collectd 0.123 s Health Tedge-Mapper-Collectd
Kill the PID 0.288 s Health Tedge-Mapper-Collectd
Recheck PID of tedge-mapper-collectd 6.936 s Health Tedge-Mapper-Collectd
Compare PID change 0.002 s Health Tedge-Mapper-Collectd
Stop watchdog service 0.33 s Health Tedge-Mapper-Collectd
Remove entry from service file 0.255 s Health Tedge-Mapper-Collectd
tedge-collectd-mapper health status 6.177 s Health Tedge-Mapper-Collectd
c8y-log-plugin health status 6.064 s MQTT health endpoints
c8y-configuration-plugin health status 6.26 s MQTT health endpoints
Publish on a local insecure broker 0.454 s Basic Pub Sub
Publish on a local secure broker 2.283 s Basic Pub Sub
Publish on a local secure broker with client authentication 3.226 s Basic Pub Sub
Wrong package name 0.339 s Improve Tedge Apt Plugin Error Messages
Wrong version 0.344 s Improve Tedge Apt Plugin Error Messages
Wrong type 0.568 s Improve Tedge Apt Plugin Error Messages
tedge_connect_test_positive 0.648 s Tedge Connect Test
tedge_connect_test_negative 3.025 s Tedge Connect Test
tedge_connect_test_sm_services 10.068 s Tedge Connect Test
tedge_disconnect_test_sm_services 2.695 s Tedge Connect Test
Install thin-edge.io 23.645 s Call Tedge
call tedge -V 0.209 s Call Tedge
call tedge -h 0.135 s Call Tedge
call tedge -h -V 0.177 s Call Tedge
call tedge help 0.139 s Call Tedge
tedge config list 0.067 s Call Tedge Config List
tedge config list --all 0.074 s Call Tedge Config List
set/unset device.type 0.382 s Call Tedge Config List
set/unset device.key.path 0.316 s Call Tedge Config List
set/unset device.cert.path 0.32 s Call Tedge Config List
set/unset c8y.root.cert.path 0.294 s Call Tedge Config List
set/unset c8y.smartrest.templates 0.307 s Call Tedge Config List
set/unset az.root.cert.path 0.333 s Call Tedge Config List
set/unset az.mapper.timestamp 0.444 s Call Tedge Config List
set/unset mqtt.bind_address 0.852 s Call Tedge Config List
set/unset mqtt.port 1.051 s Call Tedge Config List
set/unset tmp.path 0.879 s Call Tedge Config List
set/unset logs.path 0.997 s Call Tedge Config List
set/unset run.path 0.384 s Call Tedge Config List
Get Put Delete 2.8689999999999998 s Http File Transfer Api
Set keys should return value on stdout 0.214 s Tedge Config Get
Unset keys should not return anything on stdout and warnings on stderr 0.683 s Tedge Config Get
Invalid keys should not return anything on stdout and warnings on stderr 0.477 s Tedge Config Get
Set configuration via environment variables 1.782 s Tedge Config Get
Set unknown configuration via environment variables 0.251 s Tedge Config Get

@Ruadhri17 Ruadhri17 temporarily deployed to Test Pull Request April 17, 2023 09:52 — with GitHub Actions Inactive
@Ruadhri17 Ruadhri17 temporarily deployed to Test Pull Request April 17, 2023 10:47 — with GitHub Actions Inactive
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's perfect. However, for the move_file function it's better to try first a simple rename and use a copy/delete only if the filesystems are different.

Comment on lines 87 to 84
pub fn new(target_path: &Path, target_permission: PermissionEntry) -> Self {
pub fn new(target_path: &Path) -> Self {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was surprised to that parameter removed. But you are correct, this can be safely removed as never assigned other than a default value.

Copy link
Member

@rina23q rina23q Apr 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm in c8y-configuration-plugin, there is parameters user, group and mode, where user can decide which permission they want to have in a downloaded file. I feel this change is breaking it.

In crates/extensions/c8y_http_proxy/src/actor.rs, this one is used. So, you cannot remove safely.

let downloader: Downloader = Downloader::new(&request.file_path);

Comment on lines 116 to 118
// Copy source to destination including permission bits and remove source after successful copy operation
tokio::fs::copy(src_path, dest_path)
.and_then(|_| tokio::fs::remove_file(src_path))
.await?;

tokio::fs::rename(src_path, dest_path).await?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not directly copy/delete the file but try first to rename it as the copy/delete will not work if the file is larger than the free space (this is not unlikely with a firmware download on a resource-constraint device).

If would be better to fallback to a copy/delete only if the rename fails because of a cross-filesystem rename, but unfortunately this is a nightly feature. So just try to rename and copy/delete in case of some error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a doubt if target_permission can really removed from Downloader. I will run a test manually and come back.

Comment on lines 87 to 84
pub fn new(target_path: &Path, target_permission: PermissionEntry) -> Self {
pub fn new(target_path: &Path) -> Self {
Copy link
Member

@rina23q rina23q Apr 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm in c8y-configuration-plugin, there is parameters user, group and mode, where user can decide which permission they want to have in a downloaded file. I feel this change is breaking it.

In crates/extensions/c8y_http_proxy/src/actor.rs, this one is used. So, you cannot remove safely.

let downloader: Downloader = Downloader::new(&request.file_path);

Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and unfortunately my concern was correct. I would block merging until the regression fixed.

@didier-wenzek
Copy link
Contributor

Tested and unfortunately my concern was correct. I would block merging until the regression fixed.

As these permission settings are used in a single place, the simpler it's to keep the signature of the move_file as proposed by @Ruadhri17 and to let the configuration manager applies the requested permission on the file once moved.

@Ruadhri17 Ruadhri17 temporarily deployed to Test Pull Request April 18, 2023 18:59 — with GitHub Actions Inactive
@Ruadhri17
Copy link
Contributor Author

Tested and unfortunately my concern was correct. I would block merging until the regression fixed.

As these permission settings are used in a single place, the simpler it's to keep the signature of the move_file as proposed by @Ruadhri17 and to let the configuration manager applies the requested permission on the file once moved.

Oops, I just reverted changes regarding PermissionEntry 😄, though I can bring them back. If we decide to stick with my solution, should we solve the problem with c8y-configuration-plugin in this PR or create the new one?

@didier-wenzek
Copy link
Contributor

Oops, I just reverted changes regarding PermissionEntry smile, though I can bring them back. If we decide to stick with my solution, should we solve the problem with c8y-configuration-plugin in this PR or create the new one?

This must be fixed in that PR. But don't revert your revert: let @rina23q have a look first. I will do the same.

Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can keep unchanged the type signature of the move_file i.e to keep the new_file_permissions: PermissionEntry parameter. However setting these permissions must be done independently of the actual method used: rename or copy-and-delete.

crates/common/tedge_utils/src/file.rs Outdated Show resolved Hide resolved
Comment on lines 131 to 143
let file_permissions = if let Some(mode) = original_permission_mode {
// Use the same file permission as the original one
PermissionEntry::new(None, None, Some(mode))
} else {
// Set the user, group, and mode as given for a new file
new_file_permissions
};

file_permissions.apply(dest_path)?;
debug!(
"Applied permissions: {:?} to {:?}",
file_permissions, dest_path
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must also be done in the copy-and-delete case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @didier-wenzek

Copy link
Member

@rina23q rina23q left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The regression is fixed by the reverting move_file change, however, some still need to be changed before merge.

Comment on lines 131 to 143
let file_permissions = if let Some(mode) = original_permission_mode {
// Use the same file permission as the original one
PermissionEntry::new(None, None, Some(mode))
} else {
// Set the user, group, and mode as given for a new file
new_file_permissions
};

file_permissions.apply(dest_path)?;
debug!(
"Applied permissions: {:?} to {:?}",
file_permissions, dest_path
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with @didier-wenzek

@rina23q rina23q dismissed their stale review April 20, 2023 08:43

No breaking regression, therefore, stale this.

@rina23q
Copy link
Member

rina23q commented Apr 20, 2023

Note: if we had #1875, the permission issue would have been tested by pipeline...

@Ruadhri17 Ruadhri17 force-pushed the fix-rename branch 2 times, most recently from c956efd to bb2dcb2 Compare April 20, 2023 19:29
@Ruadhri17 Ruadhri17 temporarily deployed to Test Pull Request April 20, 2023 19:36 — with GitHub Actions Inactive
Copy link
Contributor

@didier-wenzek didier-wenzek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

Signed-off-by: Krzysztof Piotrowski <Krzysztof.Piotrowski@inetum.com>
Signed-off-by: Krzysztof Piotrowski <Krzysztof.Piotrowski@inetum.com>
@Ruadhri17 Ruadhri17 temporarily deployed to Test Pull Request April 24, 2023 09:42 — with GitHub Actions Inactive
@didier-wenzek didier-wenzek merged commit 489c0e8 into thin-edge:main Apr 24, 2023
@Ruadhri17 Ruadhri17 deleted the fix-rename branch June 26, 2023 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants