Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add upload file any size #3883

Merged
merged 5 commits into from
Apr 5, 2023
Merged

add upload file any size #3883

merged 5 commits into from
Apr 5, 2023

Conversation

jackzhhuang
Copy link
Collaborator

@jackzhhuang jackzhhuang commented Apr 4, 2023

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Code style update (formatting, renaming)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • Documentation content changes
  • Other (please describe):

What is the current behavior?

Issue Number: N/A

What is the new behavior?

Other information

@jackzhhuang jackzhhuang linked an issue Apr 4, 2023 that may be closed by this pull request
@jackzhhuang jackzhhuang requested review from sanlee42, welbon and nkysg April 4, 2023 17:50
@codecov
Copy link

codecov bot commented Apr 4, 2023

Codecov Report

Merging #3883 (7a8199f) into master (3190084) will decrease coverage by 0.20%.
The diff coverage is n/a.

❗ Current head 7a8199f differs from pull request most recent head 5bdd952. Consider uploading reports for the commit 5bdd952 to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3883      +/-   ##
==========================================
- Coverage   53.94%   53.73%   -0.20%     
==========================================
  Files         616      616              
  Lines       67809    67809              
==========================================
- Hits        36570    36433     -137     
- Misses      31239    31376     +137     
Flag Coverage Δ
unittests 53.73% <ø> (-0.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 16 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3190084...5bdd952. Read the comment docs.

@nkysg
Copy link
Collaborator

nkysg commented Apr 5, 2023

TypeError: upload_file_big_file() takes 2 positional arguments but 3 were given

you can test jump server dir /temp_mount_for_sync_snapshot/barnard/put.sh

def upload_file_any_size(file_name, bucket, key):
file_size = os.path.getsize(file_name)
if file_size > PER_UPLOADING:
upload_file_big_file(file_name, bucket, key)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里是三个参数, 20行是2个参数,好像有问题

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我改好了, 现在可以了,你可以试试,key那个参数是后来加的

@nkysg
Copy link
Collaborator

nkysg commented Apr 5, 2023

LGTM

@github-actions
Copy link

github-actions bot commented Apr 5, 2023

Benchmark for 5e2fb57

Click to view benchmark
Test Base PR %
accumulator_append 611.8±28.04µs 619.7±32.59µs +1.29%
block_apply/block_apply_10 273.2±0.36ms 275.6±0.37ms +0.88%
block_apply/block_apply_1000 28.6±0.02s 28.8±0.02s +0.70%
get_with_proof/db_store 36.7±0.37µs 37.1±0.26µs +1.09%
get_with_proof/mem_store 31.4±0.26µs 31.6±0.21µs +0.64%
put_and_commit/db_store/1 98.0±5.63µs 100.3±5.39µs +2.35%
put_and_commit/db_store/10 872.9±41.67µs 883.2±43.20µs +1.18%
put_and_commit/db_store/100 7.5±0.29ms 7.5±0.30ms 0.00%
put_and_commit/db_store/5 422.6±23.41µs 422.8±21.98µs +0.05%
put_and_commit/db_store/50 3.9±0.16ms 3.9±0.16ms 0.00%
put_and_commit/mem_store/1 62.3±5.98µs 62.3±5.93µs 0.00%
put_and_commit/mem_store/10 583.9±51.40µs 583.3±52.87µs -0.10%
put_and_commit/mem_store/100 5.7±0.87ms 5.7±0.88ms 0.00%
put_and_commit/mem_store/5 294.0±26.55µs 292.6±25.62µs -0.48%
put_and_commit/mem_store/50 2.9±0.19ms 2.9±0.19ms 0.00%
query_block/query_block_in(10)_times(100) 3.8±0.12ms 4.0±0.35ms +5.26%
query_block/query_block_in(10)_times(1000) 38.5±0.77ms 38.8±0.73ms +0.78%
query_block/query_block_in(10)_times(10000) 391.9±8.62ms 390.5±6.59ms -0.36%
query_block/query_block_in(1000)_times(100) 987.7±8.13µs 973.9±8.52µs -1.40%
query_block/query_block_in(1000)_times(1000) 9.8±0.09ms 9.7±0.10ms -1.02%
query_block/query_block_in(1000)_times(10000) 98.0±0.81ms 97.2±0.77ms -0.82%
storage_transaction 1001.3±293.76µs 1011.2±286.88µs +0.99%
vm/transaction_execution/1 319.2±0.67ms 320.8±1.26ms +0.50%
vm/transaction_execution/10 99.9±0.62ms 99.8±0.33ms -0.10%
vm/transaction_execution/20 93.7±2.76ms 93.3±1.30ms -0.43%
vm/transaction_execution/5 122.3±1.54ms 122.1±0.48ms -0.16%
vm/transaction_execution/50 108.2±2.09ms 108.1±1.60ms -0.09%

@nkysg nkysg self-requested a review April 5, 2023 05:29
@nkysg nkysg merged commit b5c82f5 into master Apr 5, 2023
@nkysg nkysg deleted the upload-file-any-size branch April 5, 2023 05:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Dev] scripts/upload_block_on_s3.sh can't put-object size over 5G
2 participants