Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory optimization when using aws s3 (Try 2) #549

Closed
wants to merge 1 commit into from

Conversation

ebressler
Copy link
Contributor

The changes here make it so that there are no longer references being
held to the s3 objects allocated during XMLToStruct in the s3 libraries.

In my testing, when doing a listObjectsV2 against ~383,000 objects, this
change saves about 1/2 of the memory uses. (1gb of use without these
changes and ~500mb with them)

This is an updated way of doing this. I somehow closed out #547 and can't reopen the PR.

held to the s3 objects allocated during XMLToStruct in the s3 libraries.

In my testing, when doing a listObjectsV2 against ~383,000 objects, this
change saves about 1/2 of the memory uses. (1gb of use without these
changes and ~500mb with them)
@ebressler
Copy link
Contributor Author

@kahing, I found a better way of doing the fix that should work across all backends. The other PR I accidentally closed when redoing my branch.

before:

File: __debug_bin
Type: inuse_space
Time: Aug 24, 2020 at 10:39am (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 416.55MB, 98.76% of 421.76MB total
Dropped 29 nodes (cum <= 2.11MB)
Showing top 10 nodes out of 25
      flat  flat%   sum%        cum   cum%
  237.03MB 56.20% 56.20%   320.04MB 75.88%  github.com/kahing/goofys/vendor/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.XMLToStruct
   72.50MB 17.19% 73.39%    72.50MB 17.19%  github.com/kahing/goofys/vendor/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.(*XMLNode).findNamespaces
   48.51MB 11.50% 84.89%    48.51MB 11.50%  github.com/kahing/goofys/internal.(*Inode).SetFromBlobItem
   37.50MB  8.89% 93.79%    37.50MB  8.89%  github.com/kahing/goofys/internal.NewInode
       6MB  1.42% 95.21%        6MB  1.42%  encoding/xml.(*Decoder).name
    5.78MB  1.37% 96.58%        8MB  1.90%  github.com/kahing/goofys/internal.(*Goofys).insertInode
       3MB  0.71% 97.29%        3MB  0.71%  encoding/xml.makeCopy
       3MB  0.71% 98.00%    96.01MB 22.77%  github.com/kahing/goofys/internal.(*DirHandle).ReadDir
    2.21MB  0.52% 98.53%     2.21MB  0.52%  github.com/kahing/goofys/internal.(*Inode).insertChildUnlocked
       1MB  0.24% 98.76%     7.50MB  1.78%  encoding/xml.(*Decoder).Token

After:

File: __debug_bin
Type: inuse_space
Time: Aug 24, 2020 at 10:45am (EDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 287.37MB, 97.37% of 295.12MB total
Dropped 76 nodes (cum <= 1.48MB)
Showing top 10 nodes out of 26
      flat  flat%   sum%        cum   cum%
  143.54MB 48.64% 48.64%   149.54MB 50.67%  github.com/kahing/goofys/internal.(*Inode).SetFromBlobItem
   71.51MB 24.23% 72.87%    71.51MB 24.23%  github.com/kahing/goofys/internal.NewInode
   45.50MB 15.42% 88.29%    50.04MB 16.96%  github.com/kahing/goofys/vendor/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil.XMLToStruct
    9.56MB  3.24% 91.53%    14.81MB  5.02%  github.com/kahing/goofys/internal.(*Goofys).insertInode
       6MB  2.03% 93.56%        6MB  2.03%  github.com/kahing/goofys/internal.deepCopyBlobItemOputput
    4.50MB  1.52% 95.09%   240.36MB 81.45%  github.com/kahing/goofys/internal.(*DirHandle).ReadDir
    3.25MB  1.10% 96.19%     3.25MB  1.10%  github.com/kahing/goofys/internal.(*Inode).insertChildUnlocked
    2.50MB  0.85% 97.03%     3.04MB  1.03%  encoding/xml.(*Decoder).rawToken
       1MB  0.34% 97.37%     4.04MB  1.37%  encoding/xml.(*Decoder).Token
         0     0% 97.37%   240.36MB 81.45%  github.com/kahing/goofys/api/common.FusePanicLogger.ReadDir

@ebressler
Copy link
Contributor Author

@kahing , just checking in to make sure you have not missed this update.

@ebressler
Copy link
Contributor Author

@kahing anything you need from me on this?

@ebressler
Copy link
Contributor Author

@kahing , is there anything else that I need to do here or is this good?

@kahing
Copy link
Owner

kahing commented Mar 26, 2021

b12cef3

@kahing kahing closed this Mar 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants