-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make Slicer interface more generic #466
Conversation
f889904
to
a3783d5
Compare
a3783d5
to
8358b6a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
im a bit confused that we require to pass object.Object
: imo this can complicate the code of a simple data uploading applications.
how about provide both raw interface (current) accepting object.Object
and the sugared (previous) one requiring min input? @smallhive @carpawell @roman-khimov
object/slicer/slicer.go
Outdated
|
||
// SetSession pass session object to writer. It will be used for writing object routine. | ||
// This method must be called before InitDataStream. | ||
SetSession(sess session.Object) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
requirements are pretty strict, maybe it'd be clearer to pass session.Object
parameter into InitDataStream
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I may suggest an alternative solution:
- remove
SetSession
from the interface - update
func NewDataSlicer(ctx context.Context, cli *Client) (*slicer.Slicer, error) {
tofunc NewDataSlicer(ctx context.Context, cli *Client, sess *session.Object) (*slicer.Slicer, error) {
.
In this case we hide session insideobjectWriter
and don't need to set it outside - make objectWriter public as a default implementation for the interface. It's constructor even may have
session.Object
as a parameter
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
requirements are pretty strict
we thought that we need a way to be able to create a slicer once and slice more than one object and send parts to more than one node (so more than one session will be attached). we can discuss it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do I apply my suggestions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about provide both client.NewDataSlicer
and client.NewDataSlicerWithSession
?
make objectWriter public
so we'll have
w := client.NewObjectWriter(cli)
// or
w := client.NewObjectWriterWithSession(cli, sessionToken)
s := slicer.NewSession(signer, cnr, sessionToken, w, opts)
s.Slicer(strings.NewReader("Hello, world!"))
right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. The option with two constructors for ObjectWriter
also looks fine to me and it will help us to remove SetSession(sess session.Object)
from the interface
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cthulhu-rider, how that allows us to send the parts to different nodes with only one Slicer
(the problem/wish i described above)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
send the parts to different nodes
we can do it with single session token, can't we? Afaik we don't deal with dynamic sessions when we talk about Slicer
. The app just takes user's session token and attach it to any node it send objects to
so pls clarify ur concern about different nodes
, maybe i didn't get ur point
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Afaik we don't deal with dynamic sessions when we talk about Slicer
i said smth wrong. well, the question is about reusing the Slicer
then. do we want to be able to create Slicer
once and handle any object from any user?
I consider, it doesn't make sense. User has to pass all required parameters for the header object (signer neofscrypto.Signer, cnr cid.ID, owner user.ID) and we create it inside. For comparison, we manipulate the same data, but with different ways: Before
After
Yes, the code longer for header creation, but it is deadly simple |
that's what exactly bothers me. How user will undestand which fields must be explicitly set and which are optional or even must not be set? accepting i can also suggest to implement customization through optional header construction callback that is going to be called during object formation if set: var opts Options
opts.SetCustomHeaderConstructor(func(obj *object.Object) {
obj.SetType(object.Tombstone)
obj.SetWhatever()
})
slicer := new(signer neofscrypto.Signer, cnr cid.ID, owner user.ID, w ObjectWriter, opts Options)
slicer.SliceData(bytes.NewReader(tombstone.Marshal())) P.S. me personally completely okay with raw API, but I'm not sure if it will be convenient and understandable for simple data upload applications. I'd provide helpers that transform what they have to required stuff like |
It's not a problem. We can create a helper function inside the func PrepareHeader(cnr cid.ID, owner oid.ID) object.Object Also, we will mention it inside the Slicer constructor/Slice method. Any simple upload application will manage to call the func for header construction. This will be a good starting point for them, to touch our product. But, as practice shows, latter they would want to dive deeper. In this case they shouldn't rewrite a lot their code, because PrepareHeader is just 2 rows function.
I don't agree with strings attributes. Regular client and node already use object.Attribute and have to reconvert existing attributes to string. Me and @carpawell faced the situation.
One more option var 😃 Passing a fully prepared header eliminates all misunderstandings. This callback, we would
Even the current SDK interface is quite clear and understandable. The slicer improves these factors, but I tend to think it shouldn't make such far abstraction from SDK types |
@smallhive i understand and agree with almost all ur points. Let me summarize my own in my mind, to sum it up, s := slicer.New(signer, objectWriter, opts)
err := slicer.Slice(slicer.RequiredFields(cnr, owner), strings.NewReader("Hello, world!")) looks pretty good to me. If this is the approach u suggested, i agree with it. maybe single-op function would be also useful err := slicer.UploadDataToContainer(ctx, cnr, objectWriter, dataReader) |
4a98b92
to
5ad7daf
Compare
Yes, the slicer does it, but not for everyone. Current changes just move parameters from one function to another. These changes were inspired by slicer usage in few projects
For such purposes, we have client.CreateObject. Current PR code tries to achieve unification for all client types |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me try to untangle it from the beginning.
Slicer in many ways starts with InitDataStream
API for ObjectWriter
. Why do we have it? Because we need to ObjectPutInit
and WriteHeader
at the same time. But we've moved a bit from RC9 and after e19e49a it's just no longer an issue, plain ObjectPutInit()
does everything needed. This means we don't need this interface at all and slicer
package can work with a simple ObjectPutInit()
definition (notice that it can come from client or pool). Let's caller it ObjectPutter
for now (just to differentiate from ObjectWriter
even though it's a nice name to use in slicer
).
Then IIUC we have two use cases:
- we have a complete object header (of some fancy type) and want to push some data using this header
- we have no idea what these headers are, but just want to save some data with some attributes
The first one implies that we already know how to fill in object headers, including containers/owners/sessions and all of this is just out of scope for the API. The second one is better be assisted by the API.
So the first case looks like
func Write(ctx context.Context, op ObjectPutter, header object.Object, signer neofscrypto.Signer, limit uint64, data io.Reader) (oid.ID, error)
func InitWriter(ctx context.Context, op ObjectPutter, header object.Object, signer neofscrypto.Signer, limit uint64) (PayloadWriter, error)
No need for fancy structures or anything like that here. A pair of functions and that's it. Notice that in general it follows the node's pattern of working with multiple containers/owners/sessions.
Now for the second case we want to have a slicer.Slicer
to help us. We need to have network info for that which means another
type NetworkedWriter interface {
ObjectPutter
NetworkInfo(ctx context.Context, prm PrmNetworkInfo) (netmap.NetworkInfo, error)
}
and
func New(ctx context.Context, nw NetworkedWriter, signer neofscrypto.Signer, cnr cid.ID, owner user.ID, opts Options) (*Slicer, error)
Session can be passed via Options
easily. This follows the regular application pattern of "I'm managing some data in a single container by a single user". Then it's per-object
func (x *Slicer) Write(data io.Reader, attrs []object.Attribute) (oid.ID, error)
func (x *Slicer) InitWriter(attrs []object.Attribute) (PayloadWriter, error)
That's about it. client.CreateObject
and client.NewDataSlicer
can both be dropped as useless, slicer
completely substitutes them.
Maybe
For better names and consistency with |
35e3d01
to
af3ae26
Compare
According to the comment, completely rewrite the solution |
af3ae26
to
93a0ca2
Compare
4ad0460
to
f87b699
Compare
4a5f2ee
to
a59ad2d
Compare
I've made some optimizations and divided each into separate commits. I started from this numbers
In the first optimization, all numbers look much better:
Second one improves memory allocations in case we slice object, since slice_256-128:
The last one make one more step in the same direction:
|
I made one more interesting experiment, just locally and many tradeoffs like type Object struct {
v2 *object.Object
isSet bool
id oid.ID
signature *neofscrypto.Signature
version *version.Version
payloadSize uint64
containerID cid.ID
isSetContainerID bool
owner *user.ID
epoch uint64
chsum checksum.Checksum
isSetChsum bool
homo checksum.Checksum
isHomo bool
attributes []Attribute
prevID *oid.ID
children []oid.ID
splitInfo *object.SplitHeader
sess *session.Object
tp Type
} Converting to v2/fromV2 and working with And eventually the result of the bench with these changes:
Of course, it is just for comparison and to think in the future to manage something with this (or not). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost there.
@carpawell, does it work API-wise for node?
@smallhive, it'd be nice to have benchmark diff (as in benchstat
output) in commit messages.
pool/pool.go
Outdated
} | ||
|
||
// write the last part | ||
if _, err = wObj.Write(buf[:n]); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd try to not duplicate this code below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated, also added a few tests for it
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
a59ad2d
to
b45433c
Compare
Added results to commits. But after rebase numbers are changed, slightly |
c41562a
to
a13249c
Compare
Updated commit message benchmark statistic, also final approach: master vs current branch
|
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
…d pool Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
closes #442 Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Updated after benchmark. For the header object we already made flushObjectMetadata step in _writeChild and don't need to compute this data again. │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 116.55µ ± 1% 62.61µ ± 1% -46.28% (p=0.002 n=6) SliceDataIntoObjects/slice_1-128/writer-16 121.73µ ± 2% 64.94µ ± 1% -46.65% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/reader-16 111.50µ ± 3% 65.48µ ± 1% -41.27% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 113.84µ ± 1% 67.38µ ± 2% -40.81% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 110.43µ ± 1% 61.57µ ± 2% -44.25% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 115.11µ ± 1% 63.93µ ± 1% -44.46% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 109.10µ ± 1% 64.89µ ± 1% -40.53% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 114.07µ ± 1% 67.24µ ± 1% -41.05% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 108.68µ ± 1% 61.19µ ± 1% -43.70% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 233.2µ ± 0% 236.2µ ± 2% ~ (p=0.093 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 109.04µ ± 1% 64.34µ ± 1% -41.00% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 554.6µ ± 0% 569.6µ ± 2% +2.71% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 114.62µ ± 0% 63.45µ ± 1% -44.65% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 1.855m ± 2% 1.865m ± 1% ~ (p=0.180 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 117.09µ ± 3% 67.15µ ± 0% -42.65% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 7.745m ± 2% 7.889m ± 1% +1.86% (p=0.026 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 118.90µ ± 1% 67.12µ ± 1% -43.55% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 30.71m ± 1% 31.07m ± 0% +1.20% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 124.95µ ± 2% 72.49µ ± 1% -41.98% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 124.0m ± 1% 112.7m ± 0% -9.15% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 184.8µ ± 2% 125.1µ ± 3% -32.30% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 453.8m ± 1% 518.8m ± 2% +14.32% (p=0.002 n=6) geomean 461.0µ 318.0µ -31.02% │ old.txt │ new.txt │ │ B/op │ B/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 16.880Ki ± 0% 9.453Ki ± 0% -44.00% (p=0.002 n=6) SliceDataIntoObjects/slice_1-128/writer-16 18.100Ki ± 0% 9.344Ki ± 0% -48.38% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/reader-16 14.75Ki ± 0% 10.33Ki ± 0% -29.99% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 15.91Ki ± 0% 10.22Ki ± 0% -35.77% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 14.504Ki ± 0% 8.984Ki ± 0% -38.06% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 15.724Ki ± 0% 8.968Ki ± 0% -42.97% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 13.63Ki ± 0% 10.33Ki ± 0% -24.21% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 14.85Ki ± 0% 10.22Ki ± 0% -31.17% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 14.005Ki ± 0% 8.985Ki ± 0% -35.84% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 31.33Ki ± 0% 31.18Ki ± 0% -0.47% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 14.008Ki ± 0% 9.864Ki ± 0% -29.58% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 74.32Ki ± 0% 81.16Ki ± 0% +9.21% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 16.528Ki ± 0% 9.874Ki ± 0% -40.26% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 250.8Ki ± 0% 266.2Ki ± 0% +6.13% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 16.34Ki ± 0% 11.14Ki ± 0% -31.82% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 965.2Ki ± 0% 1038.0Ki ± 0% +7.55% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 17.27Ki ± 0% 10.97Ki ± 0% -36.50% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 3.711Mi ± 0% 3.981Mi ± 0% +7.27% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 16.04Ki ± 0% 11.41Ki ± 0% -28.86% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 14.88Mi ± 0% 15.53Mi ± 0% +4.34% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 23.04Ki ± 1% 17.77Ki ± 3% -22.87% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 57.72Mi ± 0% 63.79Mi ± 0% +10.51% (p=0.002 n=6) geomean 61.36Ki 46.61Ki -24.04% │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 254.0 ± 0% 149.0 ± 0% -41.34% (p=0.002 n=6) SliceDataIntoObjects/slice_1-128/writer-16 276.0 ± 0% 148.0 ± 0% -46.38% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/reader-16 223.0 ± 0% 161.0 ± 0% -27.80% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 244.0 ± 0% 160.0 ± 0% -34.43% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 218.0 ± 0% 142.0 ± 0% -34.86% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 240.0 ± 0% 143.0 ± 0% -40.42% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 204.0 ± 0% 161.0 ± 0% -21.08% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 226.0 ± 0% 160.0 ± 0% -29.20% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 211.0 ± 0% 142.0 ± 0% -32.70% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 472.0 ± 0% 478.0 ± 0% +1.27% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 211.0 ± 0% 155.0 ± 0% -26.54% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 1.140k ± 0% 1.283k ± 0% +12.54% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 247.0 ± 0% 155.0 ± 0% -37.25% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 3.884k ± 0% 4.237k ± 0% +9.10% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 243.0 ± 0% 172.0 ± 0% -29.22% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 14.91k ± 0% 16.51k ± 0% +10.70% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 259.0 ± 0% 169.0 ± 0% -34.75% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 58.73k ± 0% 64.87k ± 0% +10.47% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 240.0 ± 0% 178.0 ± 1% -25.83% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 233.8k ± 0% 252.3k ± 0% +7.92% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 340.5 ± 1% 279.5 ± 3% -17.91% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 909.8k ± 0% 1032.7k ± 0% +13.51% (p=0.002 n=6) geomean 930.0 732.6 -21.23% Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Updated after benchmark. During object slicing we recreate PrmObjectPutInit for each part, we may reuse it. │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 62.61µ ± 1% 62.90µ ± 0% ~ (p=0.132 n=6) SliceDataIntoObjects/slice_1-128/writer-16 64.94µ ± 1% 65.30µ ± 1% ~ (p=0.394 n=6) SliceDataIntoObjects/slice_4-128/reader-16 65.48µ ± 1% 61.19µ ± 1% -6.56% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 67.38µ ± 2% 63.62µ ± 1% -5.59% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 61.57µ ± 2% 63.15µ ± 1% +2.57% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 63.93µ ± 1% 64.99µ ± 1% +1.66% (p=0.004 n=6) SliceDataIntoObjects/slice_64-128/reader-16 64.89µ ± 1% 66.61µ ± 1% +2.66% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 67.24µ ± 1% 71.96µ ± 2% +7.02% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 61.19µ ± 1% 65.92µ ± 1% +7.73% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 236.2µ ± 2% 252.7µ ± 1% +6.97% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 64.34µ ± 1% 64.30µ ± 2% ~ (p=0.937 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 569.6µ ± 2% 611.3µ ± 2% +7.32% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 63.45µ ± 1% 67.79µ ± 1% +6.85% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 1.865m ± 1% 2.073m ± 1% +11.16% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 67.15µ ± 0% 64.50µ ± 1% -3.95% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 7.889m ± 1% 7.067m ± 1% -10.42% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 67.12µ ± 1% 65.98µ ± 1% -1.70% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 31.07m ± 0% 31.06m ± 1% ~ (p=1.000 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 72.49µ ± 1% 73.88µ ± 1% +1.91% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 112.7m ± 0% 112.6m ± 0% ~ (p=0.699 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 125.1µ ± 3% 125.1µ ± 3% ~ (p=0.818 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 518.8m ± 2% 510.0m ± 1% -1.71% (p=0.026 n=6) geomean 318.0µ 321.5µ +1.09% │ old.txt │ new.txt │ │ B/op │ B/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 9.453Ki ± 0% 9.547Ki ± 0% +0.99% (p=0.002 n=6) SliceDataIntoObjects/slice_1-128/writer-16 9.344Ki ± 0% 9.438Ki ± 0% +1.00% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/reader-16 10.329Ki ± 0% 9.077Ki ± 0% -12.12% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 10.220Ki ± 0% 8.968Ki ± 0% -12.25% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 8.984Ki ± 0% 9.547Ki ± 0% +6.26% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 8.968Ki ± 0% 9.438Ki ± 0% +5.24% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 10.33Ki ± 0% 11.17Ki ± 0% +8.17% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 10.22Ki ± 0% 11.06Ki ± 0% +8.26% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 8.985Ki ± 0% 10.893Ki ± 0% +21.23% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 31.18Ki ± 0% 38.08Ki ± 0% +22.11% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 9.864Ki ± 0% 9.425Ki ± 0% -4.46% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 81.16Ki ± 0% 75.66Ki ± 0% -6.78% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 9.874Ki ± 0% 11.188Ki ± 0% +13.30% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 266.2Ki ± 0% 260.8Ki ± 0% -2.01% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 11.14Ki ± 0% 10.00Ki ± 0% -10.18% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 1038.0Ki ± 0% 936.3Ki ± 0% -9.81% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 10.97Ki ± 0% 10.10Ki ± 0% -7.89% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 3.981Mi ± 0% 3.708Mi ± 0% -6.86% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 11.41Ki ± 0% 11.92Ki ± 0% +4.41% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 15.53Mi ± 0% 14.45Mi ± 0% -6.94% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 17.77Ki ± 3% 18.17Ki ± 3% +2.24% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 63.79Mi ± 0% 59.48Mi ± 0% -6.76% (p=0.002 n=6) geomean 46.61Ki 46.55Ki -0.13% │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 149.0 ± 0% 149.0 ± 0% ~ (p=1.000 n=6) ¹ SliceDataIntoObjects/slice_1-128/writer-16 148.0 ± 0% 148.0 ± 0% ~ (p=1.000 n=6) ¹ SliceDataIntoObjects/slice_4-128/reader-16 161.0 ± 0% 142.0 ± 0% -11.80% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 160.0 ± 0% 141.0 ± 0% -11.88% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 142.0 ± 0% 149.0 ± 0% +4.93% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 143.0 ± 0% 148.0 ± 0% +3.50% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 161.0 ± 0% 172.0 ± 0% +6.83% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 160.0 ± 0% 171.0 ± 0% +6.88% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 142.0 ± 0% 167.0 ± 0% +17.61% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 478.0 ± 0% 583.0 ± 0% +21.97% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 155.0 ± 0% 147.0 ± 0% -5.16% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 1.283k ± 0% 1.155k ± 0% -9.98% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 155.0 ± 0% 172.0 ± 0% +10.97% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 4.237k ± 0% 4.021k ± 0% -5.11% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 172.0 ± 0% 155.0 ± 0% -9.88% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 16.51k ± 0% 14.51k ± 0% -12.07% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 169.0 ± 0% 157.0 ± 0% -7.10% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 64.87k ± 0% 58.66k ± 0% -9.57% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 178.0 ± 1% 182.0 ± 0% +2.25% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 252.3k ± 0% 227.8k ± 0% -9.73% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 279.5 ± 3% 279.0 ± 3% ~ (p=0.790 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 1032.7k ± 0% 934.5k ± 0% -9.51% (p=0.002 n=6) geomean 732.6 720.3 -1.67% Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Updated after benchmark. During object slicing we create object for each part, we may reuse it, just resetting some fields. │ old.txt │ new.txt │ │ sec/op │ sec/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 62.90µ ± 0% 62.77µ ± 0% ~ (p=0.589 n=6) SliceDataIntoObjects/slice_1-128/writer-16 65.30µ ± 1% 65.82µ ± 0% +0.80% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/reader-16 61.19µ ± 1% 67.40µ ± 1% +10.16% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 63.62µ ± 1% 70.42µ ± 2% +10.69% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 63.15µ ± 1% 65.74µ ± 1% +4.10% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 64.99µ ± 1% 68.16µ ± 1% +4.87% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 66.61µ ± 1% 66.09µ ± 1% ~ (p=0.093 n=6) SliceDataIntoObjects/slice_64-128/writer-16 71.96µ ± 2% 77.10µ ± 7% +7.14% (p=0.041 n=6) SliceDataIntoObjects/slice_256-128/reader-16 65.92µ ± 1% 70.01µ ± 10% ~ (p=0.394 n=6) SliceDataIntoObjects/slice_256-128/writer-16 252.7µ ± 1% 234.0µ ± 1% -7.42% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 64.30µ ± 2% 65.18µ ± 6% ~ (p=0.240 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 611.3µ ± 2% 570.0µ ± 5% -6.76% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 67.79µ ± 1% 69.01µ ± 3% +1.80% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 2.073m ± 1% 2.024m ± 3% -2.39% (p=0.026 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 64.50µ ± 1% 64.07µ ± 0% -0.66% (p=0.026 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 7.067m ± 1% 6.792m ± 8% ~ (p=0.065 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 65.98µ ± 1% 69.42µ ± 1% +5.22% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 31.06m ± 1% 30.11m ± 1% -3.07% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 73.88µ ± 1% 76.69µ ± 3% +3.81% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 112.6m ± 0% 123.6m ± 4% +9.76% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 125.1µ ± 3% 106.8µ ± 12% -14.61% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 510.0m ± 1% 444.4m ± 1% -12.85% (p=0.002 n=6) geomean 321.5µ 322.7µ +0.37% │ old.txt │ new.txt │ │ B/op │ B/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 9.547Ki ± 0% 9.547Ki ± 0% ~ (p=1.000 n=6) SliceDataIntoObjects/slice_1-128/writer-16 9.438Ki ± 0% 9.421Ki ± 0% -0.18% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/reader-16 9.077Ki ± 0% 11.298Ki ± 0% +24.46% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 8.968Ki ± 0% 11.173Ki ± 0% +24.59% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 9.547Ki ± 0% 10.548Ki ± 0% +10.48% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 9.438Ki ± 0% 10.423Ki ± 0% +10.44% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 11.17Ki ± 0% 10.55Ki ± 0% -5.59% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 11.06Ki ± 0% 10.42Ki ± 0% -5.79% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 10.893Ki ± 0% 9.204Ki ± 0% -15.50% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 38.08Ki ± 0% 29.42Ki ± 0% -22.73% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 9.425Ki ± 0% 9.675Ki ± 0% +2.65% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 75.66Ki ± 0% 69.60Ki ± 0% -8.01% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 11.19Ki ± 0% 11.31Ki ± 0% +1.11% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 260.8Ki ± 0% 235.7Ki ± 0% -9.65% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 10.004Ki ± 0% 9.716Ki ± 0% -2.88% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 936.3Ki ± 0% 831.3Ki ± 0% -11.21% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 10.10Ki ± 0% 11.50Ki ± 0% +13.79% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 3.708Mi ± 0% 3.307Mi ± 0% -10.80% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 11.92Ki ± 0% 12.32Ki ± 0% +3.36% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 14.45Mi ± 0% 13.26Mi ± 0% -8.21% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 18.17Ki ± 3% 15.16Ki ± 6% -16.53% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 59.48Mi ± 0% 51.22Mi ± 0% -13.89% (p=0.002 n=6) geomean 46.55Ki 45.35Ki -2.56% │ old.txt │ new.txt │ │ allocs/op │ allocs/op vs base │ SliceDataIntoObjects/slice_1-128/reader-16 149.0 ± 0% 149.0 ± 0% ~ (p=1.000 n=6) ¹ SliceDataIntoObjects/slice_1-128/writer-16 148.0 ± 0% 148.0 ± 0% ~ (p=1.000 n=6) ¹ SliceDataIntoObjects/slice_4-128/reader-16 142.0 ± 0% 174.0 ± 0% +22.54% (p=0.002 n=6) SliceDataIntoObjects/slice_4-128/writer-16 141.0 ± 0% 173.0 ± 0% +22.70% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/reader-16 149.0 ± 0% 163.0 ± 0% +9.40% (p=0.002 n=6) SliceDataIntoObjects/slice_16-128/writer-16 148.0 ± 0% 162.0 ± 0% +9.46% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/reader-16 172.0 ± 0% 163.0 ± 0% -5.23% (p=0.002 n=6) SliceDataIntoObjects/slice_64-128/writer-16 171.0 ± 0% 162.0 ± 0% -5.26% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/reader-16 167.0 ± 0% 144.0 ± 0% -13.77% (p=0.002 n=6) SliceDataIntoObjects/slice_256-128/writer-16 583.0 ± 0% 437.0 ± 0% -25.04% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/reader-16 147.0 ± 0% 151.0 ± 0% +2.72% (p=0.002 n=6) SliceDataIntoObjects/slice_1024-128/writer-16 1.155k ± 0% 1.030k ± 0% -10.82% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/reader-16 172.0 ± 0% 174.0 ± 0% +1.16% (p=0.002 n=6) SliceDataIntoObjects/slice_4096-128/writer-16 4.021k ± 0% 3.464k ± 0% -13.85% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/reader-16 155.0 ± 0% 151.0 ± 0% -2.58% (p=0.002 n=6) SliceDataIntoObjects/slice_16384-128/writer-16 14.51k ± 0% 12.20k ± 0% -15.96% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/reader-16 157.0 ± 0% 176.0 ± 0% +12.10% (p=0.002 n=6) SliceDataIntoObjects/slice_65536-128/writer-16 58.66k ± 0% 49.56k ± 0% -15.52% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/reader-16 182.0 ± 0% 188.0 ± 1% +3.30% (p=0.002 n=6) SliceDataIntoObjects/slice_262144-128/writer-16 227.8k ± 0% 197.1k ± 0% -13.48% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/reader-16 279.0 ± 3% 230.5 ± 6% -17.38% (p=0.002 n=6) SliceDataIntoObjects/slice_1048576-128/writer-16 934.5k ± 0% 762.4k ± 0% -18.42% (p=0.002 n=6) geomean 720.3 690.1 -4.19% Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
a13249c
to
d6858d9
Compare
closes #442