Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Slicer interface more generic #466

Merged
merged 11 commits into from
Jul 29, 2023

Conversation

smallhive
Copy link
Contributor

closes #442

@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch 2 times, most recently from f889904 to a3783d5 Compare July 10, 2023 09:13
@smallhive smallhive marked this pull request as ready for review July 10, 2023 09:15
object/slicer/slicer.go Outdated Show resolved Hide resolved
client/object_put.go Outdated Show resolved Hide resolved
object/slicer/slicer.go Show resolved Hide resolved
object/slicer/slicer.go Outdated Show resolved Hide resolved
@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch from a3783d5 to 8358b6a Compare July 11, 2023 03:34
@smallhive smallhive requested a review from carpawell July 11, 2023 03:41
Copy link
Contributor

@cthulhu-rider cthulhu-rider left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

im a bit confused that we require to pass object.Object: imo this can complicate the code of a simple data uploading applications.

how about provide both raw interface (current) accepting object.Object and the sugared (previous) one requiring min input? @smallhive @carpawell @roman-khimov


// SetSession pass session object to writer. It will be used for writing object routine.
// This method must be called before InitDataStream.
SetSession(sess session.Object)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requirements are pretty strict, maybe it'd be clearer to pass session.Object parameter into InitDataStream?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may suggest an alternative solution:

  • remove SetSession from the interface
  • update func NewDataSlicer(ctx context.Context, cli *Client) (*slicer.Slicer, error) {
    to func NewDataSlicer(ctx context.Context, cli *Client, sess *session.Object) (*slicer.Slicer, error) {.
    In this case we hide session inside objectWriter and don't need to set it outside
  • make objectWriter public as a default implementation for the interface. It's constructor even may have session.Object as a parameter

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requirements are pretty strict

we thought that we need a way to be able to create a slicer once and slice more than one object and send parts to more than one node (so more than one session will be attached). we can discuss it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I apply my suggestions?

Copy link
Contributor

@cthulhu-rider cthulhu-rider Jul 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about provide both client.NewDataSlicer and client.NewDataSlicerWithSession?

make objectWriter public

so we'll have

w := client.NewObjectWriter(cli)
// or
w := client.NewObjectWriterWithSession(cli, sessionToken)

s := slicer.NewSession(signer, cnr, sessionToken, w, opts)

s.Slicer(strings.NewReader("Hello, world!"))

right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. The option with two constructors for ObjectWriter also looks fine to me and it will help us to remove SetSession(sess session.Object) from the interface

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cthulhu-rider, how that allows us to send the parts to different nodes with only one Slicer (the problem/wish i described above)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carpawell

send the parts to different nodes

we can do it with single session token, can't we? Afaik we don't deal with dynamic sessions when we talk about Slicer. The app just takes user's session token and attach it to any node it send objects to

so pls clarify ur concern about different nodes, maybe i didn't get ur point

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Afaik we don't deal with dynamic sessions when we talk about Slicer

i said smth wrong. well, the question is about reusing the Slicer then. do we want to be able to create Slicer once and handle any object from any user?

@smallhive
Copy link
Contributor Author

im a bit confused that we require to pass object.Object: imo this can complicate the code of a simple data uploading applications.

how about provide both raw interface (current) accepting object.Object and the sugared (previous) one requiring min input? @smallhive @carpawell @roman-khimov

I consider, it doesn't make sense. User has to pass all required parameters for the header object (signer neofscrypto.Signer, cnr cid.ID, owner user.ID) and we create it inside.
Passing object.Object gives a flexible way to configure the header. In this case, the user creates the header itself with the same data.

For comparison, we manipulate the same data, but with different ways:

Before

slicer := new(signer neofscrypto.Signer, cnr cid.ID, owner user.ID, w ObjectWriter, opts Options)
slicer.Slice(data io.Reader, attributes ...string)

After

var hdr object.Object
header.SetContainerID(cnr)
header.SetOwnerID(&owner)
header.SetAttributes(...)
header.///

slicer := new(w ObjectWriter)
slicer.Slice(header object.Object, data io.Reader, signer neofscrypto.Signer, opts Options)

Yes, the code longer for header creation, but it is deadly simple

@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Jul 13, 2023

header.///

that's what exactly bothers me. How user will undestand which fields must be explicitly set and which are optional or even must not be set?

accepting cid.ID, user.ID, neofscrypto.Signer, string attibutes and io.Reader covers most of the external uploader's needs imo. And accepting header structure (object.Object as done in this PR) is good for system needs and small part of deeply understanding users. Simple splitted input can call header one within package (from simple to flexible).

i can also suggest to implement customization through optional header construction callback that is going to be called during object formation if set:

var opts Options
opts.SetCustomHeaderConstructor(func(obj *object.Object) {
  obj.SetType(object.Tombstone)
  obj.SetWhatever()
})

slicer := new(signer neofscrypto.Signer, cnr cid.ID, owner user.ID, w ObjectWriter, opts Options)
slicer.SliceData(bytes.NewReader(tombstone.Marshal()))

P.S. me personally completely okay with raw API, but I'm not sure if it will be convenient and understandable for simple data upload applications. I'd provide helpers that transform what they have to required stuff like hdr object.Object.

@smallhive
Copy link
Contributor Author

header.///

that's what exactly bothers me. How user will undestand which fields must be explicitly set and which are optional or even must not be set?

It's not a problem. We can create a helper function inside the slicer package. For instance:

func PrepareHeader(cnr cid.ID, owner oid.ID) object.Object

Also, we will mention it inside the Slicer constructor/Slice method. Any simple upload application will manage to call the func for header construction. This will be a good starting point for them, to touch our product. But, as practice shows, latter they would want to dive deeper. In this case they shouldn't rewrite a lot their code, because PrepareHeader is just 2 rows function.

accepting cid.ID, user.ID, neofscrypto.Signer, string attibutes and io.Reader covers most of the external uploader's needs imo. And accepting header structure (object.Object as done in this PR) is good for system needs and small part of deeply understanding users. Simple splitted input can call header one within package (from simple to flexible).

I don't agree with strings attributes. Regular client and node already use object.Attribute and have to reconvert existing attributes to string. Me and @carpawell faced the situation.

i can also suggest to implement customization through optional header construction callback that is going to be called during object formation if set:

var opts Options
opts.SetCustomHeaderConstructor(func(obj *object.Object) {
  obj.SetType(object.Tombstone)
  obj.SetWhatever()
})
slicer := new(signer neofscrypto.Signer, cnr cid.ID, owner user.ID, w ObjectWriter, opts Options)
slicer.SliceData(bytes.NewReader(tombstone.Marshal()))

One more option var 😃 Passing a fully prepared header eliminates all misunderstandings. This callback, we would
only call for the header, because all other parts are TypeRegular. Thus, such a callback looks excessive.

P.S. me personally completely okay with raw API, but I'm not sure if it will be convenient and understandable for simple data upload applications. I'd provide helpers that transform what they have to required stuff like hdr object.Object.

Even the current SDK interface is quite clear and understandable. The slicer improves these factors, but I tend to think it shouldn't make such far abstraction from SDK types

@cthulhu-rider
Copy link
Contributor

cthulhu-rider commented Jul 17, 2023

@smallhive i understand and agree with almost all ur points. Let me summarize my own

in my mind, slicer space is intended to simplify data upload. The min required set to upload the data into NeoFS is container ID, Signer and data itself encapsulated in io.Reader primitive. So, i'd like to provide API which allows to use this small input and perform uploading by min num of instructions (ideally - by 1 instruction). If we also want to support some more specific needs like tombstone formation, it's better to add specific API next to the simple one, implement the simple API through the specific one and not force the user to understand the specifics (if he wants to, he will figure it out and use).

to sum it up,

s := slicer.New(signer, objectWriter, opts)
err := slicer.Slice(slicer.RequiredFields(cnr, owner), strings.NewReader("Hello, world!"))

looks pretty good to me. If this is the approach u suggested, i agree with it.

maybe single-op function would be also useful

err := slicer.UploadDataToContainer(ctx, cnr, objectWriter, dataReader) 

@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch 2 times, most recently from 4a98b92 to 5ad7daf Compare July 18, 2023 09:36
@smallhive
Copy link
Contributor Author

in my mind, slicer space is intended to simplify data upload

Yes, the slicer does it, but not for everyone. Current changes just move parameters from one function to another. These changes were inspired by slicer usage in few projects

maybe single-op function would be also useful

err := slicer.UploadDataToContainer(ctx, cnr, objectWriter, dataReader) 

For such purposes, we have client.CreateObject. Current PR code tries to achieve unification for all client types

Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me try to untangle it from the beginning.

Slicer in many ways starts with InitDataStream API for ObjectWriter. Why do we have it? Because we need to ObjectPutInit and WriteHeader at the same time. But we've moved a bit from RC9 and after e19e49a it's just no longer an issue, plain ObjectPutInit() does everything needed. This means we don't need this interface at all and slicer package can work with a simple ObjectPutInit() definition (notice that it can come from client or pool). Let's caller it ObjectPutter for now (just to differentiate from ObjectWriter even though it's a nice name to use in slicer).

Then IIUC we have two use cases:

  • we have a complete object header (of some fancy type) and want to push some data using this header
  • we have no idea what these headers are, but just want to save some data with some attributes

The first one implies that we already know how to fill in object headers, including containers/owners/sessions and all of this is just out of scope for the API. The second one is better be assisted by the API.

So the first case looks like

func Write(ctx context.Context, op ObjectPutter, header object.Object, signer neofscrypto.Signer, limit uint64, data io.Reader) (oid.ID, error)
func InitWriter(ctx context.Context, op ObjectPutter, header object.Object, signer neofscrypto.Signer, limit uint64) (PayloadWriter, error)

No need for fancy structures or anything like that here. A pair of functions and that's it. Notice that in general it follows the node's pattern of working with multiple containers/owners/sessions.

Now for the second case we want to have a slicer.Slicer to help us. We need to have network info for that which means another

type NetworkedWriter interface {
    ObjectPutter
    NetworkInfo(ctx context.Context, prm PrmNetworkInfo) (netmap.NetworkInfo, error)
}

and

func New(ctx context.Context, nw NetworkedWriter, signer neofscrypto.Signer, cnr cid.ID, owner user.ID, opts Options) (*Slicer, error)

Session can be passed via Options easily. This follows the regular application pattern of "I'm managing some data in a single container by a single user". Then it's per-object

func (x *Slicer) Write(data io.Reader, attrs []object.Attribute) (oid.ID, error)
func (x *Slicer) InitWriter(attrs []object.Attribute) (PayloadWriter, error)

That's about it. client.CreateObject and client.NewDataSlicer can both be dropped as useless, slicer completely substitutes them.

object/slicer/slicer.go Outdated Show resolved Hide resolved
@roman-khimov
Copy link
Member

Maybe

s/ObjectPutter/ObjectClient/
s/NetworkedWriter/NetworkedClient/
s/Write/Put/
s/InitWrite/InitPut/

For better names and consistency with client.

@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch 2 times, most recently from 35e3d01 to af3ae26 Compare July 24, 2023 07:27
@smallhive
Copy link
Contributor Author

According to the comment, completely rewrite the solution

object/slicer/slicer.go Outdated Show resolved Hide resolved
client/object_put.go Outdated Show resolved Hide resolved
pool/pool.go Outdated Show resolved Hide resolved
pool/pool.go Outdated Show resolved Hide resolved
object/slicer/slicer.go Outdated Show resolved Hide resolved
object/slicer/slicer.go Show resolved Hide resolved
object/slicer/slicer.go Outdated Show resolved Hide resolved
object/slicer/slicer.go Show resolved Hide resolved
@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch from af3ae26 to 93a0ca2 Compare July 25, 2023 08:04
client/object_put.go Outdated Show resolved Hide resolved
client/object_put.go Show resolved Hide resolved
client/object_put.go Show resolved Hide resolved
pool/pool.go Outdated Show resolved Hide resolved
object/slicer/options.go Outdated Show resolved Hide resolved
object/slicer/slicer.go Show resolved Hide resolved
object/slicer/slicer.go Show resolved Hide resolved
@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch 2 times, most recently from 4ad0460 to f87b699 Compare July 26, 2023 07:10
@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch 3 times, most recently from 4a5f2ee to a59ad2d Compare July 27, 2023 12:07
@smallhive
Copy link
Contributor Author

I've made some optimizations and divided each into separate commits. I started from this numbers

goos: linux
goarch: amd64
pkg: github.com/nspcc-dev/neofs-sdk-go/object/slicer
cpu: Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz
slice_1-128/reader-16        10000            107744 ns/op            8767 B/op        125 allocs/op
slice_1-128/writer-16        10000            111737 ns/op           10027 B/op        147 allocs/op
slice_4-128/reader-16        10000            111991 ns/op           10432 B/op        151 allocs/op
slice_4-128/writer-16         9958            117292 ns/op           11693 B/op        173 allocs/op
slice_16-128/reader-16       10000            110385 ns/op            9920 B/op        144 allocs/op
slice_16-128/writer-16       10000            113275 ns/op           11117 B/op        165 allocs/op
slice_64-128/reader-16        9385            114386 ns/op           12097 B/op        175 allocs/op
slice_64-128/writer-16        9662            119859 ns/op           13358 B/op        197 allocs/op
slice_256-128/reader-16              10000            108500 ns/op            9152 B/op        132 allocs/op
slice_256-128/writer-16               5090            230353 ns/op           21807 B/op        316 allocs/op
slice_1024-128/reader-16              9740            114233 ns/op           11461 B/op        163 allocs/op
slice_1024-128/writer-16              1930            611461 ns/op           55955 B/op        831 allocs/op
slice_4096-128/reader-16              8979            113358 ns/op           11472 B/op        163 allocs/op
slice_4096-128/writer-16               594           2019841 ns/op          175110 B/op       2621 allocs/op
slice_16384-128/reader-16            10701            111446 ns/op           10482 B/op        151 allocs/op
slice_16384-128/writer-16              156           7628920 ns/op          650199 B/op       9750 allocs/op
slice_65536-128/reader-16            10213            117537 ns/op           12304 B/op        177 allocs/op
slice_65536-128/writer-16               39          30207695 ns/op         2559859 B/op      38417 allocs/op
slice_262144-128/reader-16            9188            126489 ns/op           12594 B/op        179 allocs/op
slice_262144-128/writer-16              10         111468257 ns/op         9840274 B/op     146713 allocs/op
slice_1048576-128/reader-16           5631            193997 ns/op           16502 B/op        231 allocs/op
slice_1048576-128/writer-16              3         492035871 ns/op        41129728 B/op     610583 allocs/op

In the first optimization, all numbers look much better:

slice_1-128/reader-16        18528             64738 ns/op            7510 B/op        115 allocs/op
slice_1-128/writer-16        17288             67094 ns/op            7390 B/op        114 allocs/op
slice_4-128/reader-16        19203             62794 ns/op            6613 B/op        102 allocs/op
slice_4-128/writer-16        18355             64467 ns/op            6493 B/op        101 allocs/op
slice_16-128/reader-16       18453             64495 ns/op            7510 B/op        115 allocs/op
slice_16-128/writer-16       18032             67136 ns/op            7390 B/op        114 allocs/op
slice_64-128/reader-16       18630             64012 ns/op            7094 B/op        109 allocs/op
slice_64-128/writer-16       18106             67160 ns/op            6974 B/op        108 allocs/op
slice_256-128/reader-16              18684             64158 ns/op            7447 B/op        114 allocs/op
slice_256-128/writer-16               4561            253376 ns/op           24835 B/op        367 allocs/op
slice_1024-128/reader-16             17294             67680 ns/op            8762 B/op        132 allocs/op
slice_1024-128/writer-16              1864            638404 ns/op           64056 B/op        984 allocs/op
slice_4096-128/reader-16             18661             65393 ns/op            7456 B/op        114 allocs/op
slice_4096-128/writer-16               528           2135134 ns/op          190523 B/op       2966 allocs/op
slice_16384-128/reader-16            19276             62407 ns/op            6649 B/op        103 allocs/op
slice_16384-128/writer-16              166           7076519 ns/op          688827 B/op      10848 allocs/op
slice_65536-128/reader-16            17730             67998 ns/op            8144 B/op        123 allocs/op
slice_65536-128/writer-16               39          28123398 ns/op         2727227 B/op      43016 allocs/op
slice_262144-128/reader-16           14220             74287 ns/op            8763 B/op        133 allocs/op
slice_262144-128/writer-16               9         112732862 ns/op        10971824 B/op     171290 allocs/op
slice_1048576-128/reader-16           9207            114910 ns/op           12756 B/op        195 allocs/op
slice_1048576-128/writer-16              3         457196455 ns/op        43819968 B/op     684384 allocs/op

Second one improves memory allocations in case we slice object, since slice_256-128:

slice_1-128/writer-16        18350             65989 ns/op            7486 B/op        114 allocs/op
slice_4-128/reader-16        20100             60763 ns/op            6709 B/op        102 allocs/op
slice_4-128/writer-16        18764             63415 ns/op            6589 B/op        101 allocs/op
slice_16-128/reader-16       18746             65253 ns/op            7606 B/op        115 allocs/op
slice_16-128/writer-16       18037             66975 ns/op            7486 B/op        114 allocs/op
slice_64-128/reader-16       19128             62007 ns/op            7190 B/op        109 allocs/op
slice_64-128/writer-16       18426             65158 ns/op            7069 B/op        108 allocs/op
slice_256-128/reader-16              19174             62161 ns/op            7543 B/op        114 allocs/op
slice_256-128/writer-16               4752            246579 ns/op           23827 B/op        343 allocs/op
slice_1024-128/reader-16             18292             65920 ns/op            8857 B/op        132 allocs/op
slice_1024-128/writer-16              1875            618800 ns/op           59733 B/op        888 allocs/op
slice_4096-128/reader-16             19152             62748 ns/op            7551 B/op        114 allocs/op
slice_4096-128/writer-16               588           2017570 ns/op          172941 B/op       2582 allocs/op
slice_16384-128/reader-16            19446             60224 ns/op            6741 B/op        102 allocs/op
slice_16384-128/writer-16              174           6892950 ns/op          618236 B/op       9313 allocs/op
slice_65536-128/reader-16            18182             64999 ns/op            8221 B/op        123 allocs/op
slice_65536-128/writer-16               42          27333753 ns/op         2444545 B/op      36875 allocs/op
slice_262144-128/reader-16           15064             71057 ns/op            8740 B/op        131 allocs/op
slice_262144-128/writer-16              10         109920692 ns/op         9840644 B/op     146724 allocs/op
slice_1048576-128/reader-16          10192            107642 ns/op           11947 B/op        179 allocs/op
slice_1048576-128/writer-16              3         441915657 ns/op        39295813 B/op     586047 allocs/op

The last one make one more step in the same direction:

slice_1-128/reader-16        18714             63692 ns/op            7735 B/op        117 allocs/op
slice_1-128/writer-16        18046             65321 ns/op            7606 B/op        116 allocs/op
slice_4-128/reader-16        19737             61630 ns/op            6837 B/op        104 allocs/op
slice_4-128/writer-16        18999             63062 ns/op            6709 B/op        103 allocs/op
slice_16-128/reader-16       19062             62944 ns/op            7734 B/op        117 allocs/op
slice_16-128/writer-16       18330             65544 ns/op            7606 B/op        116 allocs/op
slice_64-128/reader-16       19230             62301 ns/op            7318 B/op        111 allocs/op
slice_64-128/writer-16       18630             65339 ns/op            7190 B/op        110 allocs/op
slice_256-128/reader-16              19058             62822 ns/op            7671 B/op        116 allocs/op
slice_256-128/writer-16               4740            245422 ns/op           23034 B/op        326 allocs/op
slice_1024-128/reader-16             18050             65924 ns/op            8985 B/op        134 allocs/op
slice_1024-128/writer-16              1990            602852 ns/op           53921 B/op        763 allocs/op
slice_4096-128/reader-16             18918             63482 ns/op            7678 B/op        116 allocs/op
slice_4096-128/writer-16               602           1968023 ns/op          147059 B/op       2025 allocs/op
slice_16384-128/reader-16            19386             60573 ns/op            6864 B/op        104 allocs/op
slice_16384-128/writer-16              176           6736108 ns/op          512057 B/op       7027 allocs/op
slice_65536-128/reader-16            17881             66058 ns/op            8328 B/op        125 allocs/op
slice_65536-128/writer-16               44          26234372 ns/op         2017252 B/op      27672 allocs/op
slice_262144-128/reader-16           15237             72409 ns/op            8748 B/op        130 allocs/op
slice_262144-128/writer-16              10         106224264 ns/op         8128953 B/op     109876 allocs/op
slice_1048576-128/reader-16           9043            112551 ns/op           11808 B/op        172 allocs/op
slice_1048576-128/writer-16              3         429085746 ns/op        32448024 B/op     438630 allocs/op

@smallhive
Copy link
Contributor Author

I made one more interesting experiment, just locally and many tradeoffs like t.Skip etc.
I've rewritten object.Object to the regular struct type. All setters and getters worked with internal fields. Just like example:

type Object struct {
	v2 *object.Object
	isSet     bool
	id        oid.ID
	signature *neofscrypto.Signature
	version   *version.Version
	payloadSize uint64
	containerID      cid.ID
	isSetContainerID bool
	owner *user.ID
	epoch uint64
	chsum      checksum.Checksum
	isSetChsum bool
	homo   checksum.Checksum
	isHomo bool
	attributes []Attribute
	prevID *oid.ID
	children []oid.ID
	splitInfo *object.SplitHeader
	sess *session.Object
	tp Type
}

Converting to v2/fromV2 and working with api-go object.Object was only in marshaling methods and in some places where it was required to save some stability.

And eventually the result of the bench with these changes:

slice_1-128/reader-16        22546             53294 ns/op            5293 B/op         65 allocs/op
slice_1-128/writer-16        21884             53806 ns/op            5172 B/op         64 allocs/op
slice_4-128/reader-16        22683             52308 ns/op            5549 B/op         69 allocs/op
slice_4-128/writer-16        21963             54744 ns/op            5428 B/op         68 allocs/op
slice_16-128/reader-16       22779             52688 ns/op            5292 B/op         65 allocs/op
slice_16-128/writer-16       22020             53704 ns/op            5172 B/op         64 allocs/op
slice_64-128/reader-16       22689             52312 ns/op            5292 B/op         65 allocs/op
slice_64-128/writer-16       22120             54532 ns/op            5172 B/op         64 allocs/op
slice_256-128/reader-16              22698             52703 ns/op            5549 B/op         69 allocs/op
slice_256-128/writer-16               5796            201214 ns/op           14979 B/op        208 allocs/op
slice_1024-128/reader-16             22813             52194 ns/op            5294 B/op         65 allocs/op
slice_1024-128/writer-16              2508            468130 ns/op           33266 B/op        467 allocs/op
slice_4096-128/reader-16             22807             52623 ns/op            5553 B/op         69 allocs/op
slice_4096-128/writer-16               676           1747515 ns/op          113465 B/op       1607 allocs/op
slice_16384-128/reader-16            22594             52474 ns/op            5310 B/op         65 allocs/op
slice_16384-128/writer-16              199           5969513 ns/op          411857 B/op       5812 allocs/op
slice_65536-128/reader-16            18676             53830 ns/op            5379 B/op         66 allocs/op
slice_65536-128/writer-16               50          23695444 ns/op         1623365 B/op      22901 allocs/op
slice_262144-128/reader-16           19968             57536 ns/op            5622 B/op         70 allocs/op
slice_262144-128/writer-16              12          95210776 ns/op         6579025 B/op      91271 allocs/op
slice_1048576-128/reader-16          12870             86426 ns/op            7677 B/op         99 allocs/op
slice_1048576-128/writer-16              3         428006130 ns/op        27349832 B/op     381077 allocs/op

Of course, it is just for comparison and to think in the future to manage something with this (or not).
Making such changes right now will lead to disaster

Copy link
Member

@roman-khimov roman-khimov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there.

@carpawell, does it work API-wise for node?

@smallhive, it'd be nice to have benchmark diff (as in benchstat output) in commit messages.

pool/pool.go Outdated
}

// write the last part
if _, err = wObj.Write(buf[:n]); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd try to not duplicate this code below.

Copy link
Contributor Author

@smallhive smallhive Jul 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated, also added a few tests for it

object/slicer/slicer.go Show resolved Hide resolved
object/slicer/slicer.go Outdated Show resolved Hide resolved
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch from a59ad2d to b45433c Compare July 28, 2023 05:41
@smallhive
Copy link
Contributor Author

Almost there.

@carpawell, does it work API-wise for node?

@smallhive, it'd be nice to have benchmark diff (as in benchstat output) in commit messages.

Added results to commits. But after rebase numbers are changed, slightly

@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch 2 times, most recently from c41562a to a13249c Compare July 28, 2023 07:20
@smallhive
Copy link
Contributor Author

Updated commit message benchmark statistic, also final approach: master vs current branch

                                                 │   old.txt    │               new.txt               │
                                                 │    sec/op    │    sec/op     vs base               │
SliceDataIntoObjects/slice_1-128/reader-16         113.39µ ± 0%   62.77µ ±  0%  -44.64% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16         117.84µ ± 1%   65.82µ ±  0%  -44.14% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16         116.26µ ± 1%   67.40µ ±  1%  -42.02% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16         121.26µ ± 3%   70.42µ ±  2%  -41.92% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16        111.88µ ± 8%   65.74µ ±  1%  -41.24% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16        116.83µ ± 2%   68.16µ ±  1%  -41.66% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16        113.87µ ± 2%   66.09µ ±  1%  -41.96% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16        120.53µ ± 2%   77.10µ ±  7%  -36.04% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       118.88µ ± 3%   70.01µ ± 10%  -41.11% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        254.7µ ± 4%   234.0µ ±  1%   -8.14% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16      120.60µ ± 3%   65.18µ ±  6%  -45.95% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       645.1µ ± 1%   570.0µ ±  5%  -11.65% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16      118.30µ ± 3%   69.01µ ±  3%  -41.67% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       2.131m ± 5%   2.024m ±  3%   -5.03% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16     112.72µ ± 2%   64.07µ ±  0%  -43.16% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16      7.266m ± 1%   6.792m ±  8%        ~ (p=0.065 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16     115.29µ ± 3%   69.42µ ±  1%  -39.79% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      28.75m ± 1%   30.11m ±  1%   +4.74% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16    131.98µ ± 4%   76.69µ ±  3%  -41.89% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     121.1m ± 2%   123.6m ±  4%   +2.07% (p=0.041 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    218.6µ ± 8%   106.8µ ± 12%  -51.15% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16    528.9m ± 6%   444.4m ±  1%  -15.96% (p=0.002 n=6)
geomean                                             480.8µ        322.7µ        -32.89%

                                                 │    old.txt    │               new.txt               │
                                                 │     B/op      │     B/op      vs base               │
SliceDataIntoObjects/slice_1-128/reader-16         15.629Ki ± 0%   9.547Ki ± 0%  -38.92% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16         16.786Ki ± 0%   9.421Ki ± 0%  -43.88% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16          16.50Ki ± 0%   11.30Ki ± 0%  -31.55% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16          17.66Ki ± 0%   11.17Ki ± 0%  -36.74% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16         14.00Ki ± 0%   10.55Ki ± 0%  -24.67% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16         15.16Ki ± 0%   10.42Ki ± 0%  -31.25% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         15.63Ki ± 0%   10.55Ki ± 0%  -32.51% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         16.79Ki ± 0%   10.42Ki ± 0%  -37.91% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       13.630Ki ± 0%   9.204Ki ± 0%  -32.47% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        30.83Ki ± 0%   29.42Ki ± 0%   -4.57% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16      16.887Ki ± 0%   9.675Ki ± 0%  -42.71% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       81.64Ki ± 0%   69.60Ki ± 0%  -14.75% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16       15.28Ki ± 0%   11.31Ki ± 0%  -25.95% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       255.3Ki ± 0%   235.7Ki ± 0%   -7.70% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16     13.205Ki ± 0%   9.716Ki ± 0%  -26.42% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16      932.1Ki ± 0%   831.3Ki ± 0%  -10.81% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      13.48Ki ± 0%   11.50Ki ± 0%  -14.71% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      3.595Mi ± 0%   3.307Mi ± 0%   -8.01% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     16.32Ki ± 1%   12.32Ki ± 0%  -24.56% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     14.45Mi ± 0%   13.26Mi ± 0%   -8.18% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    24.55Ki ± 5%   15.16Ki ± 6%  -38.24% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16    59.47Mi ± 0%   51.22Mi ± 0%  -13.88% (p=0.002 n=6)
geomean                                             61.32Ki        45.35Ki       -26.03%

                                                 │   old.txt   │              new.txt               │
                                                 │  allocs/op  │  allocs/op   vs base               │
SliceDataIntoObjects/slice_1-128/reader-16          235.0 ± 0%    149.0 ± 0%  -36.60% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16          256.0 ± 0%    148.0 ± 0%  -42.19% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16          247.0 ± 0%    174.0 ± 0%  -29.55% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16          268.0 ± 0%    173.0 ± 0%  -35.45% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16         211.0 ± 0%    163.0 ± 0%  -22.75% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16         232.0 ± 0%    162.0 ± 0%  -30.17% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         235.0 ± 0%    163.0 ± 0%  -30.64% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         256.0 ± 0%    162.0 ± 0%  -36.72% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16        204.0 ± 0%    144.0 ± 0%  -29.41% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        458.0 ± 0%    437.0 ± 0%   -4.59% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16       254.0 ± 0%    151.0 ± 0%  -40.55% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16      1.250k ± 0%   1.030k ± 0%  -17.60% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16       230.0 ± 0%    174.0 ± 0%  -24.35% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16      3.936k ± 0%   3.464k ± 0%  -11.99% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16      198.0 ± 0%    151.0 ± 0%  -23.74% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16     14.44k ± 0%   12.20k ± 0%  -15.53% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      202.0 ± 0%    176.0 ± 0%  -12.87% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16     57.08k ± 0%   49.56k ± 0%  -13.17% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     245.0 ± 0%    188.0 ± 1%  -23.27% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16    227.7k ± 0%   197.1k ± 0%  -13.44% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    360.5 ± 5%    230.5 ± 6%  -36.06% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16   934.4k ± 0%   762.4k ± 0%  -18.40% (p=0.002 n=6)
geomean                                             928.4         690.1       -25.67%

Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
…d pool

Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
closes #442

Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Updated after benchmark. For the header object we already made flushObjectMetadata step in _writeChild and don't need to compute this data again.

                                                 │   old.txt    │              new.txt               │
                                                 │    sec/op    │   sec/op     vs base               │
SliceDataIntoObjects/slice_1-128/reader-16         116.55µ ± 1%   62.61µ ± 1%  -46.28% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16         121.73µ ± 2%   64.94µ ± 1%  -46.65% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16         111.50µ ± 3%   65.48µ ± 1%  -41.27% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16         113.84µ ± 1%   67.38µ ± 2%  -40.81% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16        110.43µ ± 1%   61.57µ ± 2%  -44.25% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16        115.11µ ± 1%   63.93µ ± 1%  -44.46% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16        109.10µ ± 1%   64.89µ ± 1%  -40.53% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16        114.07µ ± 1%   67.24µ ± 1%  -41.05% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       108.68µ ± 1%   61.19µ ± 1%  -43.70% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        233.2µ ± 0%   236.2µ ± 2%        ~ (p=0.093 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16      109.04µ ± 1%   64.34µ ± 1%  -41.00% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       554.6µ ± 0%   569.6µ ± 2%   +2.71% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16      114.62µ ± 0%   63.45µ ± 1%  -44.65% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       1.855m ± 2%   1.865m ± 1%        ~ (p=0.180 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16     117.09µ ± 3%   67.15µ ± 0%  -42.65% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16      7.745m ± 2%   7.889m ± 1%   +1.86% (p=0.026 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16     118.90µ ± 1%   67.12µ ± 1%  -43.55% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      30.71m ± 1%   31.07m ± 0%   +1.20% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16    124.95µ ± 2%   72.49µ ± 1%  -41.98% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     124.0m ± 1%   112.7m ± 0%   -9.15% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    184.8µ ± 2%   125.1µ ± 3%  -32.30% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16    453.8m ± 1%   518.8m ± 2%  +14.32% (p=0.002 n=6)
geomean                                             461.0µ        318.0µ       -31.02%

                                                 │    old.txt    │               new.txt                │
                                                 │     B/op      │     B/op       vs base               │
SliceDataIntoObjects/slice_1-128/reader-16         16.880Ki ± 0%    9.453Ki ± 0%  -44.00% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16         18.100Ki ± 0%    9.344Ki ± 0%  -48.38% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16          14.75Ki ± 0%    10.33Ki ± 0%  -29.99% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16          15.91Ki ± 0%    10.22Ki ± 0%  -35.77% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16        14.504Ki ± 0%    8.984Ki ± 0%  -38.06% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16        15.724Ki ± 0%    8.968Ki ± 0%  -42.97% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         13.63Ki ± 0%    10.33Ki ± 0%  -24.21% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         14.85Ki ± 0%    10.22Ki ± 0%  -31.17% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       14.005Ki ± 0%    8.985Ki ± 0%  -35.84% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        31.33Ki ± 0%    31.18Ki ± 0%   -0.47% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16      14.008Ki ± 0%    9.864Ki ± 0%  -29.58% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       74.32Ki ± 0%    81.16Ki ± 0%   +9.21% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16      16.528Ki ± 0%    9.874Ki ± 0%  -40.26% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       250.8Ki ± 0%    266.2Ki ± 0%   +6.13% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16      16.34Ki ± 0%    11.14Ki ± 0%  -31.82% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16      965.2Ki ± 0%   1038.0Ki ± 0%   +7.55% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      17.27Ki ± 0%    10.97Ki ± 0%  -36.50% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      3.711Mi ± 0%    3.981Mi ± 0%   +7.27% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     16.04Ki ± 0%    11.41Ki ± 0%  -28.86% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     14.88Mi ± 0%    15.53Mi ± 0%   +4.34% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    23.04Ki ± 1%    17.77Ki ± 3%  -22.87% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16    57.72Mi ± 0%    63.79Mi ± 0%  +10.51% (p=0.002 n=6)
geomean                                             61.36Ki         46.61Ki       -24.04%

                                                 │   old.txt   │               new.txt               │
                                                 │  allocs/op  │  allocs/op    vs base               │
SliceDataIntoObjects/slice_1-128/reader-16          254.0 ± 0%     149.0 ± 0%  -41.34% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16          276.0 ± 0%     148.0 ± 0%  -46.38% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16          223.0 ± 0%     161.0 ± 0%  -27.80% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16          244.0 ± 0%     160.0 ± 0%  -34.43% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16         218.0 ± 0%     142.0 ± 0%  -34.86% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16         240.0 ± 0%     143.0 ± 0%  -40.42% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         204.0 ± 0%     161.0 ± 0%  -21.08% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         226.0 ± 0%     160.0 ± 0%  -29.20% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16        211.0 ± 0%     142.0 ± 0%  -32.70% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        472.0 ± 0%     478.0 ± 0%   +1.27% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16       211.0 ± 0%     155.0 ± 0%  -26.54% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16      1.140k ± 0%    1.283k ± 0%  +12.54% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16       247.0 ± 0%     155.0 ± 0%  -37.25% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16      3.884k ± 0%    4.237k ± 0%   +9.10% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16      243.0 ± 0%     172.0 ± 0%  -29.22% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16     14.91k ± 0%    16.51k ± 0%  +10.70% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      259.0 ± 0%     169.0 ± 0%  -34.75% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16     58.73k ± 0%    64.87k ± 0%  +10.47% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     240.0 ± 0%     178.0 ± 1%  -25.83% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16    233.8k ± 0%    252.3k ± 0%   +7.92% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    340.5 ± 1%     279.5 ± 3%  -17.91% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16   909.8k ± 0%   1032.7k ± 0%  +13.51% (p=0.002 n=6)
geomean                                             930.0          732.6       -21.23%

Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Updated after benchmark. During object slicing we recreate PrmObjectPutInit for each part, we may reuse it.

                                                 │   old.txt   │              new.txt               │
                                                 │   sec/op    │   sec/op     vs base               │
SliceDataIntoObjects/slice_1-128/reader-16         62.61µ ± 1%   62.90µ ± 0%        ~ (p=0.132 n=6)
SliceDataIntoObjects/slice_1-128/writer-16         64.94µ ± 1%   65.30µ ± 1%        ~ (p=0.394 n=6)
SliceDataIntoObjects/slice_4-128/reader-16         65.48µ ± 1%   61.19µ ± 1%   -6.56% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16         67.38µ ± 2%   63.62µ ± 1%   -5.59% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16        61.57µ ± 2%   63.15µ ± 1%   +2.57% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16        63.93µ ± 1%   64.99µ ± 1%   +1.66% (p=0.004 n=6)
SliceDataIntoObjects/slice_64-128/reader-16        64.89µ ± 1%   66.61µ ± 1%   +2.66% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16        67.24µ ± 1%   71.96µ ± 2%   +7.02% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       61.19µ ± 1%   65.92µ ± 1%   +7.73% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16       236.2µ ± 2%   252.7µ ± 1%   +6.97% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16      64.34µ ± 1%   64.30µ ± 2%        ~ (p=0.937 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16      569.6µ ± 2%   611.3µ ± 2%   +7.32% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16      63.45µ ± 1%   67.79µ ± 1%   +6.85% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16      1.865m ± 1%   2.073m ± 1%  +11.16% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16     67.15µ ± 0%   64.50µ ± 1%   -3.95% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16     7.889m ± 1%   7.067m ± 1%  -10.42% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16     67.12µ ± 1%   65.98µ ± 1%   -1.70% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16     31.07m ± 0%   31.06m ± 1%        ~ (p=1.000 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16    72.49µ ± 1%   73.88µ ± 1%   +1.91% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16    112.7m ± 0%   112.6m ± 0%        ~ (p=0.699 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16   125.1µ ± 3%   125.1µ ± 3%        ~ (p=0.818 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16   518.8m ± 2%   510.0m ± 1%   -1.71% (p=0.026 n=6)
geomean                                            318.0µ        321.5µ        +1.09%

                                                 │    old.txt    │               new.txt                │
                                                 │     B/op      │     B/op       vs base               │
SliceDataIntoObjects/slice_1-128/reader-16          9.453Ki ± 0%    9.547Ki ± 0%   +0.99% (p=0.002 n=6)
SliceDataIntoObjects/slice_1-128/writer-16          9.344Ki ± 0%    9.438Ki ± 0%   +1.00% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16         10.329Ki ± 0%    9.077Ki ± 0%  -12.12% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16         10.220Ki ± 0%    8.968Ki ± 0%  -12.25% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16         8.984Ki ± 0%    9.547Ki ± 0%   +6.26% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16         8.968Ki ± 0%    9.438Ki ± 0%   +5.24% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         10.33Ki ± 0%    11.17Ki ± 0%   +8.17% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         10.22Ki ± 0%    11.06Ki ± 0%   +8.26% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16        8.985Ki ± 0%   10.893Ki ± 0%  +21.23% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        31.18Ki ± 0%    38.08Ki ± 0%  +22.11% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16       9.864Ki ± 0%    9.425Ki ± 0%   -4.46% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       81.16Ki ± 0%    75.66Ki ± 0%   -6.78% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16       9.874Ki ± 0%   11.188Ki ± 0%  +13.30% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       266.2Ki ± 0%    260.8Ki ± 0%   -2.01% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16      11.14Ki ± 0%    10.00Ki ± 0%  -10.18% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16     1038.0Ki ± 0%    936.3Ki ± 0%   -9.81% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      10.97Ki ± 0%    10.10Ki ± 0%   -7.89% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      3.981Mi ± 0%    3.708Mi ± 0%   -6.86% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     11.41Ki ± 0%    11.92Ki ± 0%   +4.41% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     15.53Mi ± 0%    14.45Mi ± 0%   -6.94% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    17.77Ki ± 3%    18.17Ki ± 3%   +2.24% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16    63.79Mi ± 0%    59.48Mi ± 0%   -6.76% (p=0.002 n=6)
geomean                                             46.61Ki         46.55Ki        -0.13%

                                                 │   old.txt    │               new.txt                │
                                                 │  allocs/op   │  allocs/op   vs base                 │
SliceDataIntoObjects/slice_1-128/reader-16           149.0 ± 0%    149.0 ± 0%        ~ (p=1.000 n=6) ¹
SliceDataIntoObjects/slice_1-128/writer-16           148.0 ± 0%    148.0 ± 0%        ~ (p=1.000 n=6) ¹
SliceDataIntoObjects/slice_4-128/reader-16           161.0 ± 0%    142.0 ± 0%  -11.80% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16           160.0 ± 0%    141.0 ± 0%  -11.88% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16          142.0 ± 0%    149.0 ± 0%   +4.93% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16          143.0 ± 0%    148.0 ± 0%   +3.50% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16          161.0 ± 0%    172.0 ± 0%   +6.83% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16          160.0 ± 0%    171.0 ± 0%   +6.88% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16         142.0 ± 0%    167.0 ± 0%  +17.61% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16         478.0 ± 0%    583.0 ± 0%  +21.97% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16        155.0 ± 0%    147.0 ± 0%   -5.16% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       1.283k ± 0%   1.155k ± 0%   -9.98% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16        155.0 ± 0%    172.0 ± 0%  +10.97% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       4.237k ± 0%   4.021k ± 0%   -5.11% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16       172.0 ± 0%    155.0 ± 0%   -9.88% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16      16.51k ± 0%   14.51k ± 0%  -12.07% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16       169.0 ± 0%    157.0 ± 0%   -7.10% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      64.87k ± 0%   58.66k ± 0%   -9.57% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16      178.0 ± 1%    182.0 ± 0%   +2.25% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     252.3k ± 0%   227.8k ± 0%   -9.73% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16     279.5 ± 3%    279.0 ± 3%        ~ (p=0.790 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16   1032.7k ± 0%   934.5k ± 0%   -9.51% (p=0.002 n=6)
geomean                                              732.6         720.3        -1.67%

Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
Updated after benchmark. During object slicing we create object for each part, we may reuse it, just resetting some fields.

                                                 │   old.txt   │               new.txt               │
                                                 │   sec/op    │    sec/op     vs base               │
SliceDataIntoObjects/slice_1-128/reader-16         62.90µ ± 0%   62.77µ ±  0%        ~ (p=0.589 n=6)
SliceDataIntoObjects/slice_1-128/writer-16         65.30µ ± 1%   65.82µ ±  0%   +0.80% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16         61.19µ ± 1%   67.40µ ±  1%  +10.16% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16         63.62µ ± 1%   70.42µ ±  2%  +10.69% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16        63.15µ ± 1%   65.74µ ±  1%   +4.10% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16        64.99µ ± 1%   68.16µ ±  1%   +4.87% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16        66.61µ ± 1%   66.09µ ±  1%        ~ (p=0.093 n=6)
SliceDataIntoObjects/slice_64-128/writer-16        71.96µ ± 2%   77.10µ ±  7%   +7.14% (p=0.041 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       65.92µ ± 1%   70.01µ ± 10%        ~ (p=0.394 n=6)
SliceDataIntoObjects/slice_256-128/writer-16       252.7µ ± 1%   234.0µ ±  1%   -7.42% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16      64.30µ ± 2%   65.18µ ±  6%        ~ (p=0.240 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16      611.3µ ± 2%   570.0µ ±  5%   -6.76% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16      67.79µ ± 1%   69.01µ ±  3%   +1.80% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16      2.073m ± 1%   2.024m ±  3%   -2.39% (p=0.026 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16     64.50µ ± 1%   64.07µ ±  0%   -0.66% (p=0.026 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16     7.067m ± 1%   6.792m ±  8%        ~ (p=0.065 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16     65.98µ ± 1%   69.42µ ±  1%   +5.22% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16     31.06m ± 1%   30.11m ±  1%   -3.07% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16    73.88µ ± 1%   76.69µ ±  3%   +3.81% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16    112.6m ± 0%   123.6m ±  4%   +9.76% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16   125.1µ ± 3%   106.8µ ± 12%  -14.61% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16   510.0m ± 1%   444.4m ±  1%  -12.85% (p=0.002 n=6)
geomean                                            321.5µ        322.7µ         +0.37%

                                                 │    old.txt    │               new.txt                │
                                                 │     B/op      │     B/op       vs base               │
SliceDataIntoObjects/slice_1-128/reader-16          9.547Ki ± 0%    9.547Ki ± 0%        ~ (p=1.000 n=6)
SliceDataIntoObjects/slice_1-128/writer-16          9.438Ki ± 0%    9.421Ki ± 0%   -0.18% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/reader-16          9.077Ki ± 0%   11.298Ki ± 0%  +24.46% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16          8.968Ki ± 0%   11.173Ki ± 0%  +24.59% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16         9.547Ki ± 0%   10.548Ki ± 0%  +10.48% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16         9.438Ki ± 0%   10.423Ki ± 0%  +10.44% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         11.17Ki ± 0%    10.55Ki ± 0%   -5.59% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         11.06Ki ± 0%    10.42Ki ± 0%   -5.79% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16       10.893Ki ± 0%    9.204Ki ± 0%  -15.50% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        38.08Ki ± 0%    29.42Ki ± 0%  -22.73% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16       9.425Ki ± 0%    9.675Ki ± 0%   +2.65% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16       75.66Ki ± 0%    69.60Ki ± 0%   -8.01% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16       11.19Ki ± 0%    11.31Ki ± 0%   +1.11% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16       260.8Ki ± 0%    235.7Ki ± 0%   -9.65% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16     10.004Ki ± 0%    9.716Ki ± 0%   -2.88% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16      936.3Ki ± 0%    831.3Ki ± 0%  -11.21% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      10.10Ki ± 0%    11.50Ki ± 0%  +13.79% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16      3.708Mi ± 0%    3.307Mi ± 0%  -10.80% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     11.92Ki ± 0%    12.32Ki ± 0%   +3.36% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16     14.45Mi ± 0%    13.26Mi ± 0%   -8.21% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    18.17Ki ± 3%    15.16Ki ± 6%  -16.53% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16    59.48Mi ± 0%    51.22Mi ± 0%  -13.89% (p=0.002 n=6)
geomean                                             46.55Ki         45.35Ki        -2.56%

                                                 │   old.txt   │               new.txt                │
                                                 │  allocs/op  │  allocs/op   vs base                 │
SliceDataIntoObjects/slice_1-128/reader-16          149.0 ± 0%    149.0 ± 0%        ~ (p=1.000 n=6) ¹
SliceDataIntoObjects/slice_1-128/writer-16          148.0 ± 0%    148.0 ± 0%        ~ (p=1.000 n=6) ¹
SliceDataIntoObjects/slice_4-128/reader-16          142.0 ± 0%    174.0 ± 0%  +22.54% (p=0.002 n=6)
SliceDataIntoObjects/slice_4-128/writer-16          141.0 ± 0%    173.0 ± 0%  +22.70% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/reader-16         149.0 ± 0%    163.0 ± 0%   +9.40% (p=0.002 n=6)
SliceDataIntoObjects/slice_16-128/writer-16         148.0 ± 0%    162.0 ± 0%   +9.46% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/reader-16         172.0 ± 0%    163.0 ± 0%   -5.23% (p=0.002 n=6)
SliceDataIntoObjects/slice_64-128/writer-16         171.0 ± 0%    162.0 ± 0%   -5.26% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/reader-16        167.0 ± 0%    144.0 ± 0%  -13.77% (p=0.002 n=6)
SliceDataIntoObjects/slice_256-128/writer-16        583.0 ± 0%    437.0 ± 0%  -25.04% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/reader-16       147.0 ± 0%    151.0 ± 0%   +2.72% (p=0.002 n=6)
SliceDataIntoObjects/slice_1024-128/writer-16      1.155k ± 0%   1.030k ± 0%  -10.82% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/reader-16       172.0 ± 0%    174.0 ± 0%   +1.16% (p=0.002 n=6)
SliceDataIntoObjects/slice_4096-128/writer-16      4.021k ± 0%   3.464k ± 0%  -13.85% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/reader-16      155.0 ± 0%    151.0 ± 0%   -2.58% (p=0.002 n=6)
SliceDataIntoObjects/slice_16384-128/writer-16     14.51k ± 0%   12.20k ± 0%  -15.96% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/reader-16      157.0 ± 0%    176.0 ± 0%  +12.10% (p=0.002 n=6)
SliceDataIntoObjects/slice_65536-128/writer-16     58.66k ± 0%   49.56k ± 0%  -15.52% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/reader-16     182.0 ± 0%    188.0 ± 1%   +3.30% (p=0.002 n=6)
SliceDataIntoObjects/slice_262144-128/writer-16    227.8k ± 0%   197.1k ± 0%  -13.48% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/reader-16    279.0 ± 3%    230.5 ± 6%  -17.38% (p=0.002 n=6)
SliceDataIntoObjects/slice_1048576-128/writer-16   934.5k ± 0%   762.4k ± 0%  -18.42% (p=0.002 n=6)
geomean                                             720.3         690.1        -4.19%

Signed-off-by: Evgenii Baidakov <evgenii@nspcc.io>
@smallhive smallhive force-pushed the 442-slicer-interface-is-not-generic branch from a13249c to d6858d9 Compare July 28, 2023 09:39
@roman-khimov roman-khimov merged commit af656fb into master Jul 29, 2023
5 checks passed
@roman-khimov roman-khimov deleted the 442-slicer-interface-is-not-generic branch July 29, 2023 06:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Slicer interface is not generic enough for objects
4 participants