[Java] Unable to update VectorSchemaRoot field vectors. #46
Labels
Type: usage
usage question
Comments
hnasrullakhan
changed the title
|
Right, so if I simplify this, you want to know how to modify |
kou
changed the title
|
You could try something like the following, but since this is not using the recommended high level APIs of Vector, user has to take responsibility in reserving memory and keeping track of buffers if they are explicitly created. A naive example of shifting try (VarCharVector v = new VarCharVector("myvec", allocator)) {
v.allocateNewSafe();
v.set(0, new Text("Hello"));
v.set(1, new Text("World"));
v.setValueCount(2);
System.out.println("Vector");
System.out.println(v);
final ArrowBuf dataBuffer = v.getDataBuffer();
byte[] resAtZero = new byte[10];
dataBuffer.getBytes(0, resAtZero);
for (int i = 0; i < resAtZero.length; i++) {
System.out.print(resAtZero[i] + ", ");
resAtZero[i] += 1;
}
dataBuffer.setBytes(0, resAtZero);
System.out.println();
System.out.println("After");
System.out.println(v);
/*
*
* Vector
[Hello, World]
72, 101, 108, 108, 111, 87, 111, 114, 108, 100,
After
[Ifmmp, Xpsme]
*
*/
} |
|
Or... try (VarCharVector v = new VarCharVector("myvec", allocator)) {
v.allocateNewSafe();
v.set(0, new Text("Hello"));
v.set(1, new Text("World"));
v.setValueCount(2);
System.out.println("Vector");
System.out.println(v);
final ArrowBuf dataBuffer = v.getDataBuffer();
final ArrowBuf validityBuffer = v.getValidityBuffer();
final ArrowBuf offsetBuffer = v.getOffsetBuffer();
try (final ArrowBuf newDataBuffer = allocator.buffer(dataBuffer.capacity())) {
byte[] resAtZero = new byte[20];
dataBuffer.getBytes(0, resAtZero);
for (int i = 0; i < resAtZero.length; i++) {
System.out.print(resAtZero[i] + ", ");
resAtZero[i] += 1;
}
newDataBuffer.writeBytes(resAtZero, 0, resAtZero.length);
newDataBuffer.writerIndex(dataBuffer.writerIndex());
// dataBuffer.setBytes(0, resAtZero);
newDataBuffer.readerIndex(0);
ArrowFieldNode fieldNode = new ArrowFieldNode(v.getValueCount(), v.getNullCount());
v.loadFieldBuffers(fieldNode, List.of(validityBuffer, offsetBuffer, newDataBuffer));
System.out.println();
System.out.println("After");
System.out.println(v);
}
/*
*
* Vector
[Hello, World]
72, 101, 108, 108, 111, 87, 111, 114, 108, 100,
After
[Ifmmp, Xpsme]
*
*/
} |
|
@vibhatha Thanks I tried this and loadFieldBuffers doesnt load newdata // I see that new databuffer shows right compression value And VectorSchemaRoot from which this fieldvectors are extracted still use old fieldVector only |
if I try to serialize using ArrowStreamWriter, it uses older offsets.
i tried to use |
|
My use case:
|
does this make a copy of field vectors and not update original ? |
|
If you change the copy of the object, then you have to set it back. I am not sure if you can do the in place setting. Also when playing with buffers, make sure to clear existing and set the updated ones. Also when mutating buffers manually which we don't recommend, the developer has to take responsibility in managing memory. If new buffers were created, they need to be released (i.e use try-with-resource). |
|
Right, I understand your problem. Just to save time for both of us. May I get a minimal code snippet with the end-to-end workflow? Then I can debug and try to get a version working? Would that be okay? |
|
|
@vibhatha need your help |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the usage question you have. Please include as many useful details as possible.
Hi community,
I have a trying to modify data buffers of string column.
if i print the buffer readable bytes after
buffers.set(index, compressedBuf);, i see the right value.When i update my fieldvector.
and i check the buffers again then buffers reflect old value.
please advise .
Component(s)
Java
The text was updated successfully, but these errors were encountered: