[FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT #10
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@itudoben why was it bumped to a non snapshot version ? |
itudoben
force-pushed
the
issue-3917-netty-memory-counter-removal
branch
from
f4b46cb
to
3629ad0
Compare
itudoben
requested review from
rendel,
a team and
wirelaisse
and removed request for
a team
|
@rendel I set the version to release. Could you approve, plz. |
|
@itudoben please let's follow the same release process than for other projects, master should always be a snapshot |
itudoben
changed the title
itudoben
changed the title
rendel
approved these changes
Nov 23, 2020
GeorgeAp
pushed a commit
that referenced
this pull request
May 28, 2021
…n-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io>
GeorgeAp
pushed a commit
that referenced
this pull request
Jun 7, 2021
…n-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io>
GeorgeAp
pushed a commit
that referenced
this pull request
Jun 7, 2021
From a deadlocked run... ``` #0 0x00007f8a5d48dccd in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f8a5d486f05 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00007f8a566e7e89 in arrow::internal::FnOnce<void ()>::FnImpl<arrow::Future<Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error> >::Callback<arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler> >::invoke() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #3 0x00007f8a5650efa0 in arrow::FutureImpl::AddCallback(arrow::internal::FnOnce<void ()>) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #4 0x00007f8a566e67a9 in arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler::SpawnListObjectsV2() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #5 0x00007f8a566e723f in arrow::fs::(anonymous namespace)::TreeWalker::WalkChild(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #6 0x00007f8a566e827d in arrow::internal::FnOnce<void ()>::FnImpl<arrow::Future<Aws::Utils::Outcome<Aws::S3::Model::ListObjectsV2Result, Aws::S3::S3Error> >::Callback<arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler> >::invoke() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #7 0x00007f8a5650efa0 in arrow::FutureImpl::AddCallback(arrow::internal::FnOnce<void ()>) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #8 0x00007f8a566e67a9 in arrow::fs::(anonymous namespace)::TreeWalker::ListObjectsV2Handler::SpawnListObjectsV2() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #9 0x00007f8a566e723f in arrow::fs::(anonymous namespace)::TreeWalker::WalkChild(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so #10 0x00007f8a566e74b1 in arrow::fs::(anonymous namespace)::TreeWalker::DoWalk() () from /arrow/r/check/arrow.Rcheck/arrow/libs/arrow.so ``` The callback `ListObjectsV2Handler` is being called recursively and the mutex is non-reentrant thus deadlock. To fix it I got rid of the mutex on `TreeWalker` by using `arrow::util::internal::TaskGroup` instead of manually tracking the #/status of in-flight requests. Closes apache#9842 from westonpace/bugfix/arrow-12040 Lead-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>
GeorgeAp
pushed a commit
that referenced
this pull request
Jun 7, 2021
…n-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io>
GeorgeAp
added a commit
that referenced
this pull request
Nov 26, 2021
* Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
added a commit
that referenced
this pull request
Jan 21, 2022
* Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
added a commit
that referenced
this pull request
Jan 21, 2022
* Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
added a commit
that referenced
this pull request
Mar 28, 2022
* Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
added a commit
that referenced
this pull request
May 11, 2022
* [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
added a commit
that referenced
this pull request
Jul 15, 2022
* [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
pushed a commit
that referenced
this pull request
Aug 31, 2022
…ation (apache#13579) When locally building the `conda-cpp-valgrind` docker image I've realised we are prompting for confirmation: ``` #10 12.35 Confirm changes: [Y/n] #10 88.01 ``` We should use the -y flag when using mamba install: ``` -y, --yes Do not ask for confirmation. ``` With this change we are no longer seeing the `Confirm changes` message from mamba. Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
GeorgeAp
added a commit
that referenced
this pull request
Aug 31, 2022
* [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com>
GeorgeAp
added a commit
that referenced
this pull request
Oct 12, 2022
* [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Release siren-7.0.0-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix rebase * Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac <issac.garcia@siren.io> * Add developer tip link to readme * Remove test scope for arrow-memory-netty * Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes * Update as per review * Update comment as per review * Use stable netty version siren-4.1.78-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> Co-authored-by: Issac <issac.garcia@siren.io>
GeorgeAp
added a commit
that referenced
this pull request
Jan 26, 2023
* [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Release siren-7.0.0-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix rebase * Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac <issac.garcia@siren.io> * Add developer tip link to readme * Remove test scope for arrow-memory-netty * Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes * Update as per review * Update comment as per review * Use stable netty version siren-4.1.78-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> Co-authored-by: Issac <issac.garcia@siren.io>
GeorgeAp
added a commit
that referenced
this pull request
May 11, 2023
* [FEDE-5623] Upgrade Siren Arrow from 7.0.0 to 9.0.0 (#35) * [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Release siren-7.0.0-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix rebase * Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac <issac.garcia@siren.io> * Add developer tip link to readme * Remove test scope for arrow-memory-netty * Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes * Update as per review * Update comment as per review * Use stable netty version siren-4.1.78-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> Co-authored-by: Issac <issac.garcia@siren.io> * Fix cherry-pick * Temporarily comment out the jdk version * chore: revert commented out jdk version in pom * update: use stable version of netty after release of siren-4.1.82-1 --------- Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> Co-authored-by: Issac <issac.garcia@siren.io>
prateeknima77
pushed a commit
that referenced
this pull request
Sep 17, 2024
* [FEDE-5623] Upgrade Siren Arrow from 7.0.0 to 9.0.0 (#35) * [FEDE-5150] Upgrade arrow version from 4.0.0 to 7.0.0 (#26) * [FEDE-4869] Upgrade Arrow version to 4.0.0 (#16) * Make BaseValueVector#MAX_ALLOCATION_SIZE configurable This closes apache#65 Some of the tests are based on the assumption that the JVM can allocate at least 2GB of memory, which is not a common occurence (JVM usually defaults at 512MB). Current Travis CI VM only have 3GB of memory total, which would have make challenging to run some of the tests on them Add a system property to change BaseValueVector.MAX_ALLOCATION_SIZE to allow to use a much smaller value during tests. * prefix arrow's version with siren * use our version of netty * updated readme about siren's changes * fixed dependency issue with our own artifactory * use our version of netty * shade the arrow memory jar * improved doc * fix readme * document the changes done to the arrow fork * ARROW-5856: [Python] [Packaging] Fix use of C++ / Cython API from wheels Author: Antoine Pitrou <antoine@python.org> Closes apache#4884 from pitrou/ARROW-5856-cython-so-version and squashes the following commits: a411d7a <Antoine Pitrou> Avoid C++ ABI issues with DictionaryMemo eaede5b <Antoine Pitrou> Revert ARROW-5082 (" Stop exporting copies of shared libraries in wheel") 4594f78 <Antoine Pitrou> ARROW-5856: Try to fix Cython API from wheels * use our version of netty * set drill's default value * use our version of netty * bumped to 0.8.0 * update to 0.14.1 * comment unneeded modules * update release procedure with unneeded modules commented out * bump version to siren-0.14.1-1 and update readme * do not allocate a new offset buffer if the slice starts at 0 since the relative offset pointer would be unchanged * fix slice bounds * improved readme * Upgraded netty dependendcy to siren-4.1.27-3 * [FEDE-3917] netty direct memory counter deprecation with bump to siren-0.14.1-6-SNAPSHOT (#10) * Release siren-0.14.1-5 * Bump to version siren-0.14.1-6-SNAPSHOT * Use Siren Netty fork version siren-4.1.27-4 to release siren-0.14.1-6 * Bump version to release siren-0.14.1-5 using Netty Siren siren-4.1.27-4 * Set the version to siren-0.14.1-6-SNAPSHOT Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> * Fix rebase * Fix pom * Fix rebase - Bump version to siren-4.0.0-1-SNAPSHOT * Clean-up * Comment out memory-netty module and remove use siren netty * Update siren netty version * Fix rebase * Comment netty * Fix TestValueVector * Remove unused imports * Remove unused import * Clean up * Clean up - fix checkstyle * Clean up - fix checkstyle 2 * WIP * WIP - add siren netty plugin * WIP - ignore failing test * Fix rebase as per peer review * Fix rebase python * Fix ZeroVector unused imports * Fix checkstyle * Fix rebase * Fix rebase * Fix checkstyle imports violation * Fix checkstyle and update netty version * Use Arrow's ArrowBuf instead of our own * Fix imports * Fix checkstyle * Tentative fix: workaround for the shading issue but creat new problems * fix missing import * Fix import * Fix imports * Revert ignored unit tests * Revert automatic format changes * Revert format to original format * Update vector pom as per peer review * Revert ignored TestArrowBufHasher.testHasherNegative * Clean up * Update as per peer review * Undo unnecessary change in tasks.yml * Undo unnecessary changes * Revert changes * Revert changes * Revert changes as per peer review * Update as per peer review * Add back class path dependency exclusion * Remove whitespace * Update memory access with default value * Remove class path exclusion * Update readme * Exclude memory-core from the shaded netty in memory-netty package * Add information for checking that Siren version of Netty is used * [FEDE-5144] Fix the static initialization of MemoryUtil (#17) * Remove unecessary setAccessible call * Catch all errors thrown by the setAccessible call New version of java can throw InaccessibleObjectException which Arrow didn't handle * Update netty version to the stable version siren-4.1.48-1 Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Bumped version number to siren-4.0.0-1 * Fix rebase * Remove scope from arrow-memory-netty to make netty available for entire project else our shading will fail. Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix readme - remove duplicate section on how to contribute * Update readme as per review * Use stable netty version siren-4.1.68-1 after the release of netty Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> * Release siren-7.0.0-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> * Fix rebase * Update services configuration (#32) * Update services configuration This is needed for the module declaration in federate * Create a uber jar with netty and arrow memory dependencies This avoids conflicts with java modules. Arrows provides different modules (memory-netty, memory-core) that export the same packages (org.apache.arrow.memory), which is forbidden with java modules. The federate code can then just dependend on the shaded jar from vector. * use reduced pom in jar Co-authored-by: Issac <issac.garcia@siren.io> * Add developer tip link to readme * Remove test scope for arrow-memory-netty * Change version from siren-9.0.0-1-SNAPSHOT to siren-9.0.0-2-SNAPSHOT since 2 contain the module changes * Update as per review * Update comment as per review * Use stable netty version siren-4.1.78-1 Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> Co-authored-by: Issac <issac.garcia@siren.io> * Fix cherry-pick * Temporarily comment out the jdk version * chore: revert commented out jdk version in pom * update: use stable version of netty after release of siren-4.1.82-1 --------- Signed-off-by: George Apaaboah <george.apaaboah@gmail.com> Co-authored-by: Laurent Goujon <laurent@dremio.com> Co-authored-by: Stéphane Campinas <stephane.campinas@gmail.com> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Martin Anseaume <martin.anseaume@siren.io> Co-authored-by: Johnny Hujol <itudoben@users.noreply.github.com> Co-authored-by: ggdupont <ger.dupont@gmail.com> Co-authored-by: Issac <issac.garcia@siren.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR provides the following
siren-changesbranch usesNetty to siren-4.1.27-4siren-0.14.1-6-SNAPSHOT