Add ZeroMQ notifications #2050

str4d · 2017-01-31T13:44:17Z

Cherry-picked from the following upstream PRs:

Closes #2020.

str4d · 2017-01-31T14:36:42Z

Before merging this, Buildbot needs to be updated to pip install zmq before running the RPC tests, as that is now a dependency.

daira · 2017-02-03T23:59:55Z

contrib/debian/copyright needs an entry for depends/sources/zeromq-*.tar.gz.

daira · 2017-02-04T00:12:04Z

Files: depends/sources/zeromq-*.tar.gz
Copyright:
 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004, 2005, 2006, 2007 Free Software Foundation, Inc.
 2007-2014 iMatix Corporation
 2009-2011 250bpm s.r.o.
 2010-2011 Miru Limited
 2011 VMware, Inc.
 2012 Spotify AB
 2013 Ericsson AB
 2014 AppDynamics Inc.
 2015-2016 Brocade Communications Systems Inc.
License: LGPL-with-ZeroMQ-exception

License: LGPL-with-ZeroMQ-exception
 GNU LESSER GENERAL PUBLIC LICENSE
 Version 3, 29 June 2007
 .
 On Debian systems the GNU Lesser General Public License (LGPL) is
 located in '/usr/share/common-licenses/LGPL'.
 .
 This program is distributed in the hope that it will be useful,
 but WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 GNU General Public License for more details.
 --------------------------------------------------------------------------------
 SPECIAL EXCEPTION GRANTED BY COPYRIGHT HOLDERS
 .
 As a special exception, copyright holders give you permission to link this
 library with independent modules to produce an executable, regardless of
 the license terms of these independent modules, and to copy and distribute
 the resulting executable under terms of your choice, provided that you also
 meet, for each linked independent module, the terms and conditions of
 the license of that module. An independent module is a module which is not
 derived from or based on this library. If you modify this library, you must
 extend this exception to your version of the library.
  
 Note: this exception relieves you of any obligations under sections 4 and 5
 of this license, and section 6 of the GNU General Public License.
Comment:
 You should have received a copy of the GNU General Public License
 along with this program.  If not, see <http://www.gnu.org/licenses/>.

daira · 2017-02-04T00:17:32Z

... and add ~~libzmq3-dev~~ libzmq5-dev (see here) to Build-Depends in contrib/debian/control. [Edit: fixed link.]

Edit: I'm not sure this is necessary. If it is not, then remove the reference to libzmq*-dev from doc/zmq.md.

daira · 2017-02-04T00:51:16Z

The package.mk for libzmq builds it with the --without-libsodium --disable-curve options, which disables security and authentication features (I'm not sure of the exact details of what is disabled). Presumably this was intended by Bitcoin upstream because it's not the default, but is it what we want?

daira · 2017-02-04T01:09:44Z

libzmq has a new version 4.1.6. Its release notes since 4.1.5 are:

0MQ version 4.1.6 stable, released on 2016/11/01

Fixed avoid asserting on getifaddrs failure zeromq/libzmq#2051 - getifaddrs can fail with ECONNREFUSED

Fixed testutils won't compile when targeting windows xp zeromq/libzmq#2091 - testutil.hpp fails to build on Windows XP

Fixed Problem: CMake build from dist tarball broken zeromq/libzmq#2096 - add tests/CMakeLists.in and version.rc.in to dist tar

Fixed Problem: zmq_connect with IPv6 "source:port;dest:port" format is broken zeromq/libzmq#2107 - zmq_connect with IPv6 "source:port;dest:port" broken

Fixed Problem: zmq_ctx_term asserts with connect-before-bind and sockets with identity over inproc transport zeromq/libzmq#2117 - ctx_term assert with inproc zmq_router connect-before-bind

Fixed Socket monitor on TCP sockets uses internal Pair socket from multiple threads zeromq/libzmq#2158 - Socket monitor uses internal Pair from multiple threads

Fixed Problem: messages dropped due to HWM race zeromq/libzmq#2161 - messages dropped due to HWM race

Fixed Alignment issue with zmq_msg_t on SPARC CPU zeromq/libzmq#1325 - alignment issue with zmq_msg_t causes SIGBUS on SPARC and ARM

Some of these look like significant correctness and Denial of Service issues.

daira · 2017-02-04T01:14:32Z

zeromq/libzmq#2158 looks like undefined behaviour.

daira · 2017-02-04T01:24:18Z

Oh, the current libzmq version is actually 4.2.1 (release notes; see also the release notes for 4.2.0). In any case we should upgrade. I'm not sure whether there were any incompatible API changes between 4.1.x and 4.2.x. Edit: the release notes claim compatibility.

str4d · 2017-02-06T13:15:12Z

@daira wrote:

... and add libzmq3-dev libzmq5-dev (see here) to Build-Depends in contrib/debian/control.

The link doesn't make sense to me - could you elaborate?

Edit: I'm not sure this is necessary. If it is not, then remove the reference to libzmq*-dev from doc/zmq.md.

The impression I got from that section was that it was listed so if a downstream packager wants to use provided dependencies instead of the depends/ system, they know which package to use. So I'm +0 on removing the reference.

str4d · 2017-02-06T14:04:57Z

@daira wrote:

The package.mk for libzmq builds it with the --without-libsodium --disable-curve options, which disables security and authentication features (I'm not sure of the exact details of what is disabled). Presumably this was intended by Bitcoin upstream because it's not the default, but is it what we want?

--without-libsodium was added while upgrading from 4.0.7 to 4.1.4 (bitcoin/bitcoin#7993) to fix a build error.

--disable-curve was added while upgrading to 4.1.5 (bitcoin/bitcoin#8238). Upstream did not need it enabled because they don't have libsodium as a dependency, and thus before 4.2 could not use CURVE security. The default security is null (no security), and upstream didn't enable plain (password-based).

From 4.2.0 --without-libsodium is the default, as tweetnacl is used instead.

Now, we could enable CURVE security, but I expect it's not really necessary because the ZMQ interface is generally used from an internal network.

daira · 2017-02-06T15:47:33Z

ACK 4a8159343b0e1db2ff16dd138ab84271d2146774 and b418aa9027d3003a55089bdbcf2e6f55b46950f0.

daira · 2017-02-06T15:59:49Z

@str4d wrote:

@daira wrote:

... and add ~~libzmq3-dev~~ libzmq5-dev (see here) to Build-Depends in contrib/debian/control.

The link doesn't make sense to me - could you elaborate?

Sorry, cut-and-paste error. The intended link was http://zeromq.org/distro:debian .

I think we should update doc/zmq.md to say libzmq5-dev and to make it clearer that this is only needed when not using the depends system.

daira · 2017-02-06T16:02:04Z

@str4d wrote:

Now, we could enable CURVE security, but I expect it's not really necessary because the ZMQ interface is generally used from an internal network.

Hmm. Maybe we should just document (both in doc/zmq.md and doc/security-warnings.md) that ZMQ provides no authentication by default.

bitcartel · 2017-02-06T20:17:57Z

Upstream still on 4.1.5. Maybe they'll upgrade to 4.2.1. bitcoin/bitcoin#8639

zkbot · 2017-02-08T19:41:02Z

☔ The latest upstream changes (presumably #2072) made this pull request unmergeable. Please resolve the merge conflicts.

Continues Johnathan Corgan's work. Publishing multipart messages Bugfix: Add missing zmq header includes Bugfix: Adjust build system to link ZeroMQ code for Qt binaries

prefix instead of postfix

str4d · 2017-02-08T22:22:01Z

Rebased on master to fix merge conflict, and addressed @daira's comments.

Hmm. Maybe we should just document (both in doc/zmq.md and doc/security-warnings.md) that ZMQ provides no authentication by default.

This is already documented in doc/zmq.md by upstream (see here and here).

bitcartel · 2017-02-09T00:30:41Z

@zkbot try

zkbot · 2017-02-09T00:30:46Z

⌛ Trying commit edcec14 with merge 36df5a9...

zkbot · 2017-02-09T01:19:18Z

💔 Test failed - zcash

bitcartel · 2017-02-09T01:28:57Z

@str4d See above:
=== Running testscript --mineblock ===
./qa/pull-tester/rpc-tests.sh: line 77: /home/admin/bbs/zcash/build/qa/rpc-tests/--mineblock: No such file or directory
!!! FAIL: --mineblock !!!

`${testScripts[@]}` now does split up `testscript --agument` in two elements pushed to the array (`testscript` and `--agument`).

str4d · 2017-02-09T11:55:33Z

@bitcartel there was an upstream PR I missed that fixed it.

@zkbot try

zkbot · 2017-02-09T11:55:38Z

⌛ Trying commit 9bbc220 with merge dd8b383...

Add ZeroMQ notifications Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6103 - bitcoin/bitcoin#6684 - bitcoin/bitcoin#6686 - bitcoin/bitcoin#6736 - bitcoin/bitcoin#6739 - bitcoin/bitcoin#6743 - bitcoin/bitcoin#6768 - bitcoin/bitcoin#6779 - bitcoin/bitcoin#6810 - bitcoin/bitcoin#6927 - bitcoin/bitcoin#6980 (only upgrading zeromq) - bitcoin/bitcoin#6680 - bitcoin/bitcoin#7058 - bitcoin/bitcoin#7621 - bitcoin/bitcoin#7335 (only parts affecting `zmq_test.py`) - bitcoin/bitcoin#7853 (only parts affecting `zmq_test.py`) - bitcoin/bitcoin#7762 - bitcoin/bitcoin#7993 (only upgrading zeromq) - bitcoin/bitcoin#8238 - bitcoin/bitcoin#8701 - bitcoin/bitcoin#6685 Closes #2020.

zkbot · 2017-02-09T12:43:46Z

☀️ Test successful - zcash

daira

utACK

bitcartel · 2017-02-09T20:05:36Z

ACK

bitcartel · 2017-02-09T20:05:42Z

@zkbot r+

zkbot · 2017-02-09T20:05:43Z

📌 Commit 9bbc220 has been approved by bitcartel

zkbot · 2017-02-09T20:05:48Z

⌛ Testing commit 9bbc220 with merge 253c610...

Add ZeroMQ notifications Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6103 - bitcoin/bitcoin#6684 - bitcoin/bitcoin#6686 - bitcoin/bitcoin#6736 - bitcoin/bitcoin#6739 - bitcoin/bitcoin#6743 - bitcoin/bitcoin#6768 - bitcoin/bitcoin#6779 - bitcoin/bitcoin#6810 - bitcoin/bitcoin#6927 - bitcoin/bitcoin#6980 (only upgrading zeromq) - bitcoin/bitcoin#6680 - bitcoin/bitcoin#7058 - bitcoin/bitcoin#7621 - bitcoin/bitcoin#7335 (only parts affecting `zmq_test.py`) - bitcoin/bitcoin#7853 (only parts affecting `zmq_test.py`) - bitcoin/bitcoin#7762 - bitcoin/bitcoin#7993 (only upgrading zeromq) - bitcoin/bitcoin#8238 - bitcoin/bitcoin#8701 - bitcoin/bitcoin#6685 Closes #2020.

zkbot · 2017-02-09T20:55:27Z

☀️ Test successful - zcash

str4d added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. A-rpc-interface Area: RPC interface C-upstream-port Category: Changes that are ported from the Bitcoin Core codebase. labels Jan 31, 2017

str4d added this to the 1.0.6 milestone Jan 31, 2017

daira added I-SECURITY Problems and improvements related to security. licensing labels Feb 4, 2017

bitcartel added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 7, 2017

João Barbosa and others added 7 commits February 8, 2017 22:10

Add UpdatedBlockTip signal to CMainSignals and CValidationInterface

7e6ec07

Depends: Add ZeroMQ package

cce7754

Add ZeroMQ support. Notify blocks and transactions via ZeroMQ

f200002

Continues Johnathan Corgan's work. Publishing multipart messages Bugfix: Add missing zmq header includes Bugfix: Adjust build system to link ZeroMQ code for Qt binaries

QA: Add ZeroMQ RPC test

da0e7aa

depends: fix platform specific packages variable

0388c23

prefix instead of postfix

Add ZMQ libs to zcash-gtest

8b99caa

[travis] add zmq python module

5c7f84c

isle2983 and others added 4 commits February 8, 2017 22:11

[copyright] add MIT License copyright header to zmq_sub.py

5dfea48

Bitcoin -> Zcash in ZMQ docs

4fbc46c

Add ZeroMQ license to contrib/debian/copyright

05e6ead

[depends] ZeroMQ 4.2.1

602407d

str4d force-pushed the 2020-zmq branch from b418aa9 to 602407d Compare February 8, 2017 22:12

str4d added 2 commits February 8, 2017 22:18

Clarify that user only needs libzmq if not using depends system

7c8845e

Bump suggested ZMQ Debian package to 4.1 series

edcec14

fix rpc-tests.sh

9bbc220

`${testScripts[@]}` now does split up `testscript --agument` in two elements pushed to the array (`testscript` and `--agument`).

daira approved these changes Feb 9, 2017

View reviewed changes

zkbot merged commit 9bbc220 into zcash:master Feb 9, 2017

This was referenced Feb 9, 2017

Upgrade googletest to 1.8.0 #1437

Closed

Remove QT code #1636

Merged

str4d mentioned this pull request Feb 9, 2017

Bitcoin Core 0.12.0 #2074

Open

str4d deleted the 2020-zmq branch January 27, 2023 16:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ZeroMQ notifications #2050

Add ZeroMQ notifications #2050

str4d commented Jan 31, 2017 •

edited

Loading

str4d commented Jan 31, 2017

daira commented Feb 3, 2017

daira commented Feb 4, 2017 •

edited

Loading

daira commented Feb 4, 2017 •

edited

Loading

daira commented Feb 4, 2017

daira commented Feb 4, 2017 •

edited

Loading

0MQ version 4.1.6 stable, released on 2016/11/01

daira commented Feb 4, 2017

daira commented Feb 4, 2017 •

edited

Loading

str4d commented Feb 6, 2017

str4d commented Feb 6, 2017

daira commented Feb 6, 2017

daira commented Feb 6, 2017

daira commented Feb 6, 2017

bitcartel commented Feb 6, 2017

zkbot commented Feb 8, 2017

str4d commented Feb 8, 2017

bitcartel commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

bitcartel commented Feb 9, 2017 •

edited

Loading

str4d commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

daira left a comment

bitcartel commented Feb 9, 2017

bitcartel commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

Add ZeroMQ notifications #2050

Add ZeroMQ notifications #2050

Conversation

str4d commented Jan 31, 2017 • edited Loading

str4d commented Jan 31, 2017

daira commented Feb 3, 2017

daira commented Feb 4, 2017 • edited Loading

daira commented Feb 4, 2017 • edited Loading

daira commented Feb 4, 2017

daira commented Feb 4, 2017 • edited Loading

0MQ version 4.1.6 stable, released on 2016/11/01

daira commented Feb 4, 2017

daira commented Feb 4, 2017 • edited Loading

str4d commented Feb 6, 2017

str4d commented Feb 6, 2017

daira commented Feb 6, 2017

daira commented Feb 6, 2017

daira commented Feb 6, 2017

bitcartel commented Feb 6, 2017

zkbot commented Feb 8, 2017

str4d commented Feb 8, 2017

bitcartel commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

bitcartel commented Feb 9, 2017 • edited Loading

str4d commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

daira left a comment

Choose a reason for hiding this comment

bitcartel commented Feb 9, 2017

bitcartel commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

zkbot commented Feb 9, 2017

str4d commented Jan 31, 2017 •

edited

Loading

daira commented Feb 4, 2017 •

edited

Loading

daira commented Feb 4, 2017 •

edited

Loading

daira commented Feb 4, 2017 •

edited

Loading

daira commented Feb 4, 2017 •

edited

Loading

bitcartel commented Feb 9, 2017 •

edited

Loading