Python3 snmpsimd.py and Redis variation

[belavina]

Hello Ilya! I am trying to migrate from py2 version of snmpsim to py3 due to python2 getting deprecated on Fedora. However, I am having some troubles running snmpsimd with python3 and redis variation module. I confirmed that this only happens with snmpsimd for python3.

Error

when querying with snmpwalk/snmpget :

Using /home/l/data/redis.snmprec controller selected by candidate b'86a1b907d54bf7010394bf316e183e67'; transport ID 1.3.6.1.6.1.1.0, source address 127.0.0.1, context name "b'86a1b907d54bf7010394bf316e183e67'" 
SNMP EngineID 0x80004fb8056c6f63616c686f73742e6c6f63616c6418cf1be0, transportDomain (1, 3, 6, 1, 6, 1, 1, 0), transportAddress ('127.0.0.1', 60710), securityModel 1, securityName 86a1b907d54bf7010394bf316e183e67, securityLevel 1 
Request var-binds: 1.3.6.1.2.1=<>, flags: EXACT, GET 
ERROR data error at /home/l/data/redis.snmprec controller for 1.3.6.1.2.1: can't concat str to bytes 
Response var-binds: 1.3.6.1.2.1=<No Such Instance currently exists at this OID> 

Full log: here

Steps to Reproduce

1. Installing snmpsim & redis

$ python3 -m virtualenv venv
$ source venv/bin/activate
$ pip install snmpsim
$ pip install redis
$ snmpsimd.py --version
SNMP Simulator version 0.4.7, written by Ilya Etingof <etingof@gmail.com>
Using foundation libraries: pysnmp 4.4.11, pyasn1 0.4.6.
Python interpreter: 3.7.4 (default, Jul  9 2019, 16:32:37) 
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]

2. Recording from snmplabs

$ snmprec.py --agent-udpv4-endpoint=demo.snmplabs.com  \
                    --start-oid=1.3.6   \
                    --output-file=data/redis.snmprec   \
                   --variation-module=redis   \
                   --variation-module-options=host:127.0.0.1,port:6379,db:0,key-spaces-id:1111

3. Running simulator

snmpsimd.py --agent-udpv4-endpoint=localhost:1025 \
                     --variation-module-options=redis:host:127.0.0.1,port:6379,db:0,key-spaces-id:1111\
                     --data-dir=./data

System Details

OS: Fedora 30
Python: Python 3.7.4
SNMPsim package: snmpsim-0.4.7
Redis Version: 5.0.5

[etingof]

Hey o/

You are right! Apparently, py-redis operates on bytes on Py3, while it used to be str on Py2. That causes all sorts of weirdness...

Let me see if I can get that back to track.

[etingof]

The latest commit to master hopefully fixes this problem.

[belavina]

Hi again, thanks for a prompt response.

I am not seeing str/bytes error, however snmpsimd.py is terminating with the following now (upon receiving snmpwalk):

Request var-binds: 1.3.6.1.2.1.4.31.1.1.3.2=<>, flags: NEXT, GET 
Response var-binds: 1.3.6.1.2.1.4.31.1.1.4.1=<22906399> 
SNMP EngineID 0x80004fb8056c6f63616c686f73742e6c6f63616c6410528500, transportDomain (1, 3, 6, 1, 6, 1, 1, 0), transportAddress ('127.0.0.1', 43272), securityModel 1, securityName 86a1b907d54bf7010394bf316e183e67, securityLevel 1 
Request var-binds: 1.3.6.1.2.1.4.31.1.1.4.1=<>, flags: NEXT, GET 
Shutting down variation modules: 
Variation module "redis" shutdown OK 
Variation module "writecache" shutdown OK 
Variation module "sql" shutdown OK 
Variation module "notification" shutdown OK 
Variation module "multiplex" shutdown OK 
Variation module "numeric" shutdown OK 
Variation module "subprocess" shutdown OK 
Variation module "error" shutdown OK 
Variation module "delay" shutdown OK 
Process terminated 
ERROR Traceback (most recent call last):; 
ERROR   File "./scripts/snmpsimd.py", line 346, in processVarBinds;    str(univ.OctetString('.'.join(['%s' % x for x in oid])))); 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/snmpsim/record/search/database.py", line 210, in lookup;    return self._db[oid]; 
ERROR KeyError: '1...3...6...1...2...1...4...3.1...1...1...4...1'; 
ERROR ;During handling of the above exception, another exception occurred:;; 
ERROR Traceback (most recent call last):; 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/asyncore/dispatch.py", line 46, in runDispatcher;    use_poll=True, map=self.__sockMap, count=1); 
ERROR   File "/usr/lib64/python3.7/asyncore.py", line 207, in loop;    poll_fun(timeout, map); 
ERROR   File "/usr/lib64/python3.7/asyncore.py", line 188, in poll2;    readwrite(obj, flags); 
ERROR   File "/usr/lib64/python3.7/asyncore.py", line 123, in readwrite;    obj.handle_error(); 
ERROR   File "/usr/lib64/python3.7/asyncore.py", line 108, in readwrite;    obj.handle_read_event(); 
ERROR   File "/usr/lib64/python3.7/asyncore.py", line 422, in handle_read_event;    self.handle_read(); 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/asyncore/dgram/base.py", line 170, in handle_read;    self._cbFun(self, transportAddress, incomingMessage); 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/base.py", line 85, in _cbFun;    self, transportDomain, transportAddress, incomingMessage; 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/entity/engine.py", line 152, in __receiveMessageCbFun;    self, transportDomain, transportAddress, wholeMsg; 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/proto/rfc3412.py", line 433, in receiveMessage;    PDU, maxSizeResponseScopedPDU, stateReference); 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/entity/rfc3413/cmdrsp.py", line 140, in processPdu;    (self.__verifyAccess, snmpEngine)); 
ERROR   File "./scripts/snmpsimd.py", line 1378, in handleMgmtOperation;    PDU, (None, snmpEngine)  # custom acInfo; 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/entity/rfc3413/cmdrsp.py", line 297, in handleMgmtOperation;    rspVarBinds = mgmtFun(varBinds, (acFun, acCtx)); 
ERROR   File "./scripts/snmpsimd.py", line 583, in readNextVars;    varBinds, **self.__getCallContext(acInfo, True)); 
ERROR   File "./scripts/snmpsimd.py", line 349, in processVarBinds;    offset = searchRecordByOid(oid, text, self.__textParser); 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/snmpsim/record/search/file.py", line 85, in searchRecordByOid;    if midval < oid:; 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pyasn1/type/base.py", line 300, in __lt__;    return self._value < other; 
ERROR TypeError: '<' not supported between instances of 'tuple' and 'str'; 
ERROR ;During handling of the above exception, another exception occurred:;; 
ERROR Traceback (most recent call last):; 
ERROR   File "./scripts/snmpsimd.py", line 1835, in <module>;    transportDispatcher.runDispatcher(); 
ERROR   File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/asyncore/dispatch.py", line 50, in runDispatcher;    raise PySnmpError('poll error: %s' % ';'.join(format_exception(*exc_info()))); 
ERROR pysnmp.error.PySnmpError: poll error: Traceback (most recent call last):;;  File "./scripts/snmpsimd.py", line 346, in processVarBinds;    str(univ.OctetString('.'.join(['%s' % x for x in oid]))));;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/snmpsim/record/search/database.py", line 210, in lookup;    return self._db[oid];;KeyError: '1...3...6...1...2...1...4...3.1...1...1...4...1';;;During handling of the above exception, another exception occurred:;;;Traceback (most recent call last):;;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/asyncore/dispatch.py", line 46, in runDispatcher;    use_poll=True, map=self.__sockMap, count=1);;  File "/usr/lib64/python3.7/asyncore.py", line 207, in loop;    poll_fun(timeout, map);;  File "/usr/lib64/python3.7/asyncore.py", line 188, in poll2;    readwrite(obj, flags);;  File "/usr/lib64/python3.7/asyncore.py", line 123, in readwrite;    obj.handle_error();;  File "/usr/lib64/python3.7/asyncore.py", line 108, in readwrite;    obj.handle_read_event();;  File "/usr/lib64/python3.7/asyncore.py", line 422, in handle_read_event;    self.handle_read();;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/asyncore/dgram/base.py", line 170, in handle_read;    self._cbFun(self, transportAddress, incomingMessage);;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/carrier/base.py", line 85, in _cbFun;    self, transportDomain, transportAddress, incomingMessage;;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/entity/engine.py", line 152, in __receiveMessageCbFun;    self, transportDomain, transportAddress, wholeMsg;;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/proto/rfc3412.py", line 433, in receiveMessage;    PDU, maxSizeResponseScopedPDU, stateReference);;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/entity/rfc3413/cmdrsp.py", line 140, in processPdu;    (self.__verifyAccess, snmpEngine));;  File "./scripts/snmpsimd.py", line 1378, in handleMgmtOperation;    PDU, (None, snmpEngine)  # custom acInfo;;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pysnmp/entity/rfc3413/cmdrsp.py", line 297, in handleMgmtOperation;    rspVarBinds = mgmtFun(varBinds, (acFun, acCtx));;  File "./scripts/snmpsimd.py", line 583, in readNextVars;    varBinds, **self.__getCallContext(acInfo, True));;  File "./scripts/snmpsimd.py", line 349, in processVarBinds;    offset = searchRecordByOid(oid, text, self.__textParser);;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/snmpsim/record/search/file.py", line 85, in searchRecordByOid;    if midval < oid:;;  File "/home/l/snmpsim/venv/lib/python3.7/site-packages/pyasn1/type/base.py", line 300, in __lt__;    return self._value < other;;TypeError: '<' not supported between instances of 'tuple' and 'str';caused by <class 'TypeError'>: '<' not supported between instances of 'tuple' and 'str';