unable to add mongodb data source

[seem8]

Issue Summary

I am unable to add MongoDB data source.

Steps to Reproduce

1. Add data source.
2. Choose MongoDB.
3. Fill mandatory fields with valid MongoDB url, or just 'aaa'.

Any other info e.g. Why do you consider this to be a bug? What did you expect to happen instead?
I would like to add MonoDB data source. It works great with Clickhouse and MySQL sources.

Technical details:

• Redash Version: 9.0.0
• Browser/OS: Firefox/Ubuntu 18.04
• How did you install Redash: https://redash.io/help/open-source/dev-guide/setup

Traceback (from virtualenv with requirements and requrements_all installed):
python3 manage.py ds new --type mongodb
Name: aaa
Connection String (required): aaa
Database Name (required): aaa
Replica Set Name (optional):
Replica Set Read Preference (optional):
Traceback (most recent call last):
File "manage.py", line 9, in
manager()
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/data/redash/app/lib/python3.6/site-packages/flask/cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/data/redash/app/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/data/redash/app/lib/python3.6/site-packages/flask/cli.py", line 426, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/data/redash/app/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/data/redash/app/gitredash/redash/cli/data_sources.py", line 155, in new
if not options.is_valid():
File "/data/redash/app/gitredash/redash/utils/configuration.py", line 45, in is_valid
self.validate()
File "/data/redash/app/gitredash/redash/utils/configuration.py", line 57, in validate
jsonschema.validate(self._config, self._schema)
File "/data/redash/app/lib/python3.6/site-packages/jsonschema/validators.py", line 936, in validate
cls.check_schema(schema)
File "/data/redash/app/lib/python3.6/site-packages/jsonschema/validators.py", line 300, in check_schema
raise exceptions.SchemaError.create_from(error)
jsonschema.exceptions.SchemaError: <map object at 0x7f9464015ac8> is not of type 'array'

Failed validating 'type' in metaschema['properties']['properties']['additionalProperties']['properties']['enum']:
{'items': True, 'type': 'array'}

On schema['properties']['readPreference']['enum']:
<map object at 0x7f9464015ac8>

I modified 2 files to get more debug data:
/data/redash/app/lib/python3.6/site-packages/jsonschema/validators.py:
@classmethod
def check_schema(cls, schema):
for error in cls(cls.META_SCHEMA).iter_errors(schema):
# ADDED DEBUG
with open('/tmp/sprjson','w+') as sprjson:
sprjson.write(str(schema))
sprjson.write(str(cls))
sprjson.write(str(error))
# / ADDED DEBUG
raise exceptions.SchemaError.create_from(error)

/data/redash/app/gitredash/redash/utils/configuration.py
def validate(self):
# ADDED DEBUG
with open('/tmp/validebug','w+') as validebug:
validebug.write(str(self._config))
validebug.write(str(self._schema))
# / ADDED DEBUG
jsonschema.validate(self._config, self._schema)

cat /tmp/sprjson
{'type': 'object', 'properties': {'connectionString': {'type': 'string', 'title': 'Connection String'}, 'dbName': {'type': 'string', 'title': 'Database Name'}, 'replicaSetName': {'type': 'string', 'title': 'Replica Set Name'}, 'readPreference': {'type': 'string', 'title': 'Replica Set Read Preference', 'enum': <map object at 0x7f9464015ac8>}}, 'required': ['connectionString', 'dbName']}<class 'jsonschema.validators.create..Validator'><map object at 0x7f9464015ac8> is not of type 'array'

Failed validating 'type' in schema['properties']['properties']['additionalProperties']['properties']['enum']:
{'items': True, 'type': 'array'}

On instance['properties']['readPreference']['enum']:
<map object at 0x7f9464015ac8>

cat /tmp/validebug
{'connectionString': 'aaa', 'dbName': 'aaa'}{'type': 'object', 'properties': {'connectionString': {'type': 'string', 'title': 'Connection String'}, 'dbName': {'type': 'string', 'title': 'Database Name'}, 'replicaSetName': {'type': 'string', 'title': 'Replica Set Name'}, 'readPreference': {'type': 'string', 'title': 'Replica Set Read Preference', 'enum': <map object at 0x7f9464015ac8>}}, 'required': ['connectionString', 'dbName']}

[seem8]

I've found it. It's a really monkey patch, but it worked.
Line 163 of a file /data/redash/app/gitredash/redash/handlers/data_sources.py:

schema = get_configuration_schema_for_query_runner_type(req["type"])
was chaned to changed to
schema = {
'type': 'object',
'properties': {'connectionString': {'type': 'string', 'title': 'Connection String'},
'dbName': {'type': 'string', 'title': 'Database Name'},
'replicaSetName': {'type': 'string', 'title': 'Replica Set Name'},
'readPreference': {'type': 'string', 'title': 'Replica Set Read Preference', 'enum': '8'}},
'required': ['connectionString', 'dbName'],
}

I need that 8, because a new data source will have id of 8 in the database.

Next is config variable below. Line:
config = ConfigurationContainer(filter_none(req["options"]), schema)
was changed to
config = {
'connectionString': "mongodb://USER:PASS@IP.IP.IP.IP/?authSource=DATABASE&gssapiServiceName=mongodb",
'dbName': 'DATABASE_NAME',
}

I also commented out this two lines:
#if not config.is_valid():
# abort(400)

I was able to add MongoDB databases.

[wtfiwtz]

Is it missing a query_runner? I had the same issue with version 10.1.0 with Athena. The abort(400) responses with a BadRequest error - The browser (or proxy) sent a request that this server could not understand. which is very generic.

[wtfiwtz]

Each Query Runner is within redash/query_runners and ends with a statement like

register(Athena)

So check your Python dependencies are loaded correctly.

Look for this error message:

        logger.debug(
            "%s query runner enabled but not supported, not registering. Either disable or install missing "
            "dependencies.",
            query_runner_class.name(),
        )

[justinclift]

Hmmm, this is a old bug report for a pretty old release of Redash.

With our latest development code (eg the redash/redash:preview image on Docker Hub), we definitely have MongoDB working as a data source.

@wtfiwtz If you're not running a production setup, it might be worth trying that out? Note that the image isn't well tested enough (yet) to consider "production ready", so be careful with it. 😄

[wtfiwtz]

Thanks @justinclift ... managed to get it working on 10.1.0 (which we have in production 😬... hopefully can upgrade this shortly)

I still get a warning about pyarrow versions but this is working with requirements_all_ds.txt

google-api-python-client==1.7.11
protobuf==3.17.3
gspread==3.1.0
impyla==0.16.0
influxdb==5.2.3
mysqlclient==1.3.13
oauth2client==4.1.3
pyhive==0.6.1
pymongo[tls,srv]==3.9.0
vertica-python==0.9.5
td-client==1.0.0
pymssql==2.1.4
dynamo3==0.4.10
sasl>=0.1.3
cassandra-driver==3.21.0
atsd_client==3.0.5
simple_salesforce==0.74.3
PyAthena==1.5.0
qds-sdk>=1.9.6
ibm-db>=2.0.9
pydruid==0.5.7
requests_aws_sign==0.1.5
phoenixdb==0.7
# certifi is needed to support MongoDB and SSL:
certifi<2021.0.0
pydgraph==2.0.2
pyexasol==0.12.0
python-rapidjson==0.8.0
pyodbc==4.0.28
trino==0.305
cmem-cmempy==21.2.3
xlrd==2.0.1
openpyxl==3.0.7

firebolt-sqlalchemy==0.0.9
firebolt-sdk==0.2.1
pytz<2021.0
croniter~=1.0
python-dateutil==2.8.0
botocore==1.13
boto3==1.10.0
paramiko==2.7.0
memsql==3.0.0
snowflake-connector-python==2.3.0
azure-kusto-data==0.0.35
dql>=0.5.26

pyarrow<0.14
pymapd>=0.19.0
thrift>=0.11.0
thrift_sasl>=0.1.0

I run docker-compose run --rm server pip install -r requirements_all_ds.txt to check

Thanks!

[justinclift]

Well, as long as it's working I suppose. 😁

[justinclift]

I'll close this issue report as it's for a very outdated Redash release, and you've managed to fix your setup as well. 😄