Love the concept: struggling with the installation

[lighthouseahoy]

I found your book on the PacktPub library and love the concept. I have struggled for ages with the Pandas interface and as someone who has to switch contexts regularly the idea of a structured interface independent of where I am working is brilliant.

But I have now spent almost a full day getting the installation to complete successfully and am still getting tedious issues with the basics of database connection.

Despite your claim of an all-in-one pip install, and despite having VS14 runtime installed, I have had to jump through hoops.

Firstly, I did not have Anaconda installed, just PyCharm. So I had to start there. Fair enough, you say it makes a good foundation, so I take that as given.

Both sparse_dot_topn and python-Levenshtein seemed to want to build rather than install from a wheel. So pyoptimus refused to install, despite the presence of the Build Tools. They also seemed to want to default to a python2 build, but I guess that is someone else's issue.

After some research I managed to preinstall the modules from the following:

conda install -c conda-forge sparse_dot_topn
  conda install -c python-Levenshtein

But this simply led me to another problem when trying to load a file. Thankfully this is in your FAQs:

  pip install git+https:\/\/github.com/hi-primus/dateinfer.git

Now I'm trying a simple connection to MS SQL and I'm back to a standstill:

  _AttributeError: MSSQL_

It appears to be right. module connect[180] references DriverProperties.MSSQL.value["name"] but module properties[5] has no attribute MSSQL, though it does have a SQLSERVER. Adjusting these to line up just produces different errors and I don't know your library well enough to fiddle.

As a newcomer landing for the first time on 21.9.0-beta1, is there another version I should try first, or am I missing something entirely? The book makes it look so straightforward and I haven't even tackled SSO yet...

[luis11011]

Hi, we're happy to read that you're interested in Optimus. About the Optimus version you should use, you're not missing anything, our team still need to cover up a bunch of specific use-cases to avoid issues like yours before uploading a stable version. Currently we're working on MSSQL connections.

Could you help us out writing down which Python Version are you using? And if you're using or not a new conda environment to install Optimus (instead of the default 'base').

Also, what library are you interested in using as an engine?

[argenisleon]

Hi @lighthouseahoy, I just solve a couple of issues with the mssql driver. Could you please update the package?
Please let me know if you need anything else.

[lighthouseahoy]

OK. I pulled the changes and it will now connect. There still appears to be no mechanism for Single Sign On, but I have bodged my copy for now. Additionally, your table_to_df method is adding LIMIT and OFFSET suffixes to the query, which is incorrect syntax. I have commented this out in my copy. Which finally means I can play with the package properly. Thanks.

sso.txt

[argenisleon]

Hi @lighthouseahoy,

Thanks for the improvements I will be merging this changes today.
Can you talk about this a little more?
your table_to_df method is adding LIMIT and OFFSET suffixes to the query
Why is this incorret?

[lighthouseahoy]

Perhaps best to let SQLAlchemy illustrate this: ./sqlalchemy/dialects/mssql/base.py[299] (v1.4.23) "LIMIT/OFFSET Support". It strikes me that you already have a dependency on SQLAlchemy and you should perhaps leverage its syntax generation techniques, which would provide a more rebust solution.

This may also help:
Equivalent of LIMIT and OFFSET for SQL Server?

[argenisleon]

Oh, thanks for the detailed explanation.
The project started just using Spark so we build the SQL query by hand.
But now that we have SQLalchemy is certainly better to create the query the way you mentioned.
Opening an issue to improve this.