Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting pytest/pluggy/colorama issues while running test file #3335

Closed
rajatkmehta opened this issue Mar 22, 2018 · 14 comments
Closed

Getting pytest/pluggy/colorama issues while running test file #3335

rajatkmehta opened this issue Mar 22, 2018 · 14 comments
Labels
status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity topic: reporting related to terminal output and user-facing messages and errors type: bug problem that needs to be addressed

Comments

@rajatkmehta
Copy link

rajatkmehta commented Mar 22, 2018

<?xml version="1.0" encoding="utf-8"?><testsuite errors="1" failures="0" name="pytest" skips="0" tests="1" time="4.943"><testcase classname="pytest" name="internal" time="0"><error message="internal error">Traceback (most recent call last):
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\main.py&quot;, line 100, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\main.py&quot;, line 138, in _main
    config.hook.pytest_runtestloop(session=session)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py&quot;, line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py&quot;, line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py&quot;, line 216, in &lt;lambda&gt;
    firstresult=hook.spec_opts.get(&apos;firstresult&apos;),
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py&quot;, line 201, in _multicall
    return outcome.get_result()
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py&quot;, line 77, in get_result
    _reraise(*ex)  # noqa
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py&quot;, line 180, in _multicall
    res = hook_impl.function(*args)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\xdist\dsession.py&quot;, line 116, in pytest_runtestloop
    self.loop_once()
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\xdist\dsession.py&quot;, line 139, in loop_once
    call(**kwargs)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\xdist\dsession.py&quot;, line 240, in worker_logstart
    nodeid=nodeid, location=location)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py&quot;, line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py&quot;, line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py&quot;, line 216, in &lt;lambda&gt;
    firstresult=hook.spec_opts.get(&apos;firstresult&apos;),
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py&quot;, line 201, in _multicall
    return outcome.get_result()
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py&quot;, line 77, in get_result
    _reraise(*ex)  # noqa
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py&quot;, line 180, in _multicall
    res = hook_impl.function(*args)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\terminal.py&quot;, line 261, in pytest_runtest_logstart
    self.write_ensure_prefix(line, &quot;&quot;)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\terminal.py&quot;, line 185, in write_ensure_prefix
    self._tw.line()
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\py\_io\terminalwriter.py&quot;, line 236, in line
    self.write(&apos;\n&apos;)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\py\_io\terminalwriter.py&quot;, line 224, in write
    write_out(self._file, markupmsg)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\py\_io\terminalwriter.py&quot;, line 369, in write_out
    fil.write(msg)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py&quot;, line 40, in write
    self.__convertor.write(text)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py&quot;, line 141, in write
    self.write_and_convert(text)
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py&quot;, line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File &quot;c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py&quot;, line 174, in write_plain_text
    self.wrapped.write(text[start:end])
IOError: [Errno 0] Error</error></testcase></testsuite>
@pytestbot
Copy link
Contributor

GitMate.io thinks possibly related issues are #2258 (Unicode issues with pytest.fail), #471 (pytest keeps running deleted tests), #1563 (All pytest tests skipped), #1591 (pytest-xdist fails when running same tests several times), and #2723 (Pytest not detecting local .ini file when running specific part of test).

@pytestbot pytestbot added the type: bug problem that needs to be addressed label Mar 22, 2018
@rajatkmehta
Copy link
Author

how to resolve this? Please help!

@rajatkmehta
Copy link
Author

When I run multipletimes then tests are running 1 time out of 10 times, but all the tests are failing locally even though those are passing in jenkins job. Rest the time i get same error which i have mentioned in my thread.... Getting pytest/pluggy/colorama issues while running test file #3335

@nicoddemus
Copy link
Member

@rajatkmehta hard to tell what might be the problem, might be a broken colorama installation. What happens if you pass --color=no to pytest? Can you provide a reproducible minimal case?

@nicoddemus nicoddemus added topic: reporting related to terminal output and user-facing messages and errors status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity labels Mar 26, 2018
@rajatkmehta
Copy link
Author

I am trying to run the script in which need to make connection from database(CONN1 and CONN2), execute queries(SQL 1 and SQL2), fetch data and compare them. But getting errors like as follows....
TExecuteStatementResp(status=TStatus(errorCode=40000, errorMessage=u"Error while compiling statement: FAILED: ParseException line 1:629 cannot recognize input near '$' 'load_date' 'where' in table source", sqlState=u'42000', infoMessages=[u"*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:629 cannot recognize input near '$' 'load_date' 'where' in table source:17:16", u'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:387', u'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:186', u'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:269', u'org.apache.hive.service.cli.operation.Operation:run:Operation.java:324',

@nicoddemus
Copy link
Member

Sorry @rajatkmehta your last message doesn't seem related to pytest at all. Please provide enough information, preferably properly formatted, otherwise it is very hard for us to try to help you.

@rajatkmehta
Copy link
Author

rajatkmehta commented Mar 28, 2018

Using script:

@pytest.mark.compare_expected_changes_tolerance
def test_expected_changes_tolerance(test_data, comparator):
    """Comparing difference between two sql returning numeric values with expected tolerance

    Note: SQL should not contain nested values - they should be flattened.
           - Only two columns are allowed in select statement where count is the second value
           - Output of queries is generated as key:value pair.
          Result of numeric fields is matched against each other with expected tolerance
    """
    comparator.global_tolerance = None
    comparator.tolerance_lookup = None
    print "*"*20
    try:
        # Load test parameters
        # convert test_params_json string into json
        test_params_json = {}
        if test_data.get("test_params_json"):
            test_params_json = json.loads(test_data["test_params_json"])

        # if data_sample_sql is present, then get data sample from second connection (expected values)
        if test_data.get("data_sample_sql", None):
            test_params_json = add_data_sample_to_test_params_json(test_data, test_params_json)

        # connect to the first connection and get results from first sql
        #   get connection
        test_conn1_dtl = get_conn_details(test_data["conn1"])
        conn1 = globals()["%s_conn" % test_conn1_dtl["conn_type"]](test_conn1_dtl)

        #   generate sql
        sql1 = test_data["sql1"].format(**test_params_json)

        #   execute first sql statement
        actual_json = fetch_records(conn1, sql1, "use_first_column_as_headers")

        # read JSON file that compares results
        test_conn2_dtl = get_conn_details(test_data["conn2"])
        conn2 = globals()["%s_conn" % test_conn2_dtl["conn_type"]](test_conn2_dtl)

        #   generate sql
        sql2 = test_data["sql2"].format(**test_params_json)

        #   execute first sql statement
        expected_json = fetch_records(conn2, sql2, "use_first_column_as_headers")

        # make sure none of the lists are empty
        if not (actual_json and expected_json):
            raise ValueError("One of the inputs returned no rows")

        if test_data.get("global_tolerance"):
            comparator.global_tolerance = float(test_data["global_tolerance"])

        if test_data.get("values_tolerance_dict"):
            comparator.tolerance_lookup = json.loads(test_data["values_tolerance_dict"])

        # compare data between actual_json and expected_json
        # print "-"*20
        # print "actual", json.dumps(actual_json, indent=2)
        # print "expected", json.dumps(expected_json, indent=2)
        # print "tolerance", str(comparator.global_tolerance)
        # print "tolerance_lookup", json.dumps(comparator.tolerance_lookup)
        # print "-"*20
        comparator.compare_json(actual_json, expected_json)

    except Exception as e:
        comparator.increment_exception_fail(e)
    finally:
        comparator.global_tolerance = None
        comparator.tolerance_lookup = None
        comparator.print_results()
    comparator.assert_pass()

To run this script using command from terminal:

pytest -m compare_expected_changes_tolerance -v --config_file_path=../config/config.dev.json --test_file_path=../data_rtp_le_tests/dev_counters_expected_changes_tolerance.txt --test_params_json="{\"date_key\":\"$load_date\"}" -n 4 --junitxml=./reports/dev_counters_expected_changes_tolerance11.xml

@nicoddemus
Copy link
Member

(@rajatkmehta I formatted your post myself, please take a look at the GH Markdown Help).

Did you try running pytest passing --color=no?

@rajatkmehta
Copy link
Author

rajatkmehta commented Mar 29, 2018

Thanks for formatting my script!
I have tried using --color=no but still no progress..

@nicoddemus
Copy link
Member

Can you please post the full output when you run with --color=no?

@rajatkmehta
Copy link
Author

rajatkmehta commented Mar 29, 2018

(venv) C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool>pytest --color=no -m compare_expected_changes_tolerance -v --config_file_path=../config/config.dev.json --test_file_path
=../data_rtp_le_tests/dev_counters_expected_changes_tolerance.txt --test_params_json="{\"date_key\":\"$load_date\"}" -n 4 --junitxml=./reports/dev_counters_expected_changes_tolerance11.x
ml
============================= test session starts =============================
platform win32 -- Python 2.7.14, pytest-3.4.2, py-1.5.2, pluggy-0.6.0 -- c:\users\outworx\pycharmprojects\rtp-tools\venv\scripts\python.exe
cachedir: .pytest_cache
metadata: {'Python': '2.7.14', 'Platform': 'Windows-10-10.0.16299', 'Packages': {'py': '1.5.2', 'pytest': '3.4.2', 'pluggy': '0.6.0'}, 'Plugins': {'xdist': '1.22.2', 'json': '0.4.0', 'ht
ml': '1.16.1', 'testrail': '1.1.0', 'flaky': '3.4.0', 'forked': '0.2', 'metadata': '1.6.0'}}
rootdir: C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool, inifile: pytest.ini
plugins: xdist-1.22.2, testrail-1.1.0, metadata-1.6.0, json-0.4.0, html-1.16.1, forked-0.2, flaky-3.4.0
[gw0] win32 Python 2.7.14 cwd: C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool
[gw1] win32 Python 2.7.14 cwd: C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool
[gw2] win32 Python 2.7.14 cwd: C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool
[gw3] win32 Python 2.7.14 cwd: C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool
[gw0] Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
[gw1] Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
[gw2] Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
[gw3] Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)]
gw0 [6] / gw1 [6] / gw2 [6] / gw3 [6]
scheduling tests via LoadScheduling

test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_listings_all]
test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_listings]
test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_active_merged_properties_to_listings_all]
test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_rental_properties_all_to_listings_all]
[gw0] [ 16%] FAILED test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_listings_all]
[gw3] [ 33%] FAILED test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_active_merged_properties_to_listings_all]
test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_merged_listings]
[gw1] [ 50%] FAILED test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_rental_properties_all_to_listings_all]
test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_active_rental_communities_to_merged_listings]
[gw2] [ 66%] FAILED test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_listings]
[gw3] [ 83%] FAILED test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_merged_listings]
[gw1] [100%] FAILED test_databases.py::test_expected_changes_tolerance[rtp_le_counters_dev_compare_active_rental_communities_to_merged_listings]

 generated xml file: C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool\reports\dev_counters_expected_changes_tolerance11.xml
===Flaky Test Report===


===End Flaky Test Report===
================================== FAILURES ===================================
 test_expected_changes_tolerance[rtp_le_counters_dev_compare_dc_mysql_active_listings_tsv_to_listings_all]
[gw0] win32 -- Python 2.7.14 c:\users\outworx\pycharmprojects\rtp-tools\venv\scripts\python.exe

test_data = {'conn1': 'hive.dwr-dev1', 'conn2': 'hive.dwr-dev1', 'global_tolerance': '0.005', 'sql1': "select concat('listing_type...alized_listing_type ,': builder:', coalesce(is_builde
r_community_listing, 'NULL')) order by listing_type_builder", ...}
comparator = <common.helper.rest_tester.RESTTester object at 0x0000000006CD1DD8>

    @pytest.mark.compare_expected_changes_tolerance
    def test_expected_changes_tolerance(test_data, comparator):
        """Comparing difference between two sql returning numeric values with expected tolerance

        Note: SQL should not contain nested values - they should be flattened.
               - Only two columns are allowed in select statement where count is the second value
               - Output of queries is generated as key:value pair.
              Result of numeric fields is matched against each other with expected tolerance
        """
        comparator.global_tolerance = None
        comparator.tolerance_lookup = None
        print "*"*20
        try:
            # Load test parameters
            # convert test_params_json string into json
            test_params_json = {}
            if test_data.get("test_params_json"):
                test_params_json = json.loads(test_data["test_params_json"])

            # if data_sample_sql is present, then get data sample from second connection (expected values)
            if test_data.get("data_sample_sql", None):
                test_params_json = add_data_sample_to_test_params_json(test_data, test_params_json)

            # connect to the first connection and get results from first sql
            #   get connection
            test_conn1_dtl = get_conn_details(test_data["conn1"])
            conn1 = globals()["%s_conn" % test_conn1_dtl["conn_type"]](test_conn1_dtl)

            #   generate sql
            sql1 = test_data["sql1"].format(**test_params_json)

            #   execute first sql statement
            actual_json = fetch_records(conn1, sql1, "use_first_column_as_headers")

            # read JSON file that compares results
            test_conn2_dtl = get_conn_details(test_data["conn2"])
            conn2 = globals()["%s_conn" % test_conn2_dtl["conn_type"]](test_conn2_dtl)

            #   generate sql
            sql2 = test_data["sql2"].format(**test_params_json)

            #   execute first sql statement
            expected_json = fetch_records(conn2, sql2, "use_first_column_as_headers")

            # make sure none of the lists are empty
            if not (actual_json and expected_json):
                raise ValueError("One of the inputs returned no rows")

            if test_data.get("global_tolerance"):
                comparator.global_tolerance = float(test_data["global_tolerance"])

            if test_data.get("values_tolerance_dict"):
                comparator.tolerance_lookup = json.loads(test_data["values_tolerance_dict"])

            # compare data between actual_json and expected_json
            # print "-"*20
            # print "actual", json.dumps(actual_json, indent=2)
            # print "expected", json.dumps(expected_json, indent=2)
            # print "tolerance", str(comparator.global_tolerance)
            # print "tolerance_lookup", json.dumps(comparator.tolerance_lookup)
            # print "-"*20
            comparator.compare_json(actual_json, expected_json)

        except Exception as e:
            comparator.increment_exception_fail(e)
        finally:
            comparator.global_tolerance = None
            comparator.tolerance_lookup = None
            comparator.print_results()
>       comparator.assert_pass()
E       AssertionError: 0 passed, 1 failed!

test_databases.py:167: AssertionError
---------------------------- Captured stdout call -----------------------------
********************
TOpenSessionResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=None, statusCode=0), sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='>\xbd
#\x95\x83\x84H\x9b\x8b\xb46V\x06\x8f\xdfX', guid='5G9\xf3;\xddB\xd9\xaf\xec\xfcq\xa1j\x8a\xc4')), configuration={u'hive.server2.thrift.resultset.default.fetch.size': u'1000'}, serverProt
ocolVersion=5)
2018-03-29 16:20:21,997 - INFO - USE `default`
TExecuteStatementReq(confOverlay=None, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='>\xbd#\x95\x83\x84H\x9b\x8b\xb46V\x06\x8f\xdfX', guid='5G9\xf3;\xddB\xd9\xaf\xec\x
fcq\xa1j\x8a\xc4')), runAsync=False, statement=u'USE `default`', queryTimeout=0)
TExecuteStatementResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=None, statusCode=0), operationHandle=TOperationHandle(hasResultSet=False, modifiedRowC
ount=None, operationType=0, operationId=THandleIdentifier(secret='\\\xfa\xf5 \xaa\x04M\x1a\xac\xb5\x01dzzZ\xb3', guid="\xd0'\x8f7\xab6B6\x8do\x81\xc3\x064\xa0\xa1")))
TCloseOperationResp(status=TStatus(errorCode=None, errorMessage=None, sqlState=None, infoMessages=None, statusCode=0))
--------------------
select concat('listing_type:', normalized_listing_type ,': builder:', coalesce(is_builder_community_listing, 'NULL')) listing_type_builder, count(*) total_rec from rtp_le_qa.active_listi
ngs_tsv_$load_date  group by concat('listing_type:', normalized_listing_type ,': builder:', coalesce(is_builder_community_listing, 'NULL')) order by listing_type_builder
--------------------
2018-03-29 16:20:22,880 - INFO - select concat('listing_type:', normalized_listing_type ,': builder:', coalesce(is_builder_community_listing, 'NULL')) listing_type_builder, count(*) tota
l_rec from rtp_le_qa.active_listings_tsv_$load_date  group by concat('listing_type:', normalized_listing_type ,': builder:', coalesce(is_builder_community_listing, 'NULL')) order by list
ing_type_builder
TExecuteStatementReq(confOverlay=None, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='>\xbd#\x95\x83\x84H\x9b\x8b\xb46V\x06\x8f\xdfX', guid='5G9\xf3;\xddB\xd9\xaf\xec\x
fcq\xa1j\x8a\xc4')), runAsync=False, statement="select concat('listing_type:', normalized_listing_type ,': builder:', coalesce(is_builder_community_listing, 'NULL')) listing_type_builder
, count(*) total_rec from rtp_le_qa.active_listings_tsv_$load_date  group by concat('listing_type:', normalized_listing_type ,': builder:', coalesce(is_builder_community_listing, 'NULL')
) order by listing_type_builder", queryTimeout=0)
TExecuteStatementResp(status=TStatus(errorCode=40000, errorMessage=u"Error while compiling statement: FAILED: ParseException line 1:194 cannot recognize input near '$' 'load_date' 'group
' in table source", sqlState=u'42000', infoMessages=[u"*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:194 cannot recognize i
nput near '$' 'load_date' 'group' in table source:17:16", u'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:380', u'org.apache.hive.service.cli.operation.SQ
LOperation:prepare:SQLOperation.java:206', u'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:290', u'org.apache.hive.service.cli.operation.Operation:run:
Operation.java:320', u'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:530', u'org.apache.hive.service.cli.session.HiveSessionImpl:execu
teStatement:HiveSessionImpl.java:506', u'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:280', u'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatem
ent:ThriftCLIService.java:531', u'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437', u'org.apache.hive.service.rpc.thrift.TCLISer
vice$Processor$ExecuteStatement:getResult:TCLIService.java:1422', u'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', u'org.apache.thrift.TBaseProcessor:process:TBasePr
ocessor.java:39', u'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', u'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoo
lServer.java:286', u'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142', u'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617', u
'java.lang.Thread:run:Thread.java:745', u"*org.apache.hadoop.hive.ql.parse.ParseException:line 1:194 cannot recognize input near '$' 'load_date' 'group' in table source:22:6", u'org.apac
he.hadoop.hive.ql.parse.ParseDriver:parse:ParseDriver.java:207', u'org.apache.hadoop.hive.ql.parse.ParseUtils:parse:ParseUtils.java:77', u'org.apache.hadoop.hive.ql.parse.ParseUtils:pars
e:ParseUtils.java:70', u'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:468', u'org.apache.hadoop.hive.ql.Driver:compileInternal:Driver.java:1316', u'org.apache.hadoop.hive.ql.Driv
er:compileAndRespond:Driver.java:1294', u'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:204'], statusCode=3), operationHandle=None)
2018-03-29 16:20:23,299 - ERROR - ERROR!
Traceback (most recent call last):
  File "C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool\test_databases.py", line 130, in test_expected_changes_tolerance
    actual_json = fetch_records(conn1, sql1, "use_first_column_as_headers")
  File "C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool\test_databases.py", line 508, in fetch_records
    cursor.execute(sql)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pyhive\hive.py", line 300, in execute
    _check_status(response)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pyhive\hive.py", line 425, in _check_status
    raise OperationalError(response)
OperationalError: TExecuteStatementResp(status=TStatus(errorCode=40000, errorMessage=u"Error while compiling statement: FAILED: ParseException line 1:194 cannot recognize input near '$'
'load_date' 'group' in table source", sqlState=u'42000', infoMessages=[u"*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:194
cannot recognize input near '$' 'load_date' 'group' in table source:17:16", u'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:380', u'org.apache.hive.servic
e.cli.operation.SQLOperation:prepare:SQLOperation.java:206', u'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:290', u'org.apache.hive.service.cli.operat
ion.Operation:run:Operation.java:320', u'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:530', u'org.apache.hive.service.cli.session.Hiv
eSessionImpl:executeStatement:HiveSessionImpl.java:506', u'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:280', u'org.apache.hive.service.cli.thrift.ThriftCLISer
vice:ExecuteStatement:ThriftCLIService.java:531', u'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437', u'org.apache.hive.service.
rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422', u'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', u'org.apache.thrift.TBaseProcess
or:process:TBaseProcessor.java:39', u'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', u'org.apache.thrift.server.TThreadPoolServer$WorkerProc
ess:run:TThreadPoolServer.java:286', u'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142', u'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExe
cutor.java:617', u'java.lang.Thread:run:Thread.java:745', u"*org.apache.hadoop.hive.ql.parse.ParseException:line 1:194 cannot recognize input near '$' 'load_date' 'group' in table source
:22:6", u'org.apache.hadoop.hive.ql.parse.ParseDriver:parse:ParseDriver.java:207', u'org.apache.hadoop.hive.ql.parse.ParseUtils:parse:ParseUtils.java:77', u'org.apache.hadoop.hive.ql.par
se.ParseUtils:parse:ParseUtils.java:70', u'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:468', u'org.apache.hadoop.hive.ql.Driver:compileInternal:Driver.java:1316', u'org.apache.h
adoop.hive.ql.Driver:compileAndRespond:Driver.java:1294', u'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:204'], statusCode=3), operationHandle=None)
2018-03-29 16:20:23,301 - ERROR - Fail!
-> Exception (1):
        OperationalError(TExecuteStatementResp(status=TStatus(errorCode=40000, errorMessage=u"Error while compiling statement: FAILED: ParseException line 1:194 cannot recognize input ne
ar '$' 'load_date' 'group' in table source", sqlState=u'42000', infoMessages=[u"*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line
 1:194 cannot recognize input near '$' 'load_date' 'group' in table source:17:16", u'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:380', u'org.apache.hive
.service.cli.operation.SQLOperation:prepare:SQLOperation.java:206', u'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:290', u'org.apache.hive.service.cli
.operation.Operation:run:Operation.java:320', u'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:530', u'org.apache.hive.service.cli.sess
ion.HiveSessionImpl:executeStatement:HiveSessionImpl.java:506', u'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:280', u'org.apache.hive.service.cli.thrift.Thrif
tCLIService:ExecuteStatement:ThriftCLIService.java:531', u'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437', u'org.apache.hive.s
ervice.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422', u'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', u'org.apache.thrift.TBase
Processor:process:TBaseProcessor.java:39', u'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', u'org.apache.thrift.server.TThreadPoolServer$Wor
kerProcess:run:TThreadPoolServer.java:286', u'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142', u'java.util.concurrent.ThreadPoolExecutor$Worker:run:Thread
PoolExecutor.java:617', u'java.lang.Thread:run:Thread.java:745', u"*org.apache.hadoop.hive.ql.parse.ParseException:line 1:194 cannot recognize input near '$' 'load_date' 'group' in table
 source:22:6", u'org.apache.hadoop.hive.ql.parse.ParseDriver:parse:ParseDriver.java:207', u'org.apache.hadoop.hive.ql.parse.ParseUtils:parse:ParseUtils.java:77', u'org.apache.hadoop.hive
.ql.parse.ParseUtils:parse:ParseUtils.java:70', u'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:468', u'org.apache.hadoop.hive.ql.Driver:compileInternal:Driver.java:1316', u'org.a
pache.hadoop.hive.ql.Driver:compileAndRespond:Driver.java:1294', u'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:204'], statusCode=3), operationHandle=None
),)
 test_expected_changes_tolerance[rtp_le_counters_dev_compare_active_merged_properties_to_listings_all]
[gw3] win32 -- Python 2.7.14 c:\users\outworx\pycharmprojects\rtp-tools\venv\scripts\python.exe

test_data = {'conn1': 'hive.dwr-dev1', 'conn2': 'hive.dwr-dev1', 'global_tolerance': '0.005', 'sql1': "select concat('listing_type...NULL') , ': builder:', coalesce(get_json_object(info,
 '$.attribs.P8[2]'), 'NULL')) order by listing_type_builder", ...}
comparator = <common.helper.rest_tester.RESTTester object at 0x00000000072ECDD8>

    @pytest.mark.compare_expected_changes_tolerance
    def test_expected_changes_tolerance(test_data, comparator):
        """Comparing difference between two sql returning numeric values with expected tolerance

        Note: SQL should not contain nested values - they should be flattened.
               - Only two columns are allowed in select statement where count is the second value
               - Output of queries is generated as key:value pair.
              Result of numeric fields is matched against each other with expected tolerance
        """
        comparator.global_tolerance = None
        comparator.tolerance_lookup = None
        print "*"*20
        try:
            # Load test parameters
            # convert test_params_json string into json
            test_params_json = {}
            if test_data.get("test_params_json"):
                test_params_json = json.loads(test_data["test_params_json"])

            # if data_sample_sql is present, then get data sample from second connection (expected values)
            if test_data.get("data_sample_sql", None):
                test_params_json = add_data_sample_to_test_params_json(test_data, test_params_json)

            # connect to the first connection and get results from first sql
            #   get connection
            test_conn1_dtl = get_conn_details(test_data["conn1"])
            conn1 = globals()["%s_conn" % test_conn1_dtl["conn_type"]](test_conn1_dtl)

            #   generate sql
            sql1 = test_data["sql1"].format(**test_params_json)

            #   execute first sql statement
            actual_json = fetch_records(conn1, sql1, "use_first_column_as_headers")

            # read JSON file that compares results
            test_conn2_dtl = get_conn_details(test_data["conn2"])
            conn2 = globals()["%s_conn" % test_conn2_dtl["conn_type"]](test_conn2_dtl)

            #   generate sql
            sql2 = test_data["sql2"].format(**test_params_json)

            #   execute first sql statement
            expected_json = fetch_records(conn2, sql2, "use_first_column_as_headers")

            # make sure none of the lists are empty
            if not (actual_json and expected_json):
                raise ValueError("One of the inputs returned no rows")

            if test_data.get("global_tolerance"):
                comparator.global_tolerance = float(test_data["global_tolerance"])

            if test_data.get("values_tolerance_dict"):
                comparator.tolerance_lookup = json.loads(test_data["values_tolerance_dict"])

            # compare data between actual_json and expected_json
            # print "-"*20
            # print "actual", json.dumps(actual_json, indent=2)
            # print "expected", json.dumps(expected_json, indent=2)
            # print "tolerance", str(comparator.global_tolerance)
Traceback (most recent call last):
  File "C:\Python27\Lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\OutworX\PycharmProjects\rtp-tools\venv\Scripts\pytest.exe\__main__.py", line 9, in <module>
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\config.py", line 59, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py", line 201, in _multicall
    return outcome.get_result()
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py", line 77, in get_result
    _reraise(*ex)  # noqa
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\main.py", line 131, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\main.py", line 125, in wrap_session
    exitstatus=session.exitstatus)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\pluggy\callers.py", line 196, in _multicall
    gen.send(outcome)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\terminal.py", line 478, in pytest_sessionfinish
    self.summary_failures()
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\terminal.py", line 602, in summary_failures
    self._outrep_summary(rep)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\terminal.py", line 626, in _outrep_summary
    rep.toterminal(self._tw)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\runner.py", line 233, in toterminal
    longrepr.toterminal(out)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\_code\code.py", line 749, in toterminal
    self.reprtraceback.toterminal(tw)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\_code\code.py", line 766, in toterminal
    entry.toterminal(tw)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\_pytest\_code\code.py", line 816, in toterminal
    tw.line(line, bold=True, red=red)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\py\_io\terminalwriter.py", line 236, in line
    self.write('\n')
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\py\_io\terminalwriter.py", line 224, in write
    write_out(self._file, markupmsg)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\py\_io\terminalwriter.py", line 369, in write_out
    fil.write(msg)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py", line 40, in write
    self.__convertor.write(text)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py", line 141, in write
    self.write_and_convert(text)
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py", line 169, in write_and_convert
    self.write_plain_text(text, cursor, len(text))
  File "c:\users\outworx\pycharmprojects\rtp-tools\venv\lib\site-packages\colorama\ansitowin32.py", line 174, in write_plain_text
    self.wrapped.write(text[start:end])
IOError: [Errno 0] Error

(venv) C:\Users\OutworX\PycharmProjects\rtp-tools\comparator-tool>

@nicoddemus
Copy link
Member

(Please edit your message and format it correctly, otherwise it is very hard to read the traceback, thanks!)

@rajatkmehta
Copy link
Author

is the format okay now?

@nicoddemus
Copy link
Member

Unfortunately no 😞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: needs information reporter needs to provide more information; can be closed after 2 or more weeks of inactivity topic: reporting related to terminal output and user-facing messages and errors type: bug problem that needs to be addressed
Projects
None yet
Development

No branches or pull requests

3 participants