🎉 New Source: Coinmarketcap [low-code CDK] #18565

haithem-souala · 2022-10-27T18:09:11Z

What

https://coinmarketcap.com/api/documentation/v1/

How

A new source connector using low-code cdk.

Streams

categories
listing
quotes
fiat
exchange

🚨 User Impact 🚨

N/A

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

marcosmarxm · 2022-10-27T22:42:45Z

Thanks for the contribution @haithem-souala, @YiyangLi will review your contribution.

airbyte-integrations/connectors/source-coinmarketcap/source_coinmarketcap/spec.yaml

docs/integrations/sources/coinmarketcap.md

airbyte-integrations/connectors/source-coinmarketcap/source_coinmarketcap/coinmarketcap.yaml

airbyte-integrations/connectors/source-coinmarketcap/source_coinmarketcap/schemas/listing.json

airbyte-integrations/connectors/source-coinmarketcap/source_coinmarketcap/spec.yaml

YiyangLi · 2022-10-28T16:06:46Z

@haithem-souala Thanks for your contribution. I am sorry for the delayed response. The implementation is excellent, I am going to run the acceptance tests and set up the CI.

Let me know if you have a question to my feedback.

YiyangLi · 2022-10-28T18:02:03Z

I run the acceptance tests locally. And there is a failure in TestFullRefresh.test_sequential_reads. This is the doc that explains the test in details.

In your case, it reads the stream categories 2 times and expects the data sets to be the same, or one to be a subset of the other.

The missing records are too big, I can't share in GitHub. You can DM me, and I can share the file with you. I list an example, AMM.

The first read returns

{
  "id": "604f2762ebccdd50cd175fcc",
  "name": "AMM",
  "title": "AMM",
  "description": "AMM",
  "num_tokens": 68,
  "avg_price_change": 0.5845045873015874,
  "market_cap": 8738092462.000002,
  "market_cap_change": 0.548777777777778,
  "volume": 814334418.447913,
  "volume_change": 72.33621111111113,
  "last_updated": "2021-11-10T11:25:14.857Z"
 },

and the second read returns

{
  "id": "604f2762ebccdd50cd175fcc",
  "name": "AMM",
  "title": "AMM",
  "description": "AMM",
  "num_tokens": 68,
  "avg_price_change": 0.621663377142857,
  "market_cap": 3383277897.08,
  "market_cap_change": 0.5988507936507937,
  "volume": 529207223.5685604,
  "volume_change": 68.82604603174602,
  "last_updated": "2021-11-10T11:25:14.857Z"
 },

I am not familiar with Crypto, but I am open to learn. Is it because the market is volatile and consequently it changes dramatically between 2 reads? Can you investigate it, and see if you could resolve the test failures by tweaking the configuration on acceptance-test-config.yml.

haithem-souala · 2022-10-28T18:16:57Z

I run the acceptance tests locally. And there is a failure in TestFullRefresh.test_sequential_reads. This is the doc that explains the test in details.

In your case, it reads the stream categories 2 times and expects the data sets to be the same, or one to be a subset of the other.

The missing records are too big, I can't share in GitHub. You can DM me, and I can share the file with you. I list an example, AMM.

The first read returns
{
  "id": "604f2762ebccdd50cd175fcc",
  "name": "AMM",
  "title": "AMM",
  "description": "AMM",
  "num_tokens": 68,
  "avg_price_change": 0.5845045873015874,
  "market_cap": 8738092462.000002,
  "market_cap_change": 0.548777777777778,
  "volume": 814334418.447913,
  "volume_change": 72.33621111111113,
  "last_updated": "2021-11-10T11:25:14.857Z"
 },
and the second read returns
{
  "id": "604f2762ebccdd50cd175fcc",
  "name": "AMM",
  "title": "AMM",
  "description": "AMM",
  "num_tokens": 68,
  "avg_price_change": 0.621663377142857,
  "market_cap": 3383277897.08,
  "market_cap_change": 0.5988507936507937,
  "volume": 529207223.5685604,
  "volume_change": 68.82604603174602,
  "last_updated": "2021-11-10T11:25:14.857Z"
 },
I am not familiar with Crypto, but I am open to learn. Is it because the market is volatile and consequently it changes dramatically between 2 reads? Can you investigate it, and see if you could resolve the test failures by tweaking the configuration on acceptance-test-config.yml.

Yes, because the market is hyper volatile, i've no idea how to tweak? any suggestion? or maybe i shoud drop the categories stream!!

YiyangLi · 2022-10-28T18:20:49Z

@haithem-souala according to the doc, you can add ignored_fields helps -- For each stream, list of fields path ignoring in sequential reads test.

This is an example: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-facebook-pages/acceptance-test-config.yml#L21-L27

See if it helps.

haithem-souala · 2022-10-28T18:44:00Z

@haithem-souala according to the doc, you can add ignored_fields helps -- For each stream, list of fields path ignoring in sequential reads test.

This is an example: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-facebook-pages/acceptance-test-config.yml#L21-L27

See if it helps.

Thank you! fixed!

YiyangLi · 2022-10-28T18:56:20Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3347825584
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3347825584
🐛 https://gradle.com/s/odbw44f4kns3s

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
=================== 2 failed, 22 passed, 3 skipped in 35.69s ===================

YiyangLi · 2022-10-28T19:26:43Z

I want to clean up some files and rebase your branch. I am not sure if it's because you are in a master branch, I am not able to do it.

> git push haithem
Enumerating objects: 107, done.
Counting objects: 100% (107/107), done.
Delta compression using up to 8 threads
Compressing objects: 100% (87/87), done.
Writing objects: 100% (100/100), 111.28 KiB | 12.36 MiB/s, done.
Total 100 (delta 61), reused 33 (delta 12), pack-reused 0
remote: Resolving deltas: 100% (61/61), completed with 7 local objects.
To github.com:haithem-souala/airbyte.git
 ! [remote rejected]       airbyte-cmc/master -> airbyte-cmc/master (permission denied)

There is a doc from Github that might help, can you give us the edit permission?

haithem-souala · 2022-10-28T19:32:46Z

Done!

YiyangLi · 2022-10-28T19:40:13Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3348075729
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3348075729
🐛 https://gradle.com/s/nte33asoicm2y

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
=================== 2 failed, 22 passed, 3 skipped in 35.03s ===================

YiyangLi · 2022-10-28T20:16:08Z

{"type": "LOG", "log": {"level": "INFO", "message": "Starting syncing SourceCoinmarketcap"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Syncing stream: quotes "}}
{"type": "LOG", "log": {"level": "ERROR", "message": "Encountered an exception while reading stream quotes\nTraceback (most recent call last):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 113, in read\n    yield from self._read_stream(\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 182, in _read_stream\n    for record in record_iterator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 285, in _read_full_refresh\n    for record in records:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 115, in read_records\n    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 358, in read_records\n    for r in records_generator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 428, in read_records\n    yield from self.parse_response(response, stream_state=stream_state, stream_slice=stream_slice)\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 314, in parse_response\n    raise ReadException(error_message)\nairbyte_cdk.sources.declarative.exceptions.ReadException: Request <PreparedRequest [GET]> failed with response <Response [400]>"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Finished syncing quotes"}}
{"type": "LOG", "log": {"level": "INFO", "message": "SourceCoinmarketcap runtimes:\nSyncing stream quotes 0:00:00.189720"}}
{"type": "LOG", "log": {"level": "FATAL", "message": "Request <PreparedRequest [GET]> failed with response <Response [400]>\nTraceback (most recent call last):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-coinmarketcap/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n    for message in generator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 127, in read\n    raise e\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 113, in read\n    yield from self._read_stream(\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 182, in _read_stream\n    for record in record_iterator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 285, in _read_full_refresh\n    for record in records:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 115, in read_records\n    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 358, in read_records\n    for r in records_generator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 428, in read_records\n    yield from self.parse_response(response, stream_state=stream_state, stream_slice=stream_slice)\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 314, in parse_response\n    raise ReadException(error_message)\nairbyte_cdk.sources.declarative.exceptions.ReadException: Request <PreparedRequest [GET]> failed with response <Response [400]>"}}
{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1666988085219.6182, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "Request <PreparedRequest [GET]> failed with response <Response [400]>", "stack_trace": "Traceback (most recent call last):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-coinmarketcap/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n    for message in generator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 127, in read\n    raise e\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 113, in read\n    yield from self._read_stream(\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 182, in _read_stream\n    for record in record_iterator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 285, in _read_full_refresh\n    for record in records:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 115, in read_records\n    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 358, in read_records\n    for r in records_generator:\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 428, in read_records\n    yield from self.parse_response(response, stream_state=stream_state, stream_slice=stream_slice)\n  File \"/Users/bartdev/Playground/airbyte/airbyte-integrations/connectors/source-public-apis/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 314, in parse_response\n    raise ReadException(error_message)\nairbyte_cdk.sources.declarative.exceptions.ReadException: Request <PreparedRequest [GET]> failed with response <Response [400]>\n", "failure_type": "system_error"}}}

I think I hit the rate limit or something, I will run the acceptance tests later. I don't plan to register a new account. Instead, I will wait for about 4 hours, which will reset the rate limit. If it's good, we are going to merge it.

YiyangLi · 2022-10-29T03:24:02Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3349867395
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3349867395
🐛 https://gradle.com/s/5u6luu4c7e674

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
=================== 2 failed, 22 passed, 3 skipped in 41.47s ===================

YiyangLi · 2022-10-31T16:43:13Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3363079095
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3363079095
🐛 https://gradle.com/s/jpjpybm75y3am

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
=================== 2 failed, 22 passed, 3 skipped in 37.83s ===================

YiyangLi · 2022-10-31T17:54:31Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3363527619
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3363527619
🐛

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============= 1 failed, 23 passed, 3 skipped in 138.82s (0:02:18) ==============

YiyangLi · 2022-10-31T20:01:09Z

/test connector=connectors/source-coinmarketcap

YiyangLi · 2022-10-31T21:14:24Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3364817555
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3364817555
🐛 https://gradle.com/s/3xiud2jdhps3q

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError:  Rec...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============== 2 failed, 22 passed, 3 skipped in 84.24s (0:01:24) ==============

YiyangLi · 2022-10-31T21:45:59Z

run out of credits to call the API again, will try it again 2 hours later. The tests are good locally.

YiyangLi · 2022-11-01T00:51:52Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3365901111
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3365901111
🐛 https://gradle.com/s/i4totn53xjhew

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError:  Rec...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============== 1 failed, 23 passed, 3 skipped in 62.17s (0:01:02) ==============

- the crypto market is VERY volatile, the data change between 2 full imports when the test is run

YiyangLi · 2022-11-01T02:05:33Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3366220287
❌ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3366220287
🐛 https://gradle.com/s/pnwmg6lrl6gvu

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============= 1 failed, 23 passed, 3 skipped in 115.86s (0:01:55) ==============

YiyangLi · 2022-11-01T02:51:19Z

/test connector=connectors/source-coinmarketcap

🕑 connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3366420243
✅ connectors/source-coinmarketcap https://github.com/airbytehq/airbyte/actions/runs/3366420243
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       133      3    98%   87, 93, 230
	 source_acceptance_test/conftest.py                     196     97    51%   35, 41-43, 48, 54, 60, 66, 72-74, 80-95, 100, 105-107, 113-115, 121-122, 127-128, 133, 139, 148-157, 163-168, 232, 238, 244-250, 258-263, 271-284, 289-295, 302-313, 320-336
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              329    106    68%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 377-379, 382, 447-455, 484-485, 491, 494, 530-540, 553-578
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/config_migration.py        23     23     0%   5-37
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1479    376    75%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: This connector does not implement incremental sync
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
======================== 24 passed, 3 skipped in 58.75s ========================

YiyangLi · 2022-11-01T04:15:01Z

/publish connector=connectors/source-coinmarketcap

🕑 Publishing the following connectors:
connectors/source-coinmarketcap
https://github.com/airbytehq/airbyte/actions/runs/3366765082

Connector	Did it publish?	Were definitions generated?
connectors/source-coinmarketcap	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* init commit * add docs * add docs * Delete logs.txt * add items * fix comments * fix comment * fix acceptance test * remove *state.json used for incremental imports test * Add ignored_fields on listing to get the acceptance test pass - the crypto market is VERY volatile, the data change between 2 full imports when the test is run * manually generate source_specs.yaml for coinmarketcap Co-authored-by: Yiyang Li <yiyangli2010@gmail.com>

init commit

b352d18

github-actions bot added the area/connectors Connector related issues label Oct 27, 2022

octavia-squidington-iv added bounty community connectors/source/coinmarketcap labels Oct 27, 2022

add docs

9fe5396

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 27, 2022

haithem-souala and others added 2 commits October 27, 2022 20:12

add docs

891e274

Delete logs.txt

9948ed3

haithem-souala mentioned this pull request Oct 27, 2022

New Source - Coinmarketcap airbytehq/connector-contest#202

Closed

RealChrisSean added the hacktober label Oct 27, 2022

marcosmarxm assigned YiyangLi Oct 27, 2022

YiyangLi reviewed Oct 28, 2022

View reviewed changes

airbyte-integrations/connectors/source-coinmarketcap/source_coinmarketcap/spec.yaml Show resolved Hide resolved

haithem-souala added 2 commits October 28, 2022 10:03

add items

822659d

Merge branch 'master' of github.com:airbyte-cmc/airbyte

e8ab6d0

haithem-souala requested a review from YiyangLi October 28, 2022 08:04

YiyangLi reviewed Oct 28, 2022

View reviewed changes

haithem-souala added 2 commits October 28, 2022 18:41

fix comments

6a1cc4a

fix comment

e412a94

haithem-souala requested a review from YiyangLi October 28, 2022 16:49

fix acceptance test

4183dfb

YiyangLi changed the title ~~🎉 New Source: Coinmarketcap~~ 🎉 New Source: Coinmarketcap [low-code CDK] Oct 28, 2022

Merge branch 'airbytehq:master' into master

3b86d66

remove *state.json used for incremental imports test

cbd0060

YiyangLi added 2 commits October 28, 2022 13:18

Merge branch 'master' into master

c708ecf

Merge branch 'master' into master

b41d8c9

Merge branch 'master' into master

022aff8

sajarin added the bounty-XL Maintainer program: claimable extra large bounty PR label Oct 31, 2022

bleonard added the reward-200 label Oct 31, 2022

YiyangLi added 2 commits October 31, 2022 18:53

Add ignored_fields on listing to get the acceptance test pass

dc13090

- the crypto market is VERY volatile, the data change between 2 full imports when the test is run

Merge branch 'master' into master

722c925

YiyangLi approved these changes Nov 1, 2022

View reviewed changes

manually generate source_specs.yaml for coinmarketcap

30494fa

marcosmarxm approved these changes Nov 1, 2022

View reviewed changes

marcosmarxm merged commit aff9963 into airbytehq:master Nov 1, 2022

octavia-squidington-iii mentioned this pull request Nov 1, 2022

Bump Airbyte version from 0.40.17 to 0.40.18 #18827

Merged

RealChrisSean added the low-code label Nov 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Coinmarketcap [low-code CDK] #18565

🎉 New Source: Coinmarketcap [low-code CDK] #18565

haithem-souala commented Oct 27, 2022 •

edited

Loading

marcosmarxm commented Oct 27, 2022

YiyangLi commented Oct 28, 2022

YiyangLi commented Oct 28, 2022 •

edited

Loading

haithem-souala commented Oct 28, 2022

YiyangLi commented Oct 28, 2022

haithem-souala commented Oct 28, 2022

YiyangLi commented Oct 28, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 28, 2022 •

edited

Loading

haithem-souala commented Oct 28, 2022

YiyangLi commented Oct 28, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 28, 2022

YiyangLi commented Oct 29, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022

YiyangLi commented Oct 31, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

🎉 New Source: Coinmarketcap [low-code CDK] #18565

🎉 New Source: Coinmarketcap [low-code CDK] #18565

Conversation

haithem-souala commented Oct 27, 2022 • edited Loading

What

How

Streams

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

marcosmarxm commented Oct 27, 2022

YiyangLi commented Oct 28, 2022

YiyangLi commented Oct 28, 2022 • edited Loading

haithem-souala commented Oct 28, 2022

YiyangLi commented Oct 28, 2022

haithem-souala commented Oct 28, 2022

YiyangLi commented Oct 28, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Oct 28, 2022 • edited Loading

haithem-souala commented Oct 28, 2022

YiyangLi commented Oct 28, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Oct 28, 2022

YiyangLi commented Oct 29, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Oct 31, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Oct 31, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Oct 31, 2022

YiyangLi commented Oct 31, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Oct 31, 2022

YiyangLi commented Nov 1, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Nov 1, 2022 • edited by github-actions bot Loading

Build Failed

YiyangLi commented Nov 1, 2022 • edited by github-actions bot Loading

Build Passed

YiyangLi commented Nov 1, 2022 • edited by github-actions bot Loading

haithem-souala commented Oct 27, 2022 •

edited

Loading

YiyangLi commented Oct 28, 2022 •

edited

Loading

YiyangLi commented Oct 28, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 28, 2022 •

edited

Loading

YiyangLi commented Oct 28, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 29, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Oct 31, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading

YiyangLi commented Nov 1, 2022 •

edited by github-actions bot

Loading