Source Jira: version 0.3.1 break change with previous one

[JCWahoo]

Environment

• Airbyte version: 0.40.26
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: JIRA 0.3.1
• Destination Connector and version: Redshift Latest
• Step where error happened: Sync

Current Behavior

Various errors - connection closes, error parsing Sprints stream, date/time validation errors, unable to complete a sync. Upgrading from 0.3.0 is a breaking change

Expected Behavior

Connector syncs successfully

[marcosmarxm]

@JCWahoo do you have logs to share?

@grubberr tagging you because last version was updated by you on #20128

[JCWahoo]

Trying to rollback and repair the connection then will attempt to retrieve logs

…

On Tue, Dec 20, 2022 at 11:52 AM Marcos Marx ***@***.***> wrote: @JCWahoo <https://github.com/JCWahoo> do you have logs to share? @grubberr <https://github.com/grubberr> tagging you because last version was updated by you on #20128 <#20128> — Reply to this email directly, view it on GitHub <#20719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUAW7TDZ6ORDUDPU3DGFTODWOHP6HANCNFSM6AAAAAATET7QEM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JCWahoo]

2022-12-20 17:44:32 �[44msource�[0m > Encountered an exception while reading stream sprints Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 111, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 180, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 290, in _read_full_refresh for record_data_or_message in record_data_or_messages: File "/airbyte/integration_code/source_jira/streams.py", line 1067, in read_records yield from super().read_records(stream_slice={"board_id": board["id"]}, **kwargs) File "/airbyte/integration_code/source_jira/streams.py", line 87, in read_records yield from super().read_records(**kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 413, in read_records yield from self._read_pages( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 432, in _read_pages next_page_token = self.next_page_token(response) File "/airbyte/integration_code/source_jira/streams.py", line 51, in next_page_token if startAt < response_json["total"]: KeyError: 'total' 2022-12-20 17:44:32 �[44msource�[0m > Finished syncing sprints 2022-12-20 17:44:32 �[44msource�[0m > SourceJira runtimes: Syncing stream groups 0:00:00.100003

[JCWahoo]

2022-12-20 17:44:32 �[44msource�[0m > 'total' Traceback (most recent call last): File "/airbyte/integration_code/main.py", line 13, in <module> launch(source, sys.argv[1:]) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch for message in source_entrypoint.run(parsed_args): File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 122, in run for message in generator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 125, in read raise e File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 111, in read yield from self._read_stream( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 180, in _read_stream for record in record_iterator: File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 290, in _read_full_refresh for record_data_or_message in record_data_or_messages: File "/airbyte/integration_code/source_jira/streams.py", line 1067, in read_records yield from super().read_records(stream_slice={"board_id": board["id"]}, **kwargs) File "/airbyte/integration_code/source_jira/streams.py", line 87, in read_records yield from super().read_records(**kwargs) File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 413, in read_records yield from self._read_pages( File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 432, in _read_pages next_page_token = self.next_page_token(response) File "/airbyte/integration_code/source_jira/streams.py", line 51, in next_page_token if startAt < response_json["total"]: KeyError: 'total' 2022-12-20 17:44:33 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):359 - Total records read: 55520 (1 GB) 2022-12-20 17:44:33 �[33mWARN�[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$5):362 - Schema validation errors found for stream _issues. Error messages: [$.fields.created is of an incorrect type. Expected it to be date-time, $.created is of an incorrect type. Expected it to be date-time, $.updated is of an incorrect type. Expected it to be date-time, $.fields.updated is of an incorrect type. Expected it to be date-time] 2022-12-20 17:44:33 �[43mdestination�[0m > Airbyte message consumer: succeeded. 2022-12-20 17:44:33 �[43mdestination�[0m > A total of 1419 record(s) of data from stream AirbyteStreamNameNamespacePair{name='issues', namespace='jira_airbyte_3'} were invalid and were ignored.

[JCWahoo]

2022-12-20 17:45:33 �[1;31mERROR�[m i.a.w.g.DefaultReplicationWorker(replicate):246 - Sync worker failed. java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.general.DefaultReplicationWorker.replicate(DefaultReplicationWorker.java:238) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:164) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:84) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.26.jar:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?] Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:144) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.replicate(DefaultReplicationWorker.java:185) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:164) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:84) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.26.jar:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?] Suppressed: io.airbyte.workers.exception.WorkerException: Destination process exit with code 143. This warning is normal if the job was cancelled. at io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:124) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.replicate(DefaultReplicationWorker.java:185) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:164) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:84) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.26.jar:?] at java.lang.Thread.run(Thread.java:1589) ~[?:?] Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1 at io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromSrcAndWriteToDstRunnable$6(DefaultReplicationWorker.java:373) ~[io.airbyte-airbyte-commons-worker-0.40.26.jar:?] at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] ... 1 more 2022-12-20 17:45:33 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(prepStateForLaterSaving):501 - Source output at least one state message 2022-12-20 17:45:33 �[33mWARN�[m i.a.w.g.DefaultReplicationWorker(prepStateForLaterSaving):514 - State capture: No state retained. 2022-12-20 17:45:33 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(getReplicationOutput):434 - sync summary: { "status" : "failed", "recordsSynced" : 55518, "bytesSynced" : 1437057856, "startTime" : 1671555273265, "endTime" : 1671558333594, "totalStats" : { "recordsEmitted" : 55518, "bytesEmitted" : 1437057856, "sourceStateMessagesEmitted" : 1, "destinationStateMessagesEmitted" : 0, "recordsCommitted" : 0, "meanSecondsBeforeSourceStateMessageEmitted" : 2962, "maxSecondsBeforeSourceStateMessageEmitted" : 2962, "maxSecondsBetweenStateMessageEmittedandCommitted" : 0, "meanSecondsBetweenStateMessageEmittedandCommitted" : 0, "replicationStartTime" : 1671555273265, "replicationEndTime" : 1671558333593, "sourceReadStartTime" : 1671555273316, "sourceReadEndTime" : 1671558273342, "destinationWriteStartTime" : 1671555273426, "destinationWriteEndTime" : 0 }, "streamStats" : [ { "streamName" : "project_versions", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 13595, "bytesEmitted" : 2258511 } }, { "streamName" : "issue_priorities", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 5, "bytesEmitted" : 1325 } }, { "streamName" : "issues", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 41235, "bytesEmitted" : 1434609156 } }, { "streamName" : "issue_fields", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 192, "bytesEmitted" : 63481 } }, { "streamName" : "projects", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 106, "bytesEmitted" : 91293 } }, { "streamName" : "project_types", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 4, "bytesEmitted" : 5704 } }, { "streamName" : "labels", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 296, "bytesEmitted" : 5995 } }, { "streamName" : "sprints", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 50, "bytesEmitted" : 18267 } }, { "streamName" : "issue_link_types", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 11, "bytesEmitted" : 1657 } }, { "streamName" : "groups", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 20, "bytesEmitted" : 1553 } }, { "streamName" : "issue_resolutions", "streamNamespace" : "jira_airbyte_3", "stats" : { "recordsEmitted" : 4, "bytesEmitted" : 914 } } ] } 2022-12-20 17:45:33 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(getReplicationOutput):435 - failures: [ { "failureOrigin" : "source", "failureType" : "system_error", "internalMessage" : "'total'", "externalMessage" : "Something went wrong in the connector. See the logs for more details.", "metadata" : { "attemptNumber" : 0, "jobId" : 21608, "from_trace_message" : true, "connector_command" : "read" }, "stacktrace" : "Traceback (most recent call last):\n File \"/airbyte/integration_code/main.py\", line 13, in <module>\n launch(source, sys.argv[1:])\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n for message in source_entrypoint.run(parsed_args):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 122, in run\n for message in generator:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 125, in read\n raise e\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 111, in read\n yield from self._read_stream(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 180, in _read_stream\n for record in record_iterator:\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py\", line 290, in _read_full_refresh\n for record_data_or_message in record_data_or_messages:\n File \"/airbyte/integration_code/source_jira/streams.py\", line 1067, in read_records\n yield from super().read_records(stream_slice={\"board_id\": board[\"id\"]}, **kwargs)\n File \"/airbyte/integration_code/source_jira/streams.py\", line 87, in read_records\n yield from super().read_records(**kwargs)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 413, in read_records\n yield from self._read_pages(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py\", line 432, in _read_pages\n next_page_token = self.next_page_token(response)\n File \"/airbyte/integration_code/source_jira/streams.py\", line 51, in next_page_token\n if startAt < response_json[\"total\"]:\nKeyError: 'total'\n", "timestamp" : 1671558272994 }, { "failureOrigin" : "source", "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1", "externalMessage" : "Something went wrong within the source connector", "metadata" : { "attemptNumber" : 0, "jobId" : 21608, "connector_command" : "read" }, "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromSrcAndWriteToDstRunnable$6(DefaultReplicationWorker.java:373)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\n", "timestamp" : 1671558273343 }, { "failureOrigin" : "replication", "internalMessage" : "java.lang.RuntimeException: java.io.UncheckedIOException: java.io.IOException: Stream closed", "externalMessage" : "Something went wrong during replication", "metadata" : { "attemptNumber" : 0, "jobId" : 21608 }, "stacktrace" : "java.util.concurrent.CompletionException: java.lang.RuntimeException: java.io.UncheckedIOException: java.io.IOException: Stream closed\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: java.lang.RuntimeException: java.io.UncheckedIOException: java.io.IOException: Stream closed\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromDstRunnable$4(DefaultReplicationWorker.java:289)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: java.io.UncheckedIOException: java.io.IOException: Stream closed\n\tat java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:685)\n\tat java.base/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1931)\n\tat java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:292)\n\tat java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)\n\tat java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161)\n\tat java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:298)\n\tat java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.isFinished(DefaultAirbyteDestination.java:150)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$readFromDstRunnable$4(DefaultReplicationWorker.java:262)\n\t... 4 more\nCaused by: java.io.IOException: Stream closed\n\tat java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:172)\n\tat java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:373)\n\tat java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:367)\n\tat java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:333)\n\tat java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:376)\n\tat java.base/sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:219)\n\tat java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:173)\n\tat java.base/java.io.InputStreamReader.read(InputStreamReader.java:189)\n\tat java.base/java.io.BufferedReader.fill(BufferedReader.java:161)\n\tat java.base/java.io.BufferedReader.implReadLine(BufferedReader.java:371)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:348)\n\tat java.base/java.io.BufferedReader.readLine(BufferedReader.java:437)\n\tat java.base/java.io.BufferedReader$1.hasNext(BufferedReader.java:682)\n\t... 12 more\n", "timestamp" : 1671558333594 } ]

[grubberr]

@JCWahoo I have prepared PR #20859 which fix pagination problem
we are waiting code unfreeze