Source Stripe: Add new stream (credit_notes)

[mrhallak]

What

This is to add a new stream called credit_notes, this should solve this issue

How

• Adding the credit_notes schema to the streams.py and schemas

Recommended reading order

1. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?
New stream

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Connector version is set to 0.0.1
  • Dockerfile has version 0.0.1
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)

• Secrets in the connector's spec are annotated with airbyte_secret

• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.

• Code reviews completed

• Connector version has been incremented

  • Version has been bumped according to our Semantic Versioning for Connectors guidelines
  • Dockerfile has updated version

• Documentation updated

  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example

• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

[mrhallak]

Hey Airbyte team, I'm not sure if I've completed all the required steps here. If not, happy to make some changes with your guidance 🙏🏻

[igrankova]

Hi, @erohmensing
The credit notes are available on our account, I don't see any restrictions in permissions.
I see that the creds to Stripe test account disappeared from the LastPass, I'm wondering whether someone deleted them mistakenly.

[erohmensing]

@igrankova Ah, I'm referring to

Permission credit_note_read, credit_note_write Grants access to Credit Notes. This permission also implies the following permissions: invoice_read

from here - not sure if that changes anything.

I see that the creds to Stripe test account disappeared from the LastPass, I'm wondering whether someone deleted them mistakenly.

That seems bad. Do you mean just the config used for the test syncs? We should be able to get that back from GSM.

[igrankova]

@erohmensing
Could you please clarify what kind of problem we are expecting with this stream? I'm sure we have all needed permissions because I see the logs where there were made 2 successful requests of the credit notes yesterday:

I've tried to find what scopes are selected in the OAuth app, but there's just a client ID and redirect URLs. Don't see the scopes in the settings.
Found this description of the Secrets and Scopes here, just 2 types: user and account scope. Account scoped secrets: “Foo API key” secret for App A, “Bar API key” for App B, user scoped secrets: “OAuth access token”, “OAuth refresh token”. Not sure how the granular scopes work for the app.

I've restored the LastPass password using GSM and my browser passwords storage, but I'm worrying that it's not the first time when someone deletes the test accounts creds. Is there any option to restrict delete permissions for the LP users for the shared-integration-tests folder?

[erohmensing]

@igrankova I don't expect anything - just wanted to point out that you had mentioned explicit permissions in the original issue. Maybe you were just being extra clear about what permissions it requires, and we already request those of the user :)

If you're confident that nothing else is needed to add the new stream, let's go forward - would be great if someone from your team could take a look! It looks like there are already 3 rows in our test instance, so we should be able to test the new stream in the acceptance tests

Is there any option to restrict delete permissions for the LP users for the shared-integration-tests folder?

Great q, I will ask this of the team who owns this

[igrankova]

@erohmensing, oh! I thought that there's a problem with the tests. I see. The permissions I mentioned were added when I've created the issue and I've tried to add as more info as I can for the dev for investigation. I wasn't familiar with the API because it was a bulk issues creation according to the MLP researches.
I'll ask someone from the team to check or try to make requests with the Postman tomorrow by myself.

[igrankova]

@erohmensing
Tested the test-mode, credit notes are available:

[marcosmarxm]

@archangelic this one has a lot of conflicts. Can you handle them and add the expected records?

[marcosmarxm]

@archangelic This class should similar to this one:

airbyte/airbyte-integrations/connectors/source-stripe/source_stripe/streams.py

Lines 240 to 257 in 5707e47

	class EarlyFraudWarnings(StripeStream):
	"""
	API docs: https://stripe.com/docs/api/radar/early_fraud_warnings/list
	"""
	def request_params(
	self,
	stream_state: Mapping[str, Any],
	stream_slice: Mapping[str, Any] = None,
	next_page_token: Mapping[str, Any] = None,
	) -> MutableMapping[str, Any]:
	params = {}
	if next_page_token:
	params.update(next_page_token)
	def path(self, **kwargs):
	return "radar/early_fraud_warnings"

because CreditNotes doesn't support incremental.

[marcosmarxm]

The difference is CreditNotes allow you to set the limit parameter so you can keep that in request_params.

[marcosmarxm]

This stream doesn't support incremental.
The sandbox record is only available for connected account config.
It's possible to retrieve the record using the credentials + curl but the read is not working.
The connector adds the header with the account id, this doesn't work using curl the response is [] so we must override this function. I made all this changes but wasn't able to retrieve the record yet, probably because the Stripe connector has a custom read_records functions using specific stream slices.

[marcosmarxm]

Closed in favor of #27132