🐛Source Pardot: Fix minor schema issues, improve data integrity by not using split-up interval, improve error retries

[justbeez]

What

This PR resolves several minor bugs related to the Pardot/Marketing Cloud Account Engagement connector:

• Fix typing/deduping warnings by refining schemas
• Remove split-up interval user input, which doesn't play nicely with pagination on very large Pardot accounts (leading to potential gaps in synced records)
• Remove unneeded conditional parameters on custom_redirects
• Adds a specific handler for the daily rate limits and adjust the default retry count/backoff strategy for other recoverable errors

How

Changes to all streams:

1. Removed split-up interval option (which could lead to small data gaps on very high-volume accounts)

2. Added a constant backoff when the daily API limit is hit

3. Adjusted the default exponential backoff try count and factor to better handle transient errors seen in testing

Stream-specific updates:

1. campaigns: added schema for missing fields to ensure type consistency across stream-level keys

2. custom_redirects: fixed type of salesforceId (which is a string, unlike the Pardot native IDs); applied date-time type to createdAt; removed unneeded conditional query parameters

3. email_clicks: switched created_at Airbyte type to timestamp_without_timezone (the v4 API doesn't return timezone)

4. folders: applied date-time type to createdAt and updatedAt

5. list_emails: applied date-time type to sentAt

6. account: remove unneeded (but non-erroring) required field, createdAt

7. custom_fields: Change fieldId to string type to prevent error; remove unneeded (but non-erroring) required field (createdAt)

8. dynamic_contents: Make basedOnProspectApiFieldId string to avoid error

Review guide

Manifest and docs also updated accordingly.

User Impact

These are all non-breaking changes, and some fix errors that occur depending on the values of certain fields. Previously enabling these streams could result in a source error.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

@justbeez is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.

[marcosmarxm]

@DanyloGL and @ChristoGrab, could you please check the contribution made by Justin? I believe you are assisting him on Slack.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 17, 2025 10:05pm

[natikgadzhi]

I'm mostly happy! Let's see if basic CI runs well and merge.

[justbeez]

@natikgadzhi Need anything from me to get this one landed?

[natikgadzhi]

Just some verbal encouragement <3

[natikgadzhi]

Hopefully kicked-off the CI, but will return to this later today and merge.

[ChristoGrab]

There was a mismatch between the manifest and base image versions that caused a build failure; just bumped the base image in metadata, will keep an eye on the results

[natikgadzhi]

Alright, there is a build error because the manifest wants CDK version that is newer than what is in the base image.

We should update the base image to 6.22+ and that will work.

[natikgadzhi]

Merging because the failure is complaining that we should have CATs