Skip to content

Commit

Permalink
Improve some logs w preamble and describe some logs in README
Browse files Browse the repository at this point in the history
  • Loading branch information
notfelineit committed Jan 15, 2025
1 parent df441e9 commit b90bf70
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 3 deletions.
44 changes: 43 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,4 +194,46 @@ fe8e2a3c-f91a-11ee-9812-82f5834c1ba7:1-46602355
```
6. Repeat this process for all your shards, if your database is sharded.

**Note**: Remember to prepend the prefix `MySQL56/` onto your starting GTIDs.
**Note**: Remember to prepend the prefix `MySQL56/` onto your starting GTIDs.

## Interpreting logs
Airbyte logs will include logs from the source (this library), the Airbyte connector, and the destination. All source logs will be prefixed with `[source]` and `PlanetScale Source ::`.

### Source-level logs

`Checking connection`
The Airbyte source is checking if it can connect to the PlanetScale database.

`state file detected, parsing provided file <file>`
A state JSON file was passed to the Airbyte source.

`Syncing from tabletType <tabletType>`
The tablet type the Airbyte source is syncing from.

### Table-level logs
`syncing stream <table> with sync mode <sync mode>`
Beginning sync for _table_ with the sync mode _sync mode_ (i.e. incremental)

### Shard-level logs
Shard-level logs will be prefixed with: `[<keyspace>:<tablet_type>:<table> shard : <shard>]`.

`peeking to see if there's any new rows`
The Airbyte source is checking if there are any new rows by comparing the _current_ cursor position with the _end_ (latest) cursor position.

`new rows found, syncing rows for 1m0s`
New rows detected (_current_ cursor position is behind _end_/latest cursor position)

`syncing rows with cursor <start cursor>`
Airbyte source will attempt to sync starting from `start cursor`.

`Syncing with cursor position : <start cursor>, using last known PK : <usingLastKnownPK>, stop cursor is : <stop cursor>`
Airbyte source will attempt to read from `start cursor` to `stop cursor`. `usingLastKnownPK` specifies whether or not the Airbyte source is syncing from a last known primary key.

`DEBUG: SyncRequest.Cells = [<cells>]`
Vitess cells that the Airbyte source will sync from.

`Finished reading <recordCount> records for table [<table>]`
Airbyte source has detected that the client has ended its connection. `recordCount` records were counted as having been sent to connector for table `table`.



4 changes: 2 additions & 2 deletions cmd/internal/planetscale_edge_database.go
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ func (p PlanetScaleEdgeDatabase) Read(ctx context.Context, w io.Writer, ps Plane
p.Logger.Log(LOGLEVEL_INFO, preamble+"no new rows found, exiting")
return TableCursorToSerializedCursor(currentPosition)
}
p.Logger.Log(LOGLEVEL_INFO, fmt.Sprintf("new rows found, syncing rows for %v", readDuration))
p.Logger.Log(LOGLEVEL_INFO, fmt.Sprintf(preamble+"new rows found, syncing rows for %v", readDuration))
p.Logger.Log(LOGLEVEL_INFO, fmt.Sprintf(preamble+"syncing rows with cursor [%v]", currentPosition))

currentPosition, recordCount, err := p.sync(ctx, currentPosition, latestCursorPosition, table, ps, tabletType, readDuration)
Expand Down Expand Up @@ -263,7 +263,7 @@ func (p PlanetScaleEdgeDatabase) sync(ctx context.Context, tc *psdbconnect.Table
tc.Position = ""
}

p.Logger.Log(LOGLEVEL_INFO, fmt.Sprintf("%sSyncing with cursor position : [%v], using last known PK : %v, stop cursor is : [%v]", preamble, tc.Position, tc.LastKnownPk != nil, stopPosition))
p.Logger.Log(LOGLEVEL_INFO, fmt.Sprintf("%sSyncing with cursor position: [%v], using last known PK: %v, stop cursor is: [%v]", preamble, tc.Position, tc.LastKnownPk != nil, stopPosition))

sReq := &psdbconnect.SyncRequest{
TableName: s.Name,
Expand Down

0 comments on commit b90bf70

Please sign in to comment.