CHANGELOG.md

v4.1.9 [2022-11-30]

Bug fixes

• Fix hydrate function caching using WithCache for aggregator connections. (#460)

v4.1.8 [2022-09-08]

Bug fixes

• Remove explicit setting of the open-files limit, which was using a default which was too small. Instead, make use of the fact that Go 1.19 now sets the soft file limit to the hard limit automatically. (#444)

v4.1.7 [2022-11-08]

Bug fixes

• Move IsCancelled back to plugin package.

v4.1.6 [2022-09-02]

Bug fixes

• Fix issue where multi-region queries with a region where-clause do not have the region in the cache key, causing incorrect cache hits. (#402)

v4.1.5 [2022-08-31]

Bug fixes

• Fix ConnectionCache.SetWithTTL. (#399)
• When connection config changes, store updated connection config before calling ConnectionConfigChangedFunc. (#401)

v4.1.4 [2022-08-26]

Remove warning logs

v4.1.3 [2022-08-26]

Bug fixes

• Fix timeout waiting for pending cached item - if pending item has error, all queries waiting for that pending item now return error rather than rerunning the query. (#396)

v4.1.2 [2022-08-25]

Bug fixes

• Fix queries sometimes hanging when multiple scans are accessing the cache. (#391)

v4.1.1 [2022-08-24]

Bug fixes

• Fix concurrent map access crash for Plugin.connectionCacheMap. (#389)

v4.1.0 [2022-08-24]

What's new

• Add Plugin propertyConnectionConfigChangedFunc. This is a callback function invoked when the connection config changes. The default implementation clears the connection cache and query cache for the changed connection. (#387)

v4.0.2 [2022-08-22]

Bug fixes

• Fix Get calls stalling due to an attempt to write to an unbuffered channel. (#382)

v4.0.1 [2022-08-11]

Bug fixes

• Fix UpdateConnectionConfigs not setting the connection config. (#375)
• Fix query results with zero rows not being cached, leading to timeouts and failure to load pending cache results. (#372)
• Fix duplicate results returned from cache. (#371)
• Remove max concurrent row semaphore. Max concurrent row limiting is disabled by default, and under certain circumstances has been seen to cause a NRE. Removed for now until more detailed benchmarking can be done to justify re-adding. (#369)
• Fix parent-child listing, which was broken in v4.0.x (#378)

v4.0.0 [2022-08-04]

What's new

• A single plugin instance now supports multiple connections, as opposed to an instance being created per connection. (#365)
• Memory usage has been substantially reduced, particularly when streaming high row counts. (#366)
• Allow control of maximum cache memory usage. (#302)
• QueryData functions StreamListItem and StreamLeafListItem are now variadic, and accept multiple items to passed in a single call, simplifying the streaming of a page of data. (#341)

Breaking changes

• Go version updated to 1.19
• Plugin property TableMapFunc has changed signature. This is the function which is called for plugins with dynamic schema to return their table schema. Note that the parameter connection has been added. This may be used in place of the removed Plugin.Connection property.

The new signature is:

func(ctx context.Context, connection *Connection) (map[string]*Table, error)

• Plugin properties Connection, and Schema have been removed, and new property ConnectionMap added.
  This is a map of ConnectionData objects, keyed by connection. This is needed as each plugin instance may support multiple connections. ConnectionData looks as follows

type ConnectionData struct {
	// TableMap is a map of all the tables in the plugin, keyed by the table name
	TableMap map[string]*Table
	// connection this plugin is instantiated for
	Connection *Connection
	// schema - this may be connection specific for dynamic schemas
	Schema map[string]*proto.TableSchema
}

• ConnectionManager has been renamed to ConnectionCache. As the plugin can support multiple connections, each connection has its own ConnectionCache, which is a wrapper round an single underlying connection data cache.

NOTE: the property QueryData.ConnectionManager has been retained for comptibility reasons - this will be deprecated in a future version

v3.3.2 [2022-07-11]

What's new

• Add MaxConcurrency to GetConfig - for use when using the Get hydrate as a column hydrate function. (#353)
• Validate table Name property matches key in plugin's TableMap. (#355)

v3.3.1 [2022-6-30]

Bug fixes

• Deprecated ShouldIgnoreError property is not being respected if defined in plugin.DefaultGetConfig. (#347)
• If cached item has limit, quals must match exactly to be considered a cache hit. (#345)

v3.3.0 [2022-06-22]

What's new

• Add support for Open Telemetry. (#337)
• Return query metadata with the scan result, such as the number of hydrate functions called and the cache status. (#338)

Bug fixes

• Incomplete results should not be added to cache if the context is cancelled. (#339)
• Avoid deadlock after panic during newQueryData. (#332)

v3.2.0 [2022-05-20]

What's new

• Deprecate ShouldIgnoreError and ShouldRetryError.

  Add instead ShouldRetryErrorFunc and a new IgnoreConfig containing ShouldIgnoreErrorFunc.

  These functions receive as args the context, QueryData and HydrateData to allow access to connection config and other context data. (#261)

Bug fixes

• Fix potential transform function casting errors caused by empty hydrate items. If no hydrate data is available, do not call transform functions. (#325)
• Fix the sdk not respecting the DefaultGetConfig when resolving the ShouldIgnoreError function. (#319)

v3.1.0 [2022-03-30]

What's new

• Add CacheMatch property to KeyColumn, to support key columns which require exact matching to be considered a cache hit. (#298)
• Add table and plugin level defaults for ShouldIgnoreError and RetryConfig. (#257)

v3.0.1 [2022-03-10]

Bug fixes

• Fix issue when executing list calls with 'in' clauses, key column values passed in Quals map are incorrect. (#294)

v3.0.0 [2022-03-09]

What's new

• Add support for ltree column type. (#248)
• Add support for inet column type. (#248)

v2.2.0 [2022-03-30]

What's new

• Add CacheMatch property to KeyColumn, to support key columns which require exact matching to be considered a cache hit. (#298)
• Add table and plugin level defaults for ShouldIgnoreError and RetryConfig. (#257)

v2.1.1 [2022-03-10]

Bug fixes

• Fix issue when executing list calls with 'in' clauses, key column values passed in Quals map are incorrect. (#294)

v2.1.0 [2022-03-04]

What's new

• Add support for is null and is not null quals. (#286)

Bug fixes

• Fix list call not respecting in qual if list config has multiple key columns. (#275)

v2.0.3 [2022-02-14]

What's new

• Update all references to use github.com/turbot/steampipe-plugin-sdk/v2. (#272)

v2.0.2 [2022-02-14]

What's new

• Update package name to github.com/turbot/steampipe-plugin-sdk/v2. (#272)

v2.0.1 [2022-02-10]

Bug fixes

• Fix requires hydrate data from xxxx but none is available error - avoid mutating Column objects, which may be shared between tables. (#259)

v2.0.0 [2022-02-04]

Changed behaviour

• A _ctx column is now added to all tables. This is a JSON field which specified the Steampipe connection name. (#246)

Bug fixes

• Fix a query cache bug which under very specific circumstances would lead to an incomplete data set being returned. (#254)

v1.8.3 [2021-12-23]

What's new

• Updated missing required quals error to include table name. (#166)
• Move setting R Limit to OS specific code to allow compilation on Windows systems.
• Update makefile and GRPCServer to support protoc-gen-go-grpc 1.1.0_2.

Bug fixes

• Fix 'in' clause not being correctly evaluated when more than one key column qual is specified. (#239)
• Fix invalid memory address error when joining on a column with null value. (#233)
• Avoid adding duplicate quals to KeyColumnQualMap. Was causing invalid key column errors. (#236)

v1.8.2 [2021-11-22]

What's new

• Query cache TTL defaults to 5 minutes and is increased to match the TTL of incoming queries. (#226)
• Set cache cost of items based on number of rows and columns inserted. (#227)
• Add logging for query cache usage. (#229)

v1.8.1 [2021-11-22]

What's new

• Query result caching now determines whether a cache request is a subset of an existing cached item, taking the quals into account. (#224)

Bug fixes

• Fix timeout waiting for pending cache transfer to complete. (#218)
• Support cancellation while waiting for pending cache transfer. (#219)

v1.8.0 [2021-11-10]

What's new

• Add support for query result caching with stampede prevention, and concurrent query execution. (#211)

v1.7.3 [2021-11-08]

Bug fixes

• FromField transform should return nil property value if property is nil, rather than nil interface value. (#212)

v1.7.2 [2021-11-03]

Bug fixes

• Fix KeyColumn Require and Operators properties not being set to default if a TableMapFunc is used. (#206)
• Remove unnecessary TableMapFunc validation from plugin - this breaks dynamic plugins when plugin manager is used. (#204)

v1.7.1 [2021-11-01]

Bug fixes

• FromValue transform should fall back to next property path if a property value is nil. (#197)
• Avoid nil data being passed to hydrate calls if plugin calls StreamListItem with a nil item. (#198)

v1.7.0 [2021-10-18]

What's new

• Add dynamic schema support - add SchemaMode property to Plugin. If this is set to dynamic, Steampipe will check for plugin schema changes on startup. (#195)

v1.6.2 [2021-10-08]

Bug fixes

• Fix in clause not working when the table has any_of key columns. (#189)
• Fix transform functions being called with null data when the Get call returns a null item but no error. (#186)

v1.6.1 [2021-09-21]

Bug fixes

• Pass context to table creation callback TableMapFunc. (#183)

v1.6.0 [2021-09-21]

What's new

• Add QueryStatus.RowsRemaining function which performs context cancellation and limit checking to determine how much more data the plugin should provide. (#177)
  • This function is used internally by StreamListItem to avoid calling hydrate functions once sufficient data has been returned.
  • It may also be called directly by the plugin to avoid retrieving unneeded data from the external API
• Enable plugin table creation to use the connection config. New plugin property has been added: TableMapFunc. This can be set to a table creation function which, when invoked, has access to the parsed connection config. (#180)

v1.5.1 [2021-09-13]

Bug fixes

• Fix get call returning nothing if there is an in clause for the key column, and matrix parameters are used. (#170)

v1.5.0 [2021-08-06]

What's new

• Add cache functions SetWithTTL and Delete. (#163)

Bug fixes

• When listing missing quals, only report required quals. (#159)

v1.4.1 [2021-07-20]

Bug fixes

• Extraneous log output removed

v1.4.0 [2021-07-20]

What's new

• Return all columns provided by hydrate functions, not just requested columns. (#156)

Bug fixes

• Fix matrix parameters not being added to the KeyColumns map passed to hydrate functions. (#151)

v1.3.1 [2021-07-15]

Bug fixes

• Fix crash caused by thread sync issue with multi-region union queries. (#149)
• When checking if StreamListItem is called from a non-list function, do not through errors for anonymous functions. (#147)

v1.3.0 [2021-07-09]

What's new

• When defining key columns it is now possible to specify supported operators for each column (defaulting to '='). (#121)
• Add support for optional key columns. (#112)
• Cancellation of GRPC stream is now reflected in the context passed to plugin operations, so plugins can easily handle cancellation by checking the context. (#17)
• Add IsCancelled() function to simplify plugins checking for a cancelled context. (#143)
• Add WithCache() function - if this is chained after a hydrate function definition, it enables plugin cache optimisation to avoid concurrent hydrate functions with same parameters. (#116)

Breaking changes

• The property QueryData.QueryContext.Quals has been renamed to QueryContext.UnsafeQuals. This property contains all quals, not just key columns. These quals should not be used for filtering data as this may break the FDW row data caching, which is keyed based on key column quals. Instead, use the new property QueryData.Qualswhich contains only key column quals. (#119)
• Plugins built with v1.3 of the sdk will only be loaded by Steampipe v0.6.2 onwards.

v0.2.10 [2021-06-09]

What's new

• Provide SDK transform to get a qual value. (#77)
• Change plugin license to Apache 2.0 (#488)

Bug fixes

• Fix Cache being recreated for every query. (#106)
• Improve error messages when hydrate function fails and when StreamListItem is called from anywhere other than a list function. (#70)
• Fix key column setting of AnyColumn only populating the first key column in the KeyColumnQuals map. (#101)
• Fix EnsureStringArray transform not working for input type *string. closes #92 (#100)
• Fix Steampipe hanging after hydrate error. (#103)
• Fix list call failing with error "get call requires an '=' qual". (#103)

v0.2.9 [2021-05-13]

What's new

• Export GetQualValue function (#98)

v0.2.8 [2021-05-06]

What's new

• Added support for retryable errors and ignorable errors inside getConfig and hydrateConfig. (#15)
• Update FromField transform to accept multiple arguments, which are tried in order. (#55)
• Add ProtocolVersion property to the plugin GetSchema response. (#94)

v0.2.7 [2021-03-31]

Bug fixes

• Multiregion queries should take region quals into account for 'get' calls. (#78)

v0.2.6 [2021-03-18]

re-tagged after pushing missing commit

v0.2.5 [2021-03-18]

What's new

• Improve the hcl diagnostic to error message conversion to improve parse failure messages. (#72)

v0.2.4 [2021-03-16]

What's new

• Include key column information in GetSchema response to support dynamic path key generation. (#57)
• Make get calls with 'in' clauses asynchronous. (#30)
• Remove need to call StreamLeafListItem for parent-child list calls. The child list function can now just cal StreamListItem. (#64)
• For parent-child list calls, store the parent list call results in RowData ParentItem property so they can be accessed by hydrate functions. (#65)

Bug fixes

• Queries with 'in' clause now work for list calls with required key columns. (#61)
• For Get or List calls with required key columns and 'in' clauses, incorrect quals are passed to hydrate calls. (#69)

v0.2.3 [2021-03-02]

Bug fixes

• Fix failure of Get calls which use ItemFromKey to provide a hydrate item. (#53)

v0.2.2 [2021-02-24]

What's new?

• Set the ulimit for plugin processes, respecting env var STEAMPIPE_ULIMIT. (#43)
• When displaying hcl errors, show the context if available. (#48)
• Only show concurrency summary if there is any summary data to show. (#47)

Bug fixes

• Fix error message not displaying when a query does not provide required get or listquals. (#42)

v0.2.1 [2021-02-18]

Bug fixes

• Remove "rc" from version number in the release branch. (#38)

v0.2.0 [2021-02-17]

What's new?

• Add support for multi-region querying. (#20)
• Add support for connection config. (#21)
• Add mechanism to limit max hydrate function concurrency. (#12)
• Update environment variables to use STEAMPIPE prefix. (#32)
• Provide dependency mechanism to allow Steampipe to know if a plugin uses a newer sdk version. (#25)

v0.1.1 [2021-02-11]

What's new?

• Add transforms StringArrayToMap and EnsureStringArray. (#3)

Bug fixes

• Fix ToLower and ToUpper transforms not working when input value is a with string pointers. (#13)
• Fix failure to report errors returned from Get function. (#23)