#I285: removed empty objects and arrays from sample data #261
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
manjudr
approved these changes
Oct 24, 2024
manjudr
pushed a commit
that referenced
this pull request
Nov 19, 2024
* #OBS-I116: Dataset update API Dedupe and denorm test cases fixes * #OBS-I116: Dataset Create api test case fixes * #OBS-I116: Dataset update extraction config api test case fixes * #OBS-I116: Dataset update api test cases * #OBS-I116: fix: linting fixes * #OBS-I116: lint fixes * #OBS-I116: Dataset status transition test cases * #OBS-I116: feat: Test cases and linting fixes * #OBS-I116: feat: Dataset status transition test cases fix * #OBS-I141: added a new metric to sum the response time * #OBS-I141: modified the url variable and access dataset_id from params * #OBS-I141: added helper function to get dataset_id for error cases * #OBS-I141: added telemetry for v2 api's * #OBS-I141: added a new metric to sum the response time * #OBS-I141: modified the url variable and access dataset_id from params * #OBS-I141: added helper function to get dataset_id for error cases * #OBS-I141: added telemetry for v2 api's * #OBS-I141: added telemetry for v2 api's * #OBS-I143: feat: dataset publish changes to deploy flink connectors * #OBS-I141: removed metric for sum of response time * #OBS-I141: removed usage of builtin kafka methods from telemetry file * #OBS-I146: feat: Retire fix * Issue #SBCOSS-12 fix: convert all SQL raw queries to prepared statements * Issue #SBCOSS-12 fix: tags is an array, so requires empty json for null case; dataset draft deletion requires deletion of transformation and source config drafts * #SBCOSS-23: feat: dataset publish changes for redeployment * #OBS-I167 : read api changes while reading connectors according to v2 structure * #OBS-I173: fix: Ready to publish schema fix to expect connector configs as object and string * #OBS-I174: fix: Dataset read api fix to expect both v1 and v2 connectors * #OBS-I146: fix: Test case fix for read api * #OBS-I146: fix: Test case fix for read api changes * #OBS-I146: fix: status transition test cases * #OBS-I146: fix: Test case script fix * #OBS-I146: fix: Type error fix * #OBS-I141: removed metric for sum of response time * #OBS-I146: fix: Dataset read api test cases fixes * #OBS-I146: fix: Hudi spec generation test cases * #OBS-I146: fix: Test case and linting fix * merge commit * #OBS-I173: fix: Dataset web console required fixes * #OBS-I173: fix: Dataset update changes to accept type changes * #OBS-I167 : dataset read api changes to read live dataset source configs * #OBS-I146: fix: linting fix * #OBS-I146: fix: linting fix * #OBS-I167 : Added string or dict as type to connector_config * #OBS-I143: dataset publish changes fixes * #OBS-I167 : if dataset is empty return with error * #OBS-I143: inswert query fix * Issue #OBS-I144 fix: icon data as string; check default version * #OBS-143: fix: dataset publish fixes * #OBS-I167 : fix: removed duplicate code. * #OBS-I181 - Updated the event structure * #OBS-I164: added jwt token vwerification and access control to api's * #OBS-I164: added jwt token vwerification and access control to api's * #OBS-I164: modified the access roles and permissions * #OBS-I164: reading public key from env file * #OBS-I186 : fix: dataset metrics api * #OBS-I185 : fix: removed duplicate code. * #OBS-I164: modified public key variable in config * flink connector helm chart updates * flink connector helm chart updates * fix: dataset publish fixes * #OBS-I164: modified public key variable in config * #OBS-I186 : added dataset mertric api controller and route and minor change in dataset transition api * install pip requirments if applicable * #OBS-I186 : removed export statement * #OBS-I164: added config for option rbac verification * #OBS-I164: changed the middleware to rbac_middleware * #OBS-I186 : Logic moved to separate function * #OBS-I164: changed import name * #OBS-I186 : Logic moved to separate function * #OBS-I185 : fix: test case fixes * #OBS-I185 : fix: linting fix * #OBS-I164: modified config and rbac middleware * #OBS-I164: added jsonwebtoken package * V2 apis (#240) * #OBS-I115: Dataset list API refactoring * #0000: adding command api * #OBS-I116: Dataset CRUD APIs test and fixes * #OBS-I115: cmd api remove addn modules * #OBS-I116: Dataset status trasition to retire check for denorm fields * #OBS-I116: Dataset CRUD APIs test cases and fixes * #OBS-I115: Dataset Transition API refactoring * #OBS-I115: Dataset Transition API refactoring * #OBS-I116: Dataset update API Dedupe and denorm test cases fixes * #OBS-I115: Dataset Transition API refactoring and error handling refactoring * #OBS-I115: Dataset Transition API refactoring * #OBS-I115: Dataset Transition API refactoring * #OBS-I115: Remove unnecessary field fields_set * #OBS-I115: Dataset publish API - update the index of hudi spec properly for publish to handle schema evolution * #OBS-I116: Dataset create and status transition api code fix * #OBS-I116: Dataset Create api test case fixes * #OBS-I116: Dataset update extraction config api test case fixes * #OBS-I116: Dataset druid ingestion spec generation fix * #OBS-I116: express version upgraded * #OBS-I116: Dataset create api fixes * #OBS-I116: Dataset ingestion spec generation fix * #OBS-I126: updated swagger documentation * #OBS-I126: updated postman collection * #OBS-I126: updated postman collection * #OBS-I116: fix: entry topic column in datasets model * #OBS-I58 feat: Minio cloud store support - Added endpoint. as optional config to support the minio * #OBS-I126: added dataset read, list, update api's documentation and updated collection * #OBS-I116: feat: Dataschema api implementation v2 * #OBS-I116: feat: Schema validation fix * #OBS-I126: swagger doc updated * #OBS-I116: Dataset update api test cases * #OBS-I116: fix: linting fixes * #OBS-I116: lint fixes * #OBS-I116: Dataset status transition test cases * #OBS-I126 : added multiple requests example * #OBS-I126 : updated order * #OBS-I126 : updated server url * #OBS-I21: feat: dataset publish changes for connectors * #OBS-I1 updated Dataset Health API code * #OBS-I116: Dataset CRUD api fixes * #OBS-I116: fix: error codes fix * #OBS-I1 Refactored as per new changes * #OBS-I101: Update the publish API for the v2 APIs * #OBS-I101: fix db models * #OBS-I115: Remove the v1 unused API code and restructure the folders * #OBS-I1 Added DatasethealthService * #OBS-I1 Updated the imports and folders * #OBS-I2 updated dataset reset * #OBS-I116: fix: Command api and schema fixes * #OBS-I116: fix: Command api fix in db query * #OBS-I1 Added Notifications and alerts APIs * #OBS-I138: added decrypted response for the read api for connectors_config field and added defaults updated date and created date to list api * #OBS-I116: fix: feat: Dataset copy and export api implementation * #OBS-I2 Refactoring as per v2 APIs * #OBS-I2 typo fix * #OBS-I116: fix: fix: Dataset copy check for dataset fix * #OBS-I21: dataset publish changes fixes * #OBS-I116: fix: feat: Feedback fixes of removing set redis db * #OBS-I138: added cors to app * #OBS-I116: fix: feat: Dataset import api implementation * #OBS-I116: fix: fix: Dataset service fix * #OBS-I116: fix: repeated Validation method removal * #OBS-I116: fix: unused code * connector list * #OBS-I142: connector list api * #OBS-I116: fix: schema validation check for v1 exported dataset. * #OBS-I116: fix: Dataset overwrite after creation failure * #OBS-I116: fix: error handling * #OBS-I116: fix: error messages fix * #OBS-I116: fix: code fixes * #OBS-I142: formatted connector list api files * #OBS-I142: formatted connector list api files * #OBS-I142: formatted connector list file * #OBS-I138: required changes for dataset for master dataset migration from v1 to v2 * #OBS-I138: removed cors package * #OBS-I138: updated package json file * #OBS-I138: indentation fix * #OBS-I138: indentation fix * #OBS-I138: fixed indentations * #OBS-I142: updated postman collection * #OBS-I142: added connector list swagger documentation * #OBS-I142: updated postman collection * #OBS-I116: feat: Dataset Import and export api integration fixes * #OBS-I138: removed comment * #OBS-I138: throwing error if dataset is undefined * #OBS-I142: added test cases for connector list * #OBS-I138: merging dataset defaults to dataset draft record before saving * #OBS-I138: adding merged event to dataset while migrating live or draft dataset * #OBS-I142: added live_date field to defaultFields * #OBS-I145: Connector Read API * #OBS-I145: updated postman collection with connector read api * #OBS-I145: updated swagger documentation * #OBS-I145: updated the connector read api * merge changes * merge changes * Resolved merge changes * Resolved merge changes * #OBS-I145: updated the connector read api * #OBS-I138: removed datakey before merging defaults to dataset * #OBS-I145: updated postman collection and swagger documentation * #OBS-I145: added the test cases for connector read api * #OBS-I145: added a test case for connector read api * #OBS-I116: feat: Dataset import api fixes * #OBS-I116: feat: Test cases and linting fixes * #OBS-I116: feat: Dataset status transition test cases fix * #OBS-I1 updated the routes * #OBS-I116: feat: Dataset migratio method fix * #OBS-I108: feat: helm modifications for flink connectors * #OBS-I108: helm chart fixes * #OBS-I108: feat: Modify volume mounts * #OBS-I108: feat: Add PVC for JobManager * #OBS-I108: feat: change args for jobmanager command * #OBS-I141: added a new metric to sum the response time * #OBS-I141: modified the url variable and access dataset_id from params * #OBS-I141: added helper function to get dataset_id for error cases * #OBS-I108: feat: Use sidecar container to submit connector flink job * #OBS-I141: added telemetry for v2 api's * #OBS-I141: added a new metric to sum the response time * #OBS-I141: modified the url variable and access dataset_id from params * #OBS-I141: added helper function to get dataset_id for error cases * #OBS-I141: added telemetry for v2 api's * #OBS-I141: added telemetry for v2 api's * #OBS-I143: feat: dataset publish changes to deploy flink connectors * #OBS-I141: removed metric for sum of response time * #OBS-I141: removed usage of builtin kafka methods from telemetry file * #OBS-I146: feat: Retire fix * Issue #SBCOSS-12 fix: convert all SQL raw queries to prepared statements * Issue #SBCOSS-12 fix: tags is an array, so requires empty json for null case; dataset draft deletion requires deletion of transformation and source config drafts * #SBCOSS-23: feat: dataset publish changes for redeployment * #OBS-I167 : read api changes while reading connectors according to v2 structure * #OBS-I173: fix: Ready to publish schema fix to expect connector configs as object and string * #OBS-I174: fix: Dataset read api fix to expect both v1 and v2 connectors * #OBS-I146: fix: Test case fix for read api * #OBS-I146: fix: Test case fix for read api changes * #OBS-I146: fix: status transition test cases * #OBS-I146: fix: Test case script fix * #OBS-I146: fix: Type error fix * #OBS-I141: removed metric for sum of response time * #OBS-I146: fix: Dataset read api test cases fixes * #OBS-I146: fix: Hudi spec generation test cases * #OBS-I146: fix: Test case and linting fix * merge commit * #OBS-I173: fix: Dataset web console required fixes * #OBS-I173: fix: Dataset update changes to accept type changes * #OBS-I167 : dataset read api changes to read live dataset source configs * #OBS-I146: fix: linting fix * #OBS-I146: fix: linting fix * #OBS-I167 : Added string or dict as type to connector_config * #OBS-I143: dataset publish changes fixes * #OBS-I167 : if dataset is empty return with error * #OBS-I143: inswert query fix * Issue #OBS-I144 fix: icon data as string; check default version * #OBS-143: fix: dataset publish fixes * #OBS-I167 : fix: removed duplicate code. * #OBS-I181 - Updated the event structure * #OBS-I164: added jwt token vwerification and access control to api's * #OBS-I164: added jwt token vwerification and access control to api's * #OBS-I164: modified the access roles and permissions * #OBS-I164: reading public key from env file * #OBS-I186 : fix: dataset metrics api * #OBS-I185 : fix: removed duplicate code. * #OBS-I164: modified public key variable in config * flink connector helm chart updates * flink connector helm chart updates * fix: dataset publish fixes * #OBS-I164: modified public key variable in config * #OBS-I186 : added dataset mertric api controller and route and minor change in dataset transition api * install pip requirments if applicable * #OBS-I186 : removed export statement * #OBS-I164: added config for option rbac verification * #OBS-I164: changed the middleware to rbac_middleware * #OBS-I186 : Logic moved to separate function * #OBS-I164: changed import name * #OBS-I186 : Logic moved to separate function * #OBS-I185 : fix: test case fixes * #OBS-I185 : fix: linting fix * #OBS-I164: modified config and rbac middleware * #OBS-I164: added jsonwebtoken package * master url fix --------- Co-authored-by: Santhosh Vasabhaktula <santhosh@sanketika.in> Co-authored-by: JeraldJF <jeraldj@sanketika.in> Co-authored-by: yashashk <yashashk@sanketika.in> Co-authored-by: harishkumar gangula <harish@sanketika.in> Co-authored-by: SurabhiAngadi <surabhia@sanketika.in> Co-authored-by: Rakshitha-D <rakshithagowda.d@gmail.com> Co-authored-by: Anand Parthasarathy <anandp504@gmail.com> Co-authored-by: Aniket Sakinala <aniket@sanketika.in> * Resolved test case issue * Merge conflicts changes * #OBS-I165: added userInfo from token to request object * #OBS-I165: updated telemetry to use user role * #OBS-I165: updated datasetCreate api to add userRole as created_by * #OBS-I165: added userRole when migrating and create table from live * #OBS-I165: added userRole for data copy api * #OBS-I165: added userRole for dataset Update * #OBS-I165: added userRole for dataset Import * #OBS-I165: added userRole for dataset status transition * #OBS-I165: modified Dataset Service to update the userRoles * #OBS-I165: added permission for queryTemplateUpdate api * #OBS-I165: added userRole for query template create and update * #OBS-I165: added userRole for alerts api * #OBS-I165: added userRole for notifications api * #OBS-I165: added userRole for silences api * #OBS-I86 : commented routes have regex * #OBS-I77 : query api changes to check datasource avialability v2 * #OBS-I77 : fix: test cases * #OBS-I165: corrections userRole access in notification * #OBS-I165: added operations_admin role and updated permissions * #OBS-I165: added rbac middleware for alert routers * #OBS-I165: modified the user permissions into a json object * #OBS-I165: removed unused code * #OBS-I203: Remove the archived and purged status transition from the API * #OBS-I165: added user permissions json * #OBS-I165: removed user roles for type * #OBS-I165: added userID to req object and importing permissions from jsonfile * #OBS-I165: handled rbac disabled scenario and updated userID instead of userRole * #OBS-I165: changed the userRole to userID * #OBS-I165: added createdby for dataset publish api * #OBS-I165: added error condition and modified status code * #OBS-I165: removed redundant code * #OBS-I222 : Resolved regexp route issue in express version 5 * #OBS-I77 : query api changes to check avialability of datasource * #OBS-I77 : fix: spell fix * #OBS-I77 : granularity spell changes * #OBS-I77 : fix: test scenarios added * #OBS-I131 : fix: stop deleting draft records if submission tasks fail * #OBS-I131 : fix: upsert if republished * #OBS-I131 : fix: test case fixes * #OBS-I165: added default userID, modified error codes, added errorhandler function * #OBS-I165: added updatedby by default when an entry created * #OBS-I165: removed redundant code * #OBS-I165: modified errorhandler * #OBS-I165: modified error message * #OBS-I227 : fix: threshold of dataset level alerts inside an array. * #OBS-I179 : feat: retire dataset related alerts and metrics on dataset retire * #OBS-I79: feat: test case fixes * #OBS-I79: feat: Alerts get fix * feat: command tag name change (#252) * feat: command tag name change (#252) (#254) * #0000 fix: ingestion spec cache fix * Ingestion spec fix (#255) * feat: command tag name change (#252) (#254) * #0000 fix: ingestion spec cache fix --------- Co-authored-by: Ravi Mula <ravismula@users.noreply.github.com> * #OBS-I230: fix: kafka command * #OBS-I230: fix: cache issues changes * #OBS-I230: fix: redis db number change on update * #OBS-I187: added model for user * #OBS-I187: added userService * #OBS-I187: modified to verify keycloak token * #OBS-I187: updated rbac middleware * #OBS-I187: add seperate function to check access * #OBS-I230: fix: integration fixes * #OBS-I230: fix: kafka command fixes (#258) * #OBS-I230: fix: kafka command fixes * #OBS-I247: fix: feedback changes. --------- Co-authored-by: Ravi Mula <ravismula@users.noreply.github.com> * #OBS-I218 update kafka connector image tags * #OBS-I288: fix: removed data_format field from connector_instance (#260) * #OBS-I285: query api fix to validate and set limit to sql queries * #I285: removed empty objects and arrays from sample data (#261) * #I285: removed empty objects and arrays from sample data * Fix code scanning alert no. 98: Loop bound injection Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * #I285: lint issues fixed --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * #OBS-I285: query api to parse sql query first then do regex check with postman * #OBS-I285: fix: linting issue fix * #OBS-I285: fix: lint fix * #OBS-I285: fix: Logging errors in the middleware (#263) * #OBS-I285: fix: Logging errors in the middleware * #OBS-I285: fix: Logging errors by excluding sensitive info * #OBS-I285: fix: linting issue fix * #OBS-I289 : added route (#265) * fix: Stop the connectors specific to the dataset. (#266) * fix: uninstall dataset specific spark jobs * fix: remove nested loops * fix: uninstall dataset specific spark jobs * fix: remove nested loops * #OBS-I289 fix: Fix flink connector deployments * #OBS-I289 fix: Fix Spark Connector deployments * #OBS-I307 - revert the code as this is fixed from the front end (#268) * #OBS-I330 : changed type from hudi to datalake * fix for object-store-connector cron (#269) * #OBS-I330 : fixed merging issue * #OBS-I330 : removed comment * #OBS-I330 : Using dataset v2 export api while publishing * #OBS-I330 : removed print statement * #OBS-I330 : Using dataset v2 export api while publishing (#271) * #OBS-I330 : Using dataset v2 export api while publishing * #OBS-I330 : removed print statement * #OBS-I330 : Replacing - with _ for datasource_ref and adding partition key and primary key and timestamp key in column spec of ingestion spec * Hudi fixes (#272) * #OBS-I330 : Using dataset v2 export api while publishing * #OBS-I330 : removed print statement * #OBS-I330 : Replacing - with _ for datasource_ref and adding partition key and primary key and timestamp key in column spec of ingestion spec * fix: update connector instance id when inserting to db * #OBS-I330 : removed merging defaults to keys_config * #OBS-I330 : omit merging defaults to draft dataset for keys_config * #OBS-I335: hudi spec fix * #OBS-I335: hudi spec fix * #OBS-I335: linting fix * #OBS-I334 - clear transformations on re-upload * #OBS-I334 - clear keys_config on re-upload of schema file * #OBS-I335: dataset update fix * #OBS-I335: loop bound issue fix * #OBS-I335: dataset updated as per feedbacks * append base64 prefix upon connector register * #OBS-I334 - Fix the schema update functionality * Data mapping fix (#278) * #OBS-I335: hudi spec fix (#279) --------- Co-authored-by: JeraldJF <jeraldj@sanketika.in> Co-authored-by: Ravi Mula <ravismula@users.noreply.github.com> Co-authored-by: Rakshitha-D <rakshithagowda.d@gmail.com> Co-authored-by: SurabhiAngadi <surabhia@sanketika.in> Co-authored-by: Aniket Sakinala <aniket@sanketika.in> Co-authored-by: yashashk <yashashk@sanketika.in> Co-authored-by: Santhosh Vasabhaktula <santhosh@sanketika.in> Co-authored-by: Anand Parthasarathy <anandp504@gmail.com> Co-authored-by: Jerald <127138957+JeraldJF@users.noreply.github.com> Co-authored-by: Rakshitha-D <115482806+Rakshitha-D@users.noreply.github.com> Co-authored-by: SurabhiAngadi <138881390+SurabhiAngadi@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: yashash <126703764+yashashkumar@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.