SNOW-1820372: (from 947) when proxy is set in Connection, driver does send traffic through the proxy to S3, but not to Azure blob / GCS bucket (only Snowflake). Works with proxy envvar (GCP) #982

sfc-gh-ext-simba-jy · 2024-12-07T01:36:17Z

Description

Please explain the changes you made here.

Checklist

Format code according to the existing code style (run npm run lint:check -- CHANGED_FILES and fix problems in changed code)
Create tests which fail without the change (if possible)
Make all tests (unit and integration) pass (npm run test:unit and npm run test:integration)
Extend the types in index.d.ts file (if necessary)
Extend the README / documentation and ensure is properly displayed (if necessary)
Provide JIRA issue id (if possible) or GitHub issue id in commit message

…ke-connector-nodejs into SNOW-1781419

…ke-connector-nodejs into SNOW-1820372

codecov · 2024-12-07T02:24:22Z

Codecov Report

Attention: Patch coverage is 85.39326% with 13 lines in your changes missing coverage. Please review.

Project coverage is 88.99%. Comparing base (d7eed90) to head (51b0702).

Files with missing lines	Patch %	Lines
lib/file_transfer_agent/gcs_util.js	79.62%	11 Missing ⚠️
lib/proxy_util.js	90.90%	1 Missing ⚠️
lib/services/sf.js	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #982      +/-   ##
==========================================
+ Coverage   88.91%   88.99%   +0.08%     
==========================================
  Files          72       72              
  Lines        6944     7006      +62     
==========================================
+ Hits         6174     6235      +61     
- Misses        770      771       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ke-connector-nodejs into SNOW-1820372

index.d.ts

lib/proxy_util.js

lib/file_transfer_agent/gcs_util.js

index.d.ts

lib/file_transfer_agent/gcs_util.js

lib/file_transfer_agent/remote_storage_util.js

lib/proxy_util.js

sfc-gh-pmotacki · 2024-12-20T11:42:31Z

lib/util.js

@@ -740,6 +740,10 @@ exports.getEnvVar = function (variable) {
  return process.env[variable.toLowerCase()] || process.env[variable.toUpperCase()];
 };

+exports.restoreEnvVar = function (envVar, value) {


Do we still need this method and its test?

No, we do not need it anymore. I removed it. Thank you!

sfc-gh-pmotacki · 2024-12-20T11:51:46Z

lib/file_transfer_agent/gcs_util.js

+        isProxyEnabled = true;
+        const proxyAgent = getProxyAgent(proxy, new URL(connectionConfig.accessUrl), endPoint || `storage.${stageInfo.region.toLowerCase()}.rep.googleapis.com` );
+        axios = require('axios').create({
+          proxy: false,


Is it correct to set here proxy = false?

Yes, we need to set proxy: false if we want to use https-proxy-agent in Axios. If not, the driver will face the 'too many direction' error. The same implementation is here: https://github.com/snowflakedb/snowflake-connector-nodejs/blob/master/lib/http/base.js#L289

sfc-gh-pmotacki · 2024-12-20T11:53:19Z

lib/file_transfer_agent/gcs_util.js

      client = { gcsToken: gcsToken, gcsClient: storage };
    } else {
      client = null;
    }
+
+    if (typeof httpClient === 'undefined') {
+      axios = require('axios');


The line could be a part of if ... else when we confirm whether the proxy is set or not.

if (proxy || Util.getEnvVar('http_proxy')) { ... } else { axios = require('axios'); }

sfc-gh-pmotacki · 2024-12-20T12:04:05Z

lib/file_transfer_agent/gcs_util.js

      client = { gcsToken: gcsToken, gcsClient: storage };
    } else {
      client = null;
    }
+
+    if (typeof httpClient === 'undefined') {


The code of setting up axios could be extracted to the separated method.

…ke-connector-nodejs into SNOW-1820372

sfc-gh-ext-simba-jy and others added 28 commits November 5, 2024 10:58

add destiantion in the cache key

02444e3

add testing cases for the agentcache

6731437

fix

57fb674

fix

c6941e3

fix testing

661d3db

fix

a31e911

add proxy settings in Azure_util

1ce6b99

Merge branch 'master' into SNOW-1781419

43cb685

fix azure testing

c7533f2

move proxy functions to proxy_util.js and made proxy_util_test.js

799c53f

add proxy_util.js and proxy_util_test.js

233643a

add copyrights

18e1065

fix the bug

90b2ff3

fix

4ba8394

add logs

a0ed255

fix

443b16b

Merge branch 'master' into SNOW-1781419

f1f5701

updated logs

8baf884

Merge branch 'SNOW-1781419' of https://github.com/snowflakedb/snowfla…

2cdaa20

…ke-connector-nodejs into SNOW-1781419

fix lint

1cea678

Merge branch 'master' into SNOW-1781419

ea8534b

Merge branch 'master' into SNOW-1781419

f5c992b

fix

2bb0e5f

save

ee58566

save

3cc9544

Merge branch 'SNOW-1820372' of https://github.com/snowflakedb/snowfla…

e634a2c

…ke-connector-nodejs into SNOW-1820372

add testing

d7e022c

add connectionConfig test and remove unnecessary codes

4233eb8

added more testing and refactored the code

cb839a6

sfc-gh-ext-simba-jy added 2 commits December 10, 2024 09:00

Merge branch 'SNOW-1820372' of https://github.com/snowflakedb/snowfla…

c531d0f

…ke-connector-nodejs into SNOW-1820372

fix bug

51b0702

sfc-gh-ext-simba-jy changed the title ~~Snow 1820372~~ SNOW-1820372: (from 947) when proxy is set in Connection, driver does send traffic through the proxy to S3, but not to Azure blob / GCS bucket (only Snowflake). Works with proxy envvar (GCP) Dec 10, 2024

sfc-gh-ext-simba-jy marked this pull request as ready for review December 10, 2024 17:52

sfc-gh-ext-simba-jy requested a review from a team as a code owner December 10, 2024 17:52

sfc-gh-ext-simba-jy requested a review from sfc-gh-pmotacki December 10, 2024 17:52

sfc-gh-dszmolka reviewed Dec 11, 2024

View reviewed changes

index.d.ts Outdated Show resolved Hide resolved

sfc-gh-dszmolka reviewed Dec 11, 2024

View reviewed changes

lib/proxy_util.js Outdated Show resolved Hide resolved

fix errors

5c3328a

sfc-gh-ext-simba-jy requested a review from sfc-gh-dszmolka December 11, 2024 20:09

Merge branch 'master' into SNOW-1820372

f12f124

sfc-gh-pmotacki reviewed Dec 12, 2024

View reviewed changes

lib/file_transfer_agent/gcs_util.js Outdated Show resolved Hide resolved

lib/file_transfer_agent/gcs_util.js Outdated Show resolved Hide resolved

lib/file_transfer_agent/gcs_util.js Outdated Show resolved Hide resolved

index.d.ts Outdated Show resolved Hide resolved

sfc-gh-ext-simba-jy and others added 4 commits December 12, 2024 14:41

Merge branch 'master' into SNOW-1820372

7bf600a

fix GCS erros and update PR based on the comments

0c5c2d0

fix

8899472

Merge branch 'master' into SNOW-1820372

47089a5

sfc-gh-dprzybysz reviewed Dec 16, 2024

View reviewed changes

add comments

f97e83c

sfc-gh-ext-simba-jy requested review from sfc-gh-dprzybysz and sfc-gh-pmotacki December 16, 2024 19:25

sfc-gh-ext-simba-jy added 3 commits December 18, 2024 10:59

fix

35fc4b0

fix

0ac468d

refactor

5e44c43

sfc-gh-pmotacki reviewed Dec 20, 2024

View reviewed changes

sfc-gh-ext-simba-jy and others added 2 commits December 20, 2024 09:26

Merge branch 'master' into SNOW-1820372

6d75374

fix

696255d

sfc-gh-ext-simba-jy requested a review from sfc-gh-pmotacki December 20, 2024 18:16

sfc-gh-ext-simba-jy and others added 3 commits December 20, 2024 10:16

Merge branch 'SNOW-1820372' of https://github.com/snowflakedb/snowfla…

b4a89fd

…ke-connector-nodejs into SNOW-1820372

Merge branch 'master' into SNOW-1820372

bf4eb28

fix errors from merging

a969e08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOW-1820372: (from 947) when proxy is set in Connection, driver does send traffic through the proxy to S3, but not to Azure blob / GCS bucket (only Snowflake). Works with proxy envvar (GCP) #982

SNOW-1820372: (from 947) when proxy is set in Connection, driver does send traffic through the proxy to S3, but not to Azure blob / GCS bucket (only Snowflake). Works with proxy envvar (GCP) #982

sfc-gh-ext-simba-jy commented Dec 7, 2024

codecov bot commented Dec 7, 2024 •

edited

Loading

sfc-gh-pmotacki Dec 20, 2024 •

edited

Loading

sfc-gh-ext-simba-jy Dec 20, 2024

sfc-gh-pmotacki Dec 20, 2024

sfc-gh-ext-simba-jy Dec 20, 2024 •

edited

Loading

sfc-gh-pmotacki Dec 20, 2024

sfc-gh-ext-simba-jy Dec 20, 2024

sfc-gh-pmotacki Dec 20, 2024

sfc-gh-ext-simba-jy Dec 20, 2024

SNOW-1820372: (from 947) when proxy is set in Connection, driver does send traffic through the proxy to S3, but not to Azure blob / GCS bucket (only Snowflake). Works with proxy envvar (GCP) #982

Are you sure you want to change the base?

SNOW-1820372: (from 947) when proxy is set in Connection, driver does send traffic through the proxy to S3, but not to Azure blob / GCS bucket (only Snowflake). Works with proxy envvar (GCP) #982

Conversation

sfc-gh-ext-simba-jy commented Dec 7, 2024

Description

Checklist

codecov bot commented Dec 7, 2024 • edited Loading

Codecov Report

sfc-gh-pmotacki Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

sfc-gh-ext-simba-jy Dec 20, 2024

Choose a reason for hiding this comment

sfc-gh-pmotacki Dec 20, 2024

Choose a reason for hiding this comment

sfc-gh-ext-simba-jy Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

sfc-gh-pmotacki Dec 20, 2024

Choose a reason for hiding this comment

sfc-gh-ext-simba-jy Dec 20, 2024

Choose a reason for hiding this comment

sfc-gh-pmotacki Dec 20, 2024

Choose a reason for hiding this comment

sfc-gh-ext-simba-jy Dec 20, 2024

Choose a reason for hiding this comment

codecov bot commented Dec 7, 2024 •

edited

Loading

sfc-gh-pmotacki Dec 20, 2024 •

edited

Loading

sfc-gh-ext-simba-jy Dec 20, 2024 •

edited

Loading