Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yq 3428 fix #7450

Merged
Merged

Conversation

evanevanevanevannnn
Copy link
Collaborator

Changelog entry

fix for https://st.yandex-team.ru/YQ-3428

now treating directory as directory/* when trying to load metadata in csv inferring mode

Changelog category

  • Bugfix

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 09:14:12 UTC Pre-commit check for cacf702 has started.
2024-08-05 09:17:07 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 09:25:04 UTC Build successful.

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 09:14:13 UTC Pre-commit check for cacf702 has started.
2024-08-05 09:17:29 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 10:57:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14053 12698 0 5 1336 14

2024-08-05 10:58:51 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 11:10:47 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
24 (only retried tests) 7 0 4 4 9

2024-08-05 11:11:55 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-05 11:19:35 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
16 (only retried tests) 8 0 0 0 8

🟢 2024-08-05 11:20:42 UTC Build successful.
🟡 2024-08-05 11:21:22 UTC ydbd size 8.1 GiB changed* by +166.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f64d670 merge: cacf702 diff diff %
ydbd size 8 665 991 720 Bytes 8 666 162 128 Bytes +166.4 KiB +0.002%
ydbd stripped size 471 452 840 Bytes 471 459 752 Bytes +6.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 09:14:48 UTC Pre-commit check for cacf702 has started.
2024-08-05 09:17:41 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-05 11:22:39 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9727 9680 0 12 23 12

🟢 2024-08-05 11:23:48 UTC Build successful.
🟢 2024-08-05 11:24:18 UTC ydbd size 5.4 GiB changed* by +5.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: a1f1ecc merge: cacf702 diff diff %
ydbd size 5 825 869 880 Bytes 5 825 875 464 Bytes +5.5 KiB +0.000%
ydbd stripped size 1 463 178 928 Bytes 1 463 179 632 Bytes +704 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@@ -307,15 +307,17 @@ struct TObjectStorageExternalSource : public IExternalSource {
structuredTokenBuilder.SetNoAuth();
}

const TString effectiveFilePattern = meta->TableLocation ? meta->TableLocation : "*";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

а разве возможно чтобы TableLocation был пустым?

Copy link
Collaborator Author

@evanevanevanevannnn evanevanevanevannnn Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

да, мы когда указывает в запросе директорию, у нас meta->DataSourceLocation это пусть к файлу (в данном случае путь директории), а meta->TableLocation это имя файла (или wildCardExpr/regExpr)

вот здесь при чтении файлов так же делается, а при сборке метаданных (в том числе схемы таблицы при выводе типов) почему-то так не сделали

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DataSourceLocation это путь к data source
А TableLocation это путь к таблице/директории/файлу. Почему он пустой если указали путь к директории?
Разве возможно написать?:

select * from connection.``

Copy link
Collaborator Author

@evanevanevanevannnn evanevanevanevannnn Aug 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

можно написать
select * from connection.'/'

@evanevanevanevannnn evanevanevanevannnn enabled auto-merge (squash) August 5, 2024 12:40
Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 16:19:43 UTC Pre-commit check for 6a8a8e9 has started.
2024-08-05 16:22:23 UTC Check linux-x86_64-release-asan is running...
2024-08-05 16:51:35 UTC Check cancelled

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 16:22:23 UTC Pre-commit check for 6a8a8e9 has started.
2024-08-05 16:25:04 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 16:30:25 UTC Build successful.

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 16:25:47 UTC Pre-commit check for 6a8a8e9 has started.
2024-08-05 16:28:36 UTC Check linux-x86_64-relwithdebinfo is running...
2024-08-05 16:51:39 UTC Check cancelled

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 16:53:54 UTC Pre-commit check for 106b0bb has started.
2024-08-05 16:57:13 UTC Check linux-x86_64-release-asan is running...
2024-08-05 18:19:13 UTC Check cancelled

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 16:55:43 UTC Pre-commit check for 106b0bb has started.
2024-08-05 16:58:36 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 17:04:42 UTC Build successful.

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 16:56:03 UTC Pre-commit check for 106b0bb has started.
2024-08-05 16:58:56 UTC Check linux-x86_64-relwithdebinfo is running...
2024-08-05 18:19:12 UTC Check cancelled

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 18:21:02 UTC Pre-commit check for 1a00ba1 has started.
2024-08-05 18:24:21 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-05 20:27:22 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9711 9665 0 10 24 12

🟢 2024-08-05 20:28:35 UTC Build successful.
🟡 2024-08-05 20:29:04 UTC ydbd size 5.4 GiB changed* by +2.0 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: b2e3ab3 merge: 1a00ba1 diff diff %
ydbd size 5 828 210 344 Bytes 5 830 290 440 Bytes +2.0 MiB +0.036%
ydbd stripped size 1 463 789 104 Bytes 1 464 318 512 Bytes +517.0 KiB +0.036%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 18:21:26 UTC Pre-commit check for 1a00ba1 has started.
2024-08-05 18:24:40 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 19:54:23 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14031 12680 0 5 1335 11

2024-08-05 19:55:58 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-05 20:07:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
20 (only retried tests) 5 0 3 4 8

2024-08-05 20:08:28 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🔴 2024-08-05 20:16:59 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
15 (only retried tests) 6 0 1 0 8

🟢 2024-08-05 20:18:07 UTC Build successful.
🔴 2024-08-05 20:18:43 UTC ydbd size 8.1 GiB changed* by +2.7 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: b2e3ab3 merge: 1a00ba1 diff diff %
ydbd size 8 668 416 128 Bytes 8 671 293 488 Bytes +2.7 MiB +0.033%
ydbd stripped size 471 630 952 Bytes 471 754 120 Bytes +120.3 KiB +0.026%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 5, 2024

2024-08-05 18:22:34 UTC Pre-commit check for 1a00ba1 has started.
2024-08-05 18:25:29 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-05 18:31:03 UTC Build successful.

Copy link

github-actions bot commented Aug 6, 2024

2024-08-06 07:57:05 UTC Pre-commit check for 6063e1d has started.
2024-08-06 08:00:06 UTC Check linux-x86_64-release-clang14 is running...
🟢 2024-08-06 08:06:27 UTC Build successful.

Copy link

github-actions bot commented Aug 6, 2024

2024-08-06 07:57:25 UTC Pre-commit check for 6063e1d has started.
2024-08-06 08:00:22 UTC Check linux-x86_64-release-asan is running...
🔴 2024-08-06 10:05:28 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9699 9634 0 22 27 16

🟢 2024-08-06 10:06:34 UTC Build successful.
🟢 2024-08-06 10:07:06 UTC ydbd size 5.4 GiB changed* by -4.2 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 99add96 merge: 6063e1d diff diff %
ydbd size 5 830 489 984 Bytes 5 830 485 680 Bytes -4.2 KiB -0.000%
ydbd stripped size 1 464 451 824 Bytes 1 464 450 352 Bytes -1.4 KiB -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Aug 6, 2024

2024-08-06 07:57:26 UTC Pre-commit check for 6063e1d has started.
2024-08-06 08:00:24 UTC Check linux-x86_64-relwithdebinfo is running...
🟡 2024-08-06 09:37:48 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14053 12704 0 3 1334 12

2024-08-06 09:39:23 UTC Failed tests rerun (try 2) linux-x86_64-relwithdebinfo is running...
🟡 2024-08-06 09:47:16 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17 (only retried tests) 8 0 1 0 8

2024-08-06 09:48:31 UTC Failed tests rerun (try 3) linux-x86_64-relwithdebinfo is running...
🟢 2024-08-06 09:55:30 UTC Tests successful.

Test history | Ya make output

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9 (only retried tests) 1 0 0 0 8

🟢 2024-08-06 09:56:42 UTC Build successful.
🟢 2024-08-06 09:57:16 UTC ydbd size 8.1 GiB changed* by -4.8 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 41cd042 merge: 6063e1d diff diff %
ydbd size 8 671 674 624 Bytes 8 671 669 664 Bytes -4.8 KiB -0.000%
ydbd stripped size 471 804 168 Bytes 471 803 784 Bytes -384 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@evanevanevanevannnn evanevanevanevannnn merged commit 38a0fb6 into ydb-platform:main Aug 6, 2024
10 of 12 checks passed
evanevanevanevannnn added a commit to evanevanevanevannnn/ydb that referenced this pull request Aug 12, 2024
@evanevanevanevannnn evanevanevanevannnn deleted the YQ_3428_fix branch August 16, 2024 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants