Skip to content

fix case removed column for stats usage #1025

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ivanmorozov333 merged 2 commits into from
Jan 16, 2024
Merged

fix case removed column for stats usage #1025

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
159797a
fix case removed column for stats usage
ivanmorozov333 Jan 16, 2024
7c8565a
additional logging
ivanmorozov333 Jan 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions ydb/core/tx/columnshard/engines/changes/general_compaction.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ namespace NKikimr::NOlap::NCompaction {

void TGeneralCompactColumnEngineChanges::BuildAppendedPortionsByFullBatches(TConstructionContext& context) noexcept {
std::vector<TPortionInfoWithBlobs> portions = TPortionInfoWithBlobs::RestorePortions(SwitchedPortions, Blobs);
Blobs.clear();
std::vector<std::shared_ptr<arrow::RecordBatch>> batchResults;
Copy link
Contributor

@nsofya nsofya Jan 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Это точно корректно? Какое это имеет отношение к проверке колонок?

Copy link
Collaborator Author

@ivanmorozov333 ivanmorozov333 Jan 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

да. никакого.

auto resultSchema = context.SchemaVersions.GetLastSchema();
{
Expand All @@ -45,7 +44,6 @@ void TGeneralCompactColumnEngineChanges::BuildAppendedPortionsByFullBatches(TCon

void TGeneralCompactColumnEngineChanges::BuildAppendedPortionsByChunks(TConstructionContext& context) noexcept {
std::vector<TPortionInfoWithBlobs> portions = TPortionInfoWithBlobs::RestorePortions(SwitchedPortions, Blobs);
Blobs.clear();
static const TString portionIdFieldName = "$$__portion_id";
static const TString portionRecordIndexFieldName = "$$__portion_record_idx";
static const std::shared_ptr<arrow::Field> portionIdField = std::make_shared<arrow::Field>(portionIdFieldName, std::make_shared<arrow::UInt16Type>());
Expand Down
8 changes: 8 additions & 0 deletions ydb/core/tx/columnshard/engines/portions/common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@ class TChunkAddress {
YDB_READONLY(ui32, ColumnId, 0);
YDB_READONLY(ui16, Chunk, 0);
public:
ui32 GetEntityId() const {
return ColumnId;
}

ui32 GetChunkIdx() const {
return Chunk;
}

TChunkAddress(const ui32 columnId, const ui16 chunk)
: ColumnId(columnId)
, Chunk(chunk) {
Expand Down
4 changes: 3 additions & 1 deletion ydb/core/tx/columnshard/engines/portions/portion_info.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,9 @@ class TPortionInfo {
TSerializationStats GetSerializationStat(const ISnapshotSchema& schema) const {
TSerializationStats result;
for (auto&& i : Records) {
result.AddStat(i.GetSerializationStat(schema.GetFieldByColumnIdVerified(i.ColumnId)->name()));
if (schema.GetFieldByColumnId(i.ColumnId)) {
result.AddStat(i.GetSerializationStat(schema.GetFieldByColumnIdVerified(i.ColumnId)->name()));
}
}
return result;
}
Expand Down
6 changes: 6 additions & 0 deletions ydb/core/tx/columnshard/engines/scheme/abstract_scheme.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,4 +110,10 @@ ui32 ISnapshotSchema::GetColumnId(const std::string& columnName) const {
return *id;
}

std::shared_ptr<arrow::Field> ISnapshotSchema::GetFieldByColumnIdVerified(const ui32 columnId) const {
auto result = GetFieldByColumnId(columnId);
AFL_VERIFY(result)("event", "unknown_column")("column_id", columnId);
return result;
}

}
6 changes: 1 addition & 5 deletions ydb/core/tx/columnshard/engines/scheme/abstract_scheme.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,7 @@ class ISnapshotSchema {
ui32 GetColumnId(const std::string& columnName) const;
std::shared_ptr<arrow::Field> GetFieldByIndex(const int index) const;
std::shared_ptr<arrow::Field> GetFieldByColumnId(const ui32 columnId) const;
std::shared_ptr<arrow::Field> GetFieldByColumnIdVerified(const ui32 columnId) const {
auto result = GetFieldByColumnId(columnId);
Y_ABORT_UNLESS(result);
return result;
}
std::shared_ptr<arrow::Field> GetFieldByColumnIdVerified(const ui32 columnId) const;

TString DebugString() const {
return DoDebugString();
Expand Down