Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable Logging functionality to both new and old Log Tables #657

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
4ada070
fix: relocated runLogsSql into setupPartitionLogsTable
Kevin101Zhang Apr 4, 2024
8b57a79
chore: add prefix for INDEXs in schema
Kevin101Zhang Apr 4, 2024
a709dae
feat: added log-entry class and test
Kevin101Zhang Apr 5, 2024
f4c7ccb
Update log-entry.test.ts
Kevin101Zhang Apr 5, 2024
9de783f
Update log-entry.test.ts
Kevin101Zhang Apr 5, 2024
89fa865
uncommented out provisioning
Kevin101Zhang Apr 8, 2024
8d0da62
feat: added additional static classes in LogEntry
Kevin101Zhang Apr 8, 2024
91a3d1b
fix: modified indexer-logger and test to use logEntry
Kevin101Zhang Apr 8, 2024
0be44dd
fix: typo
Kevin101Zhang Apr 8, 2024
8a331b2
Update log-entry.test.ts
Kevin101Zhang Apr 8, 2024
1809f7a
chore: removed console.log
Kevin101Zhang Apr 8, 2024
7254a50
fix: LogLevel and LogType are defined in log-entry
Kevin101Zhang Apr 8, 2024
749e9f2
fix: indexer-logger now matches exact query
Kevin101Zhang Apr 8, 2024
00a6e61
Merge branch '639-fix-unresolved-comments-in-original-pr-httpsgithubc…
Kevin101Zhang Apr 8, 2024
a582921
fix: added test without bh
Kevin101Zhang Apr 8, 2024
8459863
Merge branch '639-fix-unresolved-comments-in-original-pr-httpsgithubc…
Kevin101Zhang Apr 8, 2024
0301061
merge scripts
Kevin101Zhang Apr 8, 2024
97e6ebc
fix: used fake timers on indexer-logger test
Kevin101Zhang Apr 9, 2024
4e431e9
fix: writeLog exclusively accepts an array of LogEntry
Kevin101Zhang Apr 9, 2024
6193ade
Merge branch '639-fix-unresolved-comments-in-original-pr-httpsgithubc…
Kevin101Zhang Apr 9, 2024
954528e
chore: moved log-entry files to indexer-logger
Kevin101Zhang Apr 9, 2024
70c9153
rebase
Kevin101Zhang Apr 9, 2024
16e9b69
rebase
Kevin101Zhang Apr 9, 2024
030fa4c
feat: provisioning of new users
Kevin101Zhang Apr 10, 2024
f7de470
merge with main includes - feat: Provision logs for existing users
Kevin101Zhang Apr 10, 2024
03ded3f
chore: remove console.log
Kevin101Zhang Apr 10, 2024
8d48fc5
added back commented out test for metaDataTableDDL
Kevin101Zhang Apr 10, 2024
80eb9f1
uncommented skip
Kevin101Zhang Apr 10, 2024
451b5bc
Merge branch '641-introduce-provisioning-of-logs' of https://github.c…
Kevin101Zhang Apr 10, 2024
e097cbc
uncommented skip, removed redundant test cases
Kevin101Zhang Apr 10, 2024
c9fc515
chore: removed comment
Kevin101Zhang Apr 10, 2024
c862042
chore: removed comment
Kevin101Zhang Apr 10, 2024
ef4d29c
mocked provisionLogsIfNeeded for test
Kevin101Zhang Apr 10, 2024
d75bc77
feat: uncommented out logging to new log table
Kevin101Zhang Apr 10, 2024
e8de5b8
chore: commented out unused code
Kevin101Zhang Apr 10, 2024
3ee3c9a
rebase
Kevin101Zhang Apr 10, 2024
39e4078
rebase from main
Kevin101Zhang Apr 10, 2024
5efbc81
add nullish coalescing to getDatabaseConnectionParameters
Kevin101Zhang Apr 10, 2024
d68a242
feat: added unit test to check logs, removed unused line
Kevin101Zhang Apr 14, 2024
d6285fe
feat: added integration test for logs asserting log_entries are equal…
Kevin101Zhang Apr 14, 2024
2ba9ffd
rebase
Kevin101Zhang Apr 15, 2024
d5f87de
fix: corrected param for indexerMeta instantiation
Kevin101Zhang Apr 15, 2024
6ed96aa
fix: use IndexerConfig for indexerMeta
Kevin101Zhang Apr 15, 2024
11dfe35
searching for provisioning endpoints
Kevin101Zhang Apr 15, 2024
b960b2c
fix: check run functions
Kevin101Zhang Apr 15, 2024
a3a040e
chore: renamed log table
Kevin101Zhang Apr 16, 2024
845cfac
chore: spacing
Kevin101Zhang Apr 16, 2024
1ead54b
set systemLogs in buildDB context to user
Kevin101Zhang Apr 16, 2024
4e718d6
fix: use of IndexerConfig in indexerMeta
Kevin101Zhang Apr 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion runner/src/indexer-meta/indexer-meta.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ describe('IndexerMeta', () => {
database: 'test_database'
};

const indexerConfig = new IndexerConfig('', '', 'some_account/some_indexer', 0, '', '', LogLevel.INFO);
const indexerConfig = new IndexerConfig('', 'some-account', 'some-indexer', 0, '', '', LogLevel.INFO);
const schemaName = indexerConfig.schemaName();

describe('writeLog', () => {
Expand Down
22 changes: 11 additions & 11 deletions runner/src/indexer-meta/indexer-meta.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,8 @@ export default class IndexerMeta {
tracer = trace.getTracer('queryapi-runner-indexer-logger');

private readonly pgClient: PgClient;
private readonly schemaName: string;
private readonly indexerConfig: IndexerConfig;
private readonly logInsertQueryTemplate: string = 'INSERT INTO %I.__logs (block_height, date, timestamp, type, level, message) VALUES %L';
private readonly loggingLevel: number;

constructor (
indexerConfig: IndexerConfig,
Expand All @@ -33,12 +32,11 @@ export default class IndexerMeta {
const pgClient = pgClientInstance ?? new PgClient(databaseConnectionParameters);

this.pgClient = pgClient;
this.schemaName = indexerConfig.schemaName();
this.loggingLevel = indexerConfig.logLevel;
this.indexerConfig = indexerConfig;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not necessary. Using private readonly will automatically expose indexerConfig to the class as a class variable.

}

private shouldLog (logLevel: LogLevel): boolean {
return logLevel >= this.loggingLevel;
return logLevel >= this.indexerConfig.logLevel;
}

async writeLogs (
Expand All @@ -49,6 +47,7 @@ export default class IndexerMeta {

const spanMessage = `write log for ${entriesArray.length === 1 ? 'single entry' : `batch of ${entriesArray.length}`} through postgres `;
const writeLogSpan = this.tracer.startSpan(spanMessage);

await wrapError(async () => {
const values = entriesArray.map(entry => [
entry.blockHeight,
Expand All @@ -58,9 +57,10 @@ export default class IndexerMeta {
LogLevel[entry.level],
entry.message
]);
const query = format(this.logInsertQueryTemplate, this.schemaName, values);

const query = format(this.logInsertQueryTemplate, this.indexerConfig.schemaName(), values);
await this.pgClient.query(query);
}, `Failed to insert ${entriesArray.length > 1 ? 'logs' : 'log'} into the ${this.schemaName}.__logs table`)
}, `Failed to insert ${entriesArray.length > 1 ? 'logs' : 'log'} into the ${this.indexerConfig.schemaName()}.__logs table`)
.finally(() => {
writeLogSpan.end();
});
Expand All @@ -69,10 +69,10 @@ export default class IndexerMeta {
async setStatus (status: IndexerStatus): Promise<void> {
const setStatusSpan = this.tracer.startSpan(`set status of indexer to ${status} through postgres`);
const values = [[STATUS_ATTRIBUTE, status]];
const query = format(METADATA_TABLE_UPSERT, this.schemaName, values);
const query = format(METADATA_TABLE_UPSERT, this.indexerConfig.schemaName(), values);

try {
await wrapError(async () => await this.pgClient.query(query), `Failed to update status for ${this.schemaName}`);
await wrapError(async () => await this.pgClient.query(query), `Failed to update status for ${this.indexerConfig.schemaName()}`);
} finally {
setStatusSpan.end();
}
Expand All @@ -81,10 +81,10 @@ export default class IndexerMeta {
async updateBlockheight (blockHeight: number): Promise<void> {
const setLastProcessedBlockSpan = this.tracer.startSpan(`set last processed block to ${blockHeight} through postgres`);
const values = [[LAST_PROCESSED_BLOCK_HEIGHT_ATTRIBUTE, blockHeight.toString()]];
const query = format(METADATA_TABLE_UPSERT, this.schemaName, values);
const query = format(METADATA_TABLE_UPSERT, this.indexerConfig.schemaName(), values);

try {
await wrapError(async () => await this.pgClient.query(query), `Failed to update last processed block height for ${this.schemaName}`);
await wrapError(async () => await this.pgClient.query(query), `Failed to update last processed block height for ${this.indexerConfig.schemaName()}`);
} finally {
setLastProcessedBlockSpan.end();
}
Expand Down
Loading