Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable Logging functionality to both new and old Log Tables #657

Merged
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
4ada070
fix: relocated runLogsSql into setupPartitionLogsTable
Kevin101Zhang Apr 4, 2024
8b57a79
chore: add prefix for INDEXs in schema
Kevin101Zhang Apr 4, 2024
a709dae
feat: added log-entry class and test
Kevin101Zhang Apr 5, 2024
f4c7ccb
Update log-entry.test.ts
Kevin101Zhang Apr 5, 2024
9de783f
Update log-entry.test.ts
Kevin101Zhang Apr 5, 2024
89fa865
uncommented out provisioning
Kevin101Zhang Apr 8, 2024
8d0da62
feat: added additional static classes in LogEntry
Kevin101Zhang Apr 8, 2024
91a3d1b
fix: modified indexer-logger and test to use logEntry
Kevin101Zhang Apr 8, 2024
0be44dd
fix: typo
Kevin101Zhang Apr 8, 2024
8a331b2
Update log-entry.test.ts
Kevin101Zhang Apr 8, 2024
1809f7a
chore: removed console.log
Kevin101Zhang Apr 8, 2024
7254a50
fix: LogLevel and LogType are defined in log-entry
Kevin101Zhang Apr 8, 2024
749e9f2
fix: indexer-logger now matches exact query
Kevin101Zhang Apr 8, 2024
00a6e61
Merge branch '639-fix-unresolved-comments-in-original-pr-httpsgithubc…
Kevin101Zhang Apr 8, 2024
a582921
fix: added test without bh
Kevin101Zhang Apr 8, 2024
8459863
Merge branch '639-fix-unresolved-comments-in-original-pr-httpsgithubc…
Kevin101Zhang Apr 8, 2024
0301061
merge scripts
Kevin101Zhang Apr 8, 2024
97e6ebc
fix: used fake timers on indexer-logger test
Kevin101Zhang Apr 9, 2024
4e431e9
fix: writeLog exclusively accepts an array of LogEntry
Kevin101Zhang Apr 9, 2024
6193ade
Merge branch '639-fix-unresolved-comments-in-original-pr-httpsgithubc…
Kevin101Zhang Apr 9, 2024
954528e
chore: moved log-entry files to indexer-logger
Kevin101Zhang Apr 9, 2024
70c9153
rebase
Kevin101Zhang Apr 9, 2024
16e9b69
rebase
Kevin101Zhang Apr 9, 2024
030fa4c
feat: provisioning of new users
Kevin101Zhang Apr 10, 2024
f7de470
merge with main includes - feat: Provision logs for existing users
Kevin101Zhang Apr 10, 2024
03ded3f
chore: remove console.log
Kevin101Zhang Apr 10, 2024
8d48fc5
added back commented out test for metaDataTableDDL
Kevin101Zhang Apr 10, 2024
80eb9f1
uncommented skip
Kevin101Zhang Apr 10, 2024
451b5bc
Merge branch '641-introduce-provisioning-of-logs' of https://github.c…
Kevin101Zhang Apr 10, 2024
e097cbc
uncommented skip, removed redundant test cases
Kevin101Zhang Apr 10, 2024
c9fc515
chore: removed comment
Kevin101Zhang Apr 10, 2024
c862042
chore: removed comment
Kevin101Zhang Apr 10, 2024
ef4d29c
mocked provisionLogsIfNeeded for test
Kevin101Zhang Apr 10, 2024
d75bc77
feat: uncommented out logging to new log table
Kevin101Zhang Apr 10, 2024
e8de5b8
chore: commented out unused code
Kevin101Zhang Apr 10, 2024
3ee3c9a
rebase
Kevin101Zhang Apr 10, 2024
39e4078
rebase from main
Kevin101Zhang Apr 10, 2024
5efbc81
add nullish coalescing to getDatabaseConnectionParameters
Kevin101Zhang Apr 10, 2024
d68a242
feat: added unit test to check logs, removed unused line
Kevin101Zhang Apr 14, 2024
d6285fe
feat: added integration test for logs asserting log_entries are equal…
Kevin101Zhang Apr 14, 2024
2ba9ffd
rebase
Kevin101Zhang Apr 15, 2024
d5f87de
fix: corrected param for indexerMeta instantiation
Kevin101Zhang Apr 15, 2024
6ed96aa
fix: use IndexerConfig for indexerMeta
Kevin101Zhang Apr 15, 2024
11dfe35
searching for provisioning endpoints
Kevin101Zhang Apr 15, 2024
b960b2c
fix: check run functions
Kevin101Zhang Apr 15, 2024
a3a040e
chore: renamed log table
Kevin101Zhang Apr 16, 2024
845cfac
chore: spacing
Kevin101Zhang Apr 16, 2024
1ead54b
set systemLogs in buildDB context to user
Kevin101Zhang Apr 16, 2024
4e718d6
fix: use of IndexerConfig in indexerMeta
Kevin101Zhang Apr 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions runner/src/indexer-meta/indexer-meta.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import IndexerMeta, { IndexerStatus } from './indexer-meta';
import type PgClient from '../pg-client';
import LogEntry, { LogLevel } from './log-entry';
import { type PostgresConnectionParams } from '../pg-client';
import IndexerConfig from '../indexer-config/indexer-config';

describe('IndexerMeta', () => {
let genericMockPgClient: PgClient;
Expand All @@ -23,16 +24,17 @@ describe('IndexerMeta', () => {
port: 5432,
database: 'test_database'
};
const functionName = 'some_account/some_indexer';
const schemaName = functionName.replace(/[^a-zA-Z0-9]/g, '_');

const indexerConfig = new IndexerConfig('', '', 'some_account/some_indexer', 0, '', '', LogLevel.INFO);
Kevin101Zhang marked this conversation as resolved.
Show resolved Hide resolved
const schemaName = indexerConfig.schemaName();

describe('writeLog', () => {
it('should insert a single log entry into the database', async () => {
const date = new Date();
jest.useFakeTimers({ now: date.getTime() });
const formattedDate = date.toISOString().replace('T', ' ').replace('Z', '+00');

const indexerMeta = new IndexerMeta(functionName, LogLevel.INFO, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const infoEntry = LogEntry.systemInfo('Info message');
await indexerMeta.writeLogs([infoEntry]);

Expand All @@ -45,7 +47,7 @@ describe('IndexerMeta', () => {
jest.useFakeTimers({ now: date.getTime() });
const formattedDate = date.toISOString().replace('T', ' ').replace('Z', '+00');

const indexerMeta = new IndexerMeta(functionName, LogLevel.INFO, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const errorEntry = LogEntry.systemError('Error message', 12345);
await indexerMeta.writeLogs([errorEntry]);

Expand All @@ -56,13 +58,13 @@ describe('IndexerMeta', () => {
it('should handle errors when inserting a single log entry', async () => {
query.mockRejectedValueOnce(new Error('Failed to insert log'));

const indexerMeta = new IndexerMeta(functionName, LogLevel.INFO, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const errorEntry = LogEntry.systemError('Error message', 12345);
await expect(indexerMeta.writeLogs([errorEntry])).rejects.toThrow('Failed to insert log');
});

it('should insert a batch of log entries into the database', async () => {
const indexerMeta = new IndexerMeta(functionName, LogLevel.INFO, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const debugEntry = LogEntry.systemDebug('Debug message');
const infoEntry = LogEntry.systemInfo('Information message');
const logEntries: LogEntry[] = [
Expand All @@ -79,7 +81,7 @@ describe('IndexerMeta', () => {
it('should handle errors when inserting a batch of log entries', async () => {
query.mockRejectedValueOnce(new Error('Failed to insert batch of logs'));

const indexerMeta = new IndexerMeta(functionName, LogLevel.INFO, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const debugEntry = LogEntry.systemDebug('Debug message');
const infoEntry = LogEntry.systemInfo('Information message');
const logEntries: LogEntry[] = [
Expand All @@ -91,15 +93,15 @@ describe('IndexerMeta', () => {
});

it('should handle empty log entry', async () => {
const indexerMeta = new IndexerMeta(functionName, LogLevel.INFO, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const logEntries: LogEntry[] = [];
await indexerMeta.writeLogs(logEntries);

expect(query).not.toHaveBeenCalled();
});

it('should skip log entries with levels lower than the logging level specified in the constructor', async () => {
const indexerMeta = new IndexerMeta(functionName, LogLevel.ERROR, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
const debugEntry = LogEntry.systemDebug('Debug message');

await indexerMeta.writeLogs([debugEntry]);
Expand All @@ -108,15 +110,15 @@ describe('IndexerMeta', () => {
});

it('writes status for indexer', async () => {
const indexerMeta = new IndexerMeta(functionName, 5, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
await indexerMeta.setStatus(IndexerStatus.RUNNING);
expect(query).toBeCalledWith(
`INSERT INTO ${schemaName}.__metadata (attribute, value) VALUES ('STATUS', 'RUNNING') ON CONFLICT (attribute) DO UPDATE SET value = EXCLUDED.value RETURNING *`
);
});

it('writes last processed block height for indexer', async () => {
const indexerMeta = new IndexerMeta(functionName, 5, mockDatabaseConnectionParameters, genericMockPgClient);
const indexerMeta = new IndexerMeta(indexerConfig, mockDatabaseConnectionParameters, genericMockPgClient);
await indexerMeta.updateBlockheight(123);
expect(query).toBeCalledWith(
`INSERT INTO ${schemaName}.__metadata (attribute, value) VALUES ('LAST_PROCESSED_BLOCK_HEIGHT', '123') ON CONFLICT (attribute) DO UPDATE SET value = EXCLUDED.value RETURNING *`
Expand Down
9 changes: 4 additions & 5 deletions runner/src/indexer-meta/indexer-meta.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import PgClient, { type PostgresConnectionParams } from '../pg-client';
import { trace } from '@opentelemetry/api';
import type LogEntry from './log-entry';
import { LogLevel } from './log-entry';
import type IndexerConfig from '../indexer-config/indexer-config';

export enum IndexerStatus {
PROVISIONING = 'PROVISIONING',
Expand All @@ -25,16 +26,15 @@ export default class IndexerMeta {
private readonly loggingLevel: number;

constructor (
functionName: string,
loggingLevel: number,
indexerConfig: IndexerConfig,
databaseConnectionParameters: PostgresConnectionParams,
pgClientInstance: PgClient | undefined = undefined
) {
const pgClient = pgClientInstance ?? new PgClient(databaseConnectionParameters);

this.pgClient = pgClient;
this.schemaName = functionName.replace(/[^a-zA-Z0-9]/g, '_');
this.loggingLevel = loggingLevel;
this.schemaName = indexerConfig.schemaName();
this.loggingLevel = indexerConfig.logLevel;
Kevin101Zhang marked this conversation as resolved.
Show resolved Hide resolved
}

private shouldLog (logLevel: LogLevel): boolean {
Expand All @@ -58,7 +58,6 @@ export default class IndexerMeta {
LogLevel[entry.level],
entry.message
]);

const query = format(this.logInsertQueryTemplate, this.schemaName, values);
await this.pgClient.query(query);
}, `Failed to insert ${entriesArray.length > 1 ? 'logs' : 'log'} into the ${this.schemaName}.__logs table`)
Expand Down
Loading
Loading