Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tajul/keyword async processing #14

Merged

Conversation

tajul-saajan
Copy link
Owner

resolves issue #8

  • Added field boolean isProcessed in searchStat entity
  • In entity, only keyword and isProcessed are required, all other fields are optional
  • After parsing the file, save it to the database with the keyword and then add the record to the queue
  • Added queue consumer which performs a search using searcher and then updates the records to the database with the search result
  • Added support for job failure, configured queue to have attempts and backoff time

@tajul-saajan tajul-saajan linked an issue Nov 29, 2023 that may be closed by this pull request
cachedResponse: string;

@Column({ type: 'bool', default: false })
isProcessed: boolean;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not account for a failed status. How about instead using an enum:

enum SearchStatus {
    Pending = 'pending',
    Completed = 'completed',
    Failed = 'failed'
}

@Entity()
export class SearchStat {
  ...
  @Column({ type: 'text' })
  status: SearchStatus;
}

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not consider the failed status, as in the queue used max attempts and exponential backoff delay to make sure the failed job retries continue with exponential backoff. But there can be cases, the job exhausts all the retries and unable to process the search. Should have considered this scenerio

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, when the queued job exhausts all its attempts and is unable to process the search operation, I guess in that scenario I need to add a cron job scheduler to pick all failed jobs and process them again?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a production application, taking care of the failed jobs could take many forms (retrying them, or sending a report/error), but it is outside of the scope of this exercise.

As for the exponential backoff, still, the status could be marked as failed in the mean time to differentiate it from other jobs 💡

@tajul-saajan
Copy link
Owner Author

In the above commit

  • status field added in search stat entity
  • added status change based on queued job success or failure

@tajul-saajan tajul-saajan changed the base branch from main to tajul/csv-keywords-max-validation December 1, 2023 05:28
…saajan/google-searcher into tajul/keyword-async-processing

# Conflicts:
#	src/search/search.controller.ts
#	src/search/search.module.ts
…jan/google-searcher into tajul/improve-google-searcher

# Conflicts:
#	package-lock.json
#	package.json
…an/google-searcher into tajul/consolidate-ui-layouts
@tajul-saajan tajul-saajan merged commit 6e0e3c5 into tajul/csv-keywords-max-validation Dec 11, 2023
@tajul-saajan tajul-saajan deleted the tajul/keyword-async-processing branch December 11, 2023 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Chore] Brittle asynchronous processing of keywords
2 participants