[FIX] HipChat Enterprise importer fails when importing a large amount of messages (millions) #13221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hudell
changed the title
rodrigok
reviewed
Jan 30, 2019
|
|
||
| for (const item of messageListIds) { | ||
| await this._importMessageList(startedByUserId, item._id); | ||
| } | ||
| } | ||
|
|
||
| _importDirectMessages() { |
There was a problem hiding this comment.
It's sync to avoid working on too many lists at the same time, as that caused the process to use too much RAM.
rodrigok
approved these changes
Jan 30, 2019
sampaiodiego
pushed a commit
that referenced
this pull request
Feb 1, 2019
… of messages (millions) (#13221) * Added new screen to check the importer history * Fixed conditional check * Better error handling and local file import * Improved error readability * Improved error handling * Allow @here and @ALL on messages that won't trigger notifications, regardless of permissions * Fixed room owners * Fixed issue when accessing import adm for the first time * Included Imported count on history screen * Increased timeout for fetching import file data * Improved Importer Counters * Missing comma * Improved RAM usage * import in batches * Avoid hitting a mongo cursor timeout * Additional fixes for the importer * Improved code * Show room owner at admin room info * Get room owner instead of room creator in admin room info * Insert messages faster * Skip messages that were already imported * Improved perfomance for hipchat imports * Improved hipchat importer perfomance * Improved message count handling * skip import operations marked as invalid
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #ISSUE_NUMBER