Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add indexer example. NEAR Social posts and comments indexer #4

Closed
wants to merge 5 commits into from

Conversation

khorolets
Copy link
Member

@khorolets khorolets commented Feb 2, 2023

Initial version that stores Posts and Comments to the PostgreSQL database (using Prisma ORM).

At the moment it's impossible to just lunch this indexer because it depends on the future 1.1 version of Lake Framework that is under development. The only way to make it work was to use file: in package.json.

But overall you can have a look at the initial logic and judge the interfaces. The idea of introducing some helpers dedicated to NEAR Social seems to be a must.

UPDATE Feb 9

  • Clean up the code I was complaining about previously. I've split the code into separate functions improving readability
  • Add list of accounts that liked the posts and support unlikes (though I believe it is suboptimal, I am storing a JSONB field in posts and push/pop account names on like/unlike). The nature of the lists and Prisma.JsonArray makes things suboptimal.

The code is working, although it feels really slow. I am not sure it is even possible to speed it up. I can assume I do something wrong with the TypeScript. @morgsmccauley @roshaans could you please have a quick look and point me to suboptimal pieces that might be improved?

UPDATE Feb 8

  • Depend on @near-lake/primitives and @near-lake/framework from NPM
  • Change migrations to have a receipt_id field in tables posts and comments
  • Handle likes and unlike (to POSTS only)

Summary. Overall code is in working shape. Though, I don't like how messy it is. I would divide some logic into separate functions. However, I was trying to write the code as a potential developer (user) would do it. I didn't expect it to grow so much, but it has lol.

I believe we can keep the code as it is right now, but it would be better to clean that up.

@khorolets khorolets force-pushed the khorolets/near-social-posts-comments-indexer branch from 24bf4a5 to 7ed5af0 Compare February 8, 2023 16:23
const blockTimestamp = block.header().timestampNanosec;
console.log(blockHeight);
console.dir(nearSocialPosts, { depth: null })
nearSocialPosts.forEach(async postAction => {
Copy link
Collaborator

@morgsmccauley morgsmccauley Feb 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This creates an array of promises but the result of these promises are not being awaited anywhere. This means handleStreamerMessage() will not wait for these promises to finish before moving on, and will therefore start processing the next block while these DB calls are still in-flight.

There are two ways of solving this:

  1. Move execution/awaiting of these promises to handleStreamerMessage() with for of :
for (const postAction of nearSocialPosts) {
  // existing function logic
}
  1. await the promise result with Promise.all():
await Promise.all(nearSocialPosts.map(async postAction => {
  // existing function logic
}));

Both will cause handleStreamerMessage() to wait for the result before moving on, but 1 will execute each postAction in order where as 2 will not, and is therefore faster. I'm not sure which is more appropriate? Do we need to execute these in order?

@morgsmccauley morgsmccauley requested a review from a team as a code owner July 27, 2023 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants