Create Sitemap for Better SEO

[MohammadPCh]

We need to generate a sitemap for our website to search engines can find our pages better and easier. now they can only crawl pages linked to our popular pages (homepage or first load of projects).
I suggest creating the sitemap once per day. We need the devOps team's help with this.
@aminlatifi @MoeNick

[MohammadPCh]

This google doc for Sitemap:
https://developers.google.com/search/docs/advanced/sitemaps/overview

[divine-comedian]

@mhmdksh @MohammadPCh - is this issue still relevant?

I'm VERY interested in improving our SEO.

[divine-comedian]

@mhmdksh can you please provide an update on this issue?

[mhmdksh]

@mhmdksh can you please provide an update on this issue?

@divine-comedian Sorry I can't provide any update on this neither this is my field of expertise. I prefer if I get unassigned as this is a Frontend/SEO-Optimization issue, not Devops work.

Thanks

@MohammadPCh Anyone can take this??

[divine-comedian]

@MohammadPCh can this issue be passed to someone or what is the next course of action?

[MohammadPCh]

@kkatusic @lovelgeorge99 @Reshzera Hey everyone, who has the bandwidth to take this on?

We need a script to:

1. Retrieve projects, users, and campaigns from the server
2. Add the static pages
3. Create a sitemap

Let me know if you need any help.

[kkatusic]

@kkatusic @lovelgeorge99 @Reshzera Hey everyone, who has the bandwidth to take this on?

We need a script to:

1. Retrieve projects, users, and campaigns from the server
2. Add the static pages
3. Create a sitemap

Let me know if you need any help.

I will take it, thx @MohammadPCh

[kkatusic]

@divine-comedian and @MohammadPCh, cc @CarlosQ96 is it ok to expose users inside sitemap?

[divine-comedian]

I am not sure about what the implications are so I would defer to what either of them think

[kkatusic]

Let's than wait @MohammadPCh and @CarlosQ96 response ;)

[MohammadPCh]

@kkatusic Yes we can add the public profile to the sitemap. but we should disallow the dashboard and private zones.

[kkatusic]

@kkatusic Yes we can add the public profile to the sitemap. but we should disallow the dashboard and private zones.

thx @MohammadPCh , let's hear also @CarlosQ96 response about this ;)

[CarlosQ96]

Ok we should make a query just to expose the minimum required info of users.
Also how often will the sitemap be generated? Ideally we could have a ipfs updated every day. I'm not sure a open public endpoint is that efficient even with cache. It's loading too much data into memory.

[CarlosQ96]

@kkatusic Sounds good to make the cronjob once per week. Exposing public data only.

[divine-comedian]

need to provide some back-end function for listing users, need to generate user sitemap only with public data on front-end.

@kkatusic please review and add estimate of when this can be completed.

[kkatusic]

need to provide some back-end function for listing users, need to generate user sitemap only with public data on front-end.

@kkatusic please review and add estimate of when this can be completed.

I will try to finish this week, but have now problem with BE, we will try to fix that before because I need BE functions also to finish this one.

[divine-comedian]

@CarlosQ96 can you give Giveth/impact-graph#1887 a review?

once BE is merged we can do FE

[divine-comedian]

Needs PR Review by @CarlosQ96

[divine-comedian]

Check when this job when it runs on Sunday and see if it needs to be optimized.

Need to see results of the job and see if it generates as it should on FE side, should be https://staging.giveth.io/sitemap.xml

need to have all data about projects users rounds inside of sitemap. Attach XML file to google search console once it is generated.

AC

• generates xml file with correct data
• should not make too many requests to the back-end

[kkatusic]

@divine-comedian, cc @CarlosQ96 , files generated fine, but project contain some character and I improved function for escaping that characters, sitemap files:

• main one that include all: https://staging.giveth.io/sitemap.xml
• projects, this has old error: https://staging.giveth.io/sitemap/projects-sitemap.xml
• qf round: https://staging.giveth.io/sitemap/qf-sitemap.xml
• users: https://staging.giveth.io/sitemap/users-sitemap.xml

[divine-comedian]

Thanks, so what are the next steps @kkatusic - do we need to wait again until next Sunday to see the results?

[kkatusic]

Thanks, so what are the next steps @kkatusic - do we need to wait again until next Sunday to see the results?

I will check again on next Monday. I generated on locahost and it's fine, but you never know what new character can appears.

[MoeNick]

@kkatusic is on it

[divine-comedian]

@CarlosQ96 if you can verify that back-end staging errors is fixed then we can push pending PRs

We are waiting to monitor other cronJobs since they seem to be causing resource issues but we're not sure which ones.

[divine-comedian]

bump on this @CarlosQ96

[kkatusic]

@divine-comedian, cc @CarlosQ96, FE is merged, we will check BE soon

[divine-comedian]

@CarlosQ96 @kkatusic this PR still needs a merge

[divine-comedian]

need to resolve Giveth/impact-graph#1922 before we can safely merge the related PR

[divine-comedian]

@kkatusic will try to run this job tomorrow and make sure results are what we expect

[kkatusic]

@divine-comedian again I stuck with this, after update on BE side it was resolving wrong IP address, for some reason hosts file has been set up for looking same FE url on same server, Moe helped and removed that after he found problem.
Then new issue, Verel don't allow you to change file or add new file inside public folder or any other problem, files are only read-only state.
Then I tried to update files to Pinata and have draft working copy, by Pinata is changing every time uploaded file URL, this not acceptable for us, I think we will need to move this URL data to database, then I need more time, migration, database table, resolver, api url....