Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Is the gazebosim.org main page content not accessible to search engine crawlers? #20

Open
j-rivero opened this issue Mar 8, 2023 · 2 comments
Open

Is the gazebosim.org main page content not accessible to search engine crawlers? #20

j-rivero opened this issue Mar 8, 2023 · 2 comments

Comments

@j-rivero
Copy link

j-rivero commented Mar 8, 2023

Looking into why the gazebosim.org page is mostly ignores for the Google searches I found some hints that can point out that the gazebosim.org main page is totally invisible to the search crawlers:

wget http://gazebosim.org/ is not able to see any real content:

  <meta name="viewport" content="width=device-width, initial-scale=1">           
  <link rel="icon" type="image/x-icon" href="/assets/icon/favicon.ico">          
  <title>Gazebo</title>                                                          
  <meta name="description" content="Simulation Driven Robotics. Iterate quickly on design concepts and control strategies with Gazebo's rich suite of tools, libraries, and cloud services.">
                                                                                 
  <base href="/">                                                                
  <style type="text/css">@font-face{font-family:'Roboto';font-style:normal;font-weight:300;src:url(https://fonts.gstatic.com/s/roboto/v30/KFOlCnqEu92Fr1MmSU5fCRc4AMP6lbBP.woff2) format('woff2');unic  <style type="text/css">@font-face{font-family:'Montserrat';font-style:normal;font-weight:300;src:url(https://fonts.gstatic.com/s/montserrat/v25/JTUSjIg1_i6t8kCHKm459WRhyyTh89ZNpQ.woff2) format('wo  <style type="text/css">@font-face{font-family:'Material Icons';font-style:normal;font-weight:400;src:url(https://fonts.gstatic.com/s/materialicons/v139/flUhRq6tzZclQEJ-Vdg-IuiaDsNcIhQ8tQ.woff2) fo mat('                                                                           
  <style>                                                                        
  body, html {                                                                   
    background-color: #ffffff;                                                   
    height: 100%;                                                                
  }                                                                              
  </style>                                                                       
<style>html,body{margin:0;padding:0;width:100%;height:100%;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;text-rendering:optimizeLegibility}</style><link rel="stylesheet" href= style                                                                           
<body>                                                                           
  <gz-app-root></gz-app-root>                                                    
<script src="runtime.1423ce2e84ce5652.js" type="module"></script><script src="polyfills.0b2dc2934fc02624.js" type="module"></script><script src="scripts.417efceffc01cbd5.js" defer></script><script s c="ma                                                                           
</body></html>

seobility.net is reporting that there are no h1 elements in the page, while if you check the web in your browser, there are:
image

seobility.net is reporting that there is no content in the web
image

screamingfrog seo tool crawls no subpages from the main page, just scripts
image
@j-rivero j-rivero mentioned this issue Mar 8, 2023
Closed
@azeey
Copy link
Collaborator

azeey commented Mar 8, 2023

It might not be as dire as it seems from the wget command because this is an Angular app, which needs javascript to run to load the page. I don't think wget runs javascript, but I think search engines do, or at least this article makes it sound like they do. But yeah, since most of gazebosim.org is static, it would be nice to serve simple html content via pre-rendereding or changing the frontend completely. I would love to use something like jekyll, but the engineering effort to convert is probably not well justified.

On a related note, running wget https://gazebosim.org/docs gives a 404 error. /cc @nkoenig

@j-rivero
Copy link
Author

j-rivero commented Mar 9, 2023

I think search engines do, or at least this article makes it sound like they do.

Reading the article (note that is an article from upstream code, which could suggest a bias), it says:

Understanding the SEO implications of a single page app
There is one major reason why single page apps are not used everywhere so far (with two separate causes):
Single page apps don't perform well on search engines

About Google it says:

The good news is that none of these two reasons are 100% accurate anymore! Google has started to index better single page apps.

mention that "none of these two reasons are 100% accurate anymore!" Google "has started to index better". More on other search engines:

This is great news, but there are other search engines that drive significant traffic. Baidu for example drives more than half of the traffic in China (curr. 1.3 billion people).

so the problem is still there for other search engines and we don't really know how bad is affecting the Google crawler.

Note that the article suggest the use of something called "Angular Universal"

What is Angular Universal, what does it enable?
Simply put, Angular Universal enables us to build apps that have both the performance and user engagement advantages of single page apps combined with the SEO friendliness of static pages.

Which is the same that other SEO pages suggest when using Angular. I totally brainstorming here and excuse me if this is totally wrong. just found some pointers in the same direction to solve a problem that I think it is real.

@azeey azeey mentioned this issue Dec 5, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@azeey @j-rivero