Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Django check seo search in wrong content #35

Closed
corentinbettiol opened this issue Feb 28, 2020 · 2 comments
Closed

Django check seo search in wrong content #35

corentinbettiol opened this issue Feb 28, 2020 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@corentinbettiol
Copy link
Member

Currently, django-check-seo is searching the content in the <div class="container"></div> tag.

It could be a problem if the main content of the pages of the crawled website is in another tag (like <main> or <div class="cms_main"> ...).

We should provide a setting where we could select a tag/class/list of tags/classes to search content in.

@corentinbettiol corentinbettiol added the bug Something isn't working label Feb 28, 2020
@corentinbettiol corentinbettiol self-assigned this Feb 28, 2020
@corentinbettiol
Copy link
Member Author

Possible solution:

  • Add a new parameter in settings.py, e.g:
DJANGO_CHECK_SEO_SEARCH_IN = "#wrap > .container"

This css selector will only search the elements with class container directly inside elements with id wrap:

image

So the #wrap > header > .container and footer > .container elements will not be selected.

The hard part is to have a consistent layout on each page to always select the main content.

@corentinbettiol
Copy link
Member Author

corentinbettiol commented Mar 2, 2020

Using a list of selectors and the fabulous extract() function of BeautifulSoup4, we can extract the selected html nodes from our main soup, and only keep what we want in the soup (the main content).

This method will certainly be chosen, because headers, navigation and footers tend to have the same class names & id for each page of the site (content class names may vary if we use different page templates in our website).

Ex:

  • Soup:
<!doctype html>
<html>
<head>
    <title>Home</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <meta name="description" content="None">
    <meta name="keywords" content="kw1,  kw2,  kw3">
</head>
<body>
    <div class="container">
        <ul class="nav">
            <li class="child selected">
                <a href="/fr/">Home</a>
                <ul>
                    <li class="child descendant">
                        <a href="/fr/blog/">Blog</a>
                    </li>
                    <li class="child descendant">
                        <a href="/fr/page-3/">Page 3</a>
                    </li>
                </ul>
            </li>
            <li class="child sibling">
                <a href="/fr/page-2/">Page 2</a>
            </li>
        </ul>
        <div class="darksky-small">
            <div class="darksky-icon darksky-rain"></div>
            <div class="darksky-temperature">9 °C</div>
            <div class="darksky-humidity">91 %</div>
            <div class="darksky-wind">3,6 km/h</div>
        </div>
        <section class="cover-section">
            <section class="container">Container content -1</section>
        </section>
        <section class="container">Container content 0</section>
        <div id="wrap">
            <section class="container">Container content 1</section>
            <section class="container">Container content 2</section>
            <div class="footer">
                <div class="containter">Footer content 1</div>
            </div>
        </div>
    </div>
</body>
</html>
excluded_selectors = ".nav, .darksky-small, .cover-section, .footer"

for element in soup.select(excluded_selectors):
    element.extract()
    # [...]

print(soup)
<!doctype html>
<html>
<head>
    <title>Home</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <meta name="description" content="None">
    <meta name="keywords" content="kw1,  kw2,  kw3">
</head>
<body>
    <div class="container">
        <section class="container">Container content 0</section>
        <div id="wrap">
            <section class="container">Container content 1</section>
            <section class="container">Container content 2</section>
        </div>
    </div>
</body>
</html>

Edit: Here's a screenshot of the previous code:

  • green background: what we want to keep
  • red background: what we don't want to keep
  • purple/blue/orange/ping tag hover: nodes to remove

image

This was referenced Mar 2, 2020
corentinbettiol added a commit that referenced this issue Mar 2, 2020
*add new DJANGO_CHECK_SEO_SEARCH_IN parameter (fix #30, #32 & #35)
*fix error in searched_in for meta descriptions tests (fix #36)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant