-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fix #4268] Adding Documentation for upgraded Search #4467
[Fix #4268] Adding Documentation for upgraded Search #4467
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A good start. Needs some spell checking, and I think I touched on the grammar issues I found.
docs/development/search.rst
Outdated
Search | ||
============ | ||
|
||
Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search result. Documentations are indexed in Elasticsearch index and the search is made through API. All the Search Code is Opensource and lives in `Github Repository`_. Currently we are using `Elasticsearch 6.3`_ version. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
result
-> results
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentations
-> Documents
docs/development/search.rst
Outdated
|
||
Installing and running Elasticsearch | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
You must need to install and run `Elasticsearch 6.3`_ version in your local development machine. You can get installation instuction `here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
instuction
is misspelled.
docs/development/search.rst
Outdated
^^^^^^^^ | ||
For using search, you need to index data to Elasticsearch Index. Run `reindex_elasticsearch` management command:: | ||
|
||
./manage.py reindex_elasticsearch |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't this take arguments?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
./manage.py search_index
which is provided by DED takes argument. our implementation of management command do not take arguments.
c25d9db
to
c752e44
Compare
@ericholscher fixed the grammer and added some docs about architecture. Is it ready to merge? |
docs/development/search.rst
Outdated
Search | ||
============ | ||
|
||
Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...instead of the built-in Sphinx search...
docs/development/search.rst
Outdated
============ | ||
|
||
Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search | ||
results. Documents are indexed in Elasticsearch index and the search is made through API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...in the Eleasticearch index...through the API.
docs/development/search.rst
Outdated
|
||
Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search | ||
results. Documents are indexed in Elasticsearch index and the search is made through API. | ||
All the Search Code is Opensource and lives in `Github Repository`_. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...the GitHub Repository.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And "open source", not Opensource
docs/development/search.rst
Outdated
Read The Docs uses Elasticsearch_ instead of built in Sphinx search for providing better search | ||
results. Documents are indexed in Elasticsearch index and the search is made through API. | ||
All the Search Code is Opensource and lives in `Github Repository`_. | ||
Currently we are using `Elasticsearch 6.3`_ version. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
using the Elasticsearch 6.3 version.
docs/development/search.rst
Outdated
|
||
Installing and running Elasticsearch | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
You must need to install and run `Elasticsearch 6.3`_ version in your local development machine. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to install and run the Elasticsearch 6.3 version on your local development machine. You can get the installation instructions here
.
docs/development/search.rst
Outdated
You must need to install and run `Elasticsearch 6.3`_ version in your local development machine. | ||
You can get installation instruction | ||
`here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_. | ||
Otherwise, you can also start a Elasticsearch Docker container by running following command:: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
running the following command
docs/development/search.rst
Outdated
|
||
Auto Indexing | ||
^^^^^^^^^^^^^ | ||
By default, Auto Indexing is turned off in development mode. To turn in on, change the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To turn it on
docs/development/search.rst
Outdated
^^^^^^^^^^^^^ | ||
By default, Auto Indexing is turned off in development mode. To turn in on, change the | ||
`ELASTICSEARCH_DSL_AUTOSYNC` settings to `True` in `readthedocs/settings/dev.py` file. | ||
After that, whenever a documentation successfully build, or project gets added, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
builds
docs/development/search.rst
Outdated
------------ | ||
The search architecture is devided into 2 parts. | ||
One part is responsible for **indexing** the documents and projects and | ||
other part is responsible to query through the Index for showing proper result to users. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
results
docs/development/search.rst
Outdated
The search architecture is devided into 2 parts. | ||
One part is responsible for **indexing** the documents and projects and | ||
other part is responsible to query through the Index for showing proper result to users. | ||
We use `django-elasticsearch-dsl`_ package mostly for the keep the search working. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to keep the
docs/development/search.rst
Outdated
|
||
Indexing | ||
^^^^^^^^ | ||
All the Sphinx documents are indexed into elasticsearch after build gets successfully finish. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the build is successful.
docs/development/search.rst
Outdated
~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
After any build gets successfully finished, `HTMLFile` objects are created for each of the | ||
`HTML` file of the build version and delete the old version's `HTMLFile` object. Signal_ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
files and the old version's HTMLFile
object is deleted.
docs/development/search.rst
Outdated
files. Both of the signals are dispatched with the list of the instances of `HTMLFile` | ||
in `instance_list` parameter. | ||
|
||
We listen the `bulk_post_create` and `bulk_post_delete` signals in our `Search` application and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
listen to the
docs/development/search.rst
Outdated
|
||
How we index projects | ||
~~~~~~~~~~~~~~~~~~~~~ | ||
We also index project informations in our search index so that user can search for projects |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
information ... so that they user can...
docs/development/search.rst
Outdated
~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
`elasticsearch-dsl`_ provide model like wrapper for `Elasticsearch document`_. | ||
As per requirements of `django-elasticsearch-dsl`_, its stored in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's stored
docs/development/search.rst
Outdated
Elasticsearch Document | ||
~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
`elasticsearch-dsl`_ provide model like wrapper for `Elasticsearch document`_. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
provides a model-like wrapper for the
docs/development/search.rst
Outdated
As per requirements of `django-elasticsearch-dsl`_, its stored in | ||
`readthedocs/search/documents.py` file. | ||
|
||
**ProjectDocument:** Its used for indexing projects. Signal listener of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's
docs/development/search.rst
Outdated
`django-elasticsearch-dsl`_ listen the `post_save` singal of `Project` model and | ||
then index/delete into Elasticsearch. | ||
|
||
**PageDocument**: Its used for indexing documentation of projects. By default, the auto |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's
docs/development/search.rst
Outdated
**PageDocument**: Its used for indexing documentation of projects. By default, the auto | ||
indexing is turned off by `ignore_signals = settings.ES_PAGE_IGNORE_SIGNALS`. | ||
`settings.ES_PAGE_IGNORE_SIGNALS` is `False` both in development and production. | ||
As mentioned above, our `Search` app listens the `bulk_post_create` and `bulk_post_delete` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
listens to the
docs/development/search.rst
Outdated
As mentioned above, our `Search` app listens the `bulk_post_create` and `bulk_post_delete` | ||
signals and index/delete documentations into Elasticsearch. The signal listeners are in | ||
the `readthedocs/search/signals.py` file. Both of the signals are dispatched | ||
after successful documentation build. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
after a successful
@RichardLitt I have fixed all the issue you have mentioned. can you check again? |
docs/development/search.rst
Outdated
|
||
Installing and running Elasticsearch | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
You need to install and run the `Elasticsearch 6.3`_ version on your local development machine. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...run Elastichsearch version 6.3
_ on your...
This might be a bit more fluent.
docs/development/search.rst
Outdated
You need to install and run the `Elasticsearch 6.3`_ version on your local development machine. | ||
You can get the installation instructions | ||
`here <https://www.elastic.co/guide/en/elasticsearch/reference/6.3/install-elasticsearch.html>`_. | ||
Otherwise, you can also start a Elasticsearch Docker container by running the following command:: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
start an Elasticsearch
docs/development/search.rst
Outdated
|
||
Indexing into Elasticsearch | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
For using search, you need to index data to Elasticsearch Index. Run `reindex_elasticsearch` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
data to the Elasticsearch
docs/development/search.rst
Outdated
Auto Indexing | ||
^^^^^^^^^^^^^ | ||
By default, Auto Indexing is turned off in development mode. To turn it on, change the | ||
`ELASTICSEARCH_DSL_AUTOSYNC` settings to `True` in `readthedocs/settings/dev.py` file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in the readthedocs...
file
docs/development/search.rst
Outdated
------------ | ||
The search architecture is devided into 2 parts. | ||
One part is responsible for **indexing** the documents and projects and | ||
other part is responsible to query through the Index for showing proper results to users. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the other part is responsible for querying the Index to show the proper results to users.
docs/development/search.rst
Outdated
The search architecture is devided into 2 parts. | ||
One part is responsible for **indexing** the documents and projects and | ||
other part is responsible to query through the Index for showing proper results to users. | ||
We use `django-elasticsearch-dsl`_ package mostly to the keep the search working. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We use the
docs/development/search.rst
Outdated
|
||
Indexing | ||
^^^^^^^^ | ||
All the Sphinx documents are indexed into elasticsearch after the build is successful. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Elasticsearch should always be capitalized.
docs/development/search.rst
Outdated
How we index documentations | ||
~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
After any build gets successfully finished, `HTMLFile` objects are created for each of the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
any build is successfully finished
docs/development/search.rst
Outdated
~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
After any build gets successfully finished, `HTMLFile` objects are created for each of the | ||
`HTML` files and the old version's `HTMLFile` object is deleted. Signal_ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Signal_
docs/development/search.rst
Outdated
~~~~~~~~~~~~~~~~~~~~~ | ||
We also index project information in our search index so that the user can search for projects | ||
from the main site. `django-elasticsearch-dsl`_ listen `post_create` and `post_delete` signals of | ||
`Project` model and index/delte into Elasticsearch accordingly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
delete
docs/development/search.rst
Outdated
Elasticsearch Document | ||
~~~~~~~~~~~~~~~~~~~~~~ | ||
|
||
`elasticsearch-dsl`_ provides model like wrapper for the `Elasticsearch document`_. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
provides a model-like wrapper
docs/development/search.rst
Outdated
|
||
`elasticsearch-dsl`_ provides model like wrapper for the `Elasticsearch document`_. | ||
As per requirements of `django-elasticsearch-dsl`_, it is stored in | ||
`readthedocs/search/documents.py` file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stored in a
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it should be the
docs/development/search.rst
Outdated
`readthedocs/search/documents.py` file. | ||
|
||
**ProjectDocument:** It is used for indexing projects. Signal listener of | ||
`django-elasticsearch-dsl`_ listen the `post_save` singal of `Project` model and |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
listens to the
docs/development/search.rst
Outdated
indexing is turned off by `ignore_signals = settings.ES_PAGE_IGNORE_SIGNALS`. | ||
`settings.ES_PAGE_IGNORE_SIGNALS` is `False` both in development and production. | ||
As mentioned above, our `Search` app listens to the `bulk_post_create` and `bulk_post_delete` | ||
signals and index/delete documentations into Elasticsearch. The signal listeners are in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
indexes/deleted documentation into...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting there! Sorry about all of these comments.
@RichardLitt I have updated the issues you have mentioned. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great. We could definitely add a bit more detail in some places, but I think this is a solid start to work from. 👍
For using search, you need to index data to the Elasticsearch Index. Run `reindex_elasticsearch` | ||
management command:: | ||
|
||
./manage.py reindex_elasticsearch |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's probably worth noting here why we implemented our own version, instead of using the detail one provided by django-elasticsearch-dsl.
docs/development/search.rst
Outdated
in `instance_list` parameter. | ||
|
||
We listen to the `bulk_post_create` and `bulk_post_delete` signals in our `Search` application and | ||
index/delete the documentation content from the `HTMLFile` instances. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Again, it's probably worth mentioning why we designed it this way, because by default it was doing an HTTP request per object.
docs/development/search.rst
Outdated
the `readthedocs/search/signals.py` file. Both of the signals are dispatched | ||
after a successful documentation build. | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's probably useful to mention how we parse the data from the JSON files to get the actual indexed data.
@safwanrahman Thanks. :) Sorry for being so persnickety! |
@ericholscher updated with fixes. Possible to merge? |
[Fix readthedocs#4268] Adding Documentation for upgraded Search
I have added instruction for preparing the local development for search. Its first step towards documentation of search as proposed in #4268.
@ericholscher r?