Elasticsearch With DotNet

In this repo, I have developed a sample project on how we can use the Elasticsearch platform in a .Net project and make improvements.

What is Elasticsearch?

ElasticSearch is an open source, REST architecture search and analysis engine that provides users with a fast and reliable search experience. Developed by Shay Banon in 2010, ElasticSearch responds to modern data management and analysis needs.

Using a JSON-based data format and built on a powerful Apache Lucene infrastructure, this tool offers near real-time data search capacity. Users benefit from this structure in many areas such as big data applications, log management, analysis systems and content management systems.

What is Elasticsearch used for?

Elasticsearch’s speed, scalability, and ability to index many types of content enable a variety of use cases, including:

• Application search
• Website search
• Enterprise search
• Logging and log analysis
• Infrastructure metrics and container monitoring
• Application performance monitoring
• Geodata analysis and visualization
• Security analytics
• Business analytics

What are Elasticsearch Components?

Elasticsearch consists of several core components that manage the processes of storing, indexing, and searching data. These components provide the distributed structure and scalability of the system. Here are the main components of Elasticsearch:

• Cluster

A top-level structure consisting of multiple nodes. Manages data and indices, ensuring high availability and scalability. Each cluster has a unique name, and all nodes must belong to the same cluster.

• Node

  An instance of Elasticsearch that stores data, indexes, and processes queries. Can operate independently or as part of a cluster. Nodes can serve different roles, such as Master Node or Data Node

• Index

A collection of data stored in a structured format. Each index contains one or more shards and is used to store and query data.

• Document

The smallest unit of data, stored in JSON format. Represents a record and has a unique identifier (ID). Stored within an index.

Shard

A physical partition of an index, distributed across nodes for efficient data management and high availability. Types:

• Primary Shard: Original data copy.
• Replica Shard: A copy of the primary shard for redundancy.

Mapping

Defines the data structure and field types within an index, such as number, text, or date. Essential for schema definition and accurate data search.

Analyzer

Processes text data by breaking it into tokens and optimizing it for search. Customizable for different fields (e.g., title vs. description).

Query DSL

JSON-based query language used for complex searches, filters, and analyses in Elasticsearch. Includes query types like match, term, and range.

Inverted Index

Data structure used for fast full-text searches by mapping words to the documents they appear in.

Snapshot and Restore

Snapshots are backups of clusters or indices. Restoring these backups enables data recovery and long-term protection.

Elasticsearch Plugins

Extend Elasticsearch functionality with additional features, such as data processing (Ingest Node) or security (X-Pack).

Using Elasticsearch in .Net

• Elasticsearch is usually used in .NET projects with the NEST library. NEST is a high-level .NET client for Elasticsearch and makes it easy to interact with Elasticsearch. To do this, we first load the Nest library into our project.
• If we are using the 8th version of Elasticsearch, the Elastic.Clients.Elasticsearch 8.1.1 library is more suitable, if it is lower, it is more suitable to use the Nest library.
• We need to establish our connection to send a request to Elasticsearch and receive a response. For this, we create the connection in program.cs. I implemented this in program.cs by making an extension.

            var pool = new SingleNodeConnectionPool(new Uri(configuration.GetSection("Elastic")["Url"]!));
            var settings = new ConnectionSettings(pool);
            var client = new ElasticClient(settings);
            services.AddSingleton(client);

• I specify the URL information in the appsettings.json file.

 "Elastic": {
   "Url": "http://localhost:9200",
   "Username": "elastic",
   "Password": "changeme"
 }

• I define this extension in program.cs.

builder.Services.AddElasticsearch(builder.Configuration);

• For post, delete, get, update and search operations, we first create our repository, then our service class and then our controller.
• In our Repository class, we add our Elasticsearch connection to the constructor.

  //The usage here is the constructor in the Nest library usage.
        private readonly ElasticClient _elasticClient;
        private const string indexName = "books";
        public BookRepository(ElasticClient elasticClient)
        {
            _elasticClient = elasticClient;
        }

• If you do not specify an ID in Elasticsearch, Elasticsearch automatically creates a unique ID. We specify that the ID will be a guid ID in the book registration method to determine it ourselves.

        public async Task<Book?> SaveAsync(Book newBook)
        {
            var response = await _elasticClient.IndexAsync(newBook, x => x.Index(indexName).Id(Guid.NewGuid().ToString()));

            if (!response.IsSuccess()) return null;

            newBook.Id = response.Id;
            return newBook;
        }

Elasticsearch Search Feature

• The search feature in Elasticsearch allows you to search on one or more indexes. Elasticsearch's powerful search capabilities allow you to perform text search, filtering, and sorting operations quickly and effectively.

How to do a search?

• SearchRequest: An object used to define the search query. SearchRequest contains which indexes to search, the query type, search criteria, and other parameters.

• MultiMatchQuery: This query type is used to search for the search term in more than one field. The fields parameter specifies which fields to search. The Type parameter specifies the query type, and the Operator parameter defines how the search terms will be combined.

• Async/Await: Searches made with Elasticsearch are generally performed asynchronously, so the await keyword is used to wait for the result of the SearchAsync method.

        public async Task<List<Book>> Search(string searchText)
        {
            /*For the query we will make, we specify in which indexes and
               fields the incoming words will be searched with the SearchRequest class.*/
            var searchRequest = new SearchRequest(indexName)
            {
                Size = 10000,
                Query = new MultiMatchQuery
                {
                    Query = searchText,
                    Fields = Infer.Fields<Book>(p => p.Description, p => p.Name),
                    Type = TextQueryType.CrossFields,
                    Operator = Operator.And
                }
            };


            var searchResponse = await _elasticClient.SearchAsync<Book>(searchRequest);

            foreach (var hit in searchResponse.Hits)
            {
                hit.Source.Id = hit.Id;
            };

            return searchResponse.Documents.ToList();
        }