Support $search in query syntax #334

daffl · 2016-05-20T06:18:23Z

daffl · 2016-05-20T06:22:33Z

Also related:

ghost · 2016-07-20T11:15:33Z

any news on this feature ?
if it is not implemented , how to fuzzy match a property now without this special attribute ?

daffl · 2016-07-20T15:47:27Z

You can create a hook that for e.g. MongoDB converts the $search query to a $regex:

app.service('todos').before({
  find(hook) {
    const query = hook.params.query;
    if(query.name.$search) {
      query.name = { $regex: new RegExp(query.name.$search) }
    }
  }
});

ghost · 2016-07-21T07:53:30Z

the way to make fuzzy match for some field in collection before your comment was to use custom route and write native query with same service db engine like "mongo" or "nedb" .
but this way is very good and show me new use case to use hooks in featherjs

ghost · 2016-07-22T01:34:46Z

what about this hook as simple solution for search in mongodb & nedb

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search) }
      }
    }
    hook.params.query = query
    return hook
  }
}

and simply include it in the

src/services/ServiceName/hooks

like

exports.before = {
  all: [],
  find: [globalHooks.searchRegex()]
}

ruddfawcett · 2016-08-17T19:57:19Z

For those interested, you can make the above code posted by @alnour-altegani case insensitive by changing

 query[field] = { $regex: new RegExp(query[field].$search) }

to:

 query[field] = { $regex: new RegExp(query[field].$search, 'i') } // note the 'i'

beeplin · 2016-10-07T01:02:28Z

so why $search, not $regex?
hope it compatible with mongoDB's original syntax, which is available now out-of-box already, so we don't need to change it in the future.

cklmercer · 2016-11-14T20:59:52Z

Just stumbled upon this after fighting with the Mongoose adapter for half an hour. This would be a super nice addition.

Any suggestions for a work-around for the feathers-mongoose adapter?

daffl · 2016-11-14T22:31:09Z

There is a working hook two comments above. Mongoose already supports it, you just have to convert the query into a regular expression in a before hook.

By now, I am also leaning more towards not adding this. Defining an abstract search format for all databases isn't really possible. Some support %like% queries, others use regular expressions and some do not support searching at all. Fuzzy matches should be implemented with the features of the database you are using.

cklmercer · 2016-11-14T23:48:52Z

Thanks for the quick reply.

sajov · 2017-01-04T19:43:07Z

got it, my fault. Thanks

my solution with implementation of $or for multiple $search

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search, 'i') }
      }
      if(field == '$or') {
        query[field].map((action, index) => {
            let f = Object.keys(action)[0];
            if(action[f].$search) {
                action[f] = { $regex: new RegExp(action[f].$search, 'i') }
            }
            return action;
        });
      }
    }
    hook.params.query = query
    return hook
  }
}

But how you guys deal with integer?

ghost · 2017-01-06T18:48:25Z

@sajov thank you for sharing your work . but, can you give example of how to use this code to search with $or operator ?

sajov · 2017-01-06T18:51:49Z

@ruddfawcett i use it in a datatable context like this

for (var i = 0;i < opts.tableHeader.length; i++) {
                        let q = {};
                        q[opts.tableHeader[i]] = {$search: queryObj.search.value.value};
                        query.$or.push(q);
                    }

here is an working example https://github.com/sajov
I write README right now

pimvanderheijden · 2017-01-16T14:58:24Z

@sajov Were you able to write a README ? I'd be very interested to see how I could use this case in combination with an $or -array e.g. like

[
    { _id: searchInput },
    { type: searchInput }
]

sajov · 2017-01-16T20:35:42Z

@MidNightP

Hook example which treat $search as { $regex: new RegExp(value, 'i') };
https://github.com/sajov/riot-crud/blob/master/example/src/hooks/index.js#L50-L53

Client example
https://github.com/sajov/riot-crud/blob/master/tags/themes/bootstrap/views/crud-views.tag#L169-L170

query.$or = [
    {fieldA : {$search: value}},
    {fieldB : {$search: value}},
];

hope that answer your question

pimvanderheijden · 2017-01-18T14:54:00Z

@sajov Thanks!

I rewrote things a bit and ended up using the hook below for a client side query containing this:

orRegex: [
  { 
    fieldA: {
      $search: 'searchinput'
    }
  },
  { 
    fieldB: {
      $search: 'searchinput'
    }
  },
  et cetera...
]

Hook:

return function(hook) {
    const { query } = hook.params

    if( Object.keys(query).includes('orRegex') )  {
      const or = query.orRegex.map((field) => {
        console.log('Object.keys(field)[0]: ', Object.keys(field)[0] )

        const attribute = Object.keys(field)[0]

        return {
          [attribute]: {
            $regex: new RegExp(field[Object.keys(field)[0]]['$search']), $options: 'ix'
          }
        }
      })

      delete(query.orRegex)

      hook.params.query['$or'] = or
    }
    return hook
  }

Suggestions welcome of course

daffl · 2017-01-18T15:16:12Z

Maybe @eddyystop has some thoughts if this can be turned into a common hook.

eddyystop · 2017-03-23T12:18:37Z

Just saw this now. I've created a link in feathers-hooks-common feathersjs-ecosystem/feathers-hooks-common#141

arve0 · 2017-06-25T21:18:51Z

Here is a fuzzy match for NeDB, it searches all properties case insensitive:

module.exports = function (options = {}) { // eslint-disable-line no-unused-vars
  return function (hook) {
    if (hook.params.query && hook.params.query.$search) {
      hook.params.query.$where = fuzzySearch(hook.params.query.$search)
      delete hook.params.query.$search
    }
    return hook
  }
}

/**
 * Returns a $where function for NeDB. The function search all
 * properties of objects and returns true if `str` is found in
 * one of the properties. Searching is not case sensitive.
 *
 * @param {string} str search for this string
 * @return {function}
 */
function fuzzySearch (str) {
  let r = new RegExp(str, 'i')

  return function () {
    for (let key in this) {
      // do not search _id and similar fields
      if (key[0] === '_' || !this.hasOwnProperty(key)) {
        continue
      }
      if (this[key].match(r)) {
        return true
      }
    }
    return false
  }
}

I actually found this to be a bit quicker than single property regex. With fuzzy search I got 47 ms average time for service.find(), versus ~68 ms for single field $regex. Times are average of 50 times loop over 7 different search terms, and the DB contains about 14000 rows, each object has four properties, all of them text. Only one property has a lengthy, 140 chars, text field. I guess the implementation is vulnerable to DOS attacks, but $regex should also be.

arve0 · 2017-06-26T21:56:09Z

Based on my comment above I've made to plugins:

Ref feathersjs/feathers#334: > We will add documentation for searching to the adapters individually.

* Add note about searching documents Ref feathersjs/feathers#334: > We will add documentation for searching to the adapters individually. * add REST example to $search section

ulrichborchers · 2018-08-01T07:13:49Z

Hi,

stumbled into this thread from querying.md:
https://github.com/feathersjs/docs/blob/master/api/databases/querying.md

... which is referring here.

Noticed a little mistake in the querying sample URI for the $in,$nin example.

If I am not mistaken it should be:
/messages?roomId[$in][]=2&roomId[$in][]=5

instead of:
/messages?roomId[$in]=2&roomId[$in]=5

because $in is an array in the query.

sajov · 2018-08-01T07:27:59Z

Hi Ulrich,

/messages?roomId[$in]=2&roomId[$in]=5

get parsed by qs into

/messages?roomId[$in][0]=2&roomId[$in][1]=5

see the option extended

zsf3 · 2018-12-31T11:28:48Z

@sajov Thank you for the hook. Btw, what is plain array in your code? why are you concatenating it query[field]? Some comments might help in understanding this logic please?

glazs · 2019-01-06T12:46:30Z

Hi everybody! I'm trying to use feathers-nedb-fuzzy-search

My query is: http://localhost:3030/messages?$sort[date]=-1&$skip=0&$search=2

And result: {"name":"BadRequest","message":"Invalid query parameter $where","code":400,"className":"bad-request","data":{"$sort":{"date":"-1"},"$skip":"0"},"errors":{}}

error: BadRequest: Invalid query parameter $where
    at new BadRequest (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/errors/lib/index.js:86:17)
    at _.each (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/filter-query.js:48:17)
    at Object.keys.forEach.key (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/commons/lib/utils.js:12:39)
    at Array.forEach (<anonymous>)
    at Object.each (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/commons/lib/utils.js:12:24)
    at cleanQuery (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/filter-query.js:41:7)
    at filterQuery (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/filter-query.js:107:18)
    at Object.filterQuery (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/service.js:46:20)
    at Object._find (/home/glazs/workspace/lmt/server/node_modules/feathers-nedb/lib/index.js:36:47)
    at callMethod (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/adapter-commons/lib/service.js:9:20)
error: TypeError: Expected a string
    at module.exports (/home/glazs/workspace/lmt/server/node_modules/escape-string-regexp/index.js:7:9)
    at fuzzySearch (/home/glazs/workspace/lmt/server/node_modules/feathers-nedb-fuzzy-search/index.js:49:22)
    at Object.<anonymous> (/home/glazs/workspace/lmt/server/node_modules/feathers-nedb-fuzzy-search/index.js:33:34)
    at promise.then.hookObject (/home/glazs/workspace/lmt/server/node_modules/@feathersjs/commons/lib/hooks.js:142:73)

What's wrong with my Feathers/NeDB?

daffl · 2019-01-06T17:15:01Z

In the latest versions of all database adapters non-standard query parameters have to be explicitly whitelisted.

nedbService({
  whitelist: [ '$where', '$search' ]
});

jscottsf · 2019-01-06T17:18:36Z

The whitelisting seems to break feathers-vuex since it uses commons. Not related to this thread tho. I'll dig deeper on that and log an issue over there.

https://github.com/feathers-plus/feathers-vuex

1MikeMakuch · 2019-01-09T18:19:39Z

what about this hook as simple solution for search in mongodb & nedb

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search) }
      }
    }
    hook.params.query = query
    return hook
  }
}

and simply include it in the

src/services/ServiceName/hooks

like

exports.before = {
  all: [],
  find: [globalHooks.searchRegex()]
}

This from ghost works for me.

Zalasanjay · 2019-11-05T12:31:51Z

While using feathers-mongodb-fuzzy-search module If you got an error like Invalid $regex

Then you can try whitelist attribute to white list that keywords in your mongoose service configuration in <service_name>.service.js file

const options = {
    paginate,
    whitelist: ['$regex', '$options']
};

lcampanis · 2021-07-25T20:23:23Z

This is an ALL inclusive search - meaning it looks for a match in all fields:

what about this hook as simple solution for search in mongodb & nedb

exports.searchRegex = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query[field] = { $regex: new RegExp(query[field].$search) }
      }
    }
    hook.params.query = query
    return hook
  }
}

and simply include it in the

src/services/ServiceName/hooks

like

exports.before = {
  all: [],
  find: [globalHooks.searchRegex()]
}

This will match any of the fields with $or:

exports.searchRegEx = function () {
  return function (hook) {
    const query = hook.params.query;
    for (let field in query) {
      if(query[field].$search && field.indexOf('$') == -1) {
        query.$or = [...(query.$or || []), { [field]: { $regex: new RegExp(query[field].$search, 'ig') } }];
        delete query[field];
      }
    }
    hook.params.query = query
    return hook
  }
}
`

DaddyWarbucks · 2022-03-11T21:57:53Z

Here are a couple examples of how I typically handle this in both Sequelize and Mongo/Mongoose

Sequelize

export const withSearch = (searchProps, idProps) => (context) => {
  if (!context.params.query) {
    return context;
  }

  const { $search, ...query } = context.params.query;

  if (!$search) {
    context.params.query = query;
    return context;
  }

  const $or = [];

  searchProps.forEach((prop) => {
    $or.push({ [prop]: { $iLike: `%${$search}%` } });
  });

  if (idProps && isValidInteger($search)) {
    idProps.forEach((prop) => {
      $or.push({ [prop]: $search });
    });
  }

  context.params.query = {
    ...query,
    $or: [...(query.$or || []), ...$or]
  };

  return context;
};

Mongo/Mongoose

const traverse = require('traverse');

// https://stackoverflow.com/questions/3446170/escape-string-for-use-in-javascript-regex
const escapeRegExp = (string) => {
  return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
};

module.exports.searchRegex = (stringProps, numberProps) => (context) => {
  if (!context.params.query) {
    return context;
  }

  // TODO: Deepclone? Or is traverse immutable?
  const query = { ...context.params.query };
  const $search = query.$search;
  delete query.$search;

  // Find and replace any `$search` properties, even when
  // nested in query or query arrays
  traverse(query).forEach(function (value) {
    if (this.parent && this.parent.node.$search !== undefined) {
      this.parent.node.$regex = new RegExp(escapeRegExp(value), 'i');
      delete this.parent.node.$search;
    }
  });

  if ($search && stringProps) {
    query.$or = query.$or || [];
    stringProps.forEach((prop) => {
      query.$or.push({
        [prop]: { $regex: new RegExp(escapeRegExp($search), 'i') }
      });
    });
  }

  if ($search && !isNaN(Number($search)) && numberProps) {
    query.$or = query.$or || [];
    numberProps.forEach((prop) => {
      query.$or.push({
        [prop]: Number($search)
      });
    });
  }

  context.params.query = query;

  return context;
};

The Sequelize example is pretty similar to many I have seen here already, except it handles Integer fields as well. The Mongo/Mongoose example is better because it uses traverse and can handle deeply nested queries in $and, $or, etc. The Sequelize example should be updated to include that

This was referenced May 20, 2016

Support $search feathersjs-ecosystem/feathers-query-filters#12

Closed

Support $search feathersjs-ecosystem/feathers-levelup#7

Closed

daffl changed the title ~~Add $search to query syntax~~ Support $search in query syntax May 20, 2016

ekryski mentioned this issue May 20, 2016

Feathers Auk Release #299

Closed

28 tasks

ekryski added the Backlog label May 21, 2016

ekryski added this to the Auk milestone May 21, 2016

ekryski added the Feature label May 21, 2016

marshallswain mentioned this issue Jul 6, 2016

Add $regexp type filter feathersjs-ecosystem/feathers-rethinkdb#16

Closed

ekryski modified the milestones: Auk, Buzzard Oct 17, 2016

ekryski mentioned this issue Oct 17, 2016

Feathers Buzzard Release #436

Closed

18 tasks

eddyystop mentioned this issue Mar 23, 2017

Support $search in query syntax feathersjs-ecosystem/feathers-hooks-common#141

Closed

benawad mentioned this issue Apr 11, 2017

Add "How to do a search query" to FAQ feathersjs-ecosystem/docs#506

Closed

arve0 added a commit to arve0/feathers-docs that referenced this issue Jun 26, 2017

Add note about searching documents

4010845

Ref feathersjs/feathers#334: > We will add documentation for searching to the adapters individually.

arve0 mentioned this issue Jun 26, 2017

Add note about searching documents feathersjs-ecosystem/docs#707

Merged

claustres mentioned this issue Aug 30, 2017

Added management for fuzzy fields arve0/feathers-mongodb-fuzzy-search#2

Closed

claustres mentioned this issue Jan 9, 2018

[Feature Request] Multi-column fuzzy search. À la feathers-nedb-fuzzy-search arve0/feathers-mongodb-fuzzy-search#9

Open

eddyystop mentioned this issue Feb 2, 2018

Suggestion: Add DB specific hooks feathersjs-ecosystem/feathers-hooks-common#347

Closed

daffl added a commit that referenced this issue Aug 25, 2018

Do not generate the users service if it already exists (#334)

bb92f0d

eddyystop mentioned this issue Jan 10, 2019

Hook to support $search in query feathersjs-ecosystem/feathers-hooks-common#501

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support $search in query syntax #334

Support $search in query syntax #334

daffl commented May 20, 2016 •

edited by marshallswain

Loading

daffl commented May 20, 2016

ghost commented Jul 20, 2016

daffl commented Jul 20, 2016

ghost commented Jul 21, 2016

ghost commented Jul 22, 2016 •

edited by ghost

Loading

ruddfawcett commented Aug 17, 2016

beeplin commented Oct 7, 2016

cklmercer commented Nov 14, 2016 •

edited

Loading

daffl commented Nov 14, 2016 •

edited

Loading

cklmercer commented Nov 14, 2016

sajov commented Jan 4, 2017 •

edited

Loading

ghost commented Jan 6, 2017

sajov commented Jan 6, 2017 •

edited

Loading

pimvanderheijden commented Jan 16, 2017

sajov commented Jan 16, 2017 •

edited

Loading

pimvanderheijden commented Jan 18, 2017 •

edited

Loading

daffl commented Jan 18, 2017

eddyystop commented Mar 23, 2017 •

edited

Loading

arve0 commented Jun 25, 2017

arve0 commented Jun 26, 2017

ulrichborchers commented Aug 1, 2018

sajov commented Aug 1, 2018

zsf3 commented Dec 31, 2018

glazs commented Jan 6, 2019 •

edited

Loading

daffl commented Jan 6, 2019

jscottsf commented Jan 6, 2019

1MikeMakuch commented Jan 9, 2019 •

edited

Loading

Zalasanjay commented Nov 5, 2019

lcampanis commented Jul 25, 2021

DaddyWarbucks commented Mar 11, 2022

Support $search in query syntax #334

Support $search in query syntax #334

Comments

daffl commented May 20, 2016 • edited by marshallswain Loading

Service Adapter Completion

daffl commented May 20, 2016

ghost commented Jul 20, 2016

daffl commented Jul 20, 2016

ghost commented Jul 21, 2016

ghost commented Jul 22, 2016 • edited by ghost Loading

ruddfawcett commented Aug 17, 2016

beeplin commented Oct 7, 2016

cklmercer commented Nov 14, 2016 • edited Loading

daffl commented Nov 14, 2016 • edited Loading

cklmercer commented Nov 14, 2016

sajov commented Jan 4, 2017 • edited Loading

ghost commented Jan 6, 2017

sajov commented Jan 6, 2017 • edited Loading

pimvanderheijden commented Jan 16, 2017

sajov commented Jan 16, 2017 • edited Loading

pimvanderheijden commented Jan 18, 2017 • edited Loading

daffl commented Jan 18, 2017

eddyystop commented Mar 23, 2017 • edited Loading

arve0 commented Jun 25, 2017

arve0 commented Jun 26, 2017

ulrichborchers commented Aug 1, 2018

sajov commented Aug 1, 2018

zsf3 commented Dec 31, 2018

glazs commented Jan 6, 2019 • edited Loading

daffl commented Jan 6, 2019

jscottsf commented Jan 6, 2019

1MikeMakuch commented Jan 9, 2019 • edited Loading

Zalasanjay commented Nov 5, 2019

lcampanis commented Jul 25, 2021

DaddyWarbucks commented Mar 11, 2022

daffl commented May 20, 2016 •

edited by marshallswain

Loading

ghost commented Jul 22, 2016 •

edited by ghost

Loading

cklmercer commented Nov 14, 2016 •

edited

Loading

daffl commented Nov 14, 2016 •

edited

Loading

sajov commented Jan 4, 2017 •

edited

Loading

sajov commented Jan 6, 2017 •

edited

Loading

sajov commented Jan 16, 2017 •

edited

Loading

pimvanderheijden commented Jan 18, 2017 •

edited

Loading

eddyystop commented Mar 23, 2017 •

edited

Loading

glazs commented Jan 6, 2019 •

edited

Loading

1MikeMakuch commented Jan 9, 2019 •

edited

Loading