Use User-Agent header to have analytics

[curquiza]

Replaces: #58

Currently, we have no analytics about integrations.
As a first step, we would like to know the proportion of each integration.

Thanks to the new spec implementation, the User-Agent header will be sent to Segment (and then to Amplitude) via the MeiliSearch server 🎉

Here is the expected format:

User-Agent: Meilisearch <language> (<version>)

-> The version should be in parenthesis and the information must start with Meilisearch

Ex:

User-Agent: Meilisearch JS (v0.22.1)

If multiple integrations are used during the user path, they should be separated by ;.

User-Agent: Meilisearch Rails (v0.2.3) ; Meilisearch Ruby (v0.16.0)

-> In this example, it means the Ruby SDK should be able to accept a custom User-Agent that the Rails plugin can pass when initializing the Ruby client.

---

First step: do it basically in all the SDKs

The SDKs should send the expected User-Agent header. In this step, no need to implement a way to pass a custom User-Agent header.
We only need to find a smart way (if possible) to pass the version, and/or to check the version is well updated for each release (using the .github/script/check-release.sh script for example)

• meilisearch-dotnet (Feature/Analytics meilisearch-dotnet#241)
• meilisearch-dart (Feature/Analytics meilisearch-dart#129)
• meilisearch-go (Feature/Analytics meilisearch-go#279)
• meilisearch-java (Feature/Analytics meilisearch-java#503)
• meilisearch-js (Add client header with package information meilisearch-js#1272)
• meilisearch-php (Feature/Analytics meilisearch-php#294) and Add the ability to provide extra custom user-agents meilisearch-php#369
• meilisearch-python (Feature/Analytics meilisearch-python#433)
• meilisearch-ruby (Feature/Analytics meilisearch-ruby#292) and (Support custom agents meilisearch-ruby#354)
• meilisearch-rust (Feature/Analytics meilisearch-rust#254)
• meilisearch-swift (Feature/Analytics meilisearch-swift#277)
• meilisearch-strapi-plugin (Pass strapi client agent to meilisearch-js strapi-plugin-meilisearch#455)
• docs-searchbar.js (Pass client agent to meilisearch-js docs-searchbar.js#621)
• meilisearch-firebase (Pass firebase client agent to meilisearch-js firestore-meilisearch#91)
• meilisearch-rails (Feature/Analytics meilisearch-rails#179)
• meilisearch-symfony (Feature/Add analytics meilisearch-symfony#193)
• meilisearch-vuepress (Pass client agents to meilisearch vuepress-plugin-meilisearch#244)
• meilisearch-gatsby (Pass gatsby client agent to meilisearch-js gatsby-plugin-meilisearch#147)
• instant-meilisearch (Pass client agent to meilisearch-js meilisearch-js-plugins#780)
• laravel scout ([9.x] Add analytics for meilisearch engine laravel/scout#681)
• docs-scraper (Feature/add analytics docs-scraper#387)

Second step: do it in every integration

This step involves some SDKs (JS, PHP, Ruby, and probably more in the future) that can accept a custom User-Agent header. This needs an internal discussion for implementation and, why not, might deserve another issue.

[curquiza]

FYI @meilisearch/product-team

[bidoubiwa]

Overwrite initial user-agent

User-agent may already contain some content. For example when making a request from my browser, the back-end server receives the following user-agent

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36

Unfortunately, I can not append new content to the end of the user-agent. Meaning that if I need to send MeiliSearch JS (v0.22.1) as a user agent, I will override the initial user-agent value. The receiver of the request will lose these information (example above).

Code example:

Front end request:

let myHeaders = new Headers();
myHeaders.append('user-agent', 'MeiliSearch JS (v0.22.1)');

fetch('/test',  {
  headers: myHeaders
}).then(res => console.log(res))

Back-end receiver

app.get('/test', (req, res) => {
  console.log(req.headers["user-agent"])
})

The console.log will output MeiliSearch JS (v0.22.1) instead of Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36

Unable to modify user-agent in chrome

Now, this is an unfortunate problem. user-agent used to be in a list of forbidden header names. It is now allowed as per the spec.

It has not yet been implemented in chromium based browsers (thus chrome also). See issue dating from 2015. Meaning that the above code sample results in chrome silently ignoring my user-agent update thus returning the original user-agent:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36.

These tests have been done with the standard library Fetch. I tried with the standard library XMLHttpRequest as well. In this case, changes are ignored as well but an error is outputted in my browser console: Refused to set unsafe header "user-agent"

  const xhttp = new XMLHttpRequest();

  xhttp.onload = function() {
    console.log(this.responseText)
  }
  xhttp.open("GET", "/test", true);
  xhttp.setRequestHeader('user-agent', "plouf") // Refused to set unsafe header "user-agent"
  xhttp.send();

Conclusion

Do we want to keep the context of the browser? If yes, then we cannot implement this feature.

Do we accept to lose the context?
We overwrite the user-agent with information about the context in which the request is made (ex: MeiliSearch + instantSearch) and we lose the context of the browser. Exactly like described by @curquiza

What about chrome
The solution we would implement that overwrites the user-agent does not throw on chrome when using the Fetch API. It is just silently ignored and returns the default user-agent. Meanwhile, it does work on all other browsers. So it introduces an enhancement that works in some cases and is just ignored in others.

[gmourier]

Hi @bidoubiwa! Thank you for this complete and accurate feedback.

If I understand correctly, it's mostly languages running on Chrome that won't be able to send their User-Agent using the conforming header.

The solution could be to ask for a custom user-agent, we would like to avoid it for several reasons. It is possible that some HTTP clients will encounter problems by not being able to override the user-agent. It is also possible that some proxies will remove a custom header by default.

In the vast majority of cases, I have the impression that we won't have the information about the use of js SDKs via chrome. Am I wrong?

If so, we can make the assumption that it is the most used. There is no real need to know if it should be maintained or prioritized in my opinion but you tell me.

The idea behind this, if I'm not mistaken @curquiza, was to know to what extent we should prioritize efforts on SDKs that probably wouldn't be used.

[curquiza]

After the discussion with @bidoubiwa and @gmourier

We keep User-Agent header for the moment to forward the SDK information into MeiliSearch. Each SDK should fill the User-Agent header.
The other integrations should be done in a second time since it will involve more implementation in the code base (cf the current issue)

The problem on Chrome we have for front-end is still present and should be solved in the future, for example by sending the analytics information as query parameters. We need first to ensure the first implementation we've done works.

TLDR;
The current issue description is still what we need to implement, nothing has changed, except we are more aware of some limits.

[bidoubiwa]

I agree with the proposal. Even without the information from chrome we still at least receive some information :)
Just to be sure, the separation with the ; it should be prefixed and sufixed with a whitespace ?

[gmourier]

I agree with the proposal. Even without the information from chrome we still at least receive some information :) Just to be sure, the separation with the ; it should be prefixed and sufixed with a whitespace ?

No need to add whitespaces to compute it. On the core side, we separate User-Agent with ;. @irevoire do you confirm? It is rather for readability needs when inspecting HTTP headers with a naked eye. Then it's very specific. Maybe we don't need to spend time on this, maybe the libs do it automatically if you pass a user-agent array?

[irevoire]

Yes we split the user-agent on the ; and we put everything in a Set 👍

[brunoocasali]

List of the qualified names used in the headers:

• Meilisearch .NET
• Meilisearch PHP
• Meilisearch Ruby
• Meilisearch Dart
• Meilisearch JavaScript
• Meilisearch Rust
• Meilisearch Java
• Meilisearch Go
• Meilisearch Swift
• Meilisearch Python

And:

• Meilisearch Rails
• Meilisearch Symfony
• Meilisearch Firebase
• Meilisearch Strapi
• Meilisearch Gatsby
• Meilisearch Laravel Scout
• Meilisearch docs-searchbar.js
• Meilisearch instant-meilisearch
• Meilisearch DocsScraper

[gmourier]

Hi @brunoocasali! Thanks for the list!

It seems it's missing Meilisearch Python here. Am I right?

Also, what's about Laravel Scout, Firebase, and Gatsby plugins? Could be nice to know it also!

[brunoocasali]

Hi @brunoocasali! Thanks for the list!

It seems it's missing Meilisearch Python here. Am I right?

Also, what's about Laravel Scout, Firebase, and Gatsby plugins? Could be nice to know it also!

Oh yeah, I forgot to add them thanks for the reminder. About the Laravel, I'm not sure if we will be able to add the customization there.

[bidoubiwa]

I'm adding the missing docs-scraper repo

[brunoocasali]

We fuck*n made it, team. You're the best! 🎉🎉🎉