New format handling CSV NDJSON

[alallema]

New handler to accept CSV and NDJSON format.

• New functions:

def add_documents_json(string_docs, primary_key)
def add_documents_csv(string_docs, primary_key)
def add_documents_ndjson(string_docs, primary_key)
def add_documents_raw(string_docs, primary_key, type)

These functions accept document data as string.
The add_documents_raw() has been added to avoid code duplication but is not mandatory.
The add_documents() still exists and gives the possibility to add documents as List.

Example of usage:

client = Client('http://127.0.0.1:7700', 'masterKey')

# JSON
jsonfile = open('../../data/dataset/movies.json', 'r')
data = jsonfile.read()
index = client.index('movies')
udpate = index.add_documents_json(data.encode('utf-8'))

# NDJSON
ndjsonfile = open('../../data/dataset/songs.ndjson', 'r')
data = ndjsonfile.read()
index = client.index('songsND')
udpate = index.add_documents_ndjson(data.encode('utf-8'))

# CSV as file
csvfile = open('../../data/dataset/songs.csv', 'r')
data = csvfile.read()
index = client.index('songs')
udpate = index.add_documents_csv(data.encode('utf-8'))

Please note:

• The add_documents_csv() can also take file directly after open it, if the option rb is given to open(path_to_file, 'rb'). I haven't found a way to avoid this.

[curquiza]

In the integration guide issue I wrote:

The SDKs always send application/json as Content-Type to every request: this should be adapted to the concerned requests (POST and PUT)

What is your investigation about this, is it complicated to adapt?

Also, in the typesense docs, they don't seem to accept string as documents, but object or list
cf

• the code base: https://github.com/typesense/typesense-python/blob/8a07f5ba1fd286e2f109866109effc25f50acd36/typesense/documents.py#L53-L72
• they documentation (tab python): https://typesense.org/docs/0.21.0/api/documents.html#configure-batch-size

Wouldn't it be an easier way to approach it for the users, accepting object instead of string? Look like there is less parsing on the user side.

[alallema]

@curquiza,

The SDKs always send application/json as Content-Type to every request it still true I let the default Content-Type as application/json it only changes when a CSV or NDJSON is sent.

For the format:

• The real problem is the csv it really complicated to transform it to string and make it work well. And it didn't work as expected because I can send a file directly to it.
• If we ask users to send objects or lists it would work with add_documents() basic function. It could be a good option I will try it.

[curquiza]

The SDKs always send application/json as Content-Type to every request it still true I let the default Content-Type as application/json it only changes when a CSV or NDJSON is sent.

Is it possible to only send Content-type for POST and PUT then, and not for DELETE and GET, or is it too complicated? That's what I meant with "his should be adapted to the concerned requests (POST and PUT)"

[alallema]

The SDKs always send application/json as Content-Type to every request it still true I let the default Content-Type as application/json it only changes when a CSV or NDJSON is sent.

Is it possible to only send Content-type for POST and PUT then, and not for DELETE and GET, or is it too complicated? That's what I meant with "his should be adapted to the concerned requests (POST and PUT)"

Oh yes, I think so! Sorry misunderstood

[bidoubiwa]

The real problem is the csv it really complicated to transform it to string and make it work well. And it didn't work as expected because I can send a file directly to it.

I feel like I don't really understand the problem 😕 Why would the user not send his CSV in string? Is it not possible to do the following:

const dataset = fs.readFileSync('./mydataset.csv')
client.index("movies").addDocumentsCsv(dataset)

and then at our side:

request.post(dataset, { "Content-type": "application/csv" })

I'm really sorry if I'm missing something

[alallema]

The real problem is the csv it really complicated to transform it to string and make it work well. And it didn't work as expected because I can send a file directly to it.

I feel like I don't really understand the problem 😕 Why would the user not send his CSV in string? Is it not possible to do the following:

const dataset = fs.readFileSync('./mydataset.csv')
client.index("movies").addDocumentsCsv(dataset)

and then at our side:

request.post(dataset, { "Content-type": "application/csv" })

I'm really sorry if I'm missing something

You right It's work well!