In order to contribute, please follow the Contributing Guidelines.
The Hyperdrive client is managing the Hyperdrive Controller endpoint for CloudServer. It exposes a simple API for a group of HDController.
It does not change the data, it is merely a proxy to HDController.
Basically a subset of HTTP 1.1 using POST, GET and DELETE verbs to create, to get and to delete an object.
On POST, the HDController will create a key that can be used in GET / DELETE to access the data stored.
npm install --save scality/hdclient
# Check for dependencies vulnerabilities
npm auditTo use the linter and run the tests:
# All tests
npm test tests/
# Other options and Mocha help
npm test -- -h
# Code coverage
npm run coverage# Generating JSDoc, to start browsing open docs/jsdoc/index.html
npm run jsdocNode-clinic is installed by default as dev dependency. It can be used to diagnose general performance, I/O specific, event-loop issues, etc. It can also be used to generate flame graphs. Below are some usage examples usage. Note that data acquisition and visualization can be sepearated, they are not in the examples.
# Node clinic diagnosis tool - help
npm run clinic
# Clinic doctor help
npm run clinic doctor
# Diagnosing on the same machine - use --on-port to kickstart load generator
NODE_ENV=production npm run clinic doctor -- \
--on-port='for i in {..10}; do curl -XPUT --data-binary @/etc/hosts "http://localhost:6767/bucket/testobj$i" ; done' \
-- node scripts/server.js 6767 scripts/example_hdclient_proxy.conf.json
NODE_ENV=production npm run clinic doctor -- \
-- node scripts/server.js 6767 scripts/example_hdclient_proxy.conf.json &
pid=$!
# Start and wait for load generator to finish from somewhere else...
kill -SIGINT $pid # Or keep process in foreground and Ctrl-C when done
wait $pid
# Flame graph help
npm run clinic flame
# Flame graph - load-generator on different machine, to be started whenever the server is up
NODE_ENV=production npm run clinic flame -- node scripts/server.js <port> <config file>
...
Ctrl-C
Analysing data
...Because deploying the full S3 server might be too much of a hassle for your specific need, a HTTP server using HdClient is provided. The mapping between object key and internal keys (the ones actually stored on the hyperdrive) is stored in-memory only. The internal keys are not accessible on the outside, mirrorring behavior of Zenko-like deployment.
# Start Hyperdrive 'proxy'
# example conf assumes 1 hdcontroller listening on localhost:18888
NODE_ENV=production node scripts/server.js 8888 scripts/example_hdclient_proxy.conf.json &
# Have fun
curl -XPUT --data-binary @/etc/hosts -v http://localhost:8888/mybucket/testobj
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> PUT /mybucket/testobj HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 267
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 267 out of 267 bytes
< HTTP/1.1 200 OK
< Date: Wed, 27 Jun 2018 10:44:32 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
curl -v http://localhost:8888/mybucket/testobj
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /mybucket/testobj HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 267
< Date: Wed, 27 Jun 2018 10:44:39 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
<payload...>
# <url>/<bucket>/<object>/<version>
curl -XDELETE -v http://localhost:8888/mybucket/testobj/64
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> DELETE /mybucket/testobj HTTP/1.1
> Host: localhost:8888
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Wed, 27 Jun 2018 10:44:58 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intactHow to run integrated hyperdrive client inside S3 or Zenko deployment? This section is only a work in progress since actual S3 integration code is not yet merged.
# Checkout S3 repository and checkout proper hdclient integration branch
git clone https://github.com/scality/CloudServer.git
cd CloudServer/
git checkout feature/RING-28500-add-hyperdrive-client-data-backend-real
# Modify package.json to use the version of hdclient you want in case latest development/1.0 is not good
# To use a local repository
# sed s%scality/hdclient%file:<path to hdclient repository% package.json
# To use a tag or commit
# sed -i s%scality/hdclient%scality/hdclient#<tag/commit>% package.json
# Add new locationConstraints
# Region us-east-1 is mandatory, since the default config still references it
cat <<EOF > hdclient_locationConfig.json
{
"us-east-1": {
"type": "file",
"objectId": "iod1",
"legacyAwsBehavior": true,
"details": {}
},
"hyperdrive-cluster-1": {
"type": "scality",
"objectId": "oid2",
"legacyAwsBehavior": false,
"details": {
"connector": {
"hdclient" : {
"bootstraps": "localhost:18888",
"path": "/store/",
}
}
}
}
}
EOF
# Pattern match restEndpoints - haven't found a better way yet...
# Edit config.json restEndpoints section to use hyperdrive-cluster-1
# e.g. to map localhost onto hdclient: sed -i %"localhost": "us-east-1"%"localhost": "hyperdrive-cluster-1"
# e.g. to map 127.0.0.1 onto hdclient: sed -i %"127.0.0.1": "us-east-1"%"127.0.0.1": "hyperdrive-cluster-1"
# Install dependencies
npm install
# Start a Kafka instance on passed kafkaBrokers parameters (in the example 127.0.0.1:6666)
# Start CloudServer (memory backend ie metadata in-memory)
# More informations inside S3 repository documentation
NODE_ENV=production S3DATA=multiple S3_LOCATION_FILE=hdclient_locationConfig.json npm run mem_backendIn a separate tab, have fun with AWS CLI
# Running S3 server uses default accessKey and secretKey
export AWS_ACCESS_KEY_ID=accessKey1
export AWS_SECRET_ACCESS_KEY=verySecretKey1
# Create a bucket
aws --endpoint-url=http://localhost:8000 s3 mb s3://brandnewbucket
# List buckets
aws --endpoint-url=http://localhost:8000 s3 ls
# Put data
aws --endpoint-url=http://localhost:8000 s3 cp /etc/hosts s3://brandnewbucket/shiny_new_object
# List bucket content
aws --endpoint-url=http://localhost:8000 s3 ls s3://brandnewbucket
# Get data
aws --endpoint-url=http://localhost:8000 s3 cp s3://brandnewbucket/shiny_new_object /tmp/retrieved
# Delete data
aws --endpoint-url=http://localhost:8000 s3 rm s3://brandnewbucket/shiny_new_object