Redis credentials

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/workflows/docker-publish.yml

@@ -0,0 +1,56 @@
+name: Docker
+
+on:
+  push:
+    # Publish `master` as Docker `latest` image.
+    branches:
+      - master
+
+    # Publish `v1.2.3` tags as releases.
+    tags:
+      - v*
+
+  # Run tests for any PRs.
+  pull_request:
+
+env:
+  # TODO: Change variable to your image's name.
+  IMAGE_NAME: phoenix
+
+jobs:
+  # Push image to GitHub Packages.
+  # See also https://docs.docker.com/docker-hub/builds/
+  push:
+    runs-on: ubuntu-latest
+    if: github.event_name == 'push'
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Build image
+        run: docker build . --file Dockerfile --tag $IMAGE_NAME
+
+      - name: Log into registry
+        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
+
+      - name: Push image
+        run: |
+          IMAGE_ID=docker.pkg.github.com/${{ github.repository }}/$IMAGE_NAME
+
+          # Change all uppercase to lowercase
+          IMAGE_ID=$(echo $IMAGE_ID | tr '[A-Z]' '[a-z]')
+
+          # Strip git ref prefix from version
+          VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')
+
+          # Strip "v" prefix from tag name
+          [[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')
+
+          # Use Docker `latest` tag convention
+          [ "$VERSION" == "master" ] && VERSION=latest
+
+          echo IMAGE_ID=$IMAGE_ID
+          echo VERSION=$VERSION
+
+          docker tag $IMAGE_NAME $IMAGE_ID:$VERSION
+          docker push $IMAGE_ID:$VERSION

.gitignore

@@ -11,4 +11,5 @@
 .localstack/
 .vscode/
 data/
-.idea
+.idea
+.DS_Store

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at davide.berdin@rtl.nl. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,29 @@
+## How to contribute to Phoenix
+
+#### **Did you find a bug?**
+
+* **Ensure the bug was not already reported** by searching on GitHub under [Issues](https://github.com/rtlnl/phoenix/issues).
+
+* If you're unable to find an open issue addressing the problem, [open a new one](https://github.com/rtlnl/phoenix/issues/new). Be sure to include a **title and clear description**, as much relevant information as possible, and a **code sample** or an **executable test case** demonstrating the expected behavior that is not occurring.
+
+#### **Did you write a patch that fixes a bug?**
+
+* Open a new GitHub pull request with the patch.
+
+* Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable.
+
+#### **Did you fix whitespace, format code, or make a purely cosmetic patch?**
+
+Changes that are cosmetic in nature and do not add anything substantial to the stability, functionality, or testability of Phoenix will generally not be accepted.
+
+#### **Do you intend to add a new feature or change an existing one?**
+
+* Suggest your change in the [Issue](https://github.com/rtlnl/phoenix/issues) and start writing code.
+
+#### **Do you have questions about the source code?**
+
+* Ask any question about how to use Phoenix in the [Issues](https://github.com/rtlnl/phoenix/issues) section
+
+Thanks! :heart: :heart: :heart:
+
+Phoenix Team

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 RTL Nederland
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,50 +1,37 @@
-# Phoenix project
+![Docker](https://github.com/rtlnl/phoenix/workflows/Docker/badge.svg?branch=master) [![Go Report Card](https://goreportcard.com/badge/github.com/rtlnl/phoenix)](https://goreportcard.com/report/github.com/rtlnl/phoenix)
 
-The Project is divided in two main parts:
+# Phoenix
 
-- Public APIs
-- Internal APIs
-- Worker
+<a href="https://ibb.co/j6Mv7b5"><img src="https://i.ibb.co/j6Mv7b5/gopher-phoenix.png" align="right" alt="gopher-phoenix" border="0"></a>
 
-## How to start
+Phoenix is the **delivery recommendation systems** that is used at RTL Nederland. These APIs are able to deliver millions of recommendations per day. We use Phoenix for powering [Videoland](https://www.videoland.com/) and [RTL Nieuws](https://www.rtlnieuws.nl/). Our data science team works very hard to generate tailored recommendations to each user and we, as the platform team, make sure that these recommendations are actually delivered.
 
-Assuming that you have `go`, `docker` and `docker-compose` installed in your machine, run `docker-compose up -d` to spin up Redis and localstack (for local S3).
+Simple, yet powerful API for delivery recommendations
 
-After having the services up and running, assuming that you have your Go environment in your `PATH`, you should be able to start directly with `go run main.go --help`. This command should print the `help` message.
+* **Easy to understand** - push and get data from Redis very quickly
+* **Fast in deliverying** - the combination of Go, Redis and Allegro cache makes the project blazing fast
+* **Smart in storing** - the APIs avoid the overload of the Redis database by using a worker for bulk uplaod
 
-Proceed by running `go run main.go internal` for the internal APIs (or `go run main.go public` for the public APIs)
+We have being used in production since December 2019 and we haven't had a single downtime since. So far, we have delivered more than 350M recommendations to our users. The average request latency is `35ms`.
 
-If you need to upload some files to the local S3, use the following commands after `localstack` has been created:
+## Quick start
 
-- `aws --endpoint-url=http://localhost:4572 s3 mb s3://test` to create a bucket in local S3
-- `aws --endpoint-url=http://localhost:4572 s3api put-bucket-acl --bucket test --acl public-read` to set up a policy for testing with local s3
-- `aws --endpoint-url=http://localhost:4572 s3 cp ~/Desktop/data.csv s3://test/content/20190713/` to copy a file to local S3
+Assuming that you have `go`, `docker` and `docker-compose` installed in your machine, you need to have 3 terminals open that points to the directory where the project is. Do the following
 
-## How to run tests
+1. In terminal number 1, run `docker-compose up -d` to spin up Redis and localstack (for local S3)
+2. In terminal number 1 run `go run main.go worker` for the Worker service
+3. In terminal number 2 run `go run main.go internal` for the Internal APIs
+4. In terminal number 3 run `go run main.go public` for the Public APIs
 
-To run all the tests, use the following command:
+Now you are ready to go :rocket:
 
-```bash
-$: go clean -testcache && go test -race ./...
-```
+## Docs
 
-The first part is to avoid that Go will cache the result of the tests. This could lead to some evaluation errors
-if you change some tests. Better without cache.
+The documentation for developing and using Phoenix is available in the [wiki](https://github.com/rtlnl/phoenix/wiki)
 
-## How to perform manual tests
+## Join the Phoenix Community
+In order to contribute to Phoenix, see the [CONTRIBUTING](CONTRIBUTING.md) file for how to go get started.
+If your company or your product is using Phoenix, please let us know by adding yourself to the Phoenix [users](USERS.md) file.
 
-For manual testing of endpoints on the different environments, a [Postman collection](docs/postman/Phoenix.postman_collection.json) has been included in the `docs` directory.
-
-## Batch Upload
-
-The APIs gives the possibility to read a file from S3 and upload it to Redis. To avoid timeouts and having the client hanging waiting for the response, the APIs has a simple `checking` mechanism. The picture below explain how the process works
-
-![](/docs/images/batch_upload.png)
-
-The process of uploading the file from S3 to Redis is delegated to a separate `go routine`. The client should store the `batchID` that is returned from the initial request `(POST /v1/batch)` and ask for the status with `GET /v1/batch/status/:id`.
-
-Time taken to upload **1.6M unique keys** from S3 is `3m 33secs`. Check this [PR](https://github.com/rtlnl/phoenix/pull/5) for more information
-
-### Worker
-
-There can be only 1 worker working at the time. The worker is responsible for reading the `batchUpload` requests from the Redis Queue and execute them 1 by 1.
+## License
+Phoenix is licensed under MIT license as found in the [LICENSE](LICENSE.md) file.

USERS.md

@@ -0,0 +1,5 @@
+# who is using Phoenix
+
+* RTL Nederland
+* Videoland
+* RTL Nieuws

chart/Chart.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+appVersion: "1.0"
+description: A Helm chart for Phoenix APIs
+name: api
+version: 0.3.0

chart/README.md

@@ -0,0 +1,50 @@
+# Phoenix
+
+Current chart version is `0.3.0`.
+
+## Deploy Phoenix
+
+The default values will create all the services that are required to make Phoenix work out of the box. Make sure you are setting properly the `ENV` variables to connect to `S3`. To install the chart simply run the below command:
+
+```bash
+$: helm upgrade --install phoenix ./chart -f values.yaml --namespace phoenix --debug --recreate-pods
+```
+
+If you find something that doesn't work, please open up an Issue or a PR! We :heart: contibutions
+
+## Chart Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| fullnameOverride | string | `""` |  |
+| nameOverride | string | `""` |  |
+| image.pullPolicy | string | `"Always"` | Image pull policy |
+| image.repository | string | `"docker.pkg.github.com/rtlnl/phoenix/phoenix"` | Image repository name |
+| image.tag | string | `"latest"` | Image tag |
+| ingress.annotations | object | `{}` | Ingress annotations (values are templated) |
+| ingress.enabled | bool | `false` | Enables Ingress |
+| ingress.hosts | array | `[]` | Ingress accepted hostnames |
+| internal.affinity | object | `{}` | Affinity settings for pod assignment |
+| internal.data | string | `{}` | ENV Varibles to be set for the internal service |
+| internal.nodeSelector | object | `{}` | Node labels for pod assignment |
+| internal.replicaCount | int | `1` | Number of nodes |
+| internal.resources | object | `{}` | CPU/Memory resource requests/limits |
+| internal.service.port | int | `8081` | Kubernetes port where service is exposed |
+| internal.service.type | string | `"ClusterIP"` | Kubernetes service type |
+| internal.tolerations | object | `{}` | Toleration labels for pod assignment  |
+| public.affinity | object | `{}` | Affinity settings for pod assignment |
+| public.data | string | `{}` | ENV Varibles to be set for the public service |
+| public.nodeSelector | object | `{}` | Node labels for pod assignment |
+| public.replicaCount | int | `1` | Number of nodes |
+| public.resources | object | `{}` | CPU/Memory resource requests/limits |
+| public.secrets | object | `{}` | Secrets to add to the public service |
+| public.service.port | int | `8082` | Kubernetes port where service is exposed |
+| public.service.type | string | `"ClusterIP"` | Kubernetes service type |
+| public.tolerations | object | `{}` | Toleration labels for pod assignment |
+| redis.enabled | bool | `true` | Enable the deployment of a local redis instance |
+| redis.resources | object | `{}` | CPU/Memory resource requests/limits  |
+| worker.affinity | object | `{}` | Affinity settings for pod assignment |
+| worker.data | string | `{}` | ENV Varibles to be set for the worker service |
+| worker.replicaCount | int | `1` | Number of nodes |
+| worker.resources | object | `{}` | CPU/Memory resource requests/limits |
+| worker.tolerations | object | `{}` | Toleration labels for pod assignment |