Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rodrigo Alves Sarmento #81

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules
.git
45 changes: 45 additions & 0 deletions .env.dist
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
###> symfony/framework-bundle ###
APP_ENV=prod
APP_SECRET=ebfb6dfbab639815bc1da66e6bf7509e
###< symfony/framework-bundle ###

# Configuration of the mysql docker container
# Example:
# MYSQL_ROOT_PASSWORD=rootpass00
# MYSQL_DATABASE=picpay
# MYSQL_USER=user
# MYSQL_PASSWORD=userpass00

MYSQL_ROOT_PASSWORD=<root_password>
MYSQL_DATABASE=<dbname>
MYSQL_USER=<user>
MYSQL_PASSWORD=<password>

###> doctrine/doctrine-bundle ###
# Example:
# DATABASE_URL=mysql://user:userpass00@db:3306/picpay?charset=utf8mb4&serverVersion=5.7

DATABASE_URL=mysql://<user>:<password>db:3306/<dbname>?charset=utf8mb4&serverVersion=5.7
###< doctrine/doctrine-bundle ###

# Data files
# Example:
# USERS_FILE_PATH=users.csv
# PRIORITY1_FILE_PATH=lista_relevancia_1.txt
# PRIORITY2_FILE_PATH=lista_relevancia_2.txt

USERS_FILE_PATH=<users_path_file>
PRIORITY1_FILE_PATH=<priority_1_path_file>
PRIORITY2_FILE_PATH=<priority_2_path_file>

###> nelmio/cors-bundle ###
# Example:
# CORS_ALLOW_ORIGIN=^https?://localhost:?[0-9]*$
CORS_ALLOW_ORIGIN=^https?://<dominio>:?[0-9]*$
###< nelmio/cors-bundle ###

# Elasticsearch base URI
# Example:
# ELASTIC_HTTP_CLIENT_BASE_URI=es:9200

ELASTIC_HTTP_CLIENT_BASE_URI=<elastic_base_uri>
17 changes: 17 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
###> Composer ###
composer.phar
###< Composer ###

###> symfony/framework-bundle ###
/.env.local
/.env.local.php
/.env.*.local
/public/bundles/
/var/
/vendor/
/.idea/
###< symfony/framework-bundle ###

.env
*.csv
sync
43 changes: 43 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,46 @@ Faça um ***Fork*** deste repositório e abra um ***Pull Request***, **com seu n
- Ter um desempenho elevado num conjunto de dados muito grande
- Utilizar o Docker

# Solução

Foi desenvolvido um sistema que disponibiliza via URL http://<host da aplicação> um sistema com duas telas. A primeira disponibiliza consulta à base de dados MySQL e a segunda consulta a base de dados do Elasticsearch. É possível visualizar no topo da tela o tempo necessário para a consulta, permitindo ao usuário comparar as duas abordagens.

Para facilitar a comparação das consultas das duas telas, os resultados podem ser exibidos ordenados por nome. Caso seja escolhido ordenar, será possível testar o impacto da ordenação no tempo de consulta do MySQL e do Elasticsearch. Obs: Não é feita ordenação por username.

Para a construção da solução foram levadas em consideração algumas das práticas propostas pela metodologia do [The Twelve-Factor App](https://12factor.net/config)

# Limitação

O Elasticsearch possui uma limitação no que diz respeito à paginação de conjuntos de dados muito grandes. Se tentarmos, por exemplo, acessar a última página (538545) do conjunto de dados fornecido sem qualquer filtro, o Elasticsearch vai acabar dando timeout, enquanto o MySQL consegue retornar. Não consegui encontrar uma solução para isso até o momento.

Lembrando apenas que o caso acima é extremo. Para várias consultas, mesmo com a ordenação por nome, o número de páginas se restringe ao número na casa dos milhares, o que mantém o tempo de busca em poucos segundos.

### Autor
- Nome: Rodrigo Alves Sarmento
- E-mail: rasarmento@gmail.com

### Tecnologias
- Docker
- Docker Compose
- MySQL 5.7
- Elasticearch 6.6
- Symfony 4.2
- Angular 4.x
- Tema Bootstrap Smartadmin 1.8 ([Visitar](https://wrapbootstrap.com/theme/smartadmin-responsive-webapp-WB0573SK0))

### Configuração
- Criar o arquivo ***app/.env*** a partir do ***app/.env.dist***
- No arquivo ***app/.env*** preencher a URL do servidor que contém a api (servidor do docker): ***API_HTTP_CLIENT_BASE_URI***
- Criar o arquivo ***.env*** a partir do ***.env.dist***
- No arquivo ***.env*** preencher os dados de configuração do container docker ***mysql***: ***MYSQL_ROOT_PASSWORD***, ***MYSQL_DATABASE***, ***MYSQL_USER*** e ***MYSQL_PASSWORD***
- No arquivo ***.env*** preencher os dados da URL de conexão do doctrine: ***DATABASE_URL***
- No arquivo ***.env*** preencher os locais dos arquivos com os dados de usuários: ***USERS_FILE_PATH***, ***PRIORITY1_FILE_PATH*** e ***PRIORITY2_FILE_PATH***
- No arquivo ***.env*** preencher o domínio de onde serão enviadas as requisições à API: ***CORS_ALLOW_ORIGIN***
- No arquivo ***.env*** preencher a URI do Elasticsearch: ***ELASTIC_HTTP_CLIENT_BASE_URI***
- Obs: Para cada configuração será fornecido um exemplo

### Instalação
- Construir containers: ```docker-compose -f docker-compose.yml -f app/docker-compose.prod.yml up -d --build```
- Realizar deploy: ```./cli/deploy```
- Carregar base de dados MySQL: http://<host da aplicação>:8080/api/user/import
- Carregar base de dados Elasticsearch: http://<host da aplicação>:8080/api/user/es/import
56 changes: 56 additions & 0 deletions app/.angular-cli.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
{
"$schema": "./node_modules/@angular/cli/lib/config/schema.json",
"project": {
"name": "smartadmin"
},
"apps": [
{
"root": "src",
"outDir": "dist",
"assets": [
"assets"
],
"index": "index.html",
"main": "main.ts",
"polyfills": "polyfills.ts",
"test": "test.ts",
"tsconfig": "tsconfig.app.json",
"testTsconfig": "tsconfig.spec.json",
"prefix": "app",
"styles": [
"styles.css"
],
"scripts": [
],
"environmentSource": "environments/environment.ts"
}
],
"e2e": {
"protractor": {
"config": "./protractor.conf.js"
}
},
"lint": [
{
"project": "src/tsconfig.app.json"
},
{
"project": "src/tsconfig.spec.json"
},
{
"project": "e2e/tsconfig.e2e.json"
}
],
"test": {
"karma": {
"config": "./karma.conf.js"
}
},
"defaults": {
"styleExt": "css",
"component": {
"style": false,
"template": false
}
}
}
6 changes: 6 additions & 0 deletions app/.env.dist
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# API server base URI
# Example:
# API_HTTP_CLIENT_BASE_URI=http://localhost:8080

API_HTTP_CLIENT_BASE_URI=<api_base_uri>

2 changes: 2 additions & 0 deletions app/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
dist
node_modules
16 changes: 16 additions & 0 deletions app/.idea/codeStyles/Project.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions app/.idea/codeStyles/codeStyleConfig.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions app/.idea/encodings.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions app/.idea/inspectionProfiles/Project_Default.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions app/.idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions app/.idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions app/.idea/painel-frontend.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions app/.idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading