Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GDPR retrieve pubsub (clean) #2294

Merged
merged 3 commits into from
May 7, 2019
Merged

GDPR retrieve pubsub (clean) #2294

merged 3 commits into from
May 7, 2019

Conversation

fenek
Copy link
Member

@fenek fenek commented May 6, 2019

This PR adds support for retrieving personal data from pubsub.

@fenek fenek changed the base branch from master to gdpr-retrieve-clean May 6, 2019 19:19
@mongoose-im
Copy link
Collaborator

mongoose-im commented May 6, 2019 •
edited
Loading

6402.1 / Erlang 19.3 / small_tests / 8cc75ab
Reports root / small

6402.5 / Erlang 19.3 / ldap_mnesia / 8cc75ab
Reports root/ big
OK: 8 / Failed: 0 / User-skipped: 1 / Auto-skipped: 0

6402.4 / Erlang 19.3 / odbc_mssql_mnesia / 8cc75ab
Reports root/ big
OK: 9 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

6402.3 / Erlang 19.3 / mysql_redis / 8cc75ab
Reports root/ big
OK: 9 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

6402.2 / Erlang 19.3 / internal_mnesia / 8cc75ab
Reports root/ big
OK: 9 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

6402.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / 8cc75ab
Reports root/ big
OK: 469 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

6402.9 / Erlang 21.0 / riak_mnesia / 8cc75ab
Reports root/ big / small
OK: 8 / Failed: 1 / User-skipped: 0 / Auto-skipped: 0

gdpr_SUITE:retrieve_personal_data:retrieve_vcard 
{error,{{badmatch,{error,enoent}},
    [{gdpr_SUITE,request_and_unzip_personal_data,3,
           [{file,"gdpr_SUITE.erl"},{line,471}]},
     {gdpr_SUITE,retrieve_and_decode_personal_data,3,
           [{file,"gdpr_SUITE.erl"},{line,461}]},
     {gdpr_SUITE,retrieve_and_validate_personal_data,5,
           [{file,"gdpr_SUITE.erl"},{line,419}]},
     {escalus_story,story,4,
            [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1545}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1063}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,995}]}]}}

Report log

6402.8 / Erlang 20.0 / pgsql_mnesia / 8cc75ab
Reports root/ big / small
OK: 9 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

@codecov
Copy link

codecov bot commented May 7, 2019 •
edited
Loading

Codecov Report

Merging #2294 into gdpr-retrieve-clean will decrease coverage by 60.73%.
The diff coverage is 0%.

Impacted file tree graph

@@                   Coverage Diff                    @@
##           gdpr-retrieve-clean    #2294       +/-   ##
========================================================
- Coverage                78.87%   18.13%   -60.74%     
========================================================
  Files                      334      334               
  Lines                    28994    28892      -102     
========================================================
- Hits                     22870     5241    -17629     
- Misses                    6124    23651    +17527
Impacted Files Coverage Δ
src/pubsub/mod_pubsub_db.erl 60% <ø> (-40%) ⬇️
src/admin_extra/service_admin_extra_gdpr.erl 0% <ø> (-93.23%) ⬇️
src/pubsub/mod_pubsub_db_rdbms_sql.erl 0% <0%> (-93.62%) ⬇️
src/pubsub/mod_pubsub_db_rdbms.erl 0% <0%> (-94.29%) ⬇️
src/pubsub/mod_pubsub.erl 0% <0%> (-71.89%) ⬇️
src/pubsub/mod_pubsub_db_mnesia.erl 35.24% <0%> (-59.47%) ⬇️
src/event_pusher/mod_event_pusher_http.erl 0% <0%> (-100%) ⬇️
src/mongoose_router_global.erl 0% <0%> (-100%) ⬇️
src/mod_auth_token_rdbms.erl 0% <0%> (-100%) ⬇️
src/mod_aws_sns.erl 0% <0%> (-100%) ⬇️
... and 286 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9e597bf...2a09f22. Read the comment docs.

@mongoose-im
Copy link
Collaborator

mongoose-im commented May 7, 2019 •
edited
Loading

6405.1 / Erlang 19.3 / small_tests / 43b2122
Reports root / small

6405.3 / Erlang 19.3 / mysql_redis / 43b2122
Reports root/ big
OK: 3081 / Failed: 1 / User-skipped: 232 / Auto-skipped: 0

mam_SUITE:rsm04:pagination_last5 
{'EXIT',tcp_close_timeout}

Report log

6405.2 / Erlang 19.3 / internal_mnesia / 43b2122
Reports root/ big
OK: 1226 / Failed: 1 / User-skipped: 65 / Auto-skipped: 0

pubsub_SUITE:dag+basic:publish_with_existing_id_test 
{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"pubsub_tools.erl"},{line,468}]},
     {pubsub_tools,receive_response,3,
             [{file,"pubsub_tools.erl"},{line,458}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"pubsub_tools.erl"},{line,448}]},
     {escalus_story,story,4,
            [{file,"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1045}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,977}]}]}}

Report log

6405.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / 43b2122
Reports root/ big
OK: 469 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

6405.5 / Erlang 19.3 / ldap_mnesia / 43b2122
Reports root/ big
OK: 1188 / Failed: 1 / User-skipped: 101 / Auto-skipped: 0

pubsub_SUITE:dag+collection:collection_delete_makes_leaf_parentless 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"alicE59.8873@localhost/res1">>,escalus_tcp,
          <0.3886.1>,
          [{event_manager,<0.3790.1>},
           {server,<<"localhost">>},
           {username,<<"alicE59.8873">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.3790.1>},
            {server,<<"localhost">>},
            {username,<<"alicE59.8873">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"alicE59.8873">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"alicE59.8873">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"D2D4E1AB8C3E5B8A">>}]},
        5000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {pubsub_tools,receive_response,3,
       [{file,"pubsub_tools.erl"},{line,457}]},
     {pubsub_tools,receive_and_check_response,4,
       [{file,"pubsub_tools.erl"},{line,448}]},
     {pubsub_SUITE,'-collection_delete_makes_leaf_parentless/1-fun-0-',1,
       [{file,"pubsub_SUITE.erl"},{line,1214}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/defaul...

Report log

6405.4 / Erlang 19.3 / odbc_mssql_mnesia / 43b2122
Reports root/ big
OK: 3084 / Failed: 0 / User-skipped: 230 / Auto-skipped: 0

6405.8 / Erlang 20.0 / pgsql_mnesia / 43b2122
Reports root/ big / small
OK: 3116 / Failed: 0 / User-skipped: 198 / Auto-skipped: 0

6405.9 / Erlang 21.0 / riak_mnesia / 43b2122
Reports root/ big / small
OK: 1459 / Failed: 1 / User-skipped: 63 / Auto-skipped: 0

mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"eve27.75452@localhost/res1">>,escalus_tcp,
          <0.26163.1>,
          [{event_manager,<0.26154.1>},
           {server,<<"localhost">>},
           {username,<<"eve27.75452">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.26154.1>},
            {server,<<"localhost">>},
            {username,<<"eve27.75452">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"eve27.75452">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"eve27.75452">>},
           {server,<<"localhost">>},
           {password,<<"password">>},
           {port,5222},
           {stream_management,true},
           {stream_id,<<"B0D7B0D4313C981A">>}]},
        10000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {mod_global_distrib_SUITE,
       '-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
       4,
       [{file,"mod_global_distrib_SUITE.erl"},{line,610}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {tes...

Report log

@mongoose-im
Copy link
Collaborator

mongoose-im commented May 7, 2019 •
edited
Loading

6406.1 / Erlang 19.3 / small_tests / dcbd224
Reports root / small

6406.2 / Erlang 19.3 / internal_mnesia / dcbd224
Reports root/ big
OK: 1211 / Failed: 0 / User-skipped: 65 / Auto-skipped: 0

6406.3 / Erlang 19.3 / mysql_redis / dcbd224
Reports root/ big
OK: 3095 / Failed: 1 / User-skipped: 232 / Auto-skipped: 0

pubsub_SUITE:dag+collection:notify_collection_test 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"alicE75.212306@localhost/res1">>,escalus_tcp,
          <0.11000.3>,
          [{event_manager,<0.10900.3>},
           {server,<<"localhost">>},
           {username,<<"alicE75.212306">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.10900.3>},
            {server,<<"localhost">>},
            {username,<<"alicE75.212306">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"alicE75.212306">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"alicE75.212306">>},
           {server,<<"localhost">>},
           {password,<<"matygrysa">>},
           {stream_id,<<"28307D1EADBE5701">>}]},
        5000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {pubsub_tools,receive_response,3,
       [{file,"pubsub_tools.erl"},{line,457}]},
     {pubsub_tools,receive_and_check_response,4,
       [{file,"pubsub_tools.erl"},{line,448}]},
     {pubsub_SUITE,'-notify_collection_test/1-fun-0-',2,
       [{file,"pubsub_SUITE.erl"},{line,1232}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/li...

Report log

6406.4 / Erlang 19.3 / odbc_mssql_mnesia / dcbd224
Reports root/ big
OK: 3097 / Failed: 1 / User-skipped: 230 / Auto-skipped: 0

mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"eve13.786946@localhost/res1">>,escalus_tcp,
          <0.22964.3>,
          [{event_manager,<0.22955.3>},
           {server,<<"localhost">>},
           {username,<<"eve13.786946">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.22955.3>},
            {server,<<"localhost">>},
            {username,<<"eve13.786946">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"eve13.786946">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"eve13.786946">>},
           {server,<<"localhost">>},
           {password,<<"password">>},
           {port,5222},
           {stream_management,true},
           {stream_id,<<"9EFFBAD48097418F">>}]},
        10000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {mod_global_distrib_SUITE,
       '-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
       4,
       [{file,"mod_global_distrib_SUITE.erl"},{line,610}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
    ...

Report log

6406.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / dcbd224
Reports root/ big
OK: 469 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0

6406.5 / Erlang 19.3 / ldap_mnesia / dcbd224
Reports root/ big
OK: 1175 / Failed: 0 / User-skipped: 101 / Auto-skipped: 0

6406.8 / Erlang 20.0 / pgsql_mnesia / dcbd224
Reports root/ big / small
OK: 3116 / Failed: 0 / User-skipped: 198 / Auto-skipped: 0

6406.9 / Erlang 21.0 / riak_mnesia / dcbd224
Reports root/ big / small
OK: 1472 / Failed: 2 / User-skipped: 63 / Auto-skipped: 0

mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"eve48.91764@localhost/res1">>,escalus_tcp,
          <0.26157.1>,
          [{event_manager,<0.26148.1>},
           {server,<<"localhost">>},
           {username,<<"eve48.91764">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.26148.1>},
            {server,<<"localhost">>},
            {username,<<"eve48.91764">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"eve48.91764">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"eve48.91764">>},
           {server,<<"localhost">>},
           {password,<<"password">>},
           {port,5222},
           {stream_management,true},
           {stream_id,<<"82646ECAD91141E4">>}]},
        10000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {mod_global_distrib_SUITE,
       '-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
       4,
       [{file,"mod_global_distrib_SUITE.erl"},{line,610}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {tes...

Report log

mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server 
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"eve72.782779@localhost/res1">>,escalus_tcp,
          <0.26829.1>,
          [{event_manager,<0.26820.1>},
           {server,<<"localhost">>},
           {username,<<"eve72.782779">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.26820.1>},
            {server,<<"localhost">>},
            {username,<<"eve72.782779">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"eve72.782779">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"eve72.782779">>},
           {server,<<"localhost">>},
           {password,<<"password">>},
           {port,5222},
           {stream_management,true},
           {stream_id,<<"8832045CB168C32B">>}]},
        10000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {mod_global_distrib_SUITE,
       '-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
       4,
       [{file,"mod_global_distrib_SUITE.erl"},{line,610}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
    ...

Report log

aleklisi
aleklisi reviewed May 7, 2019
View reviewed changes
Copy link
Contributor

@aleklisi aleklisi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR looks good 👍

big_tests/tests/gdpr_SUITE.erl
Alice, Config, "pubsub", ExpectedHeader, ExpectedItems)
Node1 = {_Domain, NodeName1} = pubsub_tools:pubsub_node(),
Node2 = {_Domain, NodeName2} = pubsub_tools:pubsub_node(),
{BinItem1, StringItem1} = item_content(<<"Item1Data">>),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about extracting to something like that:

ItemsData = [
{<<"Item1Data">>, Node1}, 
{<<"Item2Data">>,Node1 }
...],
lists:map(
    fun({ItemData, Node}) -> 
        {BinItem, StringItem} = item_content(ItemData),
        pubsub_tools:publish(Alice, BinItem, Node, [{with_payload, BinItem}])
    end, ItemsData),

??

@fenek fenek merged commit 0e05c6f into gdpr-retrieve-clean May 7, 2019
@fenek fenek deleted the gdpr-retrieve-pubsub-clean branch May 7, 2019 12:10
DenysGonchar pushed a commit that referenced this pull request May 27, 2019
@fenek @DenysGonchar
GDPR retrieve pubsub (clean) (#2294)
8644a07 
* Implement GDPR retrieve for pubsub

* Use all tests again

* Fix Dialyzer error in pubsub
@fenek fenek added this to the MongooseIM 3.3.0++ milestone Jun 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aleklisi aleklisi aleklisi left review comments

Assignees
No one assigned
Labels
GDPR waiting-for-ci waiting-for-review
Projects
None yet
Milestone
3.4.0
Development

Successfully merging this pull request may close these issues.
4 participants
@fenek @mongoose-im @aleklisi @ludwikbukowski