-
-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature] REST API for main controller features #379 #386
[feature] REST API for main controller features #379 #386
Conversation
@atb00ker The tests cases are failing because of this
I tested it by creating a new Is there any way to fix this or am I missing or doing something wrong. 🤔 💭 |
I have not looked into the problem in detail, but could you please use Read more: https://stackoverflow.com/a/24811490/6410464 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tests cases are failing because of this
from . import views as api_views File "/home/runner/work/openwisp-controller/openwisp-controller/openwisp_controller/api/views.py", line 12, in <module> from openwisp_users.api.mixins import FilterByOrganizationManaged ModuleNotFoundError: No module named 'openwisp_users.api.mixins'
I tested it by creating a new
env
but while installing all the dependencies and requirements I found that the problem arises from theopenwisp-users
module all the files from this https://github.com/openwisp/openwisp-users/tree/master/openwisp_users/api folder gets downloaded except themixins.py
file. I created that file manually and all the test cases are passing.Is there any way to fix this or am I missing or doing something wrong.
Rebase on the latest master to pull the changes just merged from #374.
The solution to the problem lies in these lines:
7b78fa0#diff-4d7c51b1efe9043e44439a949dfd92e5827321b34082903477fd04876edb7552L8-R9
For reference see PEP 508.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR is too basic, not much logic is implemented.
Please read again the issue description and follow the instructions closely!
If there's something that is not clear about the issue description, quote each line which is not clear and ask specific questions about what is not clear.
openwisp_controller/api/urls.py
Outdated
urlpatterns = [ | ||
path( | ||
'controller/', | ||
include( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this nested include really needed or is it the best practice to do this? I've never seen it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since more than one app's API's are present in this module I tried to keep the URL in the main urls.py as api/v1/
so if I don't use nesting here then each time I write path I will be required to add controller
to it, so proceeded with this approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe, you could move these URLs in a separate file and get urls from the function there like done in openwisp-radius urls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, I will do it in this manner.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@atb00ker this is done.
class DeviceSerializer(ValidatedModelSerializer): | ||
class Meta: | ||
model = Device | ||
fields = '__all__' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all the templates will surely need to contain more logic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
85edce7
to
5760b39
Compare
4feda2a
to
a1888ff
Compare
openwisp_controller/api/urls.py
Outdated
path( | ||
'device/<str:pk>/', api_views.device_detail, name='device_detail', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quick Q: shouldn't the details pages be read-only and we should have separate edit pages.
Currently, I went to http://127.0.0.1:8000/api/v1/controller/device/<pk>/
and it gave me details and the form to edit the device! 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, @atb00ker I had a discussion about this with Federico, to have separate endpoints for GET/PATCH/PUT of the device detail and other endpoints but he suggested me to keep it in this way. 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the info, I found the comment: #379 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I highly recommend you guys to read this book: https://www.amazon.com/RESTful-Web-Services-Leonard-Richardson/dp/0596529260
Having separate endpoints for editing a resource does not follow the RESTful API design principles.
A resource can be created (POST /), retrieved (GET /<pk>
), updated (PUT /<pk>
), destroyed (DELETE /<pk>
).
This is the standard across pretty much all modern REST APIs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the recommendation, I'll check it out! 😄
Looks like it's going in the correct direction! 😄 |
806d3c6
to
fab61ba
Compare
fab61ba
to
c5a72ce
Compare
@nemesisdesign , @atb00ker , @pandafy I am having a tough time solving this error for the past few days and not able to proceed further, While trying to make a PUT/PATCH request to this I have tried this approach https://stackoverflow.com/questions/50015204/direct-assignment-to-the-forward-side-of-a-many-to-many-set-is-prohibited-use-e but didn't seem to work. Also tried overriding the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ManishShah120 I must see your attempted implementation in order to correct it. Can you point out what lines of codes are you talking about specifically that are not working out?
While we get this sorted out, can you start working on adding tests for the other endpoints which are working and also work on the README documentation?
I was able to solve the problem stackoverflow with the link that you send:
However, this will not work because you will need to create a new |
Yes @atb00ker I was stuck with the |
Provide a config with some basic data. (eg: (You'll need to check in validation and ensure user gets a more meaningful error when they provide an empty config.) |
e8b1d70
to
88a2c48
Compare
@atb00ker, @nemesisdesign , I have added the And after the device being created, I am able to add more template to any configuration but not able to remove it, But when I try to perform all the operations like assign/unassign of templates, changing the order, everything works fine with the |
0d351ce
to
88a2c48
Compare
Where? I don't see those functions, can you please push that code, it's hard to tell the error without seeing the code.
I think a better way to explain the problem would be to follow the format:
|
1 similar comment
Where? I don't see those functions, can you please push that code, it's hard to tell the error without seeing the code.
I think a better way to explain the problem would be to follow the format:
|
def create(self, validated_data): | ||
config_data = validated_data.pop('config') | ||
config_data.pop('templates') | ||
device = Device.objects.create(**validated_data) | ||
Config.objects.create(device=device, **config_data) | ||
return device |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here after removing the templates
field,I am able to create a device, with its configuration, but we should not pop this field, because while creating a device user may want to assign any number of templates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@atb00ker here is the .create
method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, I'll have to look into why do you need to pop the templates.
If I remember correctly, some UTs are saving device without poping template; I would start looking there.
I'll look into it as well! (Probably weekend!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just checked it out, looks like this is fixed.
Please update here if done.
openwisp_controller/api/urls.py
Outdated
'device-config/<str:pk>/', | ||
api_views.device_config, | ||
name='device_config', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here, I have added an extra endpoint which works fine for all operation we want from the configurations associated with any device we want to perform, but the problems arises when we try to perform operations on the nested models and serializers.
99cc1ea
to
4eacd3c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! I did some basic testing again and it's working as expected.
I found other minor details to improve and I realized something is missing: extensibility (running a modified version of the API when the SAMPLE_APP=1
environment variable is set).
See the work done by @atb00ker here:
- https://github.com/openwisp/openwisp-controller/tree/a91e9e76cc0d640f504380bb465f9508f2ff7981#other-base-classes-that-can-be-inherited-and-extended
- https://github.com/openwisp/openwisp-controller/blob/a91e9e76cc0d640f504380bb465f9508f2ff7981/tests/openwisp2/sample_config/views.py
I think we need to:
- add
sample_config.api.views
- ensure API urls are imported here: https://github.com/openwisp/openwisp-controller/blob/a91e9e76cc0d640f504380bb465f9508f2ff7981/tests/openwisp2/urls.py
- import api tests here: https://github.com/openwisp/openwisp-controller/blob/a91e9e76cc0d640f504380bb465f9508f2ff7981/tests/openwisp2/sample_config/tests.py
- update the docs in a consistent way with what we have now, linking the REST API in the sample app
@atb00ker please double check this and let me know if I forgot anything regarding extensibility.
See also my comments below.
data['vpn'] = vpn1.id | ||
response = self.client.post(path, data, content_type='application/json') | ||
validation_msg = "To select a VPN, set the template type to 'VPN-client'" | ||
self.assertIn(validation_msg, response.data['vpn']) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's always a good idea to assert for the status code as well, eg: self.assertEqual(r.status_code, 400)
data = self._get_template_data.copy() | ||
data['type'] = 'vpn' | ||
data['vpn'] = vpn1.id | ||
data['organization'] = org1.pk |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if the vpn is shared, usually the template is shared as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but aren't we testing this for the operator and operator can't create shared templates, other than that admin gets all the the vpn by default, so how are we gonna test for the shared objects. 🤔
r = self.client.get(path, {'format': 'api'}) | ||
self.assertContains(r, 'shared_ca</option>') | ||
self.assertContains(r, 'shared_cert</option>') | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see a test which ensures shared templates can be assigned to a device, was that already added before? If not, can you please add it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
openwisp_controller/config/utils.py
Outdated
@@ -199,3 +201,49 @@ def get_default_templates_queryset( | |||
if backend: | |||
queryset = queryset.filter(backend=backend) | |||
return queryset | |||
|
|||
|
|||
def get_api_urls(api_views): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is this function being added here? I think it should go in openwisp_controller.config.api.urls
.
BTW the reasons we use functions is to allow to extende the API views and URLs but I don't see anything done on that front but I think we should.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, will move it to openwisp_controller.config.api.urls
- Moved url function to api dir - Added response status code check in tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, right now, I am not doing code review just the functional review.
Legends:
[SU] - Tested with Superuser
[O1A] - Tested with Org1 Administrator
[O1O] - Tested with Org1 Operator
Status:Pass - Behaves like admin UI or deviation documented
Status:Fail - Does not behave like admin UI
Status:NTD - Need to discuss
if getattr(settings, 'OPENWISP_CONTROLLER_API', True): | ||
return [ | ||
path( | ||
'controller/template/', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SN | /api/v1/controller/template/ | Status |
---|---|---|
1 | [SU] Can view all templates | Pass |
2 | [O1A] can view only org1 templates (not even shared org) | Pass |
3 | [O1O] can view no templates | Pass |
4 | [SU] VPN-Client hidden for Generic Template | NTD |
5 | [SU][O1A] Field validation error on field | Fail |
6 | [SU][O1A] Allows creating template without config | Fail |
7 | [SU][O1A][O1O] Allows creating template with config | Pass |
8 | [SU][O1A][O1O] Allow creating VPN template | Pass |
9 | [SU][O1A][O1O] Allow creating VPN template without openvpn | NTD |
10 | [O1O] Access denied to org1 operator user | NTD |
11 | [SU] Unique template name ensured | Pass |
12 | [O1A] Trying to create template for another org gives object not found error | Pass |
SN. 4:
@nemesisdesign I think this is an acceptable deviation, can you please confirm.
SN. 5:
@ManishShah120 Can we show the error on config instead of __all__
? it would be useful for people making apps using the API.
SN. 9
@nemesisdesign is this okay?
SN. 10
@nemesisdesign Shouldn't we deny access to the page if creating and viewing is not allowed? Same for admin UI. Why is access even allowed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SN /api/v1/controller/template/ Status
1 [SU] Can view all templates Pass
2 [O1A] can view only org1 templates (not even shared org) Pass
3 [O1O] can view no templates Pass
4 [SU] VPN-Client hidden for Generic Template NTD
5 [SU][O1A] Field validation error on field Fail
6 [SU][O1A] Allows creating template without config Fail
7 [SU][O1A][O1O] Allows creating template with config Pass
8 [SU][O1A][O1O] Allow creating VPN template Pass
9 [SU][O1A][O1O] Allow creating VPN template without openvpn NTD
10 [O1O] Access denied to org1 operator user NTD
11 [SU] Unique template name ensured Pass
12 [O1A] Trying to create template for another org gives object not found error Pass
SN. 4:
@nemesisdesign I think this is an acceptable deviation, can you please confirm.
Yes it's acceptable because it's an API and not a frontend.
SN. 5:
@ManishShah120 Can we show the error on config instead of__all__
? it would be useful for people making apps using the API.
If possible this would be a good improvement.
It does look good to me. Why it shouldn't? I don't understand your point.
SN. 10
@nemesisdesign Shouldn't we deny access to the page if creating and viewing is not allowed? Same for admin UI. Why is access even allowed?
Right, good catch!
DRF doesn't implement the view permission by default:
https://www.django-rest-framework.org/api-guide/permissions/#djangomodelpermissions
We have to do it. Please @ManishShah120 can you add a DjangoModelPermissions
class which inherits from DjangoModelPermissions
and adds the view permission as indicated in the DRF docs?
Then we should use this new permission class instead of the default one.
@atb00ker regarding the admin view, what do you refer to exactly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nemesisdesign is this okay?
It does look good to me. Why it shouldn't? I don't understand your point.
The VPN servers created from UI always have a openvpn section in configuration, here we are able to make it without that! 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@atb00ker SN5, SN6 is resolved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@atb00ker regarding SN10, I think you missed one more failing case which is foralong with this [O1A] administrator but not manager i.e., (is_admin for org1 not set to True)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copying @nemesisdesign 's response on Gitter:
@ManishShah120 @atb00ker I am not sure I understand what we're discussing, can you be more explicit and provide some user stories as examples?
I am not following you.
As a general rule, for now, I'd simplify everything:
only organization managers (having is_admin=True) or superusers can use the API
non superusers have limited permissions, according to the permissions defined in django
So if user is not manager of the org, it can't do anything via the API right now.
If user is superuser, can do anything.
If user is not superuser and is manager of an org, it can do what their django permissions allow them to do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More functional checks, we have some failures, please check them. Please test all the cases and ensure they are passing before the next functional review, this will help speed up the process! 😄
I have tested VPN & Config, I expect similar failures in device create page, can you please check that as well before I do functional check of the entire thing again? :-)
-
one more problem I've noticed in multiple places is that when there are multiple errors in the form, it reports only one or two of them, no all. I think you are returning when you encounter an error, can we return after we have collected all the errors? 😄
-
After sucessful creation of device/template/vpn, can we clean the form?
-
After succesful deletion, it does not redirect me to list page.
P.S: Please wait for Fed to check the 3 points above.
name='api_template_list', | ||
), | ||
path( | ||
'controller/template/<str:pk>/', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
controller/template/str:pk/ | Status | |
---|---|---|
1 | [SU] Can view any templates | Pass |
2 | [O1A] Can view only org1 templates (not even shared org) | Pass |
3 | [O1O] Can view no templates | Pass |
4 | [SU][O1A] Correct data of the template shown | Pass |
5 | [SU][O1A][O1O] Can only delete templates if they have correct privilege | Pass |
name='api_template_detail', | ||
), | ||
path( | ||
'controller/template/<str:pk>/configuration/', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
controller/template/str:pk/configuration | Status | |
---|---|---|
1 | [SU][O1A][O1O] Can download any configuration | Pass |
2 | [SU][O1A] Correct template config downloaded | Pass |
api_views.download_template_config, | ||
name='api_download_template_config', | ||
), | ||
path('controller/vpn/', api_views.vpn_list, name='api_vpn_list',), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
controller/vpn/ | Status | Fail Reason | |
---|---|---|---|
1 | [SU] Can view any VPN | Pass | |
2 | [O1A] Can view only org1 VPN (not even shared org) | Pass | |
3 | [O1O] Cannot view any VPN | Pass | |
4 | [SU][O1A] Field validation error on field | Fail | { "config" : ["This field cannot be blank."], "all" :["Invalid configuration triggered by "#/", validator says:\n\n'openvpn' is a required property"] } |
5 | [SU] Only valid config schema accepted | Pass | |
6 | [SU][O1A] Can create VPN | Pass | |
7 | [SU][O1A] Cannot create VPN | Pass | |
8 | [SU][O1A] Unique name per org check | Pass | |
9 | [SU][O1A] Appropriate CA/Certs visible | Fail | O1A does not see shared CA/Certs |
10 | [01A] Trying to create device with another org's id gives object not found error | Pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm afraid "the invalid configuration triggered by ... " is defined at model level and we have to leave as is for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
S9, is working fine for me.
), | ||
path('controller/vpn/', api_views.vpn_list, name='api_vpn_list',), | ||
path( | ||
'controller/vpn/<str:pk>/', api_views.vpn_detail, name='api_vpn_detail', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SN | controller/vpn/str:pk/ | Status |
---|---|---|
1 | [SU] Can view any vpn | Pass |
2 | [O1A] Can view only org1 vpns (not even shared org) | Pass |
3 | [O1O] Cannot view any vpn | Pass |
4 | [SU][O1A] Correct data of the vpn shown | Pass |
5 | [SU][O1A][O1O] Can only delete templates if they have correct privilege | Pass |
'controller/vpn/<str:pk>/', api_views.vpn_detail, name='api_vpn_detail', | ||
), | ||
path( | ||
'controller/vpn/<str:pk>/configuration/', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
controller/vpn/str:pk/configuration | Status | |
---|---|---|
1 | [SU] Can download any configuration | Pass |
2 | [SU][O1A] Correct template config downloaded | Pass |
Can you point out the parts of the code you refer to? We need to be precise with our requests, we have to ensure there's no ambiguity.
Isn't this the standard behavior of DRF? |
Hi, @nemesisdesign @atb00ker all the tests are passing locally, I am not able to figure out the actual reason for build failing here. |
There were too many cases of it that I didn't document them, after @ManishShah120 spends sometime at it, if I still find a case, I'll document it! 😄 P.S: I think I am going to start a document for the functional testing -- to list all TC and failures, might be useful in the future as well! 😄 |
@ManishShah120 Have you tried P.S: also Rebase! 😄 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all the tests are passing locally, I am not able to figure out the actual reason for build failing here.
@ManishShah120 Have you tried
SAMPLE_APP=1 ./runtests.py
as well?P.S: also Rebase!
👍
Sounds good, thank you Ajay. |
Yes, I am trying to tackle each one of them. |
Hi, @atb00ker @nemesisdesign I tried to override the
Base But with the above code all the Else If I am missing something, please suggest. 😄 other than this all the issue from @atb00ker review is resolved. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok @ManishShah120, you've done great work in this PR and I think this is now ready for usage, even though there's some minor details missing for which I have created issues:
- [change] REST API: implement view permissions #454 (depends on [feature] Add class to implement view permissions in DRF openwisp-users#249)
@atb00ker let me know if you find other issues and let's create more follow up issues if needed.
We will surely find things to improve in the following weeks while testing, so @ManishShah120 stay tuned and please follow up and help us with the issues mentioned above, all those are needed for a proper implementation of a REST API in @openwisp 😊 .
Closes #379