WorkBC API Console - Industries parameter

[BK01]

The Industries parameter in the WorkBC API console does not appear to be working. Please investigate and update the api-spec.

Test performed:
Leave all parameters blank, with the exception of the Industry parameter. Enter ‘16’ and execute. Look at the total count of results returned and note the number. Now re-execute, but change the industry ID to ‘3’. The total count and the results don’t change at all.

spec:
https://github.com/bcgov/api-specs/tree/master/jobposting

However, the same parameter appears to work in the old console:
http://apps.gov.bc.ca/pub/api-explorer/?url=https://workbcjobs.api.gov.bc.ca/v1/specs/#!/Job_Feed/post_jobs

[banders]

The issue affects OAS2. It is related to parameters that are passed as form data. Seems that for OAS2 recent versions of swagger-ui are issuing requests with Content Type headers that are not consistent with form data parameters. The Content Type should be application/x-www-form-urlencoded, but is instead application/json. As such, the underlying API interprets the request as if no parameters were passed.

We have also translated the "jobposting" spec to OAS3, so I tested with that under the latest release of swagger-ui (3.4.2) as well. This version of swagger-ui appears to have buggy support for application/x-www-form-urlencoded content in the request body. I have posted a comment on an existing swagger-ui issue to make the core developers aware of the problem we're encountering.

@BK01 In summary, there are two separate problems with the latest version of swagger-ui: one affecting how form data are handled for OAS2 specs, and another affecting how form data are handled in OAS3 specs. Because we are migrating our specs to OAS3, I suggest we pursue a solution affecting OAS3 only. Thoughts?

[BK01]

@banders Thank you for identifying the issue. I agree, given the timing and progress already made on OAS3 / s-ui in BCDC, please proceed with a solution in OAS3. I have temporarily removed the link to the new viewer for WorkBC and replaced with the old viewer. Once resolved, we can switch back.

[BK01]

@BK01 - UAT with latest viewer

[BK01]

@banders In cad.data, the user input parameters appear to be missing from the /jobs POST request section.

Previous version containing input parameters can be found here for example.

[banders]

@BK01 Are you testing with the OAS3 spec?

https://raw.githubusercontent.com/banders/api-specs/upgrade-to-openapi3/jobposting/jobposting.json

The new version of swagger-ui (not yet deployed in CAD right?) should fix the issue for the /jobs POST operation using OAS3. (There is still a swagger-ui bug affecting this operation for OAS2, as described above.)

[BK01]

Will confirm status on swagger-ui, re-assign and place in QA pipeline for now.

[BK01]

DataBC to include Marina in UAT plan

[BK01]

Version of swagger-ui (3.4.4) used in cad.data still has a bug. Swagger community to be notified. Tested in cad.data and petstore.

[banders]

Re-raised this problem with the swagger-ui community:

swagger-api/swagger-ui#3788

[banders]

@BK01 I think this issue may have been fixed by the swagger-ui community. I'll plan to test it out on the next release of swagger-ui.

[banders]

@BK01 I have tested the latest swagger-ui against the WorkBC API, but there's still a problem with the way that swagger-ui encodes the industries parameter.

I have investigated the problem thoroughly, and tried various modifications to the API spec in an effort to fix the issue, but I have not been successful. I posted an a issue with the swagger-ui community.

swagger-api/swagger-ui#4071

I hope that the community can advise on whether this is a bug or a problem with our API spec.

[banders]

@BK01 The swagger-ui community believes that our difficulty with the Industries parameter in the WorkBC API is indeed due to a bug in swagger-ui:

swagger-api/swagger-ui#4071 (comment)

They are aiming to have a fix included in the Jan 12 milestone.

[BK01]

Test to be repeated next week, following by a tentative date selection for migration of viewer to PROD.

[BK01]

@banders to review release 3.11 for fix

[banders]

@ll911 @BK01 I am testing the latest swagger-ui to determine if it addresses the WorkBC API issue, but I ran into a snag:

Looks like there is a CORS issue with the https://workbcjobs.api.gov.bc.ca when testing from my local dev PC (on localhost:9090). Is the server configured to include the "Access-Control-Allow-Origin: *" header?

[ll911]

curl -I -XGET https://workbcjobs.api.gov.bc.ca/v1/regions/
HTTP/2 200
content-type: application/json; charset=utf-8
content-length: 276
x-ratelimit-limit-minute: 100
x-ratelimit-remaining-minute: 97
date: Mon, 12 Mar 2018 18:14:27 GMT
server: Microsoft-IIS/7.5
cache-control: no-cache
pragma: no-cache
expires: -1
x-opval: GLE

access-control-allow-origin: *

[banders]

@ll911 Hmm... not working for me from a browser (Chrome):

GET https://workbcjobs.api.gov.bc.ca/v1/jobTypes

I'm seeing these response headers:

content-length:61
content-type:text/html; charset=iso-8859-1
date:Mon, 12 Mar 2018 18:18:04 GMT
status:403
x-kong-proxy-latency:2
x-kong-upstream-latency:36
x-ratelimit-limit-minute:100
x-ratelimit-remaining-minute:99

[banders]

@ll911 seems to be working now. Suddenly the same request I made above is returning:

access-control-allow-origin:*
cache-control:no-cache
content-length:310
content-type:application/json; charset=utf-8
date:Mon, 12 Mar 2018 18:22:51 GMT
expires:-1
pragma:no-cache
server:Microsoft-IIS/7.5
status:200
x-kong-proxy-latency:2
x-kong-upstream-latency:64
x-opval:GLE
x-ratelimit-limit-minute:100
x-ratelimit-remaining-minute:98

Did you change something?

[banders]

@ll911 I'm getting the CORS exception again from CURL and the API Console. Looks like it might be related to an HTTP 403 error. Any idea why the request would be returning 403?

curl -I -XGET https://workbcjobs.api.gov.bc.ca/v1/regions/
HTTP/1.1 403 Forbidden
Content-Type: text/html; charset=iso-8859-1
Content-Length: 61
Connection: keep-alive
X-RateLimit-Limit-minute: 100
X-RateLimit-Remaining-minute: 96
Date: Mon, 12 Mar 2018 18:49:34 GMT
X-Kong-Upstream-Latency: 18
X-Kong-Proxy-Latency: 2

[banders]

@BK01 @ll911 The problem I mentioned above with CORS and HTTP 403 still seems to be affecting the WorkBC Jobs API, although the issue is intermittent. During those times when the API is working for me, I was able to test it with the latest version of swagger-ui (3.12.1). The new version fixes queries that use the Industries parameter on the POST /jobs endpoint.

I incorporated the new swagger-ui into ckanext-openapiviewer. The PR has been merged. Can you try deploying in CAD?

You can test the Industries parameter using the following post data:

{
  "industries": [
    12
  ]
}

[ll911]

deployed to cad

[BK01]

@banders @ll911

Test 1: In chrome on cad.data I didn't see a specific error message, but also don't see the expected response using the test case above (industry = 16) as compared with the older API console

Test 2: In Firefox on cad.data I received the following error message along with the same response as above in test 1. "TypeError: NetworkError when attempting to fetch resource."

[banders]

@BK01 @ll911

I'm seeing intermittent HTTP 403 errors for the Router API. I have tested from my PC and from a remote server. Seems like the API works for about 5 minutes, then gives 403 for 5 minutes, then works again, then 403... This is similar to what I mentioned yesterday for the WorkBC API. It is causing the API Console to report CORS errors (because the HTTP 403 response doesn't include CORS headers). Does DataBC notice intermittent 403s from the router and WorkBC APIs?

[BK01]

DataBC - Test OAS3 spec in BCDC DLV works with industries only. Migrate new console to TEST / PROD (3.12.1). @BK01 create delivery booking
WorkBC team - adjust default example

[BK01]

Repeated UAT today. Inconsistent results found. In the first test, results showed no difference in number of records by industry (2000+). TypeError: Failed to fetch for second test.

[BK01]

Tests all successful at 2:15pm 2018-03-27