Review Yodie for potential COMRADES Integration

[Shadrock]

Task: We are planning a "test integration" to ensure that tech developed by COMRADES consortium partners can be integrated with Ushahidi's work flow. To that end, please review YODIE as an example. Identify any potential issues that need to be addressed such as differences in programming language, UI/UX needs, general compatibility with platform.

The test integration will also require revisiting the API documentation Robbie recently created: Draft in Ushahidi Drive and submitted final doc in COMRADES Drive. Most of the consortium were uncertain of how to use this documentation with regards to the products they are developing.

Due Date: February 17th

Note: There is working meeting for COMRADES scheduled for March 29-30 in the Netherlands. If there's anything specific that should be done in person (e.g. training them on how to use Github) or if you'd like to attend and present something that will make your life easier, this would be a good time.

[rjmackay]

@Shadrock is the test integration due by Feb 17th? or just review for scoping?

[Shadrock]

@rjmackay that date is for scoping. To test the integration is going to be a bit of a logistics lift... we'll need to get the devs from the COMRADES consortium to enter it via Github to start with... so they'll be a call, etc. But I want to make it as easy for them as possible so that they'll actually do it the way we want to.

[Shadrock]

@rjmackay @jshorland I just love the great whooshing sound a deadline makes as it goes by... I'm setting up a call with Jess this week to talk integration: it would be nice to have this done. Robbie: would it be easier for you to just start by sitting in on that call? If so, I'll arrange a time.

[rjmackay]

Yikes. Sorry!

Initial thoughts:

• Ideally we wouldn't run or host YODIE ourselves, but use it through an API. Rather the partner that built it should run it.
• The existing YODIE api appears to be very simplistic: http://services.gate.ac.uk/yodie/api.html
  • There is no spec for the expected response format. We'll need this to start anything
  • It looks like the API returns links to any terms it could match in its database. We could probably store these as a list appended to a post, and maybe linked within the post body. Other tools might have more complex data which could be trickier for us to store in a deployment
• My architecture proposal previously suggested that analysis tools would integrate with Ushahidi receiving webhook data, that is: whenever a new post was received we would post it to an API designed to deal with Ushahidi
  • I was intentionally vague when we original wrote this because the webhook system doesn't exist yet.
  • From the previously supplied docs they'd primarily just use the Posts API to submit analysis results back.
• If they're not able to build a custom API to receive data from Ushahidi and send the analysis results back, we can build an intermediate service to do this ourselves.

Development needed on Ushahidi platform to support this:

Option 1 use webhooks and an intermediary service

• Build a webhook system that forwards any newly created Posts to registered services
• Build an intermediate service. This would sit between Ushahidi and YODIE. The webhook would send data to this service, it then forwards it to YODIE, parses any response and sends it back the Ushahidi API to amend the Post
• Service could be written in any language, and just has to understand HTTP and JSON

Option 2 build an Ushahidi plugin

• Build an internal event system, such that plugins can listen to events within Ushahidi
• Build a plugin that listens to new Post events within Ushahidi.
  • The plugin would forward data to YODIE, parse any response and save it back into Ushahidi to amend the original post
• Plugins would have to be in PHP and work with Ushahidi codebase

Both of these are reasonably feasible. Option 1 makes a lot more sense if theres a chance that other partners would want to build or contribute to the intermediate service. It doesn't force any tech choices. Option 2 is very similar but would run inside an individual Ushahidi deployment.

If we prioritize this we might be able to prepare an example plugin/service and present that at the working meeting. That would give the partners some idea how to reproduce this themselves.

Concerns:

• YODIE api doesn't describe its result format at all. Is there documentation of that somewhere?
• I couldn't see a working example of the YODIE api. Could they provide some examples that produce interesting results?
  • This random snippet of text seems to do something but without a spec I'm not sure what half of the result means http://services.gate.ac.uk/yodie/service?text=Caveats.%205.8.%20X.694%20ASN.1%20with%20PER%20+%20Fast%20Infoset;%205.9.%20...%20A%20successful%20EXI%20format%20will%20include%20facilities%20for%20helping%20computers%20encode%20and%20decode%20the%20entities%20of%20XML%20documents%20efficiently%20and%20in%20a%20compact%20form.%20The%20purpose%20of%20...

[rjmackay]

@willdoran could you take a look at this in case I missed anything?

[Shadrock]

@rjmackay @jshorland @willdoran My feeling is that we will have to build an intermediate service to do this ourselves. I've just heard back from the YODIE developers who checked in on this thread and sent me the following e-mail:

the latest version of YODIE is restricted to an approved list of users - please sign up for an account on https://cloud.gate.ac.uk and send me your username, and I will grant you the required permission.

The API presented by this service is different from the old services.gate.ac.uk/yodie mentioned in that GitHub issue. The basic API is documented at https://cloud.gate.ac.uk/info/help/online-api.html and this specific service returns its results as annotations of type Mention, with features class and inst for the relevant class and instance URIs in the underlying knowledge base.

The default quota for this service on cloud.gate.ac.uk is 800 requests per 24 hours at an average rate of no more than 2 requests per second, but these limits can be raised considerably if this is required for project partners or other research users. And if necessary we can look into providing a private deployment of the same service for your sole use.

One thing I'm unclear on is if the YODIE developers are on the COMRADES project: I was introduced to them from one of the COMRADES leads but it's unclear to me how they are connected to our work. So I want to make sure that integrating YODIE would actually "count" for COMRADES before we put too much work into it: even as a test. Edit: I've confirmed that YODIE was actually developed as part of the COMRADES deliverables... so it's really not a test integration... but a real live one!

That being said, what would the next steps for this be? Should I organize a developer chat to go through some of these issues? Is there a clear set of next steps here to pursue from our end?

[jshorland]

@willdoran @tuxpiper are one of you able to look into this next week?

[willdoran]

@Shadrock @jshorland To simplify for the intellectually challenged(me)

1. Build a webhook within platform which

• Create Webhook registration
• Forward all inbound posts to a intermediary service
  Question: does "all posts" include posts created by users via the web client?

2. Build an intermediary service which

• Exposes a webhook to Platform API
• Receives post data
• Requests Yodized post data
• Updates the Post in the API

Things I would like to know:

1. Can I have an account on Yodie? I've already registered as willdoran/will@ushahidi.com
2. What is the expected result from Yodie? Will the Post Survey contain fields that the yodie result can just be inserted into?
3. Can I make some test requests to Yodie? Is there a dev I can talk to?

Am I missing anything?

[tuxpiper]

my 2c. To me the "intermediary service" idea sounds more modern, leaning more towards the idea of microservices, which I would favour.

[willdoran]

The Platform API changes
Comrades Yodie Proxy

[willdoran]

@rjmackay So a Yodie response looks like this:

{
	"text": "The framework ships with a public/.htaccess file that is used to allow URLs without index.php. If you use Apache to serve your Lumen application, be sure to enable the mod_rewrite module.\n\nIf the .htaccess file that ships with Lumen does not work with your Apache installation, try this one:",
	"entities": {
		"Mention": [{
			"indices": [34, 43],
			"inst": "http://dbpedia.org/resource/.htaccess",
			"dbpInterestingClasses": "owl:Thing",
			"dbpSpecificClasses": "owl:Thing",
			"confidence": 0.615344877613882
		}, {
			"indices": [71, 75],
			"inst": "http://dbpedia.org/resource/Uniform_Resource_Locator",
			"dbpInterestingClasses": "owl:Thing",
			"dbpSpecificClasses": "owl:Thing",
			"confidence": 0.52
		}, {
			"indices": [127, 132],
			"inst": "http://dbpedia.org/resource/Lumen_(anatomy)",
			"dbpInterestingClasses": "owl:Thing",
			"dbpSpecificClasses": "owl:Thing",
			"confidence": -0.6288636363636364
		}, {
			"indices": [168, 179],
			"inst": "http://dbpedia.org/resource/Rewrite_engine",
			"dbpInterestingClasses": "owl:Thing",
			"dbpSpecificClasses": "owl:Thing",
			"confidence": 0.6293402799127326
		}, {
			"indices": [196, 205],
			"inst": "http://dbpedia.org/resource/.htaccess",
			"dbpInterestingClasses": "owl:Thing",
			"dbpSpecificClasses": "owl:Thing",
			"confidence": 0.615344877613882
		}, {
			"indices": [227, 232],
			"inst": "http://dbpedia.org/resource/Lumen_(anatomy)",
			"dbpInterestingClasses": "owl:Thing",
			"dbpSpecificClasses": "owl:Thing",
			"confidence": -0.66
		}]
	}
}

I do not know how they expect the platform client to display this but I have two suggestions:

1. This raw data could be stored in a standard text field as is until they decide what they want
2. If they want - for example - to display the text with the annotation as links around the matches words that would necessitate a speical Yodie field type...which isn't much use in the long run really as it would be kind of useless for anything not related to Yodie. So I would suggest that if they want the data displayed in a particular way they return Markup and we add a MK field type which would be more useful long term.

@Shadrock I think that if they want to display the data in some particular way on the platform then they need to think it out soon and I think that asking them to return pre-created markup is not a huge ask and would be more useful generally.

[Shadrock]

@rjmackay asking for pre-created markup sounds good to me. Will and I discussed creating the integration in the way that makes the most sense to us and presenting that to them. Do you and @willdoran still need me to ask them for more details? I'm under the impression that this is already taken care of but ping if not.

[willdoran]

@Shadrock I think we can present it to them as the solution and if it doesn't work for them we can revisit it but until then I think it's ok to assume it's covered - pending feedback from them when the integration is ready for them.