BuddyPress Integration

[JJJ]

BuddyPress components would hugely benefit from indexing of their content:

• Extended Member Profile fields
• Activity streams
• Username @ mentions
• Individual Groups and Members
• Private messages

The above list is loosely in priority order as I see it. Users ask for multifaceted member profile search constantly, and there just isn't a real solution out there.

[tlovett1]

We have a client who wants multifaceted member profile search right now. This would be great.

[swissspidy]

This. So much this. I could really use faceted search at the moment. How can I extend ElasticPress to integrate BP_User_Query?

[lkraav]

I'm also kicking off a BuddyPress integration project for a site. Do we have enough interested parties here to collaborate on an ElasticPress solution or?

[swissspidy]

Here's what I ended up with last year, if it helps: https://github.com/swissspidy/wptalents/blob/feature/buddypress-integration/classes/core/ElasticPress.php

[tlovett1]

If someone wants to package the BuddyPress integration into a feature, that would be awesome.

[nickchomey]

I just found this repo https://github.com/MESH-Research/elasticpress-buddypress that offers a seemingly-maintained ElasticPress-BuddyPress Feature, though it is missing a lot of key functionality.

I'm quite eager to get this working in a fairly comprehensive way with automatic index updating for Groups, Members, Activity, Messages, as well as the BuddyBoss additions of Media, Documents etc...

I'm reasonably competent with BuddyPress/bbPress/BuddyBoss, but am brand new to ElasticSearch/Press so don't really have a grasp of how to do all of this. At a glance, the MESH Feature seems to be using some out-of-date mechanisms when comparing to the Feature API

@mikethicke (from the MESH Repo), @swissspidy (from the WPTalents Repo) and anyone else here from the ElasticPress team or otherwise - if you could you give me some guidance, I'd be more than happy to do the legwork in updating and filling out the Feature!

[swissspidy]

As much as I would like to help you, I have used neither BuddyPress nor ElasticPress for over 6 years.

[nickchomey]

I've gotten some good feedback from @mikethicke over in his repo and will take a crack at writing a new feature that brings his BP-specific mechanisms to an existing core-EP Feature template. When there is something worth getting feedback on, I'll create a pull request to this repo and mention it here.

Hopefully I/we can get this to work soon enough! It really would be an enormously useful tool for either large BP networks or just any BP network that also has a lot of posts and documents to index and search!

[nickchomey]

I've made some progress on this (mostly just debugging Boone's fork/update of one of the EP-BP plugins mentioned above, such that the BP Groups indexing now works with the Indexables and Features APIs).

I then hit a roadblock when I discovered that EP doesn't automatically allow for a multi-index search (or even just a non-Posts search), as described in this issue #2844. However, I think it was fortuitous as it forced me to rethink everything and I believe I've found a fairly ideal solution to all of this!

If I'm not mistaken, the essence of EP is that it allows WordPress to remain as it is, but simply replaces a heavy SQL query with a faster, drop-in, ES query. Once the results are generated, the PHP/SQL resumes, doing whatever it needs to do to display the results page loop.

As it turns out, I'm actually a BuddyBoss user, rather than BuddyPress, and BB created a BP Global Search plugin 7+ years ago (about 6 months after this Issue was created!), which continues to be updated as part of the BB Platform. It handles all the heavy lifting of integrating with/overriding a WP search, generating the various queries and conditions for each data type, and then displaying the results in an attractive and functional search results page.

So, I don't see why I can't just allow it to do its thing, but replace the SQL query with an ES query. It would surely need some modifications to work with BuddyPress/bbPress, but that seems like a relatively minor task that someone else can figure out later.

Here's an excerpt of an SQL query generated by it:

( SELECT DISTINCT id, 'posts' as type, post_title LIKE 'test' AS relevance, post_date as entry_date FROM wp_posts p LEFT JOIN wp_term_relationships r ON p.ID = r.object_id WHERE 1=1 AND (((p.post_title LIKE 'test') OR (p.post_excerpt LIKE 'test') OR (p.post_content LIKE 'test')) OR r.term_taxonomy_id IN (SELECT tt.term_taxonomy_id FROM wp_term_taxonomy tt INNER JOIN wp_terms t ON t.term_id = tt.term_id WHERE ( t.slug LIKE 'test' OR t.name LIKE 'test' ) AND tt.taxonomy IN ('\'category\', \'post_tag\', \'post_format\'') ) OR p.ID IN ( SELECT post_id FROM wp_postmeta WHERE (( meta_value LIKE 'test' )) ) ) AND p.post_type = 'post' AND p.post_status = 'publish' ORDER BY relevance DESC, entry_date DESC LIMIT 3 ) 
UNION
( SELECT DISTINCT p.id , 'forum' as type, p.post_title LIKE 'test' AS relevance, p.post_date as entry_date FROM wp_posts p LEFT JOIN wp_postmeta pm ON pm.post_id = p.ID AND pm.meta_key = '_bbp_group_ids' LEFT JOIN wp_bp_suspend sf ON ( sf.item_type = 'forum' AND sf.item_id = p.ID ) WHERE 1=1 AND (post_title LIKE 'test' OR ExtractValue(post_content, '//text()') LIKE 'test') AND post_type = 'forum' AND ( post_status IN ('publish','private','hidden') OR pm.meta_value IN (0) ) AND ( ( sf.user_suspended = 0 OR sf.user_suspended IS NULL ) AND ( sf.hide_parent = 0 OR sf.hide_parent IS NULL ) AND ( sf.hide_sitewide = 0 OR sf.hide_sitewide IS NULL ) ) ORDER BY relevance DESC, entry_date DESC LIMIT 3 ) 
UNION
 ( SELECT DISTINCT u.id, 'members' as type, u.display_name LIKE 'test' AS relevance, a.date_recorded as entry_date FROM wp_users u LEFT JOIN wp_bp_activity a ON a.user_id=u.id AND a.component = 'members' AND a.type = 'last_activity' LEFT JOIN wp_bp_suspend s ON ( s.item_type = 'user' AND s.item_id = u.id ) WHERE 1=1 AND u.user_status = 0 AND (u.id IN ( SELECT ID FROM wp_users WHERE ( ID IN ( SELECT user_id FROM wp_usermeta WHERE ExtractValue(meta_value, '//text()') LIKE 'test' AND meta_key NOT IN( 'first_name', 'last_name', 'nickname' ) ) OR display_name LIKE 'test' OR user_email LIKE 'test' OR user_login LIKE 'test' ) ) OR u.id IN ( 1 )) AND ( ( s.user_suspended = 0 OR s.user_suspended IS NULL ) AND ( s.hide_parent = 0 OR s.hide_parent IS NULL ) AND ( s.hide_sitewide = 0 OR s.hide_sitewide IS NULL ) ) GROUP BY u.id ORDER BY relevance DESC, entry_date DESC LIMIT 3 ) 
UNION
 ( SELECT DISTINCT g.id, 'groups' as type, g.name LIKE 'test' AS relevance, gm2.meta_value as entry_date FROM wp_bp_groups_groupmeta gm1, wp_bp_groups_groupmeta gm2, wp_bp_groups g LEFT JOIN wp_bp_suspend s ON ( s.item_type = 'groups' AND s.item_id = g.id ) WHERE 1=1 AND g.id = gm1.group_id AND g.id = gm2.group_id AND gm2.meta_key = 'last_activity' AND gm1.meta_key = 'total_member_count' AND ( g.name LIKE 'test' OR g.description LIKE 'test' ) AND ( ( s.user_suspended = 0 OR s.user_suspended IS NULL ) AND ( s.hide_parent = 0 OR s.hide_parent IS NULL ) AND ( s.hide_sitewide = 0 OR s.hide_sitewide IS NULL ) ) ORDER BY relevance DESC, entry_date DESC LIMIT 3 ) 
UNION
( SELECT a.id , 'activity' as type, a.content LIKE 'test' AS relevance, a.date_recorded as entry_date FROM wp_bp_activity a LEFT JOIN wp_bp_suspend s ON ( s.item_type = 'activity' AND s.item_id = a.id ) WHERE 1=1 AND is_spam = 0 AND ExtractValue(a.content, '//text()') LIKE 'test' AND a.hide_sitewide = 0 AND a.type = 'activity_update' AND ( ( a.privacy IN ( 'public','loggedin' ) and a.component != 'groups' ) OR ( a.item_id IN ( '21','17','22','23','24','25','26','27','28','29','30','31','32','33','35','36','37','3','18','13' ) AND a.component = 'groups' ) OR ( a.user_id IN ( '15','16','21','1' ) AND a.privacy = 'friends' ) OR ( a.user_id = '1' AND a.privacy = 'onlyme' )) AND ( ( s.user_suspended = 0 OR s.user_suspended IS NULL ) AND ( s.hide_parent = 0 OR s.hide_parent IS NULL ) AND ( s.hide_sitewide = 0 OR s.hide_sitewide IS NULL ) ) ORDER BY relevance DESC, entry_date DESC LIMIT 3 ) 

As you can see, it pre-generates a bunch of search conditions (for post privacy settings (public, friends, onlyme), group privacy (public, private, hidden), suspended/moderated user/post, etc...) via SQL queries and/or persistent Object Cache. I don't see any reason why I should interfere with, let alone attempt to replicate, any of that. All of the pre-queries will likely preclude EP Autosuggest, but so be it. It has built-in AJAX search capabilities, which mostly uses the exact same mechanism, so that should suffice. (I also suspect replacing UNION with UNION ALL and getting rid of DISTINCT could improve performance...)

I assume it will be no problem at all to add the various metadata/fields that are used in the SQL queries to the ES documents via a JSON mapping. Likewise, I suspect it won't be terribly difficult to convert the SQL query clauses into ES-friendly query args. It may prove a bit trickier to properly update/sync changes with EP, but I can worry about that later.

---

My main question is the following:

Given that the BuddyBoss code will be handling the override/hijacking of the WordPress search mechanism, which EP functions and filters should I be looking at in order to initiate an ES query, create the args etc…?

Is it just query() in wp-content/plugins/elasticpress/includes/classes/Elasticsearch.php? Can I just call that directly as part of the BB Search code, without EP going through all its regular motions?

I'm not particularly concerned with how long it might take to achieve all of this - I'm enjoying learning how this all works and would also just like to give back something meaningful to BP/EP/WP. I just hope you fine folks can help nudge me in the right direction! Thanks!

(@felipeelia, I hope you don't mind me tagging you here as a follow-up from my question in #2844)