(EPIC) Bug: Many contact fields are invalid - they should be URLs but are usernames, and don’t work on the website

[fancyham]

Dependency

• Feature request: Warn when an invalid URL is the social media fields #2130
• Add tooltip for LinkedIn contact detail field #2197
• Add client-side validation for LinkedIn contact detail field #2199
• Add client-side validation and tooltip for Pinterest contact detail field #2182
• Add client-side validation and tooltip for Twitter contact detail field  #2184
• Add client-side validation, tooltip, go to link and design for Instagram contact field #2201
• Add client-side validation, tooltip, go to link and design for Facebook contact field  #2202
• Update tooltip text for all social media tooltips #2272
• Add a Go To Link for the website form field #2273
• Make sure links on public site actually work (especially if you’re modifying how links are stored on the database) (As per team discussion on Jan 6, 2025, no modification has been made on how the links are stored on the database, on the front end side within the contact details tab we are entering complete social media links.) Make sure social media links on public site actually work #2318
• Research ways to sanitize malformed social media links in the database #2294 Clean up/rationalize/fix any currently-malformed data in the database for these fields, perhaps with a batch operation.

Describe the bug

Maybe not really a bug but an issue with our data and form fields that allow invalid info.

Many of our listings have incorrect URLs for Instagram, Twitter, etc. because volunteers are entering @username in the fields rather than the properly formatted full URLs that our system is expecting

Noting this as an issue so we don’t forget it and because devs may be able to help find and fix errors, or solve a different way.

What we need to do

We need to clean these up, and prevent it from happening in the future.

Possible solutions

Some possible solutions:

1. Fix our existing data — going through each of our listings and confirming and correcting the fields

• This could be a volunteer task and people check and correct these manually (we have about 500 listings total to review, but no easy to use UI to do the review)
• Or, perhaps, we could write and run scripts to turn usernames into full URLs

2. We could allow usernames and URLs both, and the website parses the field and displays the full URL appropriately

3. We could make all fields usernames-only and run a script to replace URLs with usernames in database, then display the full URL on the website

We should also display errors when field is incorrectly formatted to help data validators enter fields correctly

• Feature request: Warn when an invalid URL is the social media fields #2130 — Indicate an error when there’s an invalid URL in these fields — perhaps “Needs to be a full URL e.g. http…" (or explicitly support and validate usernames (i.e. remove the @ if needed))

and (optionally) prevent approval (and display error message) if these aren’t valid URLs

How to recreate

For example, go to this organization (unless it’s been fixed since this bug report):
https://foodoasis.la/organizations?latitude=34.081648&longitude=-117.740882&org=god%27s_pantry_-_pomona&id=4772

In the database:
<img width="838" alt="image" src="https://github.com/hackforla/food-oasis/assets/3376957/45ded83d-f8aa-490b-af7d-8555a6c5bd10”>

Our web site - Instagram icon has malformed URL (it should be going to Instagram.com/godspantry_org)
Video: https://github.com/hackforla/food-oasis/assets/3376957/f7a8a5c2-9397-4b8a-8f1e-a186fdf0de58

[SAUMILDHANKAR]

Adding helpful comments from slack for future reference:

Saumil
10:25 PM
Hi Bryan, getting back here regarding your feedback on #2169 (review).
We discussed the requirements for the issue and below is the summary of the discussion:
There are a few options we can try which would let you see the changes before this PR is merged but that would involve some time and effort to incorporate. Best option for now would be to merge it to the dev site and then the changes could be seen and tested. However, based on today's meeting, it is not clear if the tooltip and validation should be for a URL or social media handles/usernames. I have learnt that we had decided to use handles earlier. So, maybe we need more discussion on the next steps for this. Let me know your thoughts. If Thursday meeting would be good to discuss further it would work for me. Once the next steps have been decided, I can work on finalizing the tooltip text in this issue and the validation part can be done in this issue: #2130. Looking forward to your response. Thank you.

Bryan Wu
5:40 PM
I wasn’t aware we were using handles - if that’s been working well for us then that’s fine I guess? From a Quick Look, though, I see a lot of URLs in our database but it’s very inconsistent. Sometimes there’s no http or sometimes it’s a handle only (which I think breaks our site’s link to that service) (see attached)
Off the top of my head I think the full url would be clearer to our volunteers though, because the concept of username or handles don’t really apply to some of those social medias - Facebook or LinkedIn, for example. And if we, say, add mastodon, then thee URLs will really be more than the user’s handle.
BUT if we do want to support handles for some (like instagram, and perhaps in addition to the full URL), we’d need to validate it (do we require the “@“?), parse it for display and create the links on the site correctly.
And while I don’t know for sure, I wouldn’t be surprised if some of the services use different urls for different types of organizations or people.
(BTW, can we add this to our conversation above to the GitHub issue’s comments? I think it’ll be helpful for future folks to see how we figured this out) (edited)

3 files

Bryan Wu

Note how this Facebook url has “pages” in it before the username/handle

[fancyham]

Notes from a meeting between John D and me today:

• John D mentioned that devs are leaning towards wanting to use usernames only.

• Bryan cautions: Usernames -> URLs might work for some services, but might not work for others, such as like Facebook which seems to have more complicated URLs and multiple schemas and types of pages, such as

  • https://www.facebook.com/buy.nothing.burbank.glendale.verdugo.foothills
  • https://www.facebook.com/groups/9690892508/
  • https://www.facebook.com/profile.php?id=61560417517497
  • https://www.facebook.com/profile.php?id=100063617047904

• So if going to usernames for some (or all), please research that you’re 100% sure that you can create the correct URL for the organization from a username only. Just because it works for one case doesn’t necessarily mean that we’ve covered all possible cases. I tried briefly to find documentation for URLs but didn't find it.

• If in doubt, the fallback/default should probably be to use a URL

---

• FYI: We don't currently support Mastodon, but here are some examples in case we eventually do...

  • https://xoxo.zone/@xoxo
  • https://mastodon.social/@GottaLaff

  A username would technically be "@GottaLaff@mastodon.social"... so we'd decompose that and re-create the URL? Sounds like a lot of work! :)

https://docs.joinmastodon.org/user/signup/#:~:text=Mastodon%20usernames%20actually%20consist%20of,the%20website%2C%20e.g.%20example.com

[SAUMILDHANKAR]

Adding my notes for some ideas on tooltips for LinkedIn and Twitter:

For LinkedIn (Source: https://www.linkedin.com/help/linkedin/answer/a542685/manage-your-public-profile-url?lang=en#:~:text=You%20can%20create%20your%20public,profile%20URL%20at%20a%20time.)

Examples of LinkedIn URLs and what should be entered by the person adding the contact details:

1. "https://www.linkedin.com/in/williamhgates/" for this URL, please enter: in/williamhgates/
2. "https://www.linkedin.com/company/food-oasis-la/" for this URL, please enter: company/food-oasis-la/
3. "https://www.linkedin.com/groups/6519652/" for this URL, please enter: groups/6519652/

Additional helpful notes:

- A LinkedIn URL can be 3-100 characters long but it can not have spaces, symbols, special characters or word linkedIn and it is case insensitive. 
- All the above restrictions/guidelines would also apply to the text being entered in the contact field.

For X (Source: https://help.x.com/en/managing-your-account/x-username-rules)

Examples of X URLs and what should be entered by the person adding the contact details:

1. "https://x.com/elonmusk" for this URL, please enter: elonmusk
2. "https://x.com/microsoft" for this URL, please enter: microsoft

Additional helpful notes:

- Usernames can't contain words twitter or admin.
- Usernames can't be longer than 15 characters and can only use alphanumerics and underscore, no other symbol, dash or space is allowed.
- All the above restrictions/guidelines would also apply to the text being entered in the contact field.

@fancyham Hi Bryan, the above examples do not cover all the scenarios but we can leave a note saying if the URL doesn't match the above format reach out to your admin or something else. Please share your thoughts on if the above examples could be used for tooltips or if we can add/change some more info. I will add similar info for pinterest, facebook, instagram next as well. Thanks.

[fancyham]

Thanks for doing that research!

So… I still think it’d be far simpler to just use the full URL, perhaps in all cases?

Could you help me understand why we’d not want to do that? Is it difficult for the user in some cases? A database thing? Etc?

[fancyham]

Thanks for doing that research, BTW

[SAUMILDHANKAR]

Thanks Bryan. We discussed this during our dev meeting yesterday and using the full URL would be simpler. So, we have decided to go ahead with that. I would add more notes in this issue with examples for tooltips for each of the social media we use. Also, planning to break this issue down into smaller issues for each social media contact field. Hope it works. Thank you.

[fancyham]

Hi, Saumil,

The tooltips should be changed a bit:

Change it from "Enter your Twitter username" to "Enter the organization's Twitter username"

Thanks!

(also, I'm deliberately not asking to change twitter to x :) )

[SAUMILDHANKAR]

Just adding notes from this week's dev meeting

As part of this epic issue, pending issues are:

• Add tooltip for LinkedIn contact detail field #2197 and Add client-side validation for LinkedIn contact detail field #2199
• New issue to fix tooltips as suggested by Bryan
• Maybe a new issue for adding Go to link next to website field

[SAUMILDHANKAR]

Thanks @MuthamilselvanG for the reminder to update progress for this issue. I will add links on top of this issue as well.

1. Progress: We had a team discussion about this issue on Nov 4, 2024, 7:15 PM pacific time (attended by both developers and designers). Adding the notes from the discussion below:
   a) For all social media URL fields, there should be an option to enter either:

    i) the full URL (which could be parsed, if required).
        OR
    ii) just the username.
        OR
    iii) partial URLs (for example, as Bryan mentioned, Facebook can have complicated URLs such as https://www.facebook.com/buy.nothing.burbank.glendale.verdugo.foothills)
   

   b) The other way to do the above could be just leaving the social media textfields completely blank as it is currently for LinkedIn and then validating the full URL once it is entered by the user as required.
   c) Tooltips: Currently, the tooltips read as Enter your Instagram username for the Instagram tooltip and so on for other tooltips, this should be changed to Enter the organization's Instagram username if we decide to go with letting users enter usernames, if we decide to go with full URLs the tooltip should be changed accordingly. I (Saumil) will create an issue for this.
   d) PS: This epic issue to be closed once all the dependency issues are done.

2. Blockers: No blockers.

3. Availability: 5 hours this week.

4. ETA: Not sure about completion time.

[fancyham]

Just a reminder:

make sure to also check that the links are showing up correctly on the listings themselves. I know you’re trying to rationalize to some standard format, so make sure links work correctly on the org listing pages on the public site. That’s the output end of the stuff you’re validating here.

[SAUMILDHANKAR]

Adding notes from the team discussion about this issue on Nov 11, 2024, 7:15 PM pacific time below:

• We would need to run a script to sanitize data (social media links) already stored in the database. Need an issue to figure out the best way to achieve this.
• Tooltips for the social media links to mention the organization's Instagram URL and so on for others.
• Add placeholders such as https://instagram.com/... and so on for others.
• All social media fields to be converted to plain textfields and the fixed portion would be removed.
• For the website textfield, make sure the validation allows the users to enter both http and https
• Validations should just display a warning if it doesn't satisfy the requirements, the users should still be able to proceed.
• If save the progress isn't successful, prompt a toast saying couldn't be saved.

Some of these have already been addressed in #2279, will check what is left and update accordingly.
@fancyham Hope I didn't miss what we discussed in the meeting, feel free to add or edit. Thank you.

[fancyham]

That all looks good. 👍

One thing I’d add is that people may paste in URLs that lack the “http://“ altogether. e.g. would “google.com” pass our validation and be properly rationalized into the database? It might actually work fine, I think. So does “google.com” count as a valid URL?

Not sure what the best solution is. Allow

• http://instagram.com/,
• https://instagram.com/…, and
• http://www.instagram.com/…,
• instagram.com/…,
• www.instagram.com/…

and any other valid URL. There surely must be a lightweight and permissive URL validation routine we can use.