Next features to build

[mmahalwy]

Salam!

I have made a quick survey that you can find here and sent it to people that wrote to support with the assumption they'd be more receptive. Here is the top 3 features people really care about:

1. Copy with tashkeel. This is the Rename this repo!  #1 request
2. Go to ayah, and only that ayah
3. Social media sharing (share range of ayahs or one ayah).

These are obviously somewhat missing from the new Quran.com that was in legacy.

Annoying things people are complaining about:

1. Site stability
2. Audio (I have NO idea why audio is an issue, but it seems like it is and we need to fix this)

What do people want built next on Quran.com?
It's scattered. But right now seems that website speed and stability is a common trend, then people want nice things like bookmarking and better search.

What other products next?
ask.quran.com
audio.quran.com
learn.quran.com
These are the top picks so far. ask.quran.com is a problem I personally face and wish it was solved in a better and more effective way, I think we could most definitely build a prototype for it and see where it goes.

What to do next:

1. There are things we need to address right away: speed and stability.
2. Add the requested features (copy/paste, and the single ayah). Social media needs a little more thought and we should probably consult someone
3. Put up links or beta email collection pages to gauge interest for the next products to build

Let me know if you'd like to see the results yourself

[mmahalwy]

@reshadn @ahmedre @sharabash @bshama @redhae @dsog

[reshadn]

Salams. Thanks for this breakdown. I'd love to see the results and I'm curious how many people filled out the survey, did we capture email addresses? I'm new so I apologize for the noob questions. Do we have an email list for "beta" testers or active users who are willing to test things out and give more feedback?

re: site speed and stability - have we been able to determine if the main issues are on mobile browsers? slow networks? specific browsers? heavy load times (not sure what our numbers are like today). Some of this should be easier to capture via pagespeed on the client or other tools on the BE.

Looking forward to it iA!

[mmahalwy]

@reshadn:
Now about 25% of people emailed filled it out. We don't have a list of beta testers to be honest, we previously just subdomained things. I wanted to use optimizely to feature flag things but unfortunately our traffic is so high that we fall under many top tier plans for many products.

I have a deploy coming up that will help with site speed and stability (we kept crashing cause of a search bug).

I will test it more too

[ATouhou]

I don't know how happy people will be about their emails being publicly available like that (Y) Else looking good

[mmahalwy]

Good catch. I am going to hide this then..

[mmahalwy]

Email me personally for results mmahalwy@gmail.com

[ATouhou]

1# Focus on making the site stable! The bismillah is still broken :'( it's heart breaking to see, as this is most likely the most used quran website, and having that part broken is very critical...

Feature suggestions:
#1. Memorization module. (& encouragment to learn&study X ayah a day)
#2. One guy requested being able to turn word-by-word off (this should ofcourse be possible / and / or change to bangladesh as we have this data , also we have the translitteration word-by-word data.).
#2.1 Wordmorphology integrated (the corpus.quran data).. Do this through the API so others can use it...
3. Copy arabic text (also with tashkeel)
4. Geolocation (and put the standard languages accordingly). Ask for permissions to geolocation OR use IP.
5. Maybe prompt a guide? (look at the way intro.js does it here on github) as I see some people are having a hard time changing language.
6. Maybe some kind of "share" button, where when clicked, it asks to share through, email, pinterest, facebook, twitter, etc.
7. Provide documentations for API
8. People are requesting more translations and tafseer. I really think you should all translations etc. to Github in a seperate REPO, and keep note of all typo etc. fixes there. Maybe make a site like Tanzil (i don't know if tanzil still is updating their site!!) or start working with them!
9. More reciters. Do we have all reciters from "versebyverse" ?
10. Many requests for Apps. Since you're already using angular, would it be too much of an hassle to make an iOS, Windows and Android app in a framework like ionic ?
11. A lot of requests on improving the Search feature!! Have a look at alfanous and their search engine :-)
12. Speed requests... Make sure all is gzip, minified, cached, on cdn, etc.
13. Tag and bookmark. Do loginsystem and let users signup fast with Facebook, Twitter, Steam, Github, etc. This way we can also engage them when not being active for long time (and they should be able to turn this off)
15. For now, tajweed font is not possible with the current setup (the fonts), but should not be forgot :-)
16. Maybe a info page for each surah like this: https://muflihun.com/quran/intro/25

Bug list:

1. People complaining about audio not working. Maybe this is on mac osx / safari??
2. Last word is jumping in readmode on mouseover, which is really annoying, i see another guy complained about this AND emailed you about it. I also complained about this :P
   3.When we click Search while the search bar is empty, it starts searching anyway

About new concepts:
There seem to be a majority that wants you to focus on quran untill it is completely stable and perfect.
Also I think the ask.quran.com unless we can acquire trustworthy scholars, it would be a very bad idea to facilitate bad knowledge. Marriage platform, could risk to much fitnah as well... Food & Masjid could be VERY nice . But let's focus on the quran site for now!! Please do give feedback about the feature suggestions and if it is something you plan to do! Maybe set up everything in a CRM system so there is better structure of tasks, and add team members to control who should do what (so you do not forget).
Most of feature suggestions is review of the excel sheet :-)

[mmahalwy]

@ATouhou all look good!

Perhaps let's find the most important 3 things and perhaps you can help us with getting the data and I can clean/download it?

[mmahalwy]

Bugs have been fixed... we are literally just waiting to push and deploy. Stability is because:
legacy.quran.com has seen no changes in a very very long time. Quran.com is changing every week and we dont have the proper infrastructure or time for it! We need more time :(

[ATouhou]

Indexed English tafseer ibn kathir and word-by-word audio in arabic (with tajweed and just standalone word) on recitequran.com ! I've contacted them a couple of times and they said they will consider making everything open-source. The alim.org refused to share their tafseer ibn katheer.
I think if you contact them with a @quran.com email they might share it...
I have a danish translation of the Qu'ran that was recently released and I cleaned it so that each ayah is on 1 line. But I am looking for an approval from a "scholar" in Denmark, before releasing it, so it does not contain anything wrong!! (The translation is called "Den klare koran")
We sadly have mostly only translations done by kufaar in Denmark.
Another translation by a danish "scholar" (Shahid Mehdi islamqa.dk) is awaiting approval and will then be released. I can make sure to "clean it" when it is released :-)

The turkish translitteration would be a great add. GlobalQuran.com have resources to download for free. Also word-by-word translitteration and bangladesh was provided by the other github guy :-)

Have you considered putting up all tasks on a CRM system to monitor what to do, so nothing falls "between the cracks" ?

For the 80% of words:
https://drive.google.com/file/d/0B4VHPzkqEV3KUjZNN2VEajdfXzQ/view?usp=sharing
https://drive.google.com/file/d/0B4VHPzkqEV3KOGotbUJ5enI2b2s/view?usp=sharing

[mmahalwy]

Yeah I contacted recitequran.com and no luck with that. They don't want to share it.
Alim.org sent me an email today and said contact the original authors, CC: @ahmedre

[ATouhou]

@mmahalwy Actually original author has put the site on github hhhhhhhhhhhhhhh it's based on Drupal but has everything except the tafseer!
https://github.com/alim-foundation/www.alim.org

They did not know this before I told them that!.... They then said they would not share it , but I'm free to help on alim.org ...

We can scrape it from the site if they would not share it..

I also spoke with qtafsir.com , they are behind recitequran.com and both parties refused to share the tafseer ibn kathir...??
We could scrape it and index it on the logic we wish

[mmahalwy]

i could scrape everything, i have done it before, but we like to be cool with others and not be the evil people with a flashy domain. So, I like the scratch each other's back than scrape!

And I really want the english ibn kathir :(

[ATouhou]

There is nothing wrong with scraping the public data. We share it fisabillilah and are not making 1cent on it! If you search github you will see there is already 2-3 Scrapers for sunnah.com as there is a demand for the informations even though the terms of conditions says don't scrape! .
Anyway we can copy+paste tafseer ibn kathir for each ayah / page manually from a PDF (I have it and can provide it) or do the process automaticly and scrape the data....
Really up to you.
Also the brother over at muflihun.com have some good tafseer and translations.
We should also be carefull to not add any tafseer that Ahlus sunnah wal jama3a have warned against!!

[mmahalwy]

Hmm, it's a tough decision that I don't have an answer for. On one side, we can create our own data for everything (scrapping, etc etc + paying a qari for the audio) and on the other is trying to push through and convince people to share their data. It's annoying, I mean we say and are happy to share our database, for example, only if someone personally requests it (so we have some small control over what happens). Last thing we want to do is people take it and use it in the wrong way, or take it and try to rebuild quran.com with ads!

I think it's an interesting conversation and I really hope with our good intentions we can solve this problem within the community.

As for the Ibn Kathir, I don't know if there is a good PDF scrapper? Maybe that will help? Have you searched to see if there is Ibn Kathir scrappers? I previously scrapped all the word-by-word audio for Recitethequran but now refused to use it unless they just say 'OK'. That's all I need!

[ATouhou]

Ofcourse. I really think we should continue to push the guys at recitequran and alim.org.
I really do not understand a muslim, refusing such an opportunity to gain free hasanat by others work...
I understand having some control over it, like the decision by the android app team, removing the source code to prevent people to releasing verisons with ads / malware etc.

I think all data should be so widely accessible that people can choose whatever platform they wish. At any point in Sha Allah we get hasanat even from the works of others when sharing our data.
Also providing the API for people, will ensure some authentic data for all and easy setup for other connected platforms .

The tafseer (ibn kathir) isen't , verse by verse, but it is more build up in chapters for verses/pages. The question also relies how to structure the data. Append each aya to a "chapter id" and maybe also an offset to this. -> or bind some data to a signle ayah... I think the first example is best.. We can take the data from PDF or from an existing site. I don't really care, as long as our intentions are pure and to provide a service for the ummah, fisabillilah.
I also scraped all pictures and sound from recitequran.com :-) The audio I got for verse-by-verse was is 1.10 GB on 18,079 files while the tajweed audio files is 0.99GB and 14,905 files.
Maybe we should contact Wisam Sheriff who recorded the audio and hear his opinion and make him talk to the "shareislam" team?

How is your current translation & tafseer files set up?? could you share the SQL structure so I know how to set it up if it becomes relevant?

[ahmedre]

so here are some random thoughts about scraping and what not:

1. for said people specifically, this is not an option - Mohamed tried taking some data from one of these sites before and we got a very strongly worded angry email about it (and rightfully so, since we didn't ask permission) - they asked us to remove the content. they were unwilling to negotiate anything, unless we accepted joining underneath their organization (i.e. being "acquired" by them).
2. you give the example of sunnah - since i am friends with the brothers who run it, i can tell you that they are unhappy about scraping, especially because lots of the people who scrape the data put it in apps that they add ads to (or sell for profit) - it feels unjust for them to spend all this effort getting lots of volunteers to type up these books and then give them for free, only to have someone take them and sell them and make money on their work and the work of the volunteers.
3. one of the other issues with data is that you can't easily propagate changes when people are using your data and mistakes are found (ex in translations, etc). another issue is copyright - even though i don't like the idea of islamic copyright, this is a concern i have (especially since some of the translations we have in Quran Android and quran.com have copyrights, and we weren't able to convince their owners to let us use them - we left them though because we were never asked to explicitly take them down). we have been asked to remove content from quranicaudio before, and that we did do, however.
4. for Android, we never closed the data (the repository for the pages is still open, and the rest of our data is coming from tanzil and other sites that we link to in the about) - we just closed the actual app to try to decrease the shady things people do or have done with it.

[ahmedre]

one last point which i do understand - even if you're only in it for thawab and zero worldly reasons for things, you want people to come to your site (i.e. it costs $0 to visit a website - why should they visit website x instead of our website?) - and the same with their team.

given that we are bigger than them (by virtue of having the domain alone), they have little reason to give us their data, because it basically causes them to have no use for their website.

regarding speaking with br Wessam - this was one of the things we were planning on doing, and, as a fallback, we're thinking about how we can get our own recording from a sheikh to put on quran.com.

[ATouhou]

@ahmedre

1. If we scrape the data (can be done behind VPN if anxious about being tracked), they can't track the origin of the data. Those groups shareislam and alim do not own the tafseer ibn kathir, nor the publishing right or anything. It will just save us some headache to index the data by scraping their site. The volunteers should be happy that the work is shared and their hasanat is multiplied by the billions!
2. If the brother made it open-source more people could develop applications for it, his rewards will be multiple even more! Even if the sick people who shares it does it solely to sell the data for a small price.
   2&3.Also if he made it accessibly more easy, he could make an email signup to download the data! If any changes/fixes/new releases he can just send an email to all the developers asking them to fix it ASAP.
3. Okay yes. Makes sense :-)
4. I think any muslim should strive to make it the best and collaborate in doing so. In the specific case with recitequran.com , quran.com won't be able to use the tajweed sounds nor make an coloured tajweed font due to the technology chosen to solve it (recitequran.com is using images and not fonts).
   Personally I use recitequran mostly to read and quran.com to lookup ayah fast / share ayah / light reading.
   At any point a muslim should do the best to help his "competitor" as we all do it fisabillilah and for the ummah.

[dashohoxha]

My opinion on some of the issues discussed on this thread:

• GitHub issues is the best CRM or project management system that I have seen yet, so let's stick with it.
• Let's do one thing and do it well (or perfect), instead of trying to do lots of things but none of them is OK. So, let's focus first on doing right the Quran (fixing bugs and some neccessary basic features), and then we can think about "ask", "audio", "learn", and other advanced features.
• It is a good thing that the back-end (data) and the front-end (UI) are clearly separated and communicate through an API. This allows for other applications to be built on the future, using the same API. But I think that this API should be well defined and well documented (maybe I am not properly informed about this).
• I think that the data should have such a licence that allows the others to reuse it, but forbids them to abuse it. Maybe somebody will try to abuse it, but they should know that they are violating the licence and someday they may be taken into account for it. Maybe Creative Commons or something like this can be suitable, but I am not an expert on licence issues.

[doganmeh]

I am eager to learn about and use the quran.com api mentioned in this thread, and once I figure it out, I can dedicate time to document it (if not documented already). I am very detail oriented, and can do a good job in that.

I want to build a site for quranic arabic learners, and for the sake of ease (details mentioned to br @mmahalwy in email) I plan to start out with https://github.com/idris/django-quran. But I do want to be part of this circle as well, and contribute wherever I can. I am a long time follower of the quranic corpus.

By the way, I am still looking for the word by word english translation that was used on quran.com (and many other sites). I would appreciate if you could share it, or point to me to elsewhere that I can obtain it.

Thanks for all and keep up the good work (sorry for kind'a out of subject post)

Mehmet

[dashohoxha]

I plan to start out with https://github.com/idris/django-quran

Thanks for pointing this out. I can find there the Arabic text and English translations of the Quran.

[doganmeh]

Not the word by word one.

[dashohoxha]

Have you checked this: http://corpus.quran.com/wordbyword.jsp
There is also an API: http://corpus.quran.com/java/
Maybe this is what you need.

[doganmeh]

Yes, I have. I don't want to clutter this thread (as this is out of subject), but, that seems like a downloadable java archive. Does not include data, or information on how to hook up to a live server. Thanks for trying to help.

[sharabash]

@mmattdugan I can give you a dump of the word translations and Arabic words
and stuff like that, I.e. the relevant tables in postgresql. basically the
translations we scraped from Quranic corpus many years ago so it might be
out of date/out of sync with the current corpus but in sha Allah I can
provide you with a sql dump or plain text utf8 CSV dump of a relevant view
for you to parse. will this work?

On Tue, May 17, 2016, 8:24 PM Dashamir Hoxha notifications@github.com
wrote:

Have you checked this: http://corpus.quran.com/wordbyword.jsp
There is also an API: http://corpus.quran.com/java/
Maybe this is what you need.

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#145 (comment)

[doganmeh]

Please. Thanks a lot. @sharabash SQL dump may work better.

[abumq]

Assalamu alaykum ikhwaani,

Akhi @mmahalwy, you seem to have some control over decisioins. Here are my 2 cents. I developed https://muflihun.com from scratch and collected a lot of data and displayed only authentic tafaaseer and translations (as far as I know) (as you can see on the site) so if you need any data to improve quran.com, I will in shaa Allah share it.

We are one ummah (on top I am not author of anything or anyone who can claim 100% copyright on any islamic resource) and only Allah subhaanahu wa ta'ala can reward us in true sense and on Him we truly rely.

[mmahalwy]

@mkhan3189 mashallah! That's awesome! You have a great website :) I can definitely see how we can collaborate in terms of content. Have you checked out https://www.github.com/quran/quran.com-api ?

[abumq]

@mmahalwy no not yet but will in shaa Allah check it later

[doganmeh]

OK everybody my little Quran application is live here: http://quran4.me/

It consists of two packages which are open sourced in my github account:
https://github.com/doganmeh/quranfal
https://github.com/doganmeh/django-quran

What does it do?
Helps people who try to understand quran. Mark the words that you know and don't and it gives you report of i) the frequency of the word ii) what percentage of the Quran you understand and not. Basically click UNDER the word to mark as KNOWN, and ABOVE the word to add to your STUDY list. Click ON the word itself to go to details such as grammar information as well as other places that the word is in the Quran. There is also word-by-word meaning that you see if you HOVER over the word, but of course it does not work on the touch devices such as smartphones. There is also whole-aya meaning given when hovered over the aya-end marks.

I am also working on a flash card widget that will help review words in the study list. It is at its infancy as I deployed it a few days back, but I hope it will grow up to include just about everything for Quranic Arabic learners and help people learn Quran's meaning by reading it.. Please check it out and let me know what you think..

ps. 1) It is in beta stage, but I will do my best to preserve data... so you can go ahead and start using it.. 2) quranfal stands for Quran for Arabic Learners but I might change that name as the targeted audience is everybody who reads Quran.

Mehmet