Consider event tracking

[gracjan]

At this point we are in the dark about haskell-mode usage and about which parts of haskell-mode are used and in what ways. It is hard to make decisions about priority of optimizations, feature implementation or feature removal.

Insight by automatic event tracking could help a lot knowing what features are actually used and how.

Rough plan:

1. Use Google analytics event tracking to count how many times each function is invoked.
2. Gather cumulative stats of usage while program is working and contact google only every couple minutes.
3. Send also information about other modes enabled along with haskell-mode.
4. Send also keybindings to know how people bind commands.
5. Never send any personal data.

There is prior art because Aquamacs sends general usage info to their servers while checking for updates.

Tasks:

1. Implement low level protocol to contact google analytics.
2. Implement function instrumentation and activate it only in our files.
3. See how make google analytics show useful graphs using given information.

[gracjan]

Prior art:

https://github.com/Homebrew/brew/blob/master/share/doc/homebrew/Analytics.md

[gracjan]

More prior art:

http://popcon.debian.org/

[Gabriella439]

You should not collect analytics without first asking the user permission to do so

[ankhers]

I'm not entirely sure what data plans on being collected, but it may be useful if we were able to enable / disable analytics on a per project basis.

[ojw]

Maybe implement this as a separate package? Users could opt-in by installing it, and no one would have to worry about this feature being on accidentally, even if it was intended to ship as an opt-in feature only.

[fisx]

I'm strongly opposed. I'm a big fan of haskell-mode, and I would hate to spend time on removing google analytics from the code in order to be comfortable using it. Is there any chance of just dropping this idea?

If not: what are the questions you are hoping to answer with the data? If we can steer the discussion to this question, not only may it increase support of the idea, but you may also find some bits of data that you want to collect, but would have missed. I would also be curious to know what information, if not function call traces, you consider "personal" and not to be shared.

Just my two cents. Again: great works, can't thank you enough, please keep up the good work! (-:

[gracjan]

@fisx: Haskell Mode has accumulated complexity, most of which could be simplified provided that we knew what is and what is not used. Some questions I'd like to get answer to:

1. Which defcustom variables happen to be always at default value across all people using haskell-mode? If nobody cared enough to change the default, then it makes sense to remove the complexity of the variable.
2. Which defcustom variables happen to be always changed from the default value? A wrong default is a bug that could be just fixed. Otherwise we will get tutorials saying: you have to change this var to make haskell-mode work at all.
3. Which functions are called surprisingly high number of times? There might be a roundabout workflow that could be simplified.
4. Which functions aren't called at all? A useful functionality might be blocked by some bug, lack of documentation, lack of keybinding. Or it might be useful in theory but not in practice and then the complexity can be removed.
5. Which other minor modes are used together with haskell-mode? Emacs is an art of putting minor modes together and haskell-mode could be more friendly to other Emacs packages if it only knew how. For example only by accident I learned that smartparens is used with haskell-mode. Those work somewhat together, could work much better. What other modes are there that are used with haskell-mode?
6. The usual questions: which version of haskell-mode, which version of Emacs, Windows? Linux? Mac? Terminal or GUI?

We have had situations when obviously buggy functionality was used in a larger workflow and then sadness ensued when that functionality was removed. Such events have crippling effect on development morale basically people are afraid to touch things because 'somebody might be using it'.

About the 'personal' vs everything else: common theme to stats we would like to collect is the questions 'is functionality actually used?'. It is enough for us to count calls of functions defined publicly in haskell-mode, if publicly available minor modes are used together with haskell-mode, if defcustom variables are changed to one of predefined (defined in the source code) values. We would draw the line just short of reporting the contents of free-form variables, although the one-bit info if it was changed from standard value would be useful.

Low hanging fruit is gone, to 'keep up the good work' (as you say) we need to have some real data to guide the development.

[fisx]

Thanks @gracjan. I am not any more comfortable with the idea, but you are giving a lot of really good reasons for it.

[vlatkoB]

Can you make a separate package that would collect and aggregate needed data/stats locally and save it in a simple readable pure text file that can be inspected and afterwards sent to wherever/whomever you want?

I think most of us (users) would gladly sacrifice 10 mins a week to help better haskell-mode, while being felt secure and private.

[gracjan]

@vlatkoB: Can you implement your proposal so that we can check in practice how it works?

[vlatkoB]

@gracjan Unfortunately, I will not have time in foreseeable future to take this task.

[gracjan]

Event tracking would require somebody to sit and understand what the collected data means. I do not think we have enough hands on board to do this at this point.

[gracjan]

https://mail.haskell.org/pipermail/ghc-devs/2016-December/013363.html