Configure group based sorting in agenda

[gerazov]

Does this feature exist in Emacs orgmode core?

N/A

Orgmode link

No response

Feature value

The sorting of tasks in the agenda view seems a bit arbitrary - I get a mix of NEXT and TODO items, and the files they come from are also mixed even if all the tasks are for the same date. I couldn't find how sorting is done in the help file...

For e.g. I get:

понеделник 01 ноември 2021
  work:       Scheduled:  NEXT task name
  nehm:       Scheduled:  NEXT task name
  nehm:       Scheduled:  NEXT task name
  pros:       Scheduled:  NEXT task name
  work:       Scheduled:  NEXT task name
  nehm:       Scheduled:  NEXT task name
  nehm:       Scheduled:  TODO task name
  evoc:       Scheduled:  TODO task name
  nehm:       Scheduled:  NEXT task name

It would be very helpful if there is a way to group items based on keywords and files, and possibly set different priorities for sorting them in the agenda, e.g. DONE > DOING > NEXT > TODO 👍

Additional context

No response

[kristijanhusak]

Currently this logic is used: https://orgmode.org/manual/Sorting-of-agenda-items.html

But it seems this should be configurable with this: https://orgmode.org/worg/doc.html#org-agenda-sorting-strategy. It seems that there is no a specific setting to sort by the todo keyword. Both TODO and NEXT keywords are basically a TODO state, and orgmode allows sorting by TODO -> DONE or vice versa.

[gerazov]

Ok so it says:

For the daily/weekly agenda, the items for each day are sorted. The default order is to first collect all items containing an explicit time-of-day specification. These entries are shown at the beginning of the list, as a schedule for the day. After that, items remain grouped in categories, in the sequence given by org-agenda-files. Within each category, items are sorted by priority (see Priorities), which is composed of the base priority (2000 for priority ‘A’, 1000 for ‘B’, and 0 for ‘C’), plus additional increments for overdue scheduled or deadline items.

In orgmode.nvim the specific time todos are listed after the todos without a time of day, this is less useful, i.e. the emacs way sounds better.

And the items are not grouped in categories in my case, even if I pass an explicit list of files rather than `org/*' they don't get sorted - I get the same mix as before.

👍 for sorting them by todo state too even if not emacs style 😉

[levouh]

Something like this (code here and from an article here) would be something that users would have to add manually, right? Based on the linked code, it appears that jethrokuan has items sorted based on their TODO state in separate files, then just has a title and list if items that come together to make up a given view.

This looks like it would be something like this, which would accommodate for not just this use-case but realistically any "custom agenda view" use case.

Not 100% sure of the state right now, but would something like this require changes to the core code (this repo.) @kristijanhusak?

[kristijanhusak]

@levouh If we would add some similar configuration, then yeah, it would require some changes, and probably not so small ones. I'm not even sure what's possible with the custom agenda command, I would have to investigate. There is like an AgendaItem that basically wraps all logic around determining what and where to show, so that can be reused, but rendering itself would need changes.

[levouh]

Makes sense, I'll see if I can't come up with a good summary before taking a stab at what the code might actually look like. Can open up a separate issue to track it as well, whichever you'd prefer.

The main start on the issue would be:

1. Summary of how the functionality works in Emacs
2. What is applicable change-wise here, and what changes should be a part of the core

[kristijanhusak]

Makes sense, I'll see if I can't come up with a good summary before taking a stab at what the code might actually look like. Can open up a separate issue to track it as well, whichever you'd prefer.

The main start on the issue would be:

1. Summary of how the functionality works in Emacs
2. What is applicable change-wise here, and what changes should be a part of the core

Awesome, thanks! That will help tremendously. You can open up a separate enhancement issue, since this is related only to sorting.

[gerazov]

Ok, after some playing around with the sorting in agenda/init.lua I've finally fixed this 😅

The problem was that the all the tasks were sorted based on their timestamp. This meant that date_only tasks were randomly sorted and all of them went before the tasks scheduled with times of day (I guess os.time() was defaulting to 00:00 AM).

Now tasks with time of day are only sorted and they come first in the agenda view, followed by the tasks set without.

Before (notice how the two tasks from ea without time of day get separated):

After:

One thing that I still want to improve is for the tasks to follow the order of the files as given to orgmode in the config. This would allow users to e.g. put their personal task org last so work tasks comes before or vice versa. To this end I've commented out the file sorting in parser/files.lua in Files.all(), but still the order is not the same as in the config - I guess it's the async reading of the files (every reload gives a different order in the agenda):

    vim.schedule_wrap(function(err, content)
      if err then
        return callback(nil)
      end
      return callback(File.from_content(content, category, path, ext == 'org_archive'))
    end)

[gerazov]

Ok the ordering is now fixed 🤟 and the files are still read asyncly.

Here's what I get for the same date (assuming work > ea > valence)

[kristijanhusak]

@gerazov can you double check how Emacs orgmode is doing the sorting? I think I adjusted it to work the same way as there, but maybe I missed something.
If it's the same as there, lets go with some solution to allow configuring the sorting as a user, and leave this one as default. As a first (temporary) step we could have a sorting function exposed in the configuration, which you can override to your needs. Once #135 is done, this function would be removed or changed to adapt the new functionality.

[gerazov]

Yeah it's like that in the specs as far as I can tell (and it feels right too 😅 )

For the daily/weekly agenda, the items for each day are sorted. The default order is to first collect all items containing an explicit time-of-day specification. These entries are shown at the beginning of the list, as a schedule for the day. After that, items remain grouped in categories, in the sequence given by org-agenda-files. Within each category, items are sorted by priority (see Priorities), which is composed of the base priority (2000 for priority ‘A’, 1000 for ‘B’, and 0 for ‘C’), plus additional increments for overdue scheduled or deadline items.

Plus the randomization is minimized to within category listings, which is a bit weird since they are read from the same file 🤔

I haven't added the priorities though for the no time-of-day entries, will do that and update. I'm not sure about the

additional increments for overdue of scheduled or deadline items.

It reads to me that first comes the overdue deadline, then the overdue scheduled item, then the priority items, then the regular items, no?

I can also implement this for the todo-s list.

[gerazov]

Ok, it's now taking deadlines into account, but deadlines are sorted before overdue scheduled items. I'm not sure how it is in emacs as the spec doesn't read very exact. Here's what I get for 3 deadlines (1 overdue, 1 today, 1 tomorrow), 1 overdue scheduled item and 1 today item:

I'm happy with this, how does it look to you guys?

[kristijanhusak]

@levouh @gerazov I did some refactoring on agenda to allow easier introduction of custom commands. If you find any issues with current functionality let me know.

@gerazov you will have to rebase your sort PR because I split different agenda view in different files.

[gerazov]

Ok will do 👍

I'm not sure about the functionality still - I'll have to take a closer look. I get mixing of scheduled and deadlines according to overdue date...

[kristijanhusak]

If you want I'll look into this during this week, you just need to be a bit patient :)
I want to make it 1 to 1 with Emacs orgmode so it's consistent.

[gerazov]

Yes please 😅 no hurry 😉

[levouh]

I did some refactoring on agenda to allow easier introduction of custom commands. If you find any issues with current functionality let me know

Will do. Once I'm done with the org-edit-special stuff I plan on aggregating my notes into #135 and we can go from there. This refactor looks like it will only help, so should be good to go, thanks @kristijanhusak.

[kristijanhusak]

@gerazov I pushed a change that should make agenda sorting exactly the same as Emacs in orgmode, at least it was same in my tests.
Please give it a test, and also compare it to the Emacs orgmode to confirm if it's really the same.

[gerazov]

Ok awesome 😎 I'll give it a go!

[gerazov]

I get an empty agenda ...

Maybe it doesn't find my org files 🤔 - I list them in my config as such:

  org_agenda_files = {
      '~/org/work.org',
      '~/org/ea.org',
      '~/org/bme.org',
      '~/org/personal.org',
      },

[kristijanhusak]

@gerazov did you do :TSUpdate org ? Parser got some updates recently.

[gerazov]

Tried that now - still same problem. It's fine if I use my fork, so I guess it isn't neovim doing funny business.

[kristijanhusak]

@gerazov Try doing TSUninstall org and then TSInstall org, update wasn't sufficient for some people.

[gerazov]

Yup works! 🙏

The sorting is not ok though - the tasks that do not have a time of day stamp should be sorted by category first:

For the daily/weekly agenda, the items for each day are sorted. The default order is to first collect all items containing an explicit time-of-day specification. These entries are shown at the beginning of the list, as a schedule for the day. After that, items remain grouped in categories, in the sequence given by org-agenda-files. Within each category, items are sorted by priority (see Priorities), which is composed of the base priority (2000 for priority ‘A’, 1000 for ‘B’, and 0 for ‘C’), plus additional increments for overdue scheduled or deadline items.

[kristijanhusak]

I would suggest to set up basic emacs installation and compare with that. I did 1 to 1 comparison while setting up the sorting. Maybe what is written and what really happens is not exactly the same.

[gerazov]

That doesn't seem right. Did you try with multiple categories?

Now the overdue items get sorted before today's items, which let's say is ok, but not ideal (and not following specs). Still, even with all the scheduled items for today I get a mix of categories. This is surely not useful (i.e. I can't imagine what are they prioritized with?) And things really gets messy once you have 10+ org files and the tasks from different categories get mixed.

Here's a glimpse (these are all tasks scheduled for today 😅):

I implemented category based sorting in my fork and it works well:

    -- if different categories sort by category
    if a.headline:get_category() ~= b.headline:get_category() then
      return category_inds[a.headline:get_category()] < category_inds[b.headline:get_category()]
    end

[kristijanhusak]

Is this a screenshot from Neovim or emacs Orgmode?
For me sorting by the schedule/deadline date has more sense then to sort by category. Category is just some kind of label to me, doesn't really give any value to the task itself. If I'm overdue on something, I'd like to see that first.
I didn't think of this sorting functionality myself, as I said, I did 1 to 1 copy from Emacs orgmode. They have more options to offer beside this one, but this is the default (I think called "priority-up"). I think they also support setting up custom sorting function, so I'll probably expose something similar so you can override it as you wish.

[gerazov]

Yeah I can go with the overdue priority sorting, but the sorting for tasks scheduled for the same day (with no/equal priorities) should be category based no?

The screenshot is with neovim.

[kristijanhusak]

I think it just uses the default sorting, which is as things appear in the files.
It would be best to compare both of them since you have bunch of files, and if you find any differences let me know.

[gerazov]

I've given it another go - rebasing and squashing. I can't figure out how to apply the category sorting to the same date items only 😅

The default sorting is actually the problem because the files are read asynchronously - this mixes them up. But, I would keep the async loading mosdef 👍

[gerazov]

Ok installing emacs and configuring orgmode seems quite involved - that's how I came here in the first place 😅

What I've done is create a minimum working example.

Here are three identical files having 4 tasks scheduled for today. I had to rename them to .md - GitHub doesn't like orgmode apparently 😄

* TODO Task 1
  SCHEDULED: <2021-12-17 пет>
* TODO Task 2
  SCHEDULED: <2021-12-17 пет>
* TODO Task 3
  SCHEDULED: <2021-12-17 пет>
* TODO Task 4
  SCHEDULED: <2021-12-17 пет>

test1.md
test2.md
test3.md

Now when I run the agenda it shows:

I don't know if emacs reproduces this, but this mixing of categories and tasks within categories doesn't make sense. In fact, it's counter productive.

Also based on this MWE I'm not sure if it's the async loading doing it anymore - it seems random (yet refreshing it doesn't change the ordering) 🤔

[gerazov]

I've made the category based sorting in the PR, but it follows the spec, i.e. sorts by category and then by overdueness. (within category I still get the weird mix but this doesn't bother me as the position of the task in the file is not that important).

I've tried making it work overdue > category but can't make it work ...

The expanded files have three more old tasks:

* TODO Task 1
  SCHEDULED: <2021-12-17 пет>
* TODO Task 2
  SCHEDULED: <2021-12-17 пет>
* TODO Task 3
  SCHEDULED: <2021-12-17 пет>
* TODO Task 4
  SCHEDULED: <2021-12-17 пет>
* TODO Task 5
  SCHEDULED: <2021-12-16 чет>
* TODO Task 6
  SCHEDULED: <2021-12-15 сре>
* TODO Task 7
  SCHEDULED: <2021-12-14 вто>

They are here:
test1.md
test2.md
test3.md

[gerazov]

Ok with 7ba698a it now works based on overdues + categories:

[gerazov]

I've also added deadlines in the mix. At first it's mixing them with the regular items:

But with f6678a4 it's ok 🤟

[kristijanhusak]

@gerazov I tested your files with emacs orgmode, and you were right, it sorts by category in these cases. I pushed a fix where sorting falls back to the default order of headlines being loaded. Pull latest master and give it a test.

[jgollenz]

@gerazov is there anything left to this issue or can I close it?

[gerazov]

I guess so 😅 closing ...