look into org-noter support

[bdarcus]

The question

What, if anything, do we need to better integrate citar and org-noter, regardless of whether one is also using citar-org-roam?

Details

With announcement of new maintainers for org-noter, it seems its future is more secure.

We should look into ORB-like support here, either in the code itself (say a template variable), or on the wiki.

I originally added this to the citar-org-roam tracker, but realized the below would work, either for inclusion in citar-org, or on the wiki.

(defvar citar-org-noter-title-template
  " * ${title}
- tags ::
- keywords :: ${keywords}
:properties:
:custom_id: ${=key=}
:url: ${url}
:author: ${author editor}
:noter_document: ${file}  ; <== special file keyword: if more than one filename
:noter_page:               ;     is available, the user will be prompted to choose
:end:")

Also, see if citar and citar-org-roam template approaches are, or can be, aligned, so one can mix and match across contexts?

See also #750.

[bdarcus]

@deen1 - with the caveat I'm not sure when I'll have time to implement this (depending on how complicated it is), it might be good to document here what's actually required. I don't ATM use org-noter, so am unsure.

I will add that it looks like I created this issue around the same time I added capture template support to citar-org-roam.

So the question:

What, if anything, do we need to better integrate citar and org-noter, regardless of whether one is also using citar-org-roam?

BTW, the other project I've been working on is a rust project, in which I'm also learning the language. So my coding brain cycles have been wrapped up there for the last six months. Why it's hard to switch back to elisp, and this code :-)

[deen1]

I wouldn't say I'm well-versed in org-noter, much less Emacs package
creation in general, so take the following with a grain of salt.

As well as my reference library (i.e. journal papers and the like), I
also have a separate collection of pdf files for news articles etc. I
use org-noter to take notes on these and store them in large files,
with many headings corresponding to individual articles.

As far as I can tell, the important thing is that under each org
heading, the PROPERTIES drawer contains the NOTER_DOCUMENT property,
which is the path to the file. Then setting
org-noter-notes-search-path and org-noter-default-notes-file-names
will allow org-noter to associate the pdf file and the notes files. So
I can either open the notes file, go under the appropriate org-heading
(and PROPERTIES drawer), do M-x org-noter and get an org-noter frame, OR
I could go to a pdf with an pre-existing notes file (or heading), and do M-x org-noter,
and get the same result.

This at least works for the news article pdfs. For the journal articles
and use with citar, there seems to be a slight wrinkle to do with
making sure the filename of the notes file matches the filename of the
pdf file. For instance, I used to use Zotero to manage my bibliography,
and the names for pdfs took the form %AUTHORS %YEAR %TITLE.pdf.
Creating a notes file for them via citar means that the notes file has
the form ${citekey}.org, which causes an issue when org-noter tries to find the notes
for a given pdf. (But it works vice versa, going to the ${citekey}.org file and doing M-x org-noter
still works properly.)

Finally, and I'm not 100% sure about this, but I think

(setq citar-open-note-function 'orb-citar-edit-note)

might be necessary to get this to all work together, but I'm unsure how.

(P.S. I can appreciate the difficulty toggling between different projects. I should be doing something other than yak-shaving my Emacs config, but I'm convinced it will improve my productivity :-) )

[bdarcus]

I wouldn't say I'm well-versed in org-noter, much less Emacs package creation in general, so take the following with a grain of salt.

No, this is actually useful.

As far as I can tell, the important thing is that under each org heading, the PROPERTIES drawer contains the NOTER_DOCUMENT property, which is the path to the file.

Right.

IIRC (need to check), citar-org-roam defers to org-roam to create the property drawers, so that would be something to figure out.

One additional issue alongside that is to think about workflow with all these moving parts.

With the standard citar note function, I see no problem.

For the journal articles and use with citar, there seems to be a slight wrinkle to do with making sure the filename of the notes file matches the filename of the pdf file. For instance, I used to use Zotero to manage my bibliography, and the names for pdfs took the form %AUTHORS %YEAR %TITLE.pdf. Creating a notes file for them via citar means that the notes file has the form ${citekey}.org, which causes an issue when org-noter tries to find the notes for a given pdf. (But it works vice versa, going to the ${citekey}.org file and doing M-x org-noter still works properly.)

Right, this is another thing to check.

It may be that with noter one would need to write the path in the bib entry?

Finally, and I'm not 100% sure about this, but I think

(setq citar-open-note-function 'orb-citar-edit-note)

might be necessary to get this to all work together, but I'm unsure how.

Does all of this work with orb? If so, that indeed would be another thing to check; how they do it.

[deen1]

It may be that with noter one would need to write the path in the bib entry?

I'm not sure. The bibfile that contains the Zotero-originating citations includes the path to the pdf file for each bib-entry (where one exists). Unless you meant the path to the notes .org file, which I just tried to add to one entry to see if that helped -- same negative result.

One thing that slightly confuses me, and might be a red herring, is that some of these notes files have an additional :PROPERTIES: drawer, above the main org heading, looking like:

:PROPERTIES:
:ID:       2dabb44e-1132-4fb7-b9c5-4b3842c12510
:ROAM_REFS: cite:foo
:END:

Maybe this helps? Not sure.

Does all of this work with orb? If so, that indeed would be another thing to check; how they do it.

I think so. Here are some settings that might be useful (sorry to spam):

  (setq orb-preformat-keywords
        '("citekey" "title" "url" "author-or-editor" "keywords" "file")
        orb-process-file-keyword t
        orb-file-field-extensions '("pdf"))

  (citar-register-notes-source    'orb-citar-source
				  (list :items #'citar-org-roam--get-candidates
					:hasitems #'citar-org-roam-has-notes
					:open #'citar-org-roam-open-note
					:create #'orb-citar-edit-note
					:annotate #'citar-org-roam--annotate))
  (setq citar-notes-source 'orb-citar-source)

These help enable this capture template for notes:

  (add-to-list 'org-roam-capture-templates
	       '("r" "bibliography reference" plain
		 (file "~/org-roam/org-notes/template.org")
		 :if-new
		 (file+head "~/org-roam/org-notes/${citekey}.org" "#+title: ${title}\n")
		 )))

with template.org looking like this:

* %^{title}
:PROPERTIES:
:Custom_ID: %^{citekey}
:URL: %^{url}
:AUTHOR: %^{author-or-editor}
:NOTER_DOCUMENT: %^{file}  
:END:


- tags ::
- keywords :: %^{keywords}

[bdarcus]

I think so. Here are some settings that might be useful (sorry to spam).

I mean internally in ORB. How does it support org-noter, or how do users configure it to do so?

[deen1]

I'm not sure. I think the settings I provided are a way in which ORB gets the information it needs (from citar, or from helm-bibtex) to create fill in the note template. Apart from that, you seem to have already found one piece of the puzzle here.

[bdarcus]

You mean the "file" keyword?

I think we can already do this, though there there may be some wrinkles? Like, org-noter I assume requires a single path, but citar can return multiple.

[roshanshariff]

From briefly looking at org-noter, here's my understanding of the problem:

Finding the PDF file from a given notes file should work fine, because as @deen1 said, org-noter puts the PDF filename in a property drawer under each heading.

The converse, finding the note file associated with a PDF file, requires the notes file to have the same base name as the PDF file but with a ".org" extension; this is hard-coded into org-noter. With Zotero, PDF files can have arbitrary names, so if you want your notes to be named by citation key (for example) then there's no way to get org-noter to find your notes.

In principle, Citar could help here by locating the citation key associated with a given PDF (by reverse-lookup of the file field in the bibliography file exported from Zotero). The problem is, even if such a function existed, org-noter has no way of overriding its note filename lookup logic.

I think the first step would be to get org-noter to refactor this logic out (i.e. a function to locate a note filename given a PDF filename) and make it customizable. Then we could override that function with one that does the lookup through Citar. Ideally, org-noter would have a customizable list of functions it calls in order to get the note file associated with a given PDF document.

(That said, I don't have a lot of time to work on it these days because I'm trying to get my thesis done soon.)

[bdarcus]

I think the first step would be to get org-noter to refactor this logic out (i.e. a function to locate a note filename given a PDF filename) and make it customizable. Then we could override that function with one that does the lookup through Citar. Ideally, org-noter would have a customizable list of functions it calls in order to get the note file associated with a given PDF document.

@petermao - would you be open to this feature request?

PS - I see you're in Santa Monica; I grew up there!

(That said, I don't have a lot of time to work on it these days because I'm trying to get my thesis done soon.)

Best of luck with that.

[deen1]

Thanks @roshanshariff, that helped clarify my empirical understanding of org-noter's inner workings! I've shifted away from the standalone Zotero application recently, (though I still use it from time to time due to the sizeable library I'd built up over the years). The new Zotra package fits my current needs, and the maintainer there took the time to add some functionality so that I can now grab pdfs from the arXiv via org-roam-protocol and link them to a new bibentry. (If you haven't already, maybe worth checking out? Once you've defended of course :-)) The pdfs are then given the citar-key for their filename, so the issue I described about finding the corresponding notes file doesn't occur with these new entries as far as I can tell.

@bdarcus Would it then be possible to implement the "file" keyword before handling any changes to org-noter itself? The Zotero/non cite-key filename issue is a little annoying for me, but I can work around it as it's a) a problem in one direction only b) maybe not a problem for other people who haven't had the same circuitous journey through bibliography management? Obviously no urgent need on my part for any of this to get done (and thanks again for all your prior work on this package!), just a suggestion possible course of action.

[bdarcus]

Would it then be possible to implement the "file" keyword before handling any changes to org-noter itself?

Is it not already?

[deen1]

Sorry, I meant in terms of adding it to the PROPERTIES drawer. I can't seem to get this work via templates for some reason.

[bdarcus]

Not sure what I can do when, but in the meantime, are you aware of:

https://retorque.re/zotero-better-bibtex/
http://zotfile.com/

[petermao]

I think the first step would be to get org-noter to refactor this logic out (i.e. a function to locate a note filename given a PDF filename) and make it customizable. Then we could override that function with one that does the lookup through Citar. Ideally, org-noter would have a customizable list of functions it calls in order to get the note file associated with a given PDF document.

@petermao - would you be open to this feature request?

The refactoring has sort-of happened (see this commit from June 4, 2023). See the stuff about ...create-session-from-document-file....

For the record, default org-noter does the following to find a notes-file from a document-file:

1. Look for files named <(file-name-base document-filename)>.org or in the list org-noter-default-notes-file-names (default is "Notes.org") in the directories specified in 2 and 3 below.
2. Check the directories in the list org-noter-notes-search-path (default is "~/Documents")
3. Check the current directory and all directories above it up to ~/
4. Check each file for a NOTER_DOCUMENT property that matches the document-filename.
5. If there is more than one candidate, then present them to the user, otherwise, open the file.

This is as it came from the original author, so for me and @dmitrym0, this is "found code." I could imagine expanding org-noter-default-notes-file-names to accept globs or regexes.

The recently added functions org-noter--update-doc-rename-in-notes and org-noter--update-notes-rename-in-notes use the same logic maintain the NOTER_DOCUMENT property when either the document or the notes-file gets moved or renamed. I'm not sure how widely this new feature is used because it has to be activated by the user (we don't want to create surprises).

Dmitry has an org-roam compatible ...create-session-from-document-file... in the works as the existing framework is insuffiently flexible (why? I don't know, not being a hard-core org-roam user....yet).

PS - I see you're in Santa Monica; I grew up there!
Huzzah! It's a wonderful place.

[deen1]

Not sure what I can do when, but in the meantime, are you aware of:

https://retorque.re/zotero-better-bibtex/ http://zotfile.com/

Yes -- in fact that's what I've been using before. I'm not sure how that would help add stuff to the PROPERTIES drawer in an org file.

The point about all of this was to move away from Zotero to something more contained in Emacs, and since I started using citar and citar-org-roam, to use variables defined in those packages to create the notes files. Otherwise, I could just stick with the ORB setup I was using before, which seemed to work fine.

[bdarcus]

I'm not sure how that would help add stuff to the PROPERTIES drawer in an org file.

I was mentioning them in response to your point about file names, since zotfile allows you rewrite and move them.

[bdarcus]

@petermao

Dmitry has an org-roam compatible ...create-session-from-document-file... in the works as the existing framework is insuffiently flexible ...

Do you have an issue over there for that that we can link this to?

In any case, might be useful for him to know about this.

EDIT: never mind; I see the draft PR; I added the link to this issue there.

[deen1]

I was mentioning them in response to your point about file names, since
zotfile allows you rewrite and move them.

It had crossed my mind to do that at some point, but there's a bit of a hysteresis issue as I would also have to change the filename in the NOTER_DOCUMENT field for each of the notes I've already created (and possibly their filenames as well), which would be a bit tedious to do manually, and maybe a little tricky to do automatically.

[dmitrym0]

Dmitry has an org-roam compatible ...create-session-from-document-file... in the works as the existing framework is insuffiently flexible (why? I don't know, not being a hard-core org-roam user....yet).

org-noter has a very opinionated way of deciding where notes go - typically a notes.org. I like org-roam's approach -- you get to decide at capture time where the note should go -- either an existing note, or a new node (a "node" in org-roam's parlance).

[deen1]

Think I came up with a temporary fix for the NOTER_DOCUMENT issue, could someone check this out and see if it works for them? I.e. the PROPERTIES drawer is suitably altered, and running org-noter inside the notes .org file produces an Org-noter note-taking window arrangement/session with the corresponding .pdf.:

(defun citar-add-org-noter-document-property(key &optional entry)
  (interactive)
    (let* ((file-list-temp (list (citar--select-resource key :files t)))
	(file-path-temp (alist-get 'file file-list-temp))  )
    (org-set-property "NOTER_DOCUMENT" file-path-temp) ))

(advice-add 'citar-create-note :after #'citar-add-org-noter-document-property)

My emacs-lisp is not that good, so maybe there is a nicer or safer way of doing this.

EDIT:

This function more or less matches what I had as my org-roam-bibtex template:

(defun citar-add-org-noter-document-property(key &optional entry)
  "Set various properties PROPERTIES drawer when new Citar note is created."
  (interactive)
  (let* ((file-list-temp (list (citar--select-resource key :files t)))
	 (file-path-temp (alist-get 'file file-list-temp)) ;;
	 (cite-author (cdr (citar-get-field-with-value'(author) key)))
	 (cite-url (cdr (citar-get-field-with-value '(url) key))) )

    (org-set-property "NOTER_DOCUMENT" file-path-temp)
    (org-set-property "Custom_ID" key)
    (org-set-property "AUTHOR" cite-author)
    (org-set-property "URL"    cite-url)
    (org-roam-ref-add (concat "@" key))
    (org-id-get-create) ))

(advice-add 'citar-create-note :after #'citar-add-org-noter-document-property)

[asymmetric]

What's the current state of integration?

The usecase I'd like to address is:

• call citar-open
• select an entry
• be shown an existing org-noter note (or offer to create one), similar to how it happens with org-roam

At the moment this is not possible, and I have to switch to the PDF and then run org-noter.

[asymmetric]

Also, I noticed this file exists, but I've loaded it and it doesn't seem to do what I describe above.

Plus it's not in the package on MELPA, so I guess it's old and not really meant to be used.

[bdarcus]

What's the current state of integration?

I really have no clue, other than to note that the org-noter PR that may have made at least some of this possible has been merged.

org-noter/org-noter#39

Also, I noticed this file exists, but I've loaded it and it doesn't seem to do what I describe above.

Plus it's not in the package on MELPA, so I guess it's old and not really meant to be used.

And it's a bit odd it requires both citar and org-ref.

Perhaps you can dig into the org-noter changes and report back?

[bdarcus]

What's the current state of integration?

The usecase I'd like to address is:

• call citar-open
• select an entry
• be shown an existing org-noter note (or offer to create one), similar to how it happens with org-roam

At the moment this is not possible, and I have to switch to the PDF and then run org-noter.

Would be nice too if it could elegantly work with citar-org-roam, though I'm unsure of technical details of how and where that ideally would be implemented.

[dmitrym0]

What's the current state of integration?

I really have no clue, other than to note that the org-noter PR that may have made at least some of this possible has been merged.

org-noter/org-noter#39

This PR has been merged, and while it's specifically targeting org-roam, the note file opening process is generic enough to support any workflow, I hope. There's additional documentation here: https://github.com/org-noter/org-noter/blob/update-documentation-customization-recipes/docs/customization-recipes.org#customizing-note-opening-process

Also, I noticed this file exists, but I've loaded it and it doesn't seem to do what I describe above.
Plus it's not in the package on MELPA, so I guess it's old and not really meant to be used.

And it's a bit odd it requires both citar and org-ref.

org-noter code base is 5+ years old at this point with multiple contributors and maintainers over the years so I'm not surprised that some things have gone stale.

[bdarcus]

What's the current state of integration?

I really have no clue, other than to note that the org-noter PR that may have made at least some of this possible has been merged.
org-noter/org-noter#39

This PR has been merged, and while it's specifically targeting org-roam, the note file opening process is generic enough to support any workflow, I hope. There's additional documentation here: https://github.com/org-noter/org-noter/blob/update-documentation-customization-recipes/docs/customization-recipes.org#customizing-note-opening-process

OK, thanks @dmitrym0!

For anyone that wants to look into this more (I don't really have time ATM):

It seems org-noter-create-session-from-document-hook is the key customization point.

Here's an example of a citar "notes source" configuration; in this case for org-roam integration:

https://github.com/emacs-citar/citar-org-roam/blob/main/citar-org-roam.el#L75-L81

The things we need for full integration are functions to:

1. identify which sources have associated notes (to update the citar display)
2. how to open such notes if they exist
3. how to create new notes if they don't exist

I'm just a bit unsure how, and where, it would best fit together.

EDIT: to expand on the last bit.

Citar now has a notes API that should ideally work for this use case. So my question here is where the relevant open and create functions should live for org-noter integration.

In the org-roam case, citar-org-roam has functions for that, of course. So really, I think, it comes down to the create new notes issue.

[deen1]

@asymmetric

The usecase I'd like to address is:

• call citar-open

• select an entry

• be shown an existing org-noter note (or offer to create one), similar to how it happens with org-roam

At the moment this is not possible, and I have to switch to the PDF and
then run org-noter.

The fix I mentioned above allows me to use org-noter in the way you describe. As well as that function, I needed to add a capture-template like the following:

(add-to-list 'org-roam-capture-templates
	     '("c" "citar literature note" plain "%?"
	       :target (file+head "%(expand-file-name citar-org-roam-subdir org-roam-directory)/${citar-citekey}.org"
				  "#+title: Notes on: ${citar-title}\n#+subtitle: ${citar-author}, ${citar-date}")
	       :unnarrowed t))

(setq citar-org-roam-capture-template-key "c")

The only thing I can't do is create a new note when currently viewing the PDF file for the citation. However, if the note file already exists, running org-noter when viewing the PDF works as expected. And I can create the note file by running citar-open, selecting the entry, and hitting RET on the Create Org-Roam Notes option.

[bdarcus]

Perhaps you could add that to the wiki, @deen1?

[deen1]

@bdarcus OK, I've written something up (hopefully haven't messed up the other stuff on the page).

[asymmetric]

@deen1 thank you so much for the detailed instructions. One question: why is your implementation tied to org-roam? In principle, would a direct linking of org-noter notes with citar keys (without org-roam) be possible?

[deen1]

@asymmetric Hmm... well I started using org-roam a few years ago, and had used org-noter in conjunction with org-roam-bibtex, so it seemed natural to switch to citar-org-roam when I changed my citation manager to citar from helm-bibtex. I find this workflow very useful in that my notes on particular topics are then linked in the org-roam database to the file notes created via citar, so I can look at backlinks etc.

But yes, my apologies if my "fix" does not help use-cases that do not involve org-roam. I think you would have to follow @bdarcus 's outlined program above and create this "fully integrated" functionality afresh, rather than relying on his earlier work in creating citar-org-roam.

Although maybe it's possible to look at citar-org-roam.el and re-write it for a generic .org file, so that instead of citar-org-roam-capture-template-key, you have citar-org-capture-template-key, and use my suggested capture template as a regular org-capture-template?

[dmitrym0]

Hey folks,

I'm org-noter maintainer -- following up on this. I'm at a point now, where I'm overwhelmed with the number of documents I'm juggling and Citar seems like the answer.

I'm reading the wiki entry from above, and I have some questions:

The only behaviour that the following instructions can't enable is the ability to create a new note file which automatically has all the citar data filled in (in the file header, title, section headlines or filename) when viewing a .pdf file by running M-x org-noter.

I'm not yet familiar with the bibliography workflow. For non org-noter use cases, where does the metadata listed above come from? Is it entered manually?

What is needed from org-noter to support a correct end to end Citar workflow?

[bdarcus]

I'm not yet familiar with the bibliography workflow. For non org-noter use cases, where does the metadata listed above come from? Is it entered manually?

No; it comes from the metadata that citar accesses from the bib file(s).

What is needed from org-noter to support a correct end to end Citar workflow?

Caveat: I am not an org-noter user, so others on this thread will have better understanding of that end.

My earlier post is still correct from a high-level POV. I added a bit of clarification at the end.

Along with my question there about where code lives, I'm also ultimately unsure of the ideal user workflow here, etc.

It seems to me ideally the choice of using org-noter is not mutually exclusive to the choice of general note package. For example, someone using citar-org-roam should be able to get excellent org-noter support as well, just as someone should if they're using the default citar notes source.

[deen1]

@dmitrym0 Sorry for the slow response, but just to clarify in case you are still interested in this:

That paragraph in the wiki entry (which I wrote) describes a use-case where one is viewing a .pdf which is associated with an entry in a bibfile that is in the citar-bibliography list. To do what I described there, i.e. creating an Org file with Citar metadata appropriately filled in the note file after running M-x org-noter, either Org-noter or Citar would then

• Search for the .pdf filename in the bibfiles listed in citar-bibliography

• Get the citekey for the bibitem to which the .pdf is associated

• Run citar-create-note with the citekey above passed as an argument to create an Org file using the citar-org-roam-capture-template (I forget now if citar-create-note automatically uses this template if citar-org-roam mode is enabled …)

• Once the Org note file is created (with the .pdf filename in the PROPERTIES drawer as the NOTER_DOCUMENT property), run org-noter as before create a session with the usual split frame between notes and .pdf.

[dmitrym0]

Hey @deen1, I am still interested in this. In fact, I was inspired to understand the what and/how of zotero/citar thanks to this issue.

I recently pushed a small PR (link).

It makes org-noter aware of whether there are existing notes files for a specific publication:

1. M-x citar-open a pdf file
2. M-x org-noter
3. Create a new node for notes
   ...

then:

1. open the same pdf file
2. M-x org-noter

No step 3. Org-noter identifies an existing node and opens it automatically:

flowchart TD
    A[User invokes M-x org-noter on a PDF file] --> B{Has user previously taken notes for the specific PDF file?}
    B -- Yes --> C[Open the notes file automatically]
    B -- No --> D[Ask the user to select a notes file interactively]
    D --> E[Open the selected notes file]

Loading

Essentially this ensure that your org-noter session always goes to the same literature notes file.

Re-reading your wiki entry it seems like what is missing is populating the literature note bibliography metadata?

[deen1]

@dmitrym0 Looks interesting, I'll have to check that out soon!

Re-reading your wiki entry it seems like what is missing is populating the literature note bibliography metadata?

Yes (unless I am forgetting something else). Basically, the metadata is used to fill in the #+Title:, #+Author: etc. headings, and maybe also the #+filetags. I'm guessing your PR/update already fills in the :NOTER_DOCUMENT: property for the file property drawer, and also uses the citekey for the notes filename?