Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow downloading more content from a webpage and index it #215

Closed
kamtschatka opened this issue Jun 9, 2024 · 3 comments
Closed

Allow downloading more content from a webpage and index it #215

kamtschatka opened this issue Jun 9, 2024 · 3 comments
Labels
feature request New feature or request

Comments

@kamtschatka
Copy link
Collaborator

I regularly bookmark youtube videos, Instagram videos, other videos.
It is not assured, that those videos stay online forever, so I prefer to download important videos (yes I am a real hoarder).

Would be great if you can enable downloading videos and serving them from hoarder for later viewing (Filesize does not matter to me, but I guess for some it matters).

Would be also great if the subtitles would be downloaded and indexed, so searching is possible also in the video content.
In the long run it would also be cool if it is possible to transcribe the video contents and make it searchable that way.
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jun 12, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
d84875d 
…p#215

added a new table that contains the information about assets for link bookmarks
created migration code that transfers the existing data into the new table
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jun 12, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
c917a4c 
…p#215

removed the old asset columns from the database
updated the UI to use the data from the linkBookmarkAssets array
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jun 18, 2024
@kamtschatka
Changes to have more assets hoarder-app#215
269c722 
improved the mapping to be more easily extendible
extracted out some duplicated code
MohamedBassem added a commit that referenced this issue Jun 23, 2024
@kamtschatka @MohamedBassem
refactor: extract assets into their own database table. #215 (#220)
9ce6958 
* Allow downloading more content from a webpage and index it #215
added a new table that contains the information about assets for link bookmarks
created migration code that transfers the existing data into the new table

* Allow downloading more content from a webpage and index it #215
removed the old asset columns from the database
updated the UI to use the data from the linkBookmarkAssets array

* generalize the assets table to not be linked in particular to links

* fix migrations post merge

* fix missing asset ids in the getBookmarks call

---------

Co-authored-by: MohamedBassem <me@mbassem.com>
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jun 30, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
484f422 
…p#215

Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jul 1, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
f77478d 
…p#215

Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jul 21, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
5ebb591 
…p#215

Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jul 21, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
1a5f1bd 
…p#215

Rebased onto master
replaced redis queue with new db queue
fixed some async/await issues
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jul 21, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
c4d7fb6 
…p#215

fixed additional async/await issues
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jul 21, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
a702812 
…p#215

Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Jul 21, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
f2c97ac 
…p#215

Rebased onto master
replaced redis queue with new db queue
fixed some async/await issues
@OliverLippertVw
Copy link

It is not assured, that those videos stay online forever

True. I was planning a long (1 month) trip for over half a year and even in this "short" period of time we had the one or other reel that doesn't exist any longer.

Not to start telling about my various collection of receipts (10+ years) -.-

So linking but also downloading + indexing is a huge benefit, I love to see as well.

@huyz
Copy link

huyz commented Aug 1, 2024

For media downloading, I currently use ArchiveBox.

@MohamedBassem MohamedBassem added the feature request New feature or request label Oct 5, 2024
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Oct 11, 2024
@kamtschatka
Allow downloading more content from a webpage and index it hoarder-ap…
c7f5997 
…p#215

Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation
@khronimo
Copy link

I would love to see it being able to index the transcript in Youtube links. I often bookmark interesting videos, interviews and am always frustrated when i try to find that particular resource again later.

@github-project-automation github-project-automation bot moved this from In Progress to Done in Hoarder's Roadmap Oct 28, 2024
kamtschatka added a commit to kamtschatka/hoarder-app that referenced this issue Nov 2, 2024
@kamtschatka @MohamedBassem
feature: Archive videos using yt-dlp. Fixes hoarder-app#215 (hoarder-…
80fcf5a 
…app#525)

* Allow downloading more content from a webpage and index it hoarder-app#215
Added a worker that allows downloading videos depending on the environment variables
refactored the code a bit
added new video asset
updated documentation

* Some tweaks

* Drop the dependency on the yt-dlp wrapper

* Update openapi specs

* Dont log an error when the url is not supported

* Better handle supported websites that dont download anything

---------

Co-authored-by: Mohamed Bassem <me@mbassem.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New feature or request
Projects
Hoarder's Roadmap
Status: Done
Milestone
No milestone
Development

No branches or pull requests
5 participants
@huyz @MohamedBassem @kamtschatka @khronimo @OliverLippertVw