Download "top 100 videos" from YouTube channel or playlist, sorted by views-per-day

[deldesir]

🚀 Pull Request Overview:

This PR adds views-per-day to the metadata so videos can be sorted automatically based on the playlist. If the playlist contains more than 100, a bookshelf with the top 100 videos (based on their views per day) is created. For playlists with less than 100 videos, the sorting still happens automatically.

📋 Checklist:

• Tested the changes on Ubuntu 24.04 with the following playlist: https://www.youtube.com/playlist?list=PLJicmE8fK0EinZZ6hY-RzJr8kcgIERWEc (18 videos)

📌 Testing scenarios:
See Issue #97

cc @EMG70

[holta]

1. Great Progress!!

videos can be sorted automatically

2. Are videos auto-sorted for users by views-per-day (highest-to-lowest) as soon as they look at the bookshelf? (Or not yet?)

Top 50 videos bookshelves are created for playlists with more than 50 videos and top 100 for ones with more than 100.

3. That's too confusing, for new users especially.

   Please replace this "step function" (steps at 10, 50, 100) with one fixed number (e.g. 100) that is easily changeable in a config file (to 10 or 50 or whatever) by reading instructions at https://github.com/iiab/calibre-web/wiki

   Let's strongly consider a universal default of 100 (for starters, aiming to assist educators and parents etc) ensuring everything is very easy to explain and understand — with a cautionary warning for implementers so they know ABOUT how long the channel/playlist might take to download — into their Calibre-Web bookshelf. ⏱️

4. Obviously Quality is what truly matters, not Quantity — but getting there is extremely hard. In the interim, if many communities require "Top 1GB" or "Top 10GB" subsets then that's one (of several) options that will be considered later.

[holta]

Can we force days_since_publish to be 1 at minimum (i.e. 1 day) to avoid divide-by-zero errors?

[holta]

NUMBER_OF_VIDEOS = 100

@deldesir please use a vividly descriptive variable/constant name. Possibly something like:

• MAX_VIDEOS_PER_DOWNLOAD

Likewise in future, we might need to consider:

• MAX_GB_PER_DOWNLOAD

[holta]

@deldesir thanks for clarifying:

• Does this PR work with YouTube channels yet?

• Is every video (mentioned by every playlist in a YouTube channel!) ranked by views-per-day?

  1. If yes, does this mean that the "Top 100 Videos" derived from a channel... sometimes include videos not actually created/owned/uploaded by that channel?

  2. How hard would it be to eliminate such "external" videos arising from other channels / other YouTube creators etc? (e.g. if implementer communities later decide that eliminating "external to channel" videos is important.)

[deldesir]

2. Are videos auto-sorted for users by views-per-day (highest-to-lowest) as soon as they look at the bookshelf? (Or not yet?)

Yes, this is done automatically.

3. That's too confusing, for new users especially.
   Please replace this "step function" (steps at 10, 50, 100) with one fixed number (e.g. 100) that is easily changeable in a config file (to 10 or 50 or whatever) by reading instructions at https://github.com/iiab/calibre-web/wiki
   Let's strongly consider a universal default of 100 (for starters, aiming to assist educators and parents etc) ensuring everything is very easy to explain and understand — with a cautionary warning for implementers so they know ABOUT how long the channel/playlist might take to download — into their Calibre-Web bookshelf. ⏱️

Done. Please see https://github.com/iiab/calibre-web/wiki/Home/_compare/8376933a218f164fa18f3cf378ce491501dd94e7...2222461e9ca347c017c0cf68173d4b61e3c02cc5

4. Obviously Quality is what truly matters, not Quantity — but getting there is extremely hard. In the interim, if many communities require "Top 1GB" or "Top 10GB" subsets then that's one (of several) options that will be considered later.

I agree. For this we'll need to know the approximate size of the videos before downloading them.

[deldesir]

Does this PR work with YouTube channels yet?
Is every video (mentioned by every playlist in a YouTube channel!) ranked by views-per-day?

This PR doesn't work with YouTube channels yet. I am working on it.

1. If yes, does this mean that the "Top 100 Videos" derived from a channel... sometimes include videos not actually created/owned/uploaded by that channel?

N/A

2. How hard would it be to eliminate such "external" videos arising from other channels / other YouTube creators etc? (e.g. if implementer communities later decide that eliminating "external to channel" videos is important.)

To eliminate "external" videos, I'll need to filter all videos based on their "uploader" ids. We should be covered once I am done supporting channels.

[holta]

To eliminate "external" videos, I'll need to filter all videos based on their "uploader" ids. We should be covered once I am done supporting channels.

Thanks for explaining:

We can also defer that, as videos that originate "external to the channel" are generally a rare/corner case — that individual IIAB implementers can (and probably should!) manually/quickly solve on their own.

[holta]

@deldesir what should @EMG70 be testing to confirm this PR is solid-or-close?

Also:

• What follow-up is most urgent in other (small!) PRs?
• e.g. What's needed to get "Download to IIAB" working well with YouTube channels-not-just-playlists?
• What other key workflows / aspects need to become more bulletproof?

Thanks!

[deldesir]

@holta, downloading top videos from channels now works. There is currently an issue with shorts being duplicated in xklb-metadata.db after their metadata has been updated. I work around this for now by filtering them. Hopefully, @chapmanjacobd can help resolve this so they can be included in top videos lists.

@EMG70, please test this PR following the instructions found in https://github.com/iiab/calibre-web/wiki#setting-the-maximum-number-of-videos-to-download. You can test with playlists, playlists of playlists, or channels.

[EMG70]

SUDO IIAB-DIAGNOSTICS - http://sprunge.us/mqtZDA?en
Just checking if all is OK,after running PR#139 .No video is downloading at all.Could Blondel be working on it at the same time?

[holta]

Thanks @EMG70 for the valuable feedback:

As this PR (or similar!) needs to be tightened up with solid testing, and merged quickly after ~2 weeks now 👍

[deldesir]

SUDO IIAB-DIAGNOSTICS - http://sprunge.us/mqtZDA?en
Just checking if all is OK,after running PR#139 .No video is downloading at all.Could Blondel be working on it at the same time?

I am investigating... I'll be back with an explanation.

[deldesir]

Thanks @EMG70 for pointing out the issue. I don't understand yet what background limitations causing this, but I just fixed it by structuring the code better. This PR will be closed and remade in to smaller ones for clarity. Thanks for your patience.

[holta]

Is the issue related to playlists and channels with many videos?

e.g. @EMG70's playlist with 151 videos?

In comparison with playlists containing <= 50 videos?

[deldesir]

The issue is tied to the latest commit I made to align with xklb v2.5.018. This commit ensured the database was fully updated in a single lb tubeadd run. When I reverted the code to the state where basic metadata is fetched first before updating to retrieve the key ones, top 100 videos were extracted successfully. However, I had to adjust the code to handle an "unavailable video" error for one of the playlists @EMG70 tested with (https://www.youtube.com/playlist?list=PLjxrf2q8roU23XGwz3Km7sQZFTdB996iG)

Regarding the Ed's failed test, I suspect the issue is indeed related to playlists and channels with many videos. It worked for a small playlist of 18 videos (with MAX_VIDEOS_PER_DOWNLOAD = 3).

[holta]

Thanks for explaining.

Please lay out different kinds of playlists/channels (or different kinds of sanity-tests?) that we can all help with — either revising or replacing these testing scenarios:

• Testing scenarios for multimedia downloads using iiab/calibre-web [TDD, Gherkin Stories] #97

[deldesir]

Done in #97 (comment)

[holta]

FYI this PR #139 is being refactored, starting with smaller PRs like:

• PR Refactor for better maintainability #142
• PR Add Max Download limits #143
• PR Update to retrieve key metadata #144
• PR Calculate each video's views-per-day [very rough approximation for popularity / importance / quality] #145
• PR Sort videos by views-per-day + download the top ones #146