HydrusYTDLProxy

A thing to allow to use YouTube-dlp with Hydrus Network

How to run

a) Local

Local Prerequisites

• Python 3.10+
• ffmpeg

Local Install

pip install -r /install/requirements.txt

Run

fastapi run main.py

b) Docker:

docker build -t hydrus-ytdl-proxy .
docker run -d -p 4458:4458 hydrus-ytdl-proxy

c) Coolify:

1. Add as NIXPACK
2. Set start command to fastapi run main.py --port=4458
3. Set network port to 4458.

Hydrus Network Setup

1. Open Hydrus Network
2. Go to network ➔ downloader components ➔ manage gallery url generators… and click on Add to add a new one with the following settings:
   1. name: HydrusYTDLProxy url
   2. url template: http://<YOUR_SERVER_HOST_OR_IP>:4458/meta?url={url}
      • Replace <YOUR_SERVER_HOST_OR_IP> with your server's host or IP address,
        or localhost if you are running it on the same machine as Hydrus Network.
      • Use https instead of http if you have ssl configured (Coolify does that), then you should also drop or adapt the port (:4458).
      • Adapt the port 4458 if you run the server on a different port.
   3. replacement phrase: {url}
   4. search term separator: empty
   5. initial search text (to prompt user): url to download
   6. example text search: https://www.youtube.com/watch?v=dQw4w9WgXcQ or
   7. example request url: _automatically generated: http://<YOUR_SERVER_HOST_OR_IP>:4458/meta?url=https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
   8. matches as a: automatically generated
      • Initially empty.
      • Later will be Matched HydrusYTDLProxy (meta as gallery) url class..
   9. OK
3. Go to network ➔ downloader components ➔ manage parsers… and click on add to add a new one with the following settings:
   1. name or description (optional): HydrusYTDLProxy (meta)
   2. example urls ➔ add: http://<YOUR_SERVER_HOST_OR_IP>:4458/meta?url=https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
   3. fetch test data from url: same as above
   4. click on fetch test data from url
   5. set up parsers.
   6. add
      1. ➔ name or description(optional): URL (best format)
      2. content type: urls
      3. url type: url to download/pursue (file/post url)
      4. url quality precedence (higher is better): 75
      5. edit formula (type JSON, otherwise click change formula type)
         1. name/description: URL (best format) from json
         2. add the following 3 steps:
            1. dictionary entry by key:
               • match type: fixed characters
               • fixed text: formats
            2. dictionary entry by key:
               • match type: fixed characters
               • fixed text: best
            3. dictionary entry by key:
               • match type: fixed characters
               • fixed text: url
         3. content to fetch: string
         4. no string processing.
   7. You can add more parsers if you want, e.g. for title, description, tags, other formats, etc.
      • See http://<YOUR_SERVER_HOST_OR_IP>:4458/docs for available fields for the /meta endpoint.
   8. OK
4. Go to network ➔ downloader components ➔ manage parsers… and click on add to add a new one with the following settings:
   1. name or description (optional): HydrusYTDLProxy (redirect)
   2. example urls ➔ add: https://www.pornhub.com/view_video.php?viewkey=ph63348bf2f3330
   3. fetch test data from url: same as above
   4. click on fetch test data from url
   5. set up parsers: add
      1. name or description(optional): URL: Redirect to API (click edit formula twice to edit domain)
      2. content type: urls
      3. url type: url to download/pursue (file/post url)
      4. url quality precedence (higher is better): 50
      5. edit formula (type ZIPPER, otherwise click change formula type)
         1. name/description: Zipper rule to build the new url to redirect to the API.
         2. add the following 3 zipped parts:
            • Zipper (start of url)
              1. change formula type ➔ change to a new ZIPPER formula…
              2. edit formula
                 1. name/description: EDIT HOST HERE IN "substitution phrase"
                 2. add
                    1. change formula type ➔ change to a new CONTEXT VARIABLE formula…
                    2. edit formula
                       1. name/description: url
                       2. variable name: url
                       3. no string processing.
                       4. apply
                    3. apply
                 3. substitution phrase: http://<YOUR_SERVER_HOST_OR_IP>:4458
                 4. apply
              3. apply
            • Zipper (middle part of url)
              1. change formula type ➔ change to a new ZIPPER formula…
              2. edit formula
                 1. name/description: middle part of url
                 2. add
                    1. change formula type ➔ change to a new CONTEXT VARIABLE formula…
                    2. edit formula
                       1. name/description: url
                       2. variable name: url
                       3. no string processing.
                       4. apply
                    3. apply
                 3. substitution phrase: /meta?url=
                 4. apply
              3. apply
            • Context variable (middle part of url)
              1. change formula type ➔ change to a new CONTEXT VARIABLE formula…
              2. edit formula
                 1. name/description: encoded url
                 2. variable name: url
                 3. Click no string processing.
                    1. Click add
                       1. Select String Converter
                       2. example string: https://www.pornhub.com/view_video.php?viewkey=ph63348bf2f3330
                       3. Click add
                          1. conversion type: encode
                          2. encoding type: url percent encoding
                          3. example string: https://www.pornhub.com/view_video.php?viewkey=ph63348bf2f3330
                          4. converted string: automatically generated: https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
                          5. apply
                       4. apply
                    2. apply
                 4. apply
              3. apply
              • fixed text: formats
            2. dictionary entry by key:
               • match type: fixed characters
               • fixed text: best
            3. dictionary entry by key:
               • match type: fixed characters
               • fixed text: url
         3. substitution phrase: \1\2\3
         4. no string processing.
         5. apply
      6. apply
   6. set up parsers: add
      1. name or description(optional): URL: just the given url for association
      2. content type: urls
      3. url type: POST parsers only: url to associate (source url)
      4. url quality precedence (higher is better): 100
      5. edit formula (type CONTEXT VARIABLE, otherwise click change formula type before)
         1. name/description: Original URL for association
         2. variable name: url
         3. apply
         4. apply
      6. apply
   7. apply
5. Go to network ➔ downloader components ➔ manage url classes
   1. and click on add to add a new one with the following settings:
      1. name: HydrusYTDLProxy (meta)
      2. url type: post url
      3. tab march rules
         1. preferred scheme: http (or https if you have ssl configured)
         2. network location: <YOUR_SERVER_HOST_OR_IP>:4458
         3. path components: none
         4. parameters ➔ add
            1. name: url
            2. name, %-encoded: url
            3. match type: regex
            4. regex: https?(:|%3A)(/|%2F)(/|%2F).+
            5. minimum allowed number of characters: ☑️ no limit
            6. maximum allowed number of characters: ☑️ no limit
            7. example string: https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
            8. default value: ☑️ none
            9. default value, %-encoded: ☑️ none
            10. default string processor: no string processing.
            11. apply
      4. tab options
         1. if matching by subdomain, keep it when normalising: Disabled. No ◻️
         2. alphabetise GET parameters when normalising: Yes ☑️
         3. disallow match on any extra path components: No ◻️
         4. disallow match on any extra parameters: No ◻️
         5. keep extra parameters for server: Yes ☑️
         6. keep fragment when normalising: No ◻️
         7. post page can produce multiple files: No ◻️
         8. associate a 'known url' with resulting files: Yes ☑️
         9. optional api/redirect url conversion: no string conversion.
         10. api/redirect url: none set.
         11. send referral url: send referral url if available
      5. example url: http://<YOUR_SERVER_HOST_OR_IP>:4458/meta?url=https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
      6. apply
   2. and click on add to add a new one with the following settings:
      1. name: HydrusYTDLProxy (meta as gallery)
      2. url type: gallery url
      3. tab march rules 4. preferred scheme: http (or https if you have ssl configured) 5. network location: <YOUR_SERVER_HOST_OR_IP>:4458 6. path components: none 7. parameters ➔ add 2. name: url 3. name, %-encoded: url 4. match type: regex 5. regex: https?(:|%3A)(/|%2F)(/|%2F).+ 6. minimum allowed number of characters: ☑️ no limit 7. maximum allowed number of characters: ☑️ no limit 8. example string: https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330 9. default value: ☑️ none 9. default value, %-encoded: ☑️ none 10. default string processor: no string processing. 11. apply
      4. tab options
         1. if matching by subdomain, keep it when normalising: Disabled. No ◻️
         2. alphabetise GET parameters when normalising: Yes ☑️
         3. disallow match on any extra path components: No ◻️
         4. disallow match on any extra parameters: No ◻️
         5. keep extra parameters for server: Yes ☑️
         6. keep fragment when normalising: No ◻️
         7. post page can produce multiple files: Disabled. No ◻️
         8. associate a 'known url' with resulting files: Yes ☑️
         9. optional api/redirect url conversion: no string conversion.
         10. api/redirect url: none set.
         11. send referral url: send referral url if available
      5. example url: http://<YOUR_SERVER_HOST_OR_IP>:4458/meta?url=https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
      6. apply
   3. and click on add to add a new one with the following settings:
      1. name: HydrusYTDLProxy (dl)
      2. url type: file url
      3. tab march rules
         1. preferred scheme: http (or https if you have ssl configured)
         2. network location: <YOUR_SERVER_HOST_OR_IP>:4458
         3. path components ➔ add
            1. match type: fixed characters
            2. fixed text: dl
            3. apply
         4. parameters ➔ add
            1. name: format
            2. name, %-encoded: format
            3. match type: character set
            4. match type: alphanumeric characters (a-zA-Z0-9)
            5. regex: https?(:|%3A)(/|%2F)(/|%2F).+
            6. minimum allowed number of characters: ☑️ no limit
            7. maximum allowed number of characters: ☑️ no limit
            8. example string: value
            9. default value: ☑️ none
            10. default value, %-encoded: ☑️ none
            11. default string processor: no string processing.
            12. apply
         5. parameters ➔ add
            1. name: url
            2. name, %-encoded: url
            3. match type: regex
            4. regex: https?(:|%3A)(/|%2F)(/|%2F).+
            5. minimum allowed number of characters: ☑️ no limit
            6. maximum allowed number of characters: ☑️ no limit
            7. example string: https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330
            8. default value: ☑️ none
            9. default value, %-encoded: ☑️ none
            10. default string processor: no string processing.
            11. apply
      4. tab options
         1. if matching by subdomain, keep it when normalising: Disabled. No ◻️
         2. alphabetise GET parameters when normalising: Yes ☑️
         3. disallow match on any extra path components: No ◻️
         4. disallow match on any extra parameters: No ◻️
         5. keep extra parameters for server: Yes ☑️
         6. keep fragment when normalising: No ◻️
         7. post page can produce multiple files: Disabled. No ◻️
         8. associate a 'known url' with resulting files: Yes ☑️
         9. optional api/redirect url conversion: no string conversion.
         10. api/redirect url: none set.
         11. send referral url: send referral url if available
      5. example url: http://<YOUR_SERVER_HOST_OR_IP>:4458/dl?url=https%3A%2F%2Fwww.pornhub.com%2Fview_video.php%3Fviewkey%3Dph63348bf2f3330&format=best
      6. apply
6. Now you can use the new url generator in Hydrus Network:
   1. Open a new page ➔ downloader ➔ gallery
   2. Select `HydrusYTDLProxy url from the dropdown menu.
   3. Paste any YT-DLP / Youtube-dl supported URL into the search box.