Fix fallback for comments

This reverts commit 95c6747a3e.

CHANGELOG.md

@@ -1,3 +1,54 @@
+# 0.9.0 (2018-10-08)
+
+## Week 9: Playlists
+
+Not as much to announce this week, but I'm still quite happy to announce a couple things, namely:
+
+Playback support for playlists has finally been added with [`88430a6`](https://github.com/omarroth/invidious/88430a6). You can now view playlists with the `&list=` query param, as you would on YouTube. You can also view mixes with the mentioned `&list=`, although they require some extra handling that I would like to add in the coming week, as well as adding playlist looping and shuffle. I think playback support has been a roadblock for more exciting features such as [#114](https://github.com/omarroth/invidious/issues/114), and I look forward to improving the experience.
+
+Comments have had a bit of a cosmetic upgrade with [#132](https://github.com/omarroth/invidious/issues/132), which I think helps better distinguish between Reddit and YouTube comments, as it makes them appear similarly to their respective sites. You can also now switch between YouTube and Reddit comments with a push of a button, which I think is quite an improvement, especially for newer or less popular videos with fewer comments.
+
+I've had a small breakthrough in speeding up users' subscription feeds with PostgreSQL's [materialized views](https://www.postgresql.org/docs/current/static/rules-materializedviews.html). Without going into too much detail, materialized views essentially cache the result of a query, making it possible to run resource-intensive queries once, rather than every time a user visits their feed. In the coming week I hope to push this out to users, and hopefully close [#173](https://github.com/omarroth/invidious/issues/173).
+
+I haven't had as much time to work on the project this week, but I'm quite happy to have added some new features. Have a great week everyone.
+
+# 0.8.0 (2018-10-02)
+
+## Week 8: Mixes
+
+Hello again!
+
+Mixes have been added with [`20130db`](https://github.com/omarroth/invidious/20130db), which makes it easy to create a playlist of related content. See [#188](https://github.com/omarroth/invidious/issues/188) for more info on how they work. Currently, they return the first 50 videos rather than a continuous feed to avoid tracking by Google/YouTube, which I think is a good trade-off between usability and privacy, and I hope other folks agree. You can create mixes by adding `RD` to the beginning of a video ID, an example is provided [here](https://www.invidio.us/mix?list=RDYE7VzlLtp-4) based on Big Buck Bunny. I've been quite happy with the results returned for the mixes I've tried, and it is not limited to music, which I think is a big plus. To emulate a continuous feed provided many are used to, using the last video of each mix as a new 'seed' has worked well for me. In the coming week I'd like to to add playback support in the player to listen to these easily.
+
+A very big thanks to [**@flourgaz**](https://github.com/flourgaz) for Docker support with [#186](https://github.com/omarroth/invidious/pull/186). This is an enormous improvement in portability for the project, and opens the door for Heroku support (see [#162](https://github.com/omarroth/invidious/issues/162)), and seamless support on Windows. For most users, it should be as easy as running `docker-compose up`.
+
+I've spent quite a bit of time this past week improving support for geo-bypass (see [#92](https://github.com/omarroth/invidious/issues/92)), and am happy to note that Invidious has been able to proxy ~50% of the geo-restricted videos I've tried. In addition, you can now watch geo-restricted videos if you have `dash` enabled as your `preferred quality`, for more details see [#34](https://github.com/omarroth/invidious/issues/34) and [#185](https://github.com/omarroth/invidious/issues/185), or last week's update. For folks interested in replicating these results for themselves, I'd take a look [here](https://gist.github.com/omarroth/3ce0f276c43e0c4b13e7d9cd35524688) for the script used, and [here](https://gist.github.com/omarroth/beffc4a76a7b82a422e1b36a571878ef) for a list of videos restricted in the US.
+
+1080p has seen a fairly smooth roll-out, although there have been a couple issues reported, mainly [#193](https://github.com/omarroth/invidious/issues/193), which is likely an issue in the player. I've also encountered a couple other issues myself that I would like to investigate. Although none are major, I'd like to keep 1080p opt-in for registered users another week to better address these issues.
+
+Have an excellent week everyone.
+
+# 0.7.0 (2018-09-25)
+
+## Week 7: 1080p and Search Types
+
+Hello again everyone! I've got quite a couple announcements this week:
+
+Experimental 1080p support has been added with [`b3ca392`](https://github.com/omarroth/invidious/b3ca392), and can be enabled by going to preferences and changing `preferred video quality` to `dash`. You can find more details [here](https://github.com/omarroth/invidious/issues/34#issuecomment-424171888). Currently quality and speed controls have not yet been integrated into the player, but I'd still appreciate feedback, mainly on any issues with buffering or DASH playback. I hope to integrate 1080p support into the player and push support site-wide in the coming weeks.
+
+You can now filter content types in search with the `type:TYPE` filter. Supported content types are `playlist`, `channel`, and `video`. More info is available [here](https://github.com/omarroth/invidious/issues/126#issuecomment-423823148). I think this is quite an improvement in usability and I hope others find the same.
+
+A [CHANGELOG](https://github.com/omarroth/invidious/blob/master/CHANGELOG.md) has been added to the repository, so folks will now receive a copy of all these updates when cloning. I think this is an improvement in hosting the project, as it is no longer tied to the `/releases` tab on Github or the posts on Patreon.
+
+Recently, users have been reporting 504s when attempting to access their subscriptions, which is tracked in [#173](https://github.com/omarroth/invidious/issues/173). This is most likely caused by an uptick in usage, which I am absolutely grateful for, but unfortunately has resulted in an increase in costs for hosting the site, which is why I will be bumping my goal on Patreon from $60 to $80. I would appreciate any feedback on how subscriptions could be improved.
+
+Other minor improvements include:
+- Additional regions added to bypass geo-block with [`9a78523`](https://github.com/omarroth/invidious/9a78523)
+- Fix for playlists containing less than 100 videos (previously shown as empty) with [`35ac887`](https://github.com/omarroth/invidious/35ac887)
+- Fix for `published` date for Reddit comments (previously showing negative seconds) with [`6e09202`](https://github.com/omarroth/invidious/6e09202)
+
+Thank you everyone for your support!
+
 # 0.6.0 (2018-09-18)
 
 ## Week 6: Filters and Thumbnails

README.md

@@ -28,6 +28,29 @@ BCH: qq4ptclkzej5eza6a50et5ggc58hxsq5aylqut2npk
 
 ## Installation
 
+### Docker:
+
+#### Build and start cluster:
+
+```bash
+$ docker-compose up
+```
+
+And visit `localhost:3000` in your browser.
+
+#### Rebuild cluster:
+
+```bash
+$ docker-compose build
+```
+
+#### Delete data and rebuild:
+
+```bash
+$ docker volume rm invidious_postgresdata
+$ docker-compose build
+```
+
 ### Installing [Crystal](https://github.com/crystal-lang/crystal):
 
 #### On Arch:
@@ -74,8 +97,21 @@ $ sudo pacman -S imagemagick librsvg
 ## Usage:
 
 ```bash
-$ crystal build src/invidious.cr
-$ ./invidious
+$ crystal build src/invidious.cr --release
+$ ./invidious -h
+Usage: invidious [arguments]
+    -b HOST, --bind HOST             Host to bind (defaults to 0.0.0.0)
+    -p PORT, --port PORT             Port to listen for connections (defaults to 3000)
+    -s, --ssl                        Enables SSL
+    --ssl-key-file FILE              SSL key file
+    --ssl-cert-file FILE             SSL certificate file
+    -h, --help                       Shows this help
+    -t THREADS, --crawl-threads=THREADS
+                                     Number of threads for crawling (default: 1)
+    -c THREADS, --channel-threads=THREADS
+                                     Number of threads for refreshing channels (default: 1)
+    -v THREADS, --video-threads=THREADS
+                                     Number of threads for refreshing videos (default: 1)
 ```
 
 Or for development:

assets/css/default.css

@@ -17,6 +17,15 @@ div {
   animation: spin 2s linear infinite;
 }
 
+.playlist-restricted {
+  height: 20em;
+  padding-right: 10px;
+}
+
+.pure-button-primary {
+  background: rgba(0, 182, 240, 1);
+}
+
 /*
  * Navbar
  */

assets/js/watch.js

@@ -0,0 +1,59 @@
+function toggle_parent(target) {
+  body = target.parentNode.parentNode.children[1];
+  if (body.style.display === null || body.style.display === "") {
+    target.innerHTML = "[ + ]";
+    body.style.display = "none";
+  } else {
+    target.innerHTML = "[ - ]";
+    body.style.display = "";
+  }
+}
+
+function toggle_comments(target) {
+  body = target.parentNode.parentNode.parentNode.children[1];
+  if (body.style.display === null || body.style.display === "") {
+    target.innerHTML = "[ + ]";
+    body.style.display = "none";
+  } else {
+    target.innerHTML = "[ - ]";
+    body.style.display = "";
+  }
+}
+
+function swap_comments(source) {
+  if (source == "youtube") {
+    get_youtube_comments();
+  } else if (source == "reddit") {
+    get_reddit_comments();
+  }
+}
+
+function commaSeparateNumber(val) {
+  while (/(\d+)(\d{3})/.test(val.toString())) {
+    val = val.toString().replace(/(\d+)(\d{3})/, "$1" + "," + "$2");
+  }
+  return val;
+}
+
+String.prototype.supplant = function(o) {
+  return this.replace(/{([^{}]*)}/g, function(a, b) {
+    var r = o[b];
+    return typeof r === "string" || typeof r === "number" ? r : a;
+  });
+};
+
+function show_youtube_replies(target) {
+  body = target.parentNode.parentNode.children[1];
+  body.style.display = "";
+
+  target.innerHTML = "Hide replies";
+  target.setAttribute("onclick", "hide_youtube_replies(this)");
+}
+
+function hide_youtube_replies(target) {
+  body = target.parentNode.parentNode.children[1];
+  body.style.display = "none";
+
+  target.innerHTML = "Show replies";
+  target.setAttribute("onclick", "show_youtube_replies(this)");
+}

config/config.yml

@@ -1,5 +1,6 @@
 crawl_threads: 1
 channel_threads: 1
+feed_threads: 1
 video_threads: 1
 db:
   user: kemal
@@ -9,3 +10,4 @@ db:
   dbname: invidious
 full_refresh: false
 https_only: false
+geo_bypass: true

config/sql/videos.sql

@@ -22,6 +22,8 @@ CREATE TABLE public.videos
     genre text COLLATE pg_catalog."default",
     genre_url text COLLATE pg_catalog."default",
     license text COLLATE pg_catalog."default",
+    sub_count_text text COLLATE pg_catalog."default",
+    author_thumbnail text COLLATE pg_catalog."default",
     CONSTRAINT videos_pkey PRIMARY KEY (id)
 )
 WITH (

docker-compose.yml

@@ -0,0 +1,21 @@
+version: '3'
+services:
+  postgres:
+    build:
+      context: .
+      dockerfile: docker/Dockerfile.postgres
+    restart: unless-stopped
+    volumes:
+      - postgresdata:/var/lib/postgresql/data
+  invidious:
+    build:
+      context: .
+      dockerfile: docker/Dockerfile
+    restart: unless-stopped
+    ports:
+      - "3000:3000"
+    depends_on:
+      - postgres
+
+volumes:
+  postgresdata:

docker/Dockerfile

@@ -0,0 +1,15 @@
+FROM archlinux/base
+
+RUN pacman -Sy --noconfirm shards crystal imagemagick librsvg \
+    which pkgconf gcc ttf-liberation
+# base-devel contains many other basic packages, that are normally assumed to already exist on a clean arch system
+
+ADD . /invidious
+
+WORKDIR /invidious
+
+RUN sed -i 's/host: localhost/host: postgres/' config/config.yml && \
+    shards && \
+    crystal build src/invidious.cr
+
+CMD [ "/invidious/invidious" ]

docker/Dockerfile.postgres

@@ -0,0 +1,10 @@
+FROM postgres:10
+
+ENV POSTGRES_USER postgres
+
+ADD ./setup.sh /setup.sh
+ADD ./config/sql /config/sql
+ADD ./docker/entrypoint.postgres.sh /entrypoint.sh
+
+ENTRYPOINT [ "/entrypoint.sh" ]
+CMD [ "postgres" ]

docker/entrypoint.postgres.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+CMD="$@"
+if [ ! -f /var/lib/postgresql/data/setupFinished ]; then
+    echo "### first run - setting up invidious database"
+    /usr/local/bin/docker-entrypoint.sh postgres &
+    sleep 10
+    until runuser -l postgres -c 'pg_isready' 2>/dev/null; do
+        >&2 echo "### Postgres is unavailable - waiting"
+        sleep 5
+    done
+    >&2 echo "### importing table schemas"
+    su postgres -c "/setup.sh" && touch /var/lib/postgresql/data/setupFinished
+    echo "### invidious database setup finished"
+    exit
+fi
+
+echo "running postgres /usr/local/bin/docker-entrypoint.sh $CMD"
+exec /usr/local/bin/docker-entrypoint.sh $CMD

setup.sh

@@ -1,7 +1,8 @@
 #!/bin/bash
 
 createdb invidious
-createuser kemal
+#createuser kemal
+psql -c "CREATE USER kemal WITH PASSWORD 'kemal';"
 psql invidious < config/sql/channels.sql
 psql invidious < config/sql/videos.sql
 psql invidious < config/sql/channel_videos.sql

shard.yml

@@ -1,5 +1,5 @@
 name: invidious
-version: 0.7.0
+version: 0.9.0
 
 authors:
   - Omar Roth <omarroth@hotmail.com>

src/invidious.cr

@@ -31,6 +31,7 @@ HMAC_KEY = CONFIG.hmac_key || Random::Secure.random_bytes(32)
 
 crawl_threads = CONFIG.crawl_threads
 channel_threads = CONFIG.channel_threads
+feed_threads = CONFIG.feed_threads
 video_threads = CONFIG.video_threads
 
 Kemal.config.extra_options do |parser|
@@ -51,6 +52,14 @@ Kemal.config.extra_options do |parser|
       exit
     end
   end
+  parser.on("-f THREADS", "--feed-threads=THREADS", "Number of threads for refreshing feeds (default: #{feed_threads})") do |number|
+    begin
+      feed_threads = number.to_i
+    rescue ex
+      puts "THREADS must be integer"
+      exit
+    end
+  end
   parser.on("-v THREADS", "--video-threads=THREADS", "Number of threads for refreshing videos (default: #{video_threads})") do |number|
     begin
       video_threads = number.to_i
@@ -85,6 +94,8 @@ end
 
 refresh_channels(PG_DB, channel_threads, CONFIG.full_refresh)
 
+refresh_feeds(PG_DB, feed_threads)
+
 video_threads.times do |i|
   spawn do
     refresh_videos(PG_DB)
@@ -105,6 +116,17 @@ spawn do
   end
 end
 
+proxies = {} of String => Array({ip: String, port: Int32})
+if CONFIG.geo_bypass
+  spawn do
+    find_working_proxies(BYPASS_REGIONS) do |region, list|
+      if !list.empty?
+        proxies[region] = list
+      end
+    end
+  end
+end
+
 before_all do |env|
   env.response.headers["X-XSS-Protection"] = "1; mode=block;"
   env.response.headers["X-Content-Type-Options"] = "nosniff"
@@ -206,6 +228,8 @@ get "/watch" do |env|
     next env.redirect "/"
   end
 
+  plid = env.params.query["list"]?
+
   user = env.get? "user"
   if user
     user = user.as(User)
@@ -225,7 +249,9 @@ get "/watch" do |env|
   end
 
   begin
-    video = get_video(id, PG_DB)
+    video = get_video(id, PG_DB, proxies)
+  rescue ex : VideoRedirect
+    next env.redirect "/watch?v=#{ex.message}"
   rescue ex
     error_message = ex.message
     STDOUT << id << " : " << ex.message << "\n"
@@ -325,7 +351,9 @@ get "/embed/:id" do |env|
   params = process_video_params(env.params.query, nil)
 
   begin
-    video = get_video(id, PG_DB)
+    video = get_video(id, PG_DB, proxies)
+  rescue ex : VideoRedirect
+    next env.redirect "/embed/#{ex.message}"
   rescue ex
     error_message = ex.message
     next templated "error"
@@ -381,6 +409,7 @@ get "/embed/:id" do |env|
 end
 
 # Playlists
+
 get "/playlist" do |env|
   plid = env.params.query["list"]?
   if !plid
@@ -390,17 +419,45 @@ get "/playlist" do |env|
   page = env.params.query["page"]?.try &.to_i?
   page ||= 1
 
+  if plid.starts_with? "RD"
+    next env.redirect "/mix?list=#{plid}"
+  end
+
   begin
     playlist = fetch_playlist(plid)
-    videos = fetch_playlist_videos(plid, page, playlist.video_count)
   rescue ex
     error_message = ex.message
     next templated "error"
   end
 
+  begin
+    videos = fetch_playlist_videos(plid, page, playlist.video_count)
+  rescue ex
+    videos = [] of PlaylistVideo
+  end
+
   templated "playlist"
 end
 
+get "/mix" do |env|
+  rdid = env.params.query["list"]?
+  if !rdid
+    next env.redirect "/"
+  end
+
+  continuation = env.params.query["continuation"]?
+  continuation ||= rdid.lchop("RD")
+
+  begin
+    mix = fetch_mix(rdid, continuation)
+  rescue ex
+    error_message = ex.message
+    next templated "error"
+  end
+
+  templated "mix"
+end
+
 # Search
 
 get "/results" do |env|
@@ -429,9 +486,8 @@ get "/search" do |env|
   user = env.get? "user"
   if user
     user = user.as(User)
-    ucids = user.subscriptions
+    view_name = "subscriptions_#{sha256(user.email)[0..7]}"
   end
-  ucids ||= [] of String
 
   channel = nil
   content_type = "all"
@@ -468,14 +524,19 @@ get "/search" do |env|
   if channel
     count, videos = channel_search(search_query, page, channel)
   elsif subscriptions
+    if view_name
       videos = PG_DB.query_all("SELECT id,title,published,updated,ucid,author FROM (
       SELECT *,
-      to_tsvector(channel_videos.title) ||
-      to_tsvector(channel_videos.author)
+      to_tsvector(#{view_name}.title) ||
+      to_tsvector(#{view_name}.author)
       as document
-      FROM channel_videos WHERE ucid IN (#{arg_array(ucids, 3)})
-      ) v_search WHERE v_search.document @@ plainto_tsquery($1) LIMIT 20 OFFSET $2;", [search_query, (page - 1) * 20] + ucids, as: ChannelVideo)
+      FROM #{view_name}
+      ) v_search WHERE v_search.document @@ plainto_tsquery($1) LIMIT 20 OFFSET $2;", search_query, (page - 1) * 20, as: ChannelVideo)
       count = videos.size
+    else
+      videos = [] of ChannelVideo
+      count = 0
+    end
   else
     begin
       search_params = produce_search_params(sort: sort, date: date, content_type: content_type,
@@ -709,7 +770,7 @@ post "/login" do |env|
     end
 
     if action == "signin"
-      user = PG_DB.query_one?("SELECT * FROM users WHERE email = $1 AND password IS NOT NULL", email, as: User)
+      user = PG_DB.query_one?("SELECT * FROM users WHERE LOWER(email) = LOWER($1) AND password IS NOT NULL", email, as: User)
 
       if !user
         error_message = "Invalid username or password"
@@ -723,7 +784,7 @@ post "/login" do |env|
 
       if Crypto::Bcrypt::Password.new(user.password.not_nil!) == password
         sid = Base64.urlsafe_encode(Random::Secure.random_bytes(32))
-        PG_DB.exec("UPDATE users SET id = id || $1 WHERE email = $2", [sid], email)
+        PG_DB.exec("UPDATE users SET id = id || $1 WHERE LOWER(email) = LOWER($2)", [sid], email)
 
         if Kemal.config.ssl || CONFIG.https_only
           secure = true
@@ -738,7 +799,7 @@ post "/login" do |env|
         next templated "error"
       end
     elsif action == "register"
-      user = PG_DB.query_one?("SELECT * FROM users WHERE email = $1 AND password IS NOT NULL", email, as: User)
+      user = PG_DB.query_one?("SELECT * FROM users WHERE LOWER(email) = LOWER($1) AND password IS NOT NULL", email, as: User)
       if user
         error_message = "Please sign in"
         next templated "error"
@@ -753,6 +814,12 @@ post "/login" do |env|
 
       PG_DB.exec("INSERT INTO users VALUES (#{args})", user_array)
 
+      view_name = "subscriptions_#{sha256(user.email)[0..7]}"
+      PG_DB.exec("CREATE MATERIALIZED VIEW #{view_name} AS \
+        SELECT * FROM channel_videos WHERE \
+        ucid = ANY ((SELECT subscriptions FROM users WHERE email = '#{user.email}')::text[]) \
+      ORDER BY published DESC;")
+
       if Kemal.config.ssl || CONFIG.https_only
         secure = true
       else
@@ -1079,12 +1146,14 @@ post "/data_control" do |env|
         body = JSON.parse(body)
         body["subscriptions"].as_a.each do |ucid|
           ucid = ucid.as_s
-          if !user.subscriptions.includes? ucid
-            PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE id = $2", ucid, user.id)
 
+          if !user.subscriptions.includes? ucid
             begin
               client = make_client(YT_URL)
               get_channel(ucid, client, PG_DB, false, false)
+
+              PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
+              user.subscriptions << ucid
             rescue ex
               next
             end
@@ -1093,8 +1162,10 @@ post "/data_control" do |env|
 
         body["watch_history"].as_a.each do |id|
           id = id.as_s
+
           if !user.watched.includes? id
             PG_DB.exec("UPDATE users SET watched = array_append(watched,$1) WHERE email = $2", id, user.email)
+            user.watched << id
           end
         end
 
@@ -1105,11 +1176,12 @@ post "/data_control" do |env|
           ucid = channel["xmlUrl"].match(/UC[a-zA-Z0-9_-]{22}/).not_nil![0]
 
           if !user.subscriptions.includes? ucid
-            PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
-
             begin
               client = make_client(YT_URL)
               get_channel(ucid, client, PG_DB, false, false)
+
+              PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
+              user.subscriptions << ucid
             rescue ex
               next
             end
@@ -1120,11 +1192,12 @@ post "/data_control" do |env|
           ucid = md["channel_id"]
 
           if !user.subscriptions.includes? ucid
-            PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
-
             begin
               client = make_client(YT_URL)
               get_channel(ucid, client, PG_DB, false, false)
+
+              PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
+              user.subscriptions << ucid
             rescue ex
               next
             end
@@ -1136,11 +1209,12 @@ post "/data_control" do |env|
           ucid = channel["url"].as_s.match(/UC[a-zA-Z0-9_-]{22}/).not_nil![0]
 
           if !user.subscriptions.includes? ucid
-            PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
-
             begin
               client = make_client(YT_URL)
               get_channel(ucid, client, PG_DB, false, false)
+
+              PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
+              user.subscriptions << ucid
             rescue ex
               next
             end
@@ -1156,19 +1230,24 @@ post "/data_control" do |env|
 
               db = entry.io.gets_to_end
               db.scan(/youtube\.com\/watch\?v\=(?<id>[a-zA-Z0-9_-]{11})/) do |md|
-                if !user.watched.includes? md["id"]
-                  PG_DB.exec("UPDATE users SET watched = array_append(watched,$1) WHERE email = $2", md["id"], user.email)
+                id = md["id"]
+
+                if !user.watched.includes? id
+                  PG_DB.exec("UPDATE users SET watched = array_append(watched,$1) WHERE email = $2", id, user.email)
+                  user.watched << id
                 end
               end
 
               db.scan(/youtube\.com\/channel\/(?<ucid>[a-zA-Z0-9_-]{22})/) do |md|
                 ucid = md["ucid"]
-                if !user.subscriptions.includes? ucid
-                  PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
 
+                if !user.subscriptions.includes? ucid
                   begin
                     client = make_client(YT_URL)
                     get_channel(ucid, client, PG_DB, false, false)
+
+                    PG_DB.exec("UPDATE users SET subscriptions = array_append(subscriptions,$1) WHERE email = $2", ucid, user.email)
+                    user.subscriptions << ucid
                   rescue ex
                     next
                   end
@@ -1306,6 +1385,8 @@ get "/feed/subscriptions" do |env|
 
     notifications = PG_DB.query_one("SELECT notifications FROM users WHERE email = $1", user.email,
       as: Array(String))
+    view_name = "subscriptions_#{sha256(user.email)[0..7]}"
+
     if preferences.notifications_only && !notifications.empty?
       args = arg_array(notifications)
 
@@ -1328,39 +1409,35 @@ get "/feed/subscriptions" do |env|
     else
       if preferences.latest_only
         if preferences.unseen_only
-          ucids = arg_array(user.subscriptions)
           if user.watched.empty?
             watched = "'{}'"
           else
-            watched = arg_array(user.watched, user.subscriptions.size + 1)
+            watched = arg_array(user.watched)
           end
 
-          videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM channel_videos WHERE \
-            ucid IN (#{ucids}) AND id NOT IN (#{watched}) ORDER BY ucid, published DESC",
-            user.subscriptions + user.watched, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM #{view_name} WHERE \
+            id NOT IN (#{watched}) ORDER BY ucid, published DESC",
+            user.watched, as: ChannelVideo)
         else
-          args = arg_array(user.subscriptions)
-          videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM channel_videos WHERE \
-          ucid IN (#{args}) ORDER BY ucid, published DESC", user.subscriptions, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM #{view_name} \
+          ORDER BY ucid, published DESC", as: ChannelVideo)
         end
 
         videos.sort_by! { |video| video.published }.reverse!
       else
         if preferences.unseen_only
-          ucids = arg_array(user.subscriptions, 3)
           if user.watched.empty?
             watched = "'{}'"
           else
-            watched = arg_array(user.watched, user.subscriptions.size + 3)
+            watched = arg_array(user.watched, 3)
           end
 
-          videos = PG_DB.query_all("SELECT * FROM channel_videos WHERE ucid IN (#{ucids}) \
-          AND id NOT IN (#{watched}) ORDER BY published DESC LIMIT $1 OFFSET $2",
-            [limit, offset] + user.subscriptions + user.watched, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT * FROM #{view_name} WHERE \
+          id NOT IN (#{watched}) LIMIT $1 OFFSET $2",
+            [limit, offset] + user.watched, as: ChannelVideo)
         else
-          args = arg_array(user.subscriptions, 3)
-          videos = PG_DB.query_all("SELECT * FROM channel_videos WHERE ucid IN (#{args}) \
-          ORDER BY published DESC LIMIT $1 OFFSET $2", [limit, offset] + user.subscriptions, as: ChannelVideo)
+          videos = PG_DB.query_all("SELECT * FROM #{view_name} \
+          ORDER BY published DESC LIMIT $1 OFFSET $2", limit, offset, as: ChannelVideo)
         end
       end
 
@@ -1409,29 +1486,8 @@ get "/feed/channel/:ucid" do |env|
     halt env, status_code: 404, response: error_message
   end
 
-  client = make_client(YT_URL)
-
   page = 1
-
-  videos = [] of SearchVideo
-  2.times do |i|
-    url = produce_channel_videos_url(ucid, page * 2 + (i - 1), auto_generated: auto_generated)
-    response = client.get(url)
-    json = JSON.parse(response.body)
-
-    if json["content_html"]? && !json["content_html"].as_s.empty?
-      document = XML.parse_html(json["content_html"].as_s)
-      nodeset = document.xpath_nodes(%q(//li[contains(@class, "feed-item-container")]))
-
-      if auto_generated
-        videos += extract_videos(nodeset)
-      else
-        videos += extract_videos(nodeset, ucid)
-      end
-    else
-      break
-    end
-  end
+  videos, count = get_60_videos(ucid, page, auto_generated)
 
   host_url = make_host_url(Kemal.config.ssl || CONFIG.https_only, env.request.headers["Host"]?)
   path = env.request.path
@@ -1518,15 +1574,14 @@ get "/feed/private" do |env|
   latest_only ||= 0
   latest_only = latest_only == 1
 
+  view_name = "subscriptions_#{sha256(user.email)[0..7]}"
+
   if latest_only
-    args = arg_array(user.subscriptions)
-    videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM channel_videos WHERE \
-    ucid IN (#{args}) ORDER BY ucid, published DESC", user.subscriptions, as: ChannelVideo)
+    videos = PG_DB.query_all("SELECT DISTINCT ON (ucid) * FROM #{view_name} ORDER BY ucid, published DESC", as: ChannelVideo)
     videos.sort_by! { |video| video.published }.reverse!
   else
-    args = arg_array(user.subscriptions, 3)
-    videos = PG_DB.query_all("SELECT * FROM channel_videos WHERE ucid IN (#{args}) \
-  ORDER BY published DESC LIMIT $1 OFFSET $2", [limit, offset] + user.subscriptions, as: ChannelVideo)
+    videos = PG_DB.query_all("SELECT * FROM #{view_name} \
+    ORDER BY published DESC LIMIT $1 OFFSET $2", limit, offset, as: ChannelVideo)
   end
 
   sort = env.params.query["sort"]?
@@ -1663,7 +1718,7 @@ get "/channel/:ucid" do |env|
   page ||= 1
 
   begin
-    author, ucid, auto_generated = get_about_info(ucid)
+    author, ucid, auto_generated, sub_count = get_about_info(ucid)
   rescue ex
     error_message = "User does not exist"
     next templated "error"
@@ -1677,27 +1732,7 @@ get "/channel/:ucid" do |env|
     end
   end
 
-  client = make_client(YT_URL)
-
-  videos = [] of SearchVideo
-  2.times do |i|
-    url = produce_channel_videos_url(ucid, page * 2 + (i - 1), auto_generated: auto_generated)
-    response = client.get(url)
-    json = JSON.parse(response.body)
-
-    if json["content_html"]? && !json["content_html"].as_s.empty?
-      document = XML.parse_html(json["content_html"].as_s)
-      nodeset = document.xpath_nodes(%q(//li[contains(@class, "feed-item-container")]))
-
-      if auto_generated
-        videos += extract_videos(nodeset)
-      else
-        videos += extract_videos(nodeset, ucid)
-      end
-    else
-      break
-    end
-  end
+  videos, count = get_60_videos(ucid, page, auto_generated)
 
   templated "channel"
 end
@@ -1718,11 +1753,15 @@ end
 # API Endpoints
 
 get "/api/v1/captions/:id" do |env|
+  env.response.content_type = "application/json"
+
   id = env.params.url["id"]
 
   client = make_client(YT_URL)
   begin
-    video = get_video(id, PG_DB)
+    video = get_video(id, PG_DB, proxies)
+  rescue ex : VideoRedirect
+    next env.redirect "/api/v1/captions/#{ex.message}"
   rescue ex
     halt env, status_code: 403
   end
@@ -1730,9 +1769,10 @@ get "/api/v1/captions/:id" do |env|
   captions = video.captions
 
   label = env.params.query["label"]?
-  if !label
-    env.response.content_type = "application/json"
+  lang = env.params.query["lang"]?
+  tlang = env.params.query["tlang"]?
 
+  if !label && !lang
     response = JSON.build do |json|
       json.object do
         json.field "captions" do
@@ -1752,22 +1792,27 @@ get "/api/v1/captions/:id" do |env|
     next response
   end
 
+  env.response.content_type = "text/vtt"
+
   caption = captions.select { |caption| caption.name.simpleText == label }
 
-  env.response.content_type = "text/vtt"
+  if lang
+    caption = captions.select { |caption| caption.languageCode == lang }
+  end
+
   if caption.empty?
-    halt env, status_code: 403
+    halt env, status_code: 404
   else
     caption = caption[0]
   end
 
-  caption_xml = client.get(caption.baseUrl).body
+  caption_xml = client.get(caption.baseUrl + "&tlang=#{tlang}").body
   caption_xml = XML.parse(caption_xml)
 
   webvtt = <<-END_VTT
   WEBVTT
   Kind: captions
-  Language: #{caption.languageCode}
+  Language: #{tlang || caption.languageCode}
 
 
   END_VTT
@@ -1806,6 +1851,8 @@ get "/api/v1/captions/:id" do |env|
 end
 
 get "/api/v1/comments/:id" do |env|
+  env.response.content_type = "application/json"
+
   id = env.params.url["id"]
 
   source = env.params.query["source"]?
@@ -1816,26 +1863,66 @@ get "/api/v1/comments/:id" do |env|
 
   if source == "youtube"
     client = make_client(YT_URL)
+    html = client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1")
     headers = HTTP::Headers.new
-    html = client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&disable_polymer=1")
-
     headers["cookie"] = html.cookies.add_request_headers(headers)["cookie"]
-    headers["content-type"] = "application/x-www-form-urlencoded"
-
-    headers["x-client-data"] = "CIi2yQEIpbbJAQipncoBCNedygEIqKPKAQ=="
-    headers["x-spf-previous"] = "https://www.youtube.com/watch?v=#{id}"
-    headers["x-spf-referer"] = "https://www.youtube.com/watch?v=#{id}"
-
-    headers["x-youtube-client-name"] = "1"
-    headers["x-youtube-client-version"] = "2.20180719"
-
     body = html.body
+
     session_token = body.match(/'XSRF_TOKEN': "(?<session_token>[A-Za-z0-9\_\-\=]+)"/).not_nil!["session_token"]
+    itct = body.match(/itct=(?<itct>[^"]+)"/).not_nil!["itct"]
     ctoken = body.match(/'COMMENTS_TOKEN': "(?<ctoken>[^"]+)"/)
 
-    if !ctoken
-      env.response.content_type = "application/json"
+    if body.match(/<meta itemprop="regionsAllowed" content="">/)
+      bypass_channel = Channel({String, HTTPClient, HTTP::Headers} | Nil).new
 
+      proxies.each do |region, list|
+        spawn do
+          proxy_html = %(<meta itemprop="regionsAllowed" content="">)
+
+          list.each do |proxy|
+            begin
+              proxy_client = HTTPClient.new(YT_URL)
+              proxy_client.read_timeout = 10.seconds
+              proxy_client.connect_timeout = 10.seconds
+
+              proxy = HTTPProxy.new(proxy_host: proxy[:ip], proxy_port: proxy[:port])
+              proxy_client.set_proxy(proxy)
+
+              response = proxy_client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1")
+              proxy_headers = HTTP::Headers.new
+              proxy_headers["cookie"] = response.cookies.add_request_headers(headers)["cookie"]
+              proxy_html = response.body
+
+              if !proxy_html.match(/<meta itemprop="regionsAllowed" content="">/)
+                bypass_channel.send({proxy_html, proxy_client, proxy_headers})
+                break
+              end
+            rescue ex
+            end
+          end
+
+          # If none of the proxies we tried returned a valid response
+          if proxy_html.match(/<meta itemprop="regionsAllowed" content="">/)
+            bypass_channel.send(nil)
+          end
+        end
+      end
+
+      proxies.size.times do
+        response = bypass_channel.receive
+        if response
+          session_token = response[0].match(/'XSRF_TOKEN': "(?<session_token>[A-Za-z0-9\_\-\=]+)"/).not_nil!["session_token"]
+          itct = response[0].match(/itct=(?<itct>[^"]+)"/).not_nil!["itct"]
+          ctoken = response[0].match(/'COMMENTS_TOKEN': "(?<ctoken>[^"]+)"/)
+
+          client = response[1]
+          headers = response[2]
+          break
+        end
+      end
+    end
+
+    if !ctoken
       if format == "json"
         next {"comments" => [] of String}.to_json
       else
@@ -1843,7 +1930,6 @@ get "/api/v1/comments/:id" do |env|
       end
     end
     ctoken = ctoken["ctoken"]
-    itct = body.match(/itct=(?<itct>[^"]+)"/).not_nil!["itct"]
 
     if env.params.query["continuation"]? && !env.params.query["continuation"].empty?
       continuation = env.params.query["continuation"]
@@ -1857,10 +1943,16 @@ get "/api/v1/comments/:id" do |env|
     }
     post_req = HTTP::Params.encode(post_req)
 
-    response = client.post("/comment_service_ajax?action_get_comments=1&pbj=1&ctoken=#{ctoken}&continuation=#{continuation}&itct=#{itct}", headers, post_req).body
-    response = JSON.parse(response)
+    headers["content-type"] = "application/x-www-form-urlencoded"
 
-    env.response.content_type = "application/json"
+    headers["x-client-data"] = "CIi2yQEIpbbJAQipncoBCNedygEIqKPKAQ=="
+    headers["x-spf-previous"] = "https://www.youtube.com/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1"
+    headers["x-spf-referer"] = "https://www.youtube.com/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1"
+
+    headers["x-youtube-client-name"] = "1"
+    headers["x-youtube-client-version"] = "2.20180719"
+    response = client.post("/comment_service_ajax?action_get_comments=1&pbj=1&ctoken=#{ctoken}&continuation=#{continuation}&itct=#{itct}&hl=en&gl=US", headers, post_req)
+    response = JSON.parse(response.body)
 
     if !response["response"]["continuationContents"]?
       halt env, status_code: 403
@@ -2016,8 +2108,6 @@ get "/api/v1/comments/:id" do |env|
       halt env, status_code: 404
     end
 
-    env.response.content_type = "application/json"
-
     if format == "json"
       reddit_thread = JSON.parse(reddit_thread.to_json).as_h
       reddit_thread["comments"] = JSON.parse(comments.to_json)
@@ -2038,7 +2128,7 @@ get "/api/v1/insights/:id" do |env|
 
   client = make_client(YT_URL)
   headers = HTTP::Headers.new
-  html = client.get("/watch?v=#{id}&disable_polymer=1")
+  html = client.get("/watch?v=#{id}&gl=US&hl=en&disable_polymer=1")
 
   headers["cookie"] = html.cookies.add_request_headers(headers)["cookie"]
   headers["content-type"] = "application/x-www-form-urlencoded"
@@ -2113,12 +2203,15 @@ get "/api/v1/insights/:id" do |env|
 end
 
 get "/api/v1/videos/:id" do |env|
+  env.response.content_type = "application/json"
+
   id = env.params.url["id"]
 
   begin
-    video = get_video(id, PG_DB)
+    video = get_video(id, PG_DB, proxies)
+  rescue ex : VideoRedirect
+    next env.redirect "/api/v1/videos/#{ex.message}"
   rescue ex
-    env.response.content_type = "application/json"
     error_message = {"error" => ex.message}.to_json
     halt env, status_code: 500, response: error_message
   end
@@ -2128,7 +2221,6 @@ get "/api/v1/videos/:id" do |env|
 
   captions = video.captions
 
-  env.response.content_type = "application/json"
   video_info = JSON.build do |json|
     json.object do
       json.field "title", video.title
@@ -2162,6 +2254,22 @@ get "/api/v1/videos/:id" do |env|
       json.field "authorId", video.ucid
       json.field "authorUrl", "/channel/#{video.ucid}"
 
+      json.field "authorThumbnails" do
+        json.array do
+          qualities = [32, 48, 76, 100, 176, 512]
+
+          qualities.each do |quality|
+            json.object do
+              json.field "url", video.author_thumbnail.gsub("=s48-", "=s#{quality}-")
+              json.field "width", quality
+              json.field "height", quality
+            end
+          end
+        end
+      end
+
+      json.field "subCountText", video.sub_count_text
+
       json.field "lengthSeconds", video.info["length_seconds"].to_i
       if video.info["allow_ratings"]?
         json.field "allowRatings", video.info["allow_ratings"] == "1"
@@ -2380,30 +2488,10 @@ get "/api/v1/channels/:ucid" do |env|
     halt env, status_code: 404, response: error_message
   end
 
-  client = make_client(YT_URL)
-
   page = 1
+  videos, count = get_60_videos(ucid, page, auto_generated)
 
-  videos = [] of SearchVideo
-  2.times do |i|
-    url = produce_channel_videos_url(ucid, page * 2 + (i - 1), auto_generated: auto_generated)
-    response = client.get(url)
-    json = JSON.parse(response.body)
-
-    if json["content_html"]? && !json["content_html"].as_s.empty?
-      document = XML.parse_html(json["content_html"].as_s)
-      nodeset = document.xpath_nodes(%q(//li[contains(@class, "feed-item-container")]))
-
-      if auto_generated
-        videos += extract_videos(nodeset)
-      else
-        videos += extract_videos(nodeset, ucid)
-      end
-    else
-      break
-    end
-  end
-
+  client = make_client(YT_URL)
   channel_html = client.get("/channel/#{ucid}/about?disable_polymer=1").body
   channel_html = XML.parse_html(channel_html)
   banner = channel_html.xpath_node(%q(//div[@id="gh-banner"]/style)).not_nil!.content
@@ -2539,27 +2627,7 @@ end
       halt env, status_code: 404, response: error_message
     end
 
-    client = make_client(YT_URL)
-
-    videos = [] of SearchVideo
-    2.times do |i|
-      url = produce_channel_videos_url(ucid, page * 2 + (i - 1), auto_generated: auto_generated)
-      response = client.get(url)
-      json = JSON.parse(response.body)
-
-      if json["content_html"]? && !json["content_html"].as_s.empty?
-        document = XML.parse_html(json["content_html"].as_s)
-        nodeset = document.xpath_nodes(%q(//li[contains(@class, "feed-item-container")]))
-
-        if auto_generated
-          videos += extract_videos(nodeset)
-        else
-          videos += extract_videos(nodeset, ucid)
-        end
-      else
-        break
-      end
-    end
+    videos, count = get_60_videos(ucid, page, auto_generated)
 
     result = JSON.build do |json|
       json.array do
@@ -2822,14 +2890,28 @@ get "/api/v1/playlists/:plid" do |env|
   page = env.params.query["page"]?.try &.to_i?
   page ||= 1
 
+  format = env.params.query["format"]?
+  format ||= "json"
+
+  continuation = env.params.query["continuation"]?
+
+  if plid.starts_with? "RD"
+    next env.redirect "/api/v1/mixes/#{plid}"
+  end
+
   begin
     playlist = fetch_playlist(plid)
-    videos = fetch_playlist_videos(plid, page, playlist.video_count)
   rescue ex
     error_message = {"error" => "Playlist is empty"}.to_json
     halt env, status_code: 404, response: error_message
   end
 
+  begin
+    videos = fetch_playlist_videos(plid, page, playlist.video_count, continuation)
+  rescue ex
+    videos = [] of PlaylistVideo
+  end
+
   response = JSON.build do |json|
     json.object do
       json.field "title", playlist.title
@@ -2839,6 +2921,20 @@ get "/api/v1/playlists/:plid" do |env|
       json.field "authorId", playlist.ucid
       json.field "authorUrl", "/channel/#{playlist.ucid}"
 
+      json.field "authorThumbnails" do
+        json.array do
+          qualities = [32, 48, 76, 100, 176, 512]
+
+          qualities.each do |quality|
+            json.object do
+              json.field "url", playlist.author_thumbnail.gsub("=s100-", "=s#{quality}-")
+              json.field "width", quality
+              json.field "height", quality
+            end
+          end
+        end
+      end
+
       json.field "description", playlist.description
       json.field "descriptionHtml", playlist.description_html
       json.field "videoCount", playlist.video_count
@@ -2870,6 +2966,80 @@ get "/api/v1/playlists/:plid" do |env|
     end
   end
 
+  if format == "html"
+    response = JSON.parse(response)
+    playlist_html = template_playlist(response)
+    next_video = response["videos"].as_a[1]?.try &.["videoId"]
+
+    response = {
+      "playlistHtml" => playlist_html,
+      "nextVideo"    => next_video,
+    }.to_json
+  end
+
+  response
+end
+
+get "/api/v1/mixes/:rdid" do |env|
+  env.response.content_type = "application/json"
+
+  rdid = env.params.url["rdid"]
+
+  continuation = env.params.query["continuation"]?
+  continuation ||= rdid.lchop("RD")
+
+  format = env.params.query["format"]?
+  format ||= "json"
+
+  begin
+    mix = fetch_mix(rdid, continuation)
+  rescue ex
+    error_message = {"error" => ex.message}.to_json
+    halt env, status_code: 500, response: error_message
+  end
+
+  response = JSON.build do |json|
+    json.object do
+      json.field "title", mix.title
+      json.field "mixId", mix.id
+
+      json.field "videos" do
+        json.array do
+          mix.videos.each do |video|
+            json.object do
+              json.field "title", video.title
+              json.field "videoId", video.id
+              json.field "author", video.author
+
+              json.field "authorId", video.ucid
+              json.field "authorUrl", "/channel/#{video.ucid}"
+
+              json.field "videoThumbnails" do
+                json.array do
+                  generate_thumbnails(json, video.id)
+                end
+              end
+
+              json.field "index", video.index
+              json.field "lengthSeconds", video.length_seconds
+            end
+          end
+        end
+      end
+    end
+  end
+
+  if format == "html"
+    response = JSON.parse(response)
+    playlist_html = template_mix(response)
+    next_video = response["videos"].as_a[1]?.try &.["videoId"]
+
+    response = {
+      "playlistHtml" => playlist_html,
+      "nextVideo"    => next_video,
+    }.to_json
+  end
+
   response
 end
 
@@ -2892,7 +3062,9 @@ get "/api/manifest/dash/id/:id" do |env|
 
   client = make_client(YT_URL)
   begin
-    video = get_video(id, PG_DB)
+    video = get_video(id, PG_DB, proxies)
+  rescue ex : VideoRedirect
+    next env.redirect "/api/manifest/dash/id/#{ex.message}"
   rescue ex
     halt env, status_code: 403
   end
@@ -3078,8 +3250,40 @@ get "/videoplayback" do |env|
   host = "https://r#{fvip}---#{mn}.googlevideo.com"
   url = "/videoplayback?#{query_params.to_s}"
 
+  if query_params["region"]?
+    client = make_client(URI.parse(host))
+    response = HTTP::Client::Response.new(status_code: 403)
+
+    if !proxies[query_params["region"]]?
+      halt env, status_code: 403
+    end
+
+    proxies[query_params["region"]].each do |proxy|
+      begin
+        client = HTTPClient.new(URI.parse(host))
+        client.read_timeout = 10.seconds
+        client.connect_timeout = 10.seconds
+
+        proxy = HTTPProxy.new(proxy_host: proxy[:ip], proxy_port: proxy[:port])
+        client.set_proxy(proxy)
+
+        response = client.head(url)
+        if response.status_code == 200
+          # For whatever reason the proxy needs to be set again
+          client.set_proxy(proxy)
+          break
+        end
+      rescue ex
+      end
+    end
+  else
     client = make_client(URI.parse(host))
     response = client.head(url)
+  end
+
+  if response.status_code != 200
+    halt env, status_code: 403
+  end
 
   if response.headers["Location"]?
     url = URI.parse(response.headers["Location"])
@@ -3224,6 +3428,24 @@ get "/vi/:id/:name" do |env|
 end
 
 error 404 do |env|
+  if md = env.request.path.match(/^\/(?<id>[a-zA-Z0-9_-]{11})/)
+    id = md["id"]
+
+    params = [] of String
+    env.params.query.each do |k, v|
+      params << "#{k}=#{v}"
+    end
+    params = params.join("&")
+
+    url = "/watch?v=#{id}"
+    if !params.empty?
+      url += "&#{params}"
+    end
+
+    env.response.headers["Location"] = url
+    halt env, status_code: 302
+  end
+
   error_message = "404 Page not found"
   templated "error"
 end

src/invidious/channels.cr

@@ -104,6 +104,9 @@ def fetch_channel(ucid, client, db, pull_all_videos = true)
 
       videos.each do |video|
         ids << video.id
+
+        # FIXME: Red videos don't provide published date, so the best we can do is ignore them
+        if Time.now - video.published > 1.minute
           db.exec("UPDATE users SET notifications = notifications || $1 \
           WHERE updated < $2 AND $3 = ANY(subscriptions) AND $1 <> ALL(notifications)", video.id, video.published, video.ucid)
 
@@ -112,6 +115,7 @@ def fetch_channel(ucid, client, db, pull_all_videos = true)
           db.exec("INSERT INTO channel_videos VALUES (#{args}) ON CONFLICT (id) DO UPDATE SET title = $2, \
           published = $3, updated = $4, ucid = $5, author = $6", video_array)
         end
+      end
 
       if count < 30
         break
@@ -172,7 +176,7 @@ def produce_channel_videos_url(ucid, page = 1, auto_generated = nil)
   continuation = Base64.urlsafe_encode(continuation)
   continuation = URI.escape(continuation)
 
-  url = "/browse_ajax?continuation=#{continuation}"
+  url = "/browse_ajax?continuation=#{continuation}&gl=US&hl=en"
 
   return url
 end
@@ -192,6 +196,12 @@ def get_about_info(ucid)
     raise "User does not exist."
   end
 
+  sub_count = about.xpath_node(%q(//span[contains(text(), "subscribers")]))
+  if sub_count
+    sub_count = sub_count.content.delete(", subscribers").to_i?
+  end
+  sub_count ||= 0
+
   author = about.xpath_node(%q(//span[@class="qualified-channel-title-text"]/a)).not_nil!.content
   ucid = about.xpath_node(%q(//link[@rel="canonical"])).not_nil!["href"].split("/")[-1]
 
@@ -203,5 +213,37 @@ def get_about_info(ucid)
     auto_generated = true
   end
 
-  return {author, ucid, auto_generated}
+  return {author, ucid, auto_generated, sub_count}
+end
+
+def get_60_videos(ucid, page, auto_generated)
+  count = 0
+  videos = [] of SearchVideo
+
+  client = make_client(YT_URL)
+
+  2.times do |i|
+    url = produce_channel_videos_url(ucid, page * 2 + (i - 1), auto_generated: auto_generated)
+    response = client.get(url)
+    json = JSON.parse(response.body)
+
+    if json["content_html"]? && !json["content_html"].as_s.empty?
+      document = XML.parse_html(json["content_html"].as_s)
+      nodeset = document.xpath_nodes(%q(//li[contains(@class, "feed-item-container")]))
+
+      if !json["load_more_widget_html"]?.try &.as_s.empty?
+        count += 30
+      end
+
+      if auto_generated
+        videos += extract_videos(nodeset)
+      else
+        videos += extract_videos(nodeset, ucid)
+      end
+    else
+      break
+    end
+  end
+
+  return videos, count
 end

src/invidious/comments.cr

@@ -104,22 +104,22 @@ def template_youtube_comments(comments)
 
     html += <<-END_HTML
     <div class="pure-g">
-      <div class="pure-u-2-24">
+      <div class="pure-u-4-24 pure-u-md-2-24">
         <img style="width:90%; padding-right:1em; padding-top:1em;" src="#{author_thumbnail}">
       </div>
-      <div class="pure-u-22-24">
+      <div class="pure-u-20-24 pure-u-md-22-24">
         <p>
-            <a href="javascript:void(0)" onclick="toggle(this)">[ - ]</a> 
-            <i class="icon ion-ios-thumbs-up"></i> #{child["likeCount"]} 
-            <b><a href="#{child["authorUrl"]}">#{child["author"]}</a></b> 
-            - #{recode_date(Time.epoch(child["published"].as_i64))} ago
-          </p>
-          <div>
+          <b>
+            <a href="#{child["authorUrl"]}">#{child["author"]}</a>
+          </b> 
           <p style="white-space:pre-wrap">#{child["contentHtml"]}</p>
+          #{recode_date(Time.epoch(child["published"].as_i64))} ago
+          | 
+          <i class="icon ion-ios-thumbs-up"></i> #{child["likeCount"]} 
+        </p>
         #{replies_html}
       </div>
     </div>
-    </div>
     END_HTML
   end
 
@@ -156,10 +156,10 @@ def template_reddit_comments(root)
 
       content = <<-END_HTML
       <p>
-        <a href="javascript:void(0)" onclick="toggle(this)">[ - ]</a> 
-        <i class="icon ion-ios-thumbs-up"></i> #{score} 
+        <a href="javascript:void(0)" onclick="toggle_parent(this)">[ - ]</a> 
         <b><a href="https://www.reddit.com/user/#{author}">#{author}</a></b> 
-        - #{recode_date(child.created_utc)} ago
+        #{score} points 
+        #{recode_date(child.created_utc)} ago
       </p>
       <div>
       #{body_html}

src/invidious/helpers/helpers.cr

@@ -2,6 +2,7 @@ class Config
   YAML.mapping({
     crawl_threads:   Int32,
     channel_threads: Int32,
+    feed_threads:    Int32,
     video_threads:   Int32,
     db:              NamedTuple(
       user: String,
@@ -14,6 +15,7 @@ class Config
     https_only:   Bool?,
     hmac_key:     String?,
     full_refresh: Bool,
+    geo_bypass:   Bool,
   })
 end
 
@@ -244,11 +246,22 @@ def extract_items(nodeset, ucid = nil)
       plid = HTTP::Params.parse(URI.parse(id).query.not_nil!)["list"]
 
       anchor = node.xpath_node(%q(.//div[contains(@class, "yt-lockup-meta")]/a))
+
       if !anchor
         anchor = node.xpath_node(%q(.//ul[@class="yt-lockup-meta-info"]/li/a))
       end
-      if anchor
-        video_count = anchor.content.match(/View full playlist \((?<count>\d+)/).try &.["count"].to_i?
+
+      video_count = node.xpath_node(%q(.//span[@class="formatted-video-count-label"]/b))
+      if video_count
+        video_count = video_count.content
+
+        if video_count == "50+"
+          author = "YouTube"
+          author_id = "UC-9-kyTW8ZkZNDHQJ6FgpwQ"
+          video_count = video_count.rchop("+")
+        end
+
+        video_count = video_count.to_i?
       end
       video_count ||= 0
 

src/invidious/helpers/proxy.cr

@@ -89,6 +89,68 @@ class HTTPClient < HTTP::Client
 end
 
 def get_proxies(country_code = "US")
+  # return get_spys_proxies(country_code)
+  return get_nova_proxies(country_code)
+end
+
+def filter_proxies(proxies)
+  proxies.select! do |proxy|
+    begin
+      client = HTTPClient.new(YT_URL)
+      client.read_timeout = 10.seconds
+      client.connect_timeout = 10.seconds
+
+      proxy = HTTPProxy.new(proxy_host: proxy[:ip], proxy_port: proxy[:port])
+      client.set_proxy(proxy)
+
+      client.head("/").status_code == 200
+    rescue ex
+      false
+    end
+  end
+
+  return proxies
+end
+
+def get_nova_proxies(country_code = "US")
+  country_code = country_code.downcase
+  client = HTTP::Client.new(URI.parse("https://www.proxynova.com"))
+  client.read_timeout = 10.seconds
+  client.connect_timeout = 10.seconds
+
+  headers = HTTP::Headers.new
+  headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
+  headers["Accept"] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
+  headers["Accept-Language"] = "Accept-Language: en-US,en;q=0.9"
+  headers["Host"] = "www.proxynova.com"
+  headers["Origin"] = "https://www.proxynova.com"
+  headers["Referer"] = "https://www.proxynova.com/proxy-server-list/country-#{country_code}/"
+
+  response = client.get("/proxy-server-list/country-#{country_code}/", headers)
+  document = XML.parse_html(response.body)
+
+  proxies = [] of {ip: String, port: Int32, score: Float64}
+  document.xpath_nodes(%q(//tr[@data-proxy-id])).each do |node|
+    ip = node.xpath_node(%q(.//td/abbr/script)).not_nil!.content
+    ip = ip.match(/document\.write\('(?<sub1>[^']+)'.substr\(8\) \+ '(?<sub2>[^']+)'/).not_nil!
+    ip = "#{ip["sub1"][8..-1]}#{ip["sub2"]}"
+    port = node.xpath_node(%q(.//td[2])).not_nil!.content.strip.to_i
+
+    anchor = node.xpath_node(%q(.//td[4]/div)).not_nil!
+    speed = anchor["data-value"].to_f
+    latency = anchor["title"].to_f
+    uptime = node.xpath_node(%q(.//td[5]/span)).not_nil!.content.rchop("%").to_f
+
+    # TODO: Tweak me
+    score = (uptime*4 + speed*2 + latency)/7
+    proxies << {ip: ip, port: port, score: score}
+  end
+
+  # proxies = proxies.sort_by { |proxy| proxy[:score] }.reverse
+  return proxies
+end
+
+def get_spys_proxies(country_code = "US")
   client = HTTP::Client.new(URI.parse("http://spys.one"))
   client.read_timeout = 10.seconds
   client.connect_timeout = 10.seconds
@@ -108,7 +170,15 @@ def get_proxies(country_code = "US")
     "xf4" => "0",
     "xf5" => "1",
   }
+
   response = client.post("/free-proxy-list/#{country_code}/", headers, form: body)
+  20.times do
+    if response.status_code == 200
+      break
+    end
+    response = client.post("/free-proxy-list/#{country_code}/", headers, form: body)
+  end
+
   response = XML.parse_html(response.body)
 
   mapping = response.xpath_node(%q(.//body/script)).not_nil!.content

src/invidious/helpers/utils.cr

@@ -238,3 +238,9 @@ def write_var_int(value : Int)
 
   return bytes
 end
+
+def sha256(text)
+  digest = OpenSSL::Digest.new("SHA256")
+  digest << text
+  return digest.hexdigest
+end

src/invidious/jobs.cr

@@ -104,6 +104,44 @@ def refresh_videos(db)
   end
 end
 
+def refresh_feeds(db, max_threads = 1)
+  max_channel = Channel(Int32).new
+
+  spawn do
+    max_threads = max_channel.receive
+    active_threads = 0
+    active_channel = Channel(Bool).new
+
+    loop do
+      db.query("SELECT email FROM users") do |rs|
+        rs.each do
+          email = rs.read(String)
+          view_name = "subscriptions_#{sha256(email)[0..7]}"
+
+          if active_threads >= max_threads
+            if active_channel.receive
+              active_threads -= 1
+            end
+          end
+
+          active_threads += 1
+          spawn do
+            begin
+              db.exec("REFRESH MATERIALIZED VIEW #{view_name}")
+            rescue ex
+              STDOUT << "REFRESH " << email << " : " << ex.message << "\n"
+            end
+
+            active_channel.send(true)
+          end
+        end
+      end
+    end
+  end
+
+  max_channel.send(max_threads)
+end
+
 def pull_top_videos(config, db)
   if config.dl_api_key
     DetectLanguage.configure do |dl_config|
@@ -154,3 +192,16 @@ def update_decrypt_function
     Fiber.yield
   end
 end
+
+def find_working_proxies(regions)
+  loop do
+    regions.each do |region|
+      proxies = get_proxies(region).first(20)
+      proxies = proxies.map { |proxy| {ip: proxy[:ip], port: proxy[:port]} }
+      # proxies = filter_proxies(proxies)
+
+      yield region, proxies
+      Fiber.yield
+    end
+  end
+end

src/invidious/mixes.cr

@@ -0,0 +1,114 @@
+class MixVideo
+  add_mapping({
+    title:          String,
+    id:             String,
+    author:         String,
+    ucid:           String,
+    length_seconds: Int32,
+    index:          Int32,
+    mixes:          Array(String),
+  })
+end
+
+class Mix
+  add_mapping({
+    title:  String,
+    id:     String,
+    videos: Array(MixVideo),
+  })
+end
+
+def fetch_mix(rdid, video_id, cookies = nil)
+  client = make_client(YT_URL)
+  headers = HTTP::Headers.new
+  headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"
+
+  if cookies
+    headers = cookies.add_request_headers(headers)
+  end
+  response = client.get("/watch?v=#{video_id}&list=#{rdid}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en", headers)
+
+  yt_data = response.body.match(/window\["ytInitialData"\] = (?<data>.*);/)
+  if yt_data
+    yt_data = JSON.parse(yt_data["data"].rchop(";"))
+  else
+    raise "Could not create mix."
+  end
+
+  if !yt_data["contents"]["twoColumnWatchNextResults"]["playlist"]?
+    raise "Could not create mix."
+  end
+
+  playlist = yt_data["contents"]["twoColumnWatchNextResults"]["playlist"]["playlist"]
+  mix_title = playlist["title"].as_s
+
+  contents = playlist["contents"].as_a
+  until contents[0]["playlistPanelVideoRenderer"]["videoId"].as_s == video_id
+    contents.shift
+  end
+
+  videos = [] of MixVideo
+  contents.each do |item|
+    item = item["playlistPanelVideoRenderer"]
+
+    id = item["videoId"].as_s
+    title = item["title"]["simpleText"].as_s
+    author = item["longBylineText"]["runs"][0]["text"].as_s
+    ucid = item["longBylineText"]["runs"][0]["navigationEndpoint"]["browseEndpoint"]["browseId"].as_s
+    length_seconds = decode_length_seconds(item["lengthText"]["simpleText"].as_s)
+    index = item["navigationEndpoint"]["watchEndpoint"]["index"].as_i
+
+    videos << MixVideo.new(
+      title,
+      id,
+      author,
+      ucid,
+      length_seconds,
+      index,
+      [rdid]
+    )
+  end
+
+  if !cookies
+    next_page = fetch_mix(rdid, videos[-1].id, response.cookies)
+    videos += next_page.videos
+  end
+
+  videos.uniq! { |video| video.id }
+  videos = videos.first(50)
+  return Mix.new(mix_title, rdid, videos)
+end
+
+def template_mix(mix)
+  html = <<-END_HTML
+  <h3>
+    <a href="/mix?list=#{mix["mixId"]}">
+      #{mix["title"]}
+    </a>
+  </h3>
+  <div class="pure-menu pure-menu-scrollable playlist-restricted">
+    <ol class="pure-menu-list">
+  END_HTML
+
+  mix["videos"].as_a.each do |video|
+    html += <<-END_HTML
+      <li class="pure-menu-item">
+        <a href="/watch?v=#{video["videoId"]}&list=#{mix["mixId"]}">
+          <img style="width:100%;" src="/vi/#{video["videoId"]}/mqdefault.jpg">
+          <p style="width:100%">#{video["title"]}</p>
+          <p>
+              <b style="width: 100%">#{video["author"]}</b>
+          </p>
+        </a>
+      </li>
+    END_HTML
+  end
+
+  html += <<-END_HTML
+    </ol>
+  </div>
+  <hr>
+  END_HTML
+
+  html
+end

src/invidious/playlists.cr

@@ -1,17 +1,3 @@
-class Playlist
-  add_mapping({
-    title:            String,
-    id:               String,
-    author:           String,
-    ucid:             String,
-    description:      String,
-    description_html: String,
-    video_count:      Int32,
-    views:            Int64,
-    updated:          Time,
-  })
-end
-
 class PlaylistVideo
   add_mapping({
     title:          String,
@@ -25,11 +11,38 @@ class PlaylistVideo
   })
 end
 
-def fetch_playlist_videos(plid, page, video_count)
+class Playlist
+  add_mapping({
+    title:            String,
+    id:               String,
+    author:           String,
+    author_thumbnail: String,
+    ucid:             String,
+    description:      String,
+    description_html: String,
+    video_count:      Int32,
+    views:            Int64,
+    updated:          Time,
+  })
+end
+
+def fetch_playlist_videos(plid, page, video_count, continuation = nil)
   client = make_client(YT_URL)
 
-  if video_count > 100
+  if continuation
+    html = client.get("/watch?v=#{continuation}&list=#{plid}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1")
+    html = XML.parse_html(html.body)
+
+    index = html.xpath_node(%q(//span[@id="playlist-current-index"])).try &.content.to_i?
+    if index
+      index -= 1
+    end
+    index ||= 0
+  else
     index = (page - 1) * 100
+  end
+
+  if video_count > 100
     url = produce_playlist_url(plid, index)
 
     response = client.get(url)
@@ -42,14 +55,21 @@ def fetch_playlist_videos(plid, page, video_count)
     nodeset = document.xpath_nodes(%q(.//tr[contains(@class, "pl-video")]))
     videos = extract_playlist(plid, nodeset, index)
   else
+    # Playlist has less than one page of videos, so subsequent pages will be empty
     if page > 1
       videos = [] of PlaylistVideo
     else
-      response = client.get("/playlist?list=#{plid}&disable_polymer=1")
+      # Extract first page of videos
+      response = client.get("/playlist?list=#{plid}&gl=US&hl=en&disable_polymer=1")
       document = XML.parse_html(response.body)
       nodeset = document.xpath_nodes(%q(.//tr[contains(@class, "pl-video")]))
 
       videos = extract_playlist(plid, nodeset, 0)
+      if continuation
+        until videos[0].id == continuation
+          videos.shift
+        end
+      end
     end
   end
 
@@ -105,7 +125,8 @@ def produce_playlist_url(id, index)
   end
   ucid = "VL" + id
 
-  meta = "\x08#{write_var_int(index).join}"
+  meta = [0x08_u8] + write_var_int(index)
+  meta = Slice.new(meta.to_unsafe, meta.size)
   meta = Base64.urlsafe_encode(meta, false)
   meta = "PT:#{meta}"
 
@@ -141,7 +162,7 @@ def fetch_playlist(plid)
     plid = "UU#{plid.lchop("UC")}"
   end
 
-  response = client.get("/playlist?list=#{plid}&disable_polymer=1")
+  response = client.get("/playlist?list=#{plid}&hl=en&disable_polymer=1")
   if response.status_code != 200
     raise "Invalid playlist."
   end
@@ -160,10 +181,13 @@ def fetch_playlist(plid)
   title = title.content.strip(" \n")
 
   description_html = document.xpath_node(%q(//span[@class="pl-header-description-text"]/div/div[1]))
+  description_html ||= document.xpath_node(%q(//span[@class="pl-header-description-text"]))
   description_html, description = html_to_content(description_html)
 
   anchor = document.xpath_node(%q(//ul[@class="pl-header-details"])).not_nil!
   author = anchor.xpath_node(%q(.//li[1]/a)).not_nil!.content
+  author_thumbnail = document.xpath_node(%q(//img[@class="channel-header-profile-image"])).try &.["src"]
+  author_thumbnail ||= ""
   ucid = anchor.xpath_node(%q(.//li[1]/a)).not_nil!["href"].split("/")[2]
 
   video_count = anchor.xpath_node(%q(.//li[2])).not_nil!.content.delete("videos, ").to_i
@@ -181,6 +205,7 @@ def fetch_playlist(plid)
     title,
     plid,
     author,
+    author_thumbnail,
     ucid,
     description,
     description_html,
@@ -191,3 +216,37 @@ def fetch_playlist(plid)
 
   return playlist
 end
+
+def template_playlist(playlist)
+  html = <<-END_HTML
+  <h3>
+    <a href="/playlist?list=#{playlist["playlistId"]}">
+      #{playlist["title"]}
+    </a>
+  </h3>
+  <div class="pure-menu pure-menu-scrollable playlist-restricted">
+    <ol class="pure-menu-list">
+  END_HTML
+
+  playlist["videos"].as_a.each do |video|
+    html += <<-END_HTML
+      <li class="pure-menu-item">
+        <a href="/watch?v=#{video["videoId"]}&list=#{playlist["playlistId"]}">
+          <img style="width:100%;" src="/vi/#{video["videoId"]}/mqdefault.jpg">
+          <p style="width:100%">#{video["title"]}</p>
+          <p>
+              <b style="width: 100%">#{video["author"]}</b>
+          </p>
+        </a>
+      </li>
+    END_HTML
+  end
+
+  html += <<-END_HTML
+    </ol>
+  </div>
+  <hr>
+  END_HTML
+
+  html
+end

src/invidious/search.cr

@@ -89,7 +89,7 @@ def search(query, page = 1, search_params = produce_search_params(content_type:
     return {0, [] of SearchItem}
   end
 
-  html = client.get("/results?q=#{URI.escape(query)}&page=#{page}&sp=#{search_params}&disable_polymer=1").body
+  html = client.get("/results?q=#{URI.escape(query)}&page=#{page}&sp=#{search_params}&hl=en&disable_polymer=1").body
   if html.empty?
     return {0, [] of SearchItem}
   end

src/invidious/users.cr

@@ -119,6 +119,15 @@ def get_user(sid, client, headers, db, refresh = true)
 
       db.exec("INSERT INTO users VALUES (#{args}) \
       ON CONFLICT (email) DO UPDATE SET id = users.id || $1, updated = $2, subscriptions = $4", user_array)
+
+      begin
+        view_name = "subscriptions_#{sha256(user.email)[0..7]}"
+        PG_DB.exec("CREATE MATERIALIZED VIEW #{view_name} AS \
+        SELECT * FROM channel_videos WHERE \
+        ucid = ANY ((SELECT subscriptions FROM users WHERE email = '#{user.email}')::text[]) \
+        ORDER BY published DESC;")
+      rescue ex
+      end
     end
   else
     user = fetch_user(sid, client, headers, db)
@@ -129,6 +138,15 @@ def get_user(sid, client, headers, db, refresh = true)
 
     db.exec("INSERT INTO users VALUES (#{args}) \
     ON CONFLICT (email) DO UPDATE SET id = users.id || $1, updated = $2, subscriptions = $4", user_array)
+
+    begin
+      view_name = "subscriptions_#{sha256(user.email)[0..7]}"
+      PG_DB.exec("CREATE MATERIALIZED VIEW #{view_name} AS \
+      SELECT * FROM channel_videos WHERE \
+      ucid = ANY ((SELECT subscriptions FROM users WHERE email = '#{user.email}')::text[]) \
+      ORDER BY published DESC;")
+    rescue ex
+    end
   end
 
   return user

src/invidious/videos.cr

@@ -110,7 +110,7 @@ CAPTION_LANGUAGES = {
 
 REGIONS        = {"AD", "AE", "AF", "AG", "AI", "AL", "AM", "AO", "AQ", "AR", "AS", "AT", "AU", "AW", "AX", "AZ", "BA", "BB", "BD", "BE", "BF", "BG", "BH", "BI", "BJ", "BL", "BM", "BN", "BO", "BQ", "BR", "BS", "BT", "BV", "BW", "BY", "BZ", "CA", "CC", "CD", "CF", "CG", "CH", "CI", "CK", "CL", "CM", "CN", "CO", "CR", "CU", "CV", "CW", "CX", "CY", "CZ", "DE", "DJ", "DK", "DM", "DO", "DZ", "EC", "EE", "EG", "EH", "ER", "ES", "ET", "FI", "FJ", "FK", "FM", "FO", "FR", "GA", "GB", "GD", "GE", "GF", "GG", "GH", "GI", "GL", "GM", "GN", "GP", "GQ", "GR", "GS", "GT", "GU", "GW", "GY", "HK", "HM", "HN", "HR", "HT", "HU", "ID", "IE", "IL", "IM", "IN", "IO", "IQ", "IR", "IS", "IT", "JE", "JM", "JO", "JP", "KE", "KG", "KH", "KI", "KM", "KN", "KP", "KR", "KW", "KY", "KZ", "LA", "LB", "LC", "LI", "LK", "LR", "LS", "LT", "LU", "LV", "LY", "MA", "MC", "MD", "ME", "MF", "MG", "MH", "MK", "ML", "MM", "MN", "MO", "MP", "MQ", "MR", "MS", "MT", "MU", "MV", "MW", "MX", "MY", "MZ", "NA", "NC", "NE", "NF", "NG", "NI", "NL", "NO", "NP", "NR", "NU", "NZ", "OM", "PA", "PE", "PF", "PG", "PH", "PK", "PL", "PM", "PN", "PR", "PS", "PT", "PW", "PY", "QA", "RE", "RO", "RS", "RU", "RW", "SA", "SB", "SC", "SD", "SE", "SG", "SH", "SI", "SJ", "SK", "SL", "SM", "SN", "SO", "SR", "SS", "ST", "SV", "SX", "SY", "SZ", "TC", "TD", "TF", "TG", "TH", "TJ", "TK", "TL", "TM", "TN", "TO", "TR", "TT", "TV", "TW", "TZ", "UA", "UG", "UM", "US", "UY", "UZ", "VA", "VC", "VE", "VG", "VI", "VN", "VU", "WF", "WS", "YE", "YT", "ZA", "ZM", "ZW"}
 BYPASS_REGIONS = {
-  "UK",
+  "GB",
   "DE",
   "FR",
   "IN",
@@ -129,7 +129,6 @@ BYPASS_REGIONS = {
   "ID",
   "BD",
   "MX",
-  "ET",
   "PH",
   "EG",
   "VN",
@@ -274,6 +273,12 @@ class Video
     streams.each { |s| s.add("label", "#{s["quality"]} - #{s["type"].split(";")[0].split("/")[1]}") }
     streams = streams.uniq { |s| s["label"] }
 
+    if self.info["region"]?
+      streams.each do |fmt|
+        fmt["url"] += "&region=" + self.info["region"]
+      end
+    end
+
     if streams[0]? && streams[0]["s"]?
       streams.each do |fmt|
         fmt["url"] += "&signature=" + decrypt_signature(fmt["s"], decrypt_function)
@@ -363,6 +368,12 @@ class Video
       end
     end
 
+    if self.info["region"]?
+      adaptive_fmts.each do |fmt|
+        fmt["url"] += "&region=" + self.info["region"]
+      end
+    end
+
     if adaptive_fmts[0]? && adaptive_fmts[0]["s"]?
       adaptive_fmts.each do |fmt|
         fmt["url"] += "&signature=" + decrypt_signature(fmt["s"], decrypt_function)
@@ -445,7 +456,9 @@ class Video
     is_family_friendly: Bool,
     genre:              String,
     genre_url:          String,
-    license:            {
+    license:            String,
+    sub_count_text:     String,
+    author_thumbnail:   {
       type:    String,
       default: "",
     },
@@ -466,21 +479,24 @@ class CaptionName
   )
 end
 
-def get_video(id, db, refresh = true)
+class VideoRedirect < Exception
+end
+
+def get_video(id, db, proxies = {} of String => Array({ip: String, port: Int32}), refresh = true)
   if db.query_one?("SELECT EXISTS (SELECT true FROM videos WHERE id = $1)", id, as: Bool)
     video = db.query_one("SELECT * FROM videos WHERE id = $1", id, as: Video)
 
     # If record was last updated over 10 minutes ago, refresh (expire param in response lasts for 6 hours)
     if refresh && Time.now - video.updated > 10.minutes
       begin
-        video = fetch_video(id)
+        video = fetch_video(id, proxies)
         video_array = video.to_a
 
         args = arg_array(video_array[1..-1], 2)
 
         db.exec("UPDATE videos SET (info,updated,title,views,likes,dislikes,wilson_score,\
-          published,description,language,author,ucid, allowed_regions, is_family_friendly,\
-          genre, genre_url, license)\
+          published,description,language,author,ucid,allowed_regions,is_family_friendly,\
+          genre,genre_url,license,sub_count_text,author_thumbnail)\
           = (#{args}) WHERE id = $1", video_array)
       rescue ex
         db.exec("DELETE FROM videos * WHERE id = $1", id)
@@ -488,7 +504,7 @@ def get_video(id, db, refresh = true)
       end
     end
   else
-    video = fetch_video(id)
+    video = fetch_video(id, proxies)
     video_array = video.to_a
 
     args = arg_array(video_array)
@@ -499,15 +515,19 @@ def get_video(id, db, refresh = true)
   return video
 end
 
-def fetch_video(id)
-  html_channel = Channel(XML::Node).new
+def fetch_video(id, proxies)
+  html_channel = Channel(XML::Node | String).new
   info_channel = Channel(HTTP::Params).new
 
   spawn do
     client = make_client(YT_URL)
-    html = client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&disable_polymer=1")
-    html = XML.parse_html(html.body)
+    html = client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1")
 
+    if md = html.headers["location"]?.try &.match(/v=(?<id>[a-zA-Z0-9_-]{11})/)
+      next html_channel.send(md["id"])
+    end
+
+    html = XML.parse_html(html.body)
     html_channel.send(html)
   end
 
@@ -525,48 +545,72 @@ def fetch_video(id)
   end
 
   html = html_channel.receive
+  if html.as?(String)
+    raise VideoRedirect.new("#{html.as(String)}")
+  end
+  html = html.as(XML::Node)
+
   info = info_channel.receive
 
   if info["reason"]? && info["reason"].includes? "your country"
-    bypass_channel = Channel({HTTP::Params | Nil, XML::Node | Nil}).new
+    bypass_channel = Channel(HTTPProxy | Nil).new
 
-    BYPASS_REGIONS.each do |country_code|
+    proxies.each do |region, list|
       spawn do
+        info = HTTP::Params.new({
+          "reason" => [info["reason"]],
+        })
+
+        list.each do |proxy|
           begin
-          proxies = get_proxies(country_code)
+            client = HTTPClient.new(YT_URL)
+            client.read_timeout = 10.seconds
+            client.connect_timeout = 10.seconds
 
-          # Try not to overload single proxy
-          proxy = proxies[0, 5].sample(1)[0]
             proxy = HTTPProxy.new(proxy_host: proxy[:ip], proxy_port: proxy[:port])
+            client.set_proxy(proxy)
 
-          client = HTTPClient.new(URI.parse("https://www.youtube.com"))
+            info = HTTP::Params.parse(client.get("/get_video_info?video_id=#{id}&ps=default&eurl=&gl=US&hl=en&disable_polymer=1").body)
+            if !info["reason"]?
+              bypass_channel.send(proxy)
+              break
+            end
+          rescue ex
+          end
+        end
+
+        # If none of the proxies we tried returned a valid response
+        if info["reason"]?
+          bypass_channel.send(nil)
+        end
+      end
+    end
+
+    proxies.size.times do
+      proxy = bypass_channel.receive
+      if proxy
+        begin
+          client = HTTPClient.new(YT_URL)
           client.read_timeout = 10.seconds
           client.connect_timeout = 10.seconds
           client.set_proxy(proxy)
 
-          proxy_info = client.get("/get_video_info?video_id=#{id}&el=detailpage&ps=default&eurl=&gl=US&hl=en&disable_polymer=1")
-          proxy_info = HTTP::Params.parse(proxy_info.body)
+          html = XML.parse_html(client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1").body)
+          info = HTTP::Params.parse(client.get("/get_video_info?video_id=#{id}&el=detailpage&ps=default&eurl=&gl=US&hl=en&disable_polymer=1").body)
 
-          if !proxy_info["reason"]?
-            proxy_html = client.get("/watch?v=#{id}&bpctr=#{Time.new.epoch + 2000}&gl=US&hl=en&disable_polymer=1")
-            proxy_html = XML.parse_html(proxy_html.body)
-
-            bypass_channel.send({proxy_info, proxy_html})
-          else
-            bypass_channel.send({nil, nil})
-          end
-        rescue ex
-          bypass_channel.send({nil, nil})
-        end
-      end
+          if info["reason"]?
+            info = HTTP::Params.parse(client.get("/get_video_info?video_id=#{id}&ps=default&eurl=&gl=US&hl=en&disable_polymer=1").body)
+          end
+
+          proxy = {ip: proxy.proxy_host, port: proxy.proxy_port}
+          region = proxies.select { |region, list| list.includes? proxy }
+          if !region.empty?
+            info["region"] = region.keys[0]
           end
 
-    BYPASS_REGIONS.size.times do
-      response = bypass_channel.receive
-      if response[0] || response[1]
-        info = response[0].not_nil!
-        html = response[1].not_nil!
           break
+        rescue ex
+        end
       end
     end
   end
@@ -603,20 +647,39 @@ def fetch_video(id)
 
   genre = html.xpath_node(%q(//meta[@itemprop="genre"])).not_nil!["content"]
   genre_url = html.xpath_node(%(//a[text()="#{genre}"])).try &.["href"]
-  if genre == "Movies"
-    genre_url ||= "/channel/UClgRkhTL3_hImCAmdLfDE4g"
-  end
+  case genre
+  when "Movies"
+    genre_url = "/channel/UClgRkhTL3_hImCAmdLfDE4g"
+  when "Education"
+    # Education channel is linked but does not exist
+    # genre_url = "/channel/UC3yA8nDwraeOfnYfBWun83g"
     genre_url = ""
+  end
+  genre_url ||= ""
 
   license = html.xpath_node(%q(//h4[contains(text(),"License")]/parent::*/ul/li))
   if license
     license = license.content
   else
-    license ||= ""
+    license = ""
+  end
+
+  sub_count_text = html.xpath_node(%q(//span[contains(@class, "yt-subscriber-count")]))
+  if sub_count_text
+    sub_count_text = sub_count_text["title"]
+  else
+    sub_count_text = "0"
+  end
+
+  author_thumbnail = html.xpath_node(%(//img[@alt="#{author}"]))
+  if author_thumbnail
+    author_thumbnail = author_thumbnail["data-thumb"]
+  else
+    author_thumbnail = ""
   end
 
   video = Video.new(id, info, Time.now, title, views, likes, dislikes, wilson_score, published, description,
-    nil, author, ucid, allowed_regions, is_family_friendly, genre, genre_url, license)
+    nil, author, ucid, allowed_regions, is_family_friendly, genre, genre_url, license, sub_count_text, author_thumbnail)
 
   return video
 end

src/invidious/views/channel.ecr

@@ -13,23 +13,32 @@
     </div>
 </div>
 
-<p class="h-box">
+<div class="h-box">
 <% if user %>
     <% if subscriptions.includes? ucid %>
-    <a href="/subscription_ajax?action_remove_subscriptions=1&c=<%= ucid %>&referer=<%= env.get("current_page") %>">
-        <b>Unsubscribe from <%= author %></b>
+    <p>
+        <a id="subscribe" onclick="unsubscribe()" class="pure-button pure-button-primary" 
+            href="/subscription_ajax?action_remove_subscriptions=1&c=<%= ucid %>&referer=<%= env.get("current_page") %>">
+            <b>Unsubscribe from <%= author %> <%= number_with_separator(sub_count) %></b>
         </a>
+    </p>
     <% else %>
-    <a href="/subscription_ajax?action_create_subscription_to_channel=1&c=<%= ucid %>&referer=<%= env.get("current_page") %>">
-        <b>Subscribe to <%= author %></b>
+    <p>
+        <a id="subscribe" onclick="subscribe()" class="pure-button pure-button-primary" 
+            href="/subscription_ajax?action_create_subscription_to_channel=1&c=<%= ucid %>&referer=<%= env.get("current_page") %>">
+            <b>Subscribe to <%= author %> <%= number_with_separator(sub_count) %></b>
         </a>
+    </p>
     <% end %>
 <% else %>
-    <a href="/login?referer=<%= env.get("current_page") %>">
+    <p>
+        <a id="subscribe" class="pure-button pure-button-primary" 
+            href="/login?referer=<%= env.get("current_page") %>">
             <b>Login to subscribe to <%= author %></b>
         </a>
+    </p>
 <% end %>
-</p>
+</div>
 
 <p class="h-box">
     <a href="https://www.youtube.com/channel/<%= ucid %>">View channel on YouTube</a>
@@ -51,8 +60,50 @@
     </div>
     <div class="pure-u-1 pure-u-md-3-5"></div>
     <div style="text-align:right;" class="pure-u-1 pure-u-md-1-5">
-    <% if videos.size == 60 %>
+    <% if count == 60 %>
         <a href="/channel/<%= ucid %>?page=<%= page + 1 %>">Next page</a>
     <% end %>
     </div>
 </div>
+
+<script>
+document.getElementById("subscribe")["href"] = "javascript:void(0);"
+
+function subscribe() {
+    var url = "/subscription_ajax?action_create_subscription_to_channel=1&c=<%= ucid %>&referer=<%= env.get("current_page") %>";
+    var xhr = new XMLHttpRequest();
+    xhr.responseType = "json";
+    xhr.timeout = 20000;
+    xhr.open("GET", url, true);
+    xhr.send();
+
+    xhr.onreadystatechange = function() {
+        if (xhr.readyState == 4) {
+            if (xhr.status == 200) {
+                subscribe_button = document.getElementById("subscribe");
+                subscribe_button.onclick = unsubscribe;
+                subscribe_button.innerHTML = '<b>Unsubscribe from <%= author %> <%= number_with_separator(sub_count + 1) %></b>'
+            }
+        }
+    }
+}
+
+function unsubscribe() {
+    var url = "/subscription_ajax?action_remove_subscriptions=1&c=<%= ucid %>&referer=<%= env.get("current_page") %>";
+    var xhr = new XMLHttpRequest();
+    xhr.responseType = "json";
+    xhr.timeout = 20000;
+    xhr.open("GET", url, true);
+    xhr.send();
+
+    xhr.onreadystatechange = function() {
+        if (xhr.readyState == 4) {
+            if (xhr.status == 200) {
+                subscribe_button = document.getElementById("subscribe");
+                subscribe_button.onclick = subscribe;
+                subscribe_button.innerHTML = '<b>Subscribe to <%= author %> <%= number_with_separator(sub_count) %></b>'
+            }
+        }
+    }
+}
+</script>

src/invidious/views/components/item.ecr

@@ -14,7 +14,12 @@
         <p><%= number_with_separator(item.subscriber_count) %> subscribers</p>
         <h5><%= item.description_html %></h5>
     <% when SearchPlaylist %>
-        <a style="width:100%;" href="/playlist?list=<%= item.id %>">
+        <% if item.id.starts_with? "RD" %>
+        <% url = "/mix?list=#{item.id}&continuation=#{item.videos[0]?.try &.id}" %>
+        <% else %>
+        <% url = "/playlist?list=#{item.id}" %>
+        <% end %>
+        <a style="width:100%;" href="<%= url %>">
             <% if env.get?("user") && env.get("user").as(User).preferences.thin_mode %>
             <% else %>
             <img style="width:100%;" src="/vi/<%= item.videos[0]?.try &.id %>/mqdefault.jpg"/>
@@ -26,6 +31,17 @@
         </p>
         <p><%= number_with_separator(item.video_count) %> videos</p>
         <p>PLAYLIST</p>
+    <% when MixVideo %>
+        <a style="width:100%;" href="/watch?v=<%= item.id %>&list=<%= item.mixes[0] %>">
+            <% if env.get?("user") && env.get("user").as(User).preferences.thin_mode %>
+            <% else %>
+            <img style="width:100%;" src="/vi/<%= item.id %>/mqdefault.jpg"/>
+            <% end %>
+            <p><%= item.title %></p>
+        </a>
+        <p>
+            <b><a style="width:100%;" href="/channel/<%= item.ucid %>"><%= item.author %></a></b>
+        </p>
     <% else %>
         <% if item.responds_to?(:playlists) && !item.playlists.empty? %>
         <% params = "&list=#{item.playlists[0]}" %>

src/invidious/views/data_control.ecr

@@ -13,7 +13,7 @@
             </div>
 
             <div class="pure-control-group">
-                <label for="import_youtube">Import <a target="_blank"
+                <label for="import_youtube">Import <a rel="noopener" target="_blank"
                         href="https://support.google.com/youtube/answer/6224202?hl=en-GB">YouTube subscriptions</a></label>
                 <input type="file" id="import_youtube" name="import_youtube">
             </div>

src/invidious/views/mix.ecr

@@ -0,0 +1,22 @@
+<% content_for "header" do %>
+<title><%= mix.title %> - Invidious</title>
+<% end %>
+
+<div class="pure-g h-box">
+    <div class="pure-u-2-3">
+        <h3><%= mix.title %></h3>
+    </div>
+    <div class="pure-u-1-3" style="text-align:right;">
+        <h3>
+            <a href="/feed/playlist/<%= mix.id %>"><i class="icon ion-logo-rss"></i></a>
+        </h3>
+    </div>
+</div>
+
+<% mix.videos.each_slice(4) do |slice| %>
+<div class="pure-g">
+    <% slice.each do |item| %>
+        <%= rendered "components/item" %>
+    <% end %>
+</div> 
+<% end %>

src/invidious/views/playlist.ecr

@@ -35,7 +35,7 @@
 <div class="pure-g h-box">
     <div class="pure-u-1 pure-u-md-1-5">
     <% if page >= 2 %>
-        <a href="/playlist?list=<%= playlist.id %>&page=<%= page - 1 %>">Next page</a>
+        <a href="/playlist?list=<%= playlist.id %>&page=<%= page - 1 %>">Previous page</a>
     <% end %>
     </div>
     <div class="pure-u-1 pure-u-md-3-5"></div>

src/invidious/views/search.ecr

@@ -13,13 +13,13 @@
 <div class="pure-g h-box">
     <div class="pure-u-1 pure-u-md-1-5">
     <% if page >= 2 %>
-        <a href="/search?q=<%= query %>&page=<%= page - 1 %>">Previous page</a>
+        <a href="/search?q=<%= HTML.escape(query.not_nil!) %>&page=<%= page - 1 %>">Previous page</a>
     <% end %>
     </div>
     <div class="pure-u-1 pure-u-md-3-5"></div>
     <div style="text-align:right;" class="pure-u-1 pure-u-md-1-5">
     <% if count >= 20 %>
-        <a href="/search?q=<%= query %>&page=<%= page + 1 %>">Next page</a>
+        <a href="/search?q=<%= HTML.escape(query.not_nil!) %>&page=<%= page + 1 %>">Next page</a>
     <% end %>
     </div>
 </div>

src/invidious/views/template.ecr

@@ -28,7 +28,7 @@
         <div class="pure-u-1 pure-u-md-12-24 searchbar">
           <form class="pure-form" action="/search" method="get">
             <fieldset>
-              <input type="search" style="width:100%;" name="q" placeholder="search" value="<%= env.params.query["q"]? || env.get? "search" %>">
+              <input type="search" style="width:100%;" name="q" placeholder="search" value="<%= env.params.query["q"]?.try {|x| HTML.escape(x)} || env.get?("search").try {|x| HTML.escape(x.as(String)) }  %>">
             </fieldset>
           </form>
         </div>

src/invidious/views/watch.ecr

@@ -22,6 +22,7 @@
 <meta name="twitter:player" content="<%= host_url %>/embed/<%= video.id %>">
 <meta name="twitter:player:width" content="1280">
 <meta name="twitter:player:height" content="720">
+<script src="/js/watch.js"></script>
 <%= rendered "components/player_sources" %>
 <title><%= HTML.escape(video.title) %> - Invidious</title>
 <% end %>
@@ -55,7 +56,13 @@
             <p><i class="icon ion-ios-eye"></i> <%= number_with_separator(video.views) %></p>
             <p><i class="icon ion-ios-thumbs-up"></i> <%= number_with_separator(video.likes) %></p>
             <p><i class="icon ion-ios-thumbs-down"></i> <%= number_with_separator(video.dislikes) %></p>
-            <p id="Genre">Genre: <a href="<%= video.genre_url %>"><%= video.genre %></a></p>
+            <p id="Genre">Genre: 
+            <% if video.genre_url.empty? %>
+            <%= video.genre %>
+            <% else %>
+            <a href="<%= video.genre_url %>"><%= video.genre %></a>
+            <% end %>
+            </p>
             <% if !video.license.empty? %>
             <p id="License">License: <%= video.license %></p>
             <% end %>
@@ -85,20 +92,23 @@
         <% if user %>
             <% if subscriptions.includes? video.ucid %>
             <p>
-                <a href="/subscription_ajax?action_remove_subscriptions=1&c=<%= video.ucid %>&referer=<%= env.get("current_page") %>">
-                    <b>Unsubscribe from <%= video.author %></b>
+                <a id="subscribe" onclick="unsubscribe()" class="pure-button pure-button-primary" 
+                    href="/subscription_ajax?action_remove_subscriptions=1&c=<%= video.ucid %>&referer=<%= env.get("current_page") %>">
+                    <b>Unsubscribe from <%= video.author %> <%= video.sub_count_text %></b>
                 </a>
             </p>
             <% else %>
             <p>
-                <a href="/subscription_ajax?action_create_subscription_to_channel=1&c=<%= video.ucid %>&referer=<%= env.get("current_page") %>">
-                    <b>Subscribe to <%= video.author %></b>
+                <a id="subscribe" onclick="subscribe()" class="pure-button pure-button-primary" 
+                    href="/subscription_ajax?action_create_subscription_to_channel=1&c=<%= video.ucid %>&referer=<%= env.get("current_page") %>">
+                    <b>Subscribe to <%= video.author %> <%= video.sub_count_text %></b>
                 </a>
             </p>
             <% end %>
         <% else %>
             <p>
-                <a href="/login?referer=<%= env.get("current_page") %>">
+                <a id="subscribe" class="pure-button pure-button-primary" 
+                    href="/login?referer=<%= env.get("current_page") %>">
                     <b>Login to subscribe to <%= video.author %></b>
                 </a>
             </p>
@@ -111,11 +121,15 @@
             </div>
             <hr>
             <div id="comments">
-                <h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3>
             </div>
         </div>
     </div>
     <div class="pure-u-1 pure-u-md-1-5">
+        <% if plid %>
+        <div id="playlist" class="h-box">
+        </div>
+        <% end %>
+
         <% if !preferences || preferences && preferences.related_videos %>
         <div class="h-box">
         <% rvs.each do |rv| %>
@@ -138,38 +152,13 @@
 </div>
 
 <script>
-function toggle(target) {
-  body = target.parentNode.parentNode.children[1];
-  if (body.style.display === null || body.style.display === "") {
-    target.innerHTML = "[ + ]";
-    body.style.display = "none";
-  } else {
-    target.innerHTML = "[ - ]";
-    body.style.display = "";
-  }
+subscribe_button = document.getElementById("subscribe");
+if (subscribe_button.getAttribute('onclick')) {
+    subscribe_button["href"] = "javascript:void(0);";
 }
 
-function toggle_comments(target) {
-  body = target.parentNode.parentNode.parentNode.children[1];
-  if (body.style.display === null || body.style.display === "") {
-    target.innerHTML = "[ + ]";
-    body.style.display = "none";
-  } else {
-    target.innerHTML = "[ - ]";
-    body.style.display = "";
-  }
-}
-
-function get_youtube_replies(target) {
-  var continuation = target.getAttribute("data-continuation");
-
-  var body = target.parentNode.parentNode;
-  var fallback = body.innerHTML;
-  body.innerHTML =
-    '<h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3>';
-
-  var url =
-    "/api/v1/comments/<%= video.id %>?format=html&continuation=" + continuation;
+function subscribe() {
+    var url = "/subscription_ajax?action_create_subscription_to_channel=1&c=<%= video.ucid %>&referer=<%= env.get("current_page") %>";
     var xhr = new XMLHttpRequest();
     xhr.responseType = "json";
     xhr.timeout = 20000;
@@ -179,21 +168,101 @@ function get_youtube_replies(target) {
     xhr.onreadystatechange = function() {
         if (xhr.readyState == 4) {
             if (xhr.status == 200) {
-        body.innerHTML = xhr.response.contentHtml;
+                subscribe_button = document.getElementById("subscribe");
+                subscribe_button.onclick = unsubscribe;
+                subscribe_button.innerHTML = '<b>Unsubscribe from <%= video.author %> <%= video.sub_count_text %></b>'
+            }
+        }
+    }
+}
+
+function unsubscribe() {
+    var url = "/subscription_ajax?action_remove_subscriptions=1&c=<%= video.ucid %>&referer=<%= env.get("current_page") %>";
+    var xhr = new XMLHttpRequest();
+    xhr.responseType = "json";
+    xhr.timeout = 20000;
+    xhr.open("GET", url, true);
+    xhr.send();
+
+    xhr.onreadystatechange = function() {
+        if (xhr.readyState == 4) {
+            if (xhr.status == 200) {
+                subscribe_button = document.getElementById("subscribe");
+                subscribe_button.onclick = subscribe;
+                subscribe_button.innerHTML = '<b>Subscribe to <%= video.author %> <%= video.sub_count_text %></b>'
+            }
+        }
+    }
+}
+
+<% if plid %>
+function get_playlist() {
+    playlist = document.getElementById("playlist");
+    playlist.innerHTML = ' \
+      <h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3> \
+      <hr>'
+
+    var plid = "<%= plid %>"
+
+    if (plid.startsWith("RD")) {
+        var plid_url = "/api/v1/mixes/<%= plid %>?continuation=<%= video.id %>&format=html";
     } else {
-        body.innerHTML = fallback;
+        var plid_url = "/api/v1/playlists/<%= plid %>?continuation=<%= video.id %>&format=html";
+    }
+
+    var xhr = new XMLHttpRequest();
+    xhr.responseType = "json";
+    xhr.timeout = 20000;
+    xhr.open("GET", plid_url, true);
+    xhr.send();
+
+    xhr.onreadystatechange = function() {
+        if (xhr.readyState == 4) {
+            if (xhr.status == 200) {
+                playlist.innerHTML = xhr.response.playlistHtml;
+                
+                if (xhr.response.nextVideo) {
+                    player.on('ended', function() {
+                        window.location.replace("/watch?v=" 
+                            + xhr.response.nextVideo
+                            + "&list=<%= plid %>"
+                            <% if params[:listen] %>
+                            + "&listen=1"
+                            <% end %>
+                            <% if params[:autoplay] %>
+                            + "&autoplay=1"
+                            <% end %>
+                            <% if params[:speed] %>
+                            + "&speed=<%= params[:speed] %>"
+                            <% end %>
+                        );
+                    });
+                }
+            } else {
+                playlist.innerHTML = "";
             }
         }
     };
 
     xhr.ontimeout = function() {
-    console.log("Pulling comments timed out.");
+        console.log("Pulling playlist timed out.");
 
-    body.innerHTML = fallback;
+        comments = document.getElementById("playlist");
+        comments.innerHTML =
+            '<h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3><hr>';
+        get_playlist();
     };
 }
 
+get_playlist();
+<% end %>
+
 function get_reddit_comments() {
+  comments = document.getElementById("comments");
+  var fallback = comments.innerHTML;
+  comments.innerHTML =
+    '<h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3>';
+
   var url = "/api/v1/comments/<%= video.id %>?source=reddit&format=html";
   var xhr = new XMLHttpRequest();
   xhr.responseType = "json";
@@ -202,17 +271,23 @@ function get_reddit_comments() {
   xhr.send();
 
   xhr.onreadystatechange = function() {
-    if (xhr.readyState == 4)
+    if (xhr.readyState == 4) {
       if (xhr.status == 200) {
-        comments = document.getElementById("comments");
         comments.innerHTML = ' \
             <div> \
                 <h3> \
                     <a href="javascript:void(0)" onclick="toggle_comments(this)">[ - ]</a> \
                     {title} \
                 </h3> \
+                <p> \
                     <b> \
-                    <a target="_blank" href="https://reddit.com{permalink}">View more comments on Reddit</a> \
+                        <a href="javascript:void(0)" onclick="swap_comments(\'youtube\')"> \
+                            View YouTube comments \
+                        </a> \
+                    </b> \
+                </p> \
+                <b> \
+                    <a rel="noopener" target="_blank" href="https://reddit.com{permalink}">View more comments on Reddit</a> \
                 </b> \
             </div> \
             <div>{contentHtml}</div> \
@@ -225,10 +300,10 @@ function get_reddit_comments() {
         <% if preferences && preferences.comments[1] == "youtube" %> 
         get_youtube_comments();
         <% else %>
-        comments = document.getElementById("comments");
-        comments.innerHTML = "";
+        comments.innerHTML = fallback;
         <% end %>
       }
+    }
   };
 
   xhr.ontimeout = function() {
@@ -239,6 +314,11 @@ function get_reddit_comments() {
 }
 
 function get_youtube_comments() {
+  comments = document.getElementById("comments");
+  var fallback = comments.innerHTML;
+  comments.innerHTML =
+    '<h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3>';
+
   var url = "/api/v1/comments/<%= video.id %>?format=html";
   var xhr = new XMLHttpRequest();
   xhr.responseType = "json";
@@ -247,9 +327,8 @@ function get_youtube_comments() {
   xhr.send();
 
   xhr.onreadystatechange = function() {
-    if (xhr.readyState == 4)
+    if (xhr.readyState == 4) {
       if (xhr.status == 200) {
-        comments = document.getElementById("comments");
         if (xhr.response.commentCount > 0)  {
             comments.innerHTML = ' \
                 <div> \
@@ -257,6 +336,11 @@ function get_youtube_comments() {
                         <a href="javascript:void(0)" onclick="toggle_comments(this)">[ - ]</a> \
                         View {commentCount} comments \
                     </h3> \
+                    <b> \
+                        <a href="javascript:void(0)" onclick="swap_comments(\'reddit\')"> \
+                            View Reddit comments \
+                        </a> \
+                    </b> \
                 </div> \
                 <div>{contentHtml}</div> \
                 <hr>'.supplant({
@@ -270,35 +354,59 @@ function get_youtube_comments() {
         <% if preferences && preferences.comments[1] == "youtube" %> 
         get_youtube_comments();
         <% else %>
-        comments = document.getElementById("comments");
         comments.innerHTML = "";
         <% end %>
       }
+    }
   };
 
   xhr.ontimeout = function() {
     console.log("Pulling comments timed out.");
 
-    comments = document.getElementById("comments");
     comments.innerHTML =
       '<h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3>';
     get_youtube_comments();
   };
 }
 
-function commaSeparateNumber(val){
-  while (/(\d+)(\d{3})/.test(val.toString())){
-    val = val.toString().replace(/(\d+)(\d{3})/, '$1'+','+'$2');
-  }
-  return val;
-}
+function get_youtube_replies(target) {
+  var continuation = target.getAttribute('data-continuation');
 
-String.prototype.supplant = function(o) {
-  return this.replace(/{([^{}]*)}/g, function(a, b) {
-    var r = o[b];
-    return typeof r === "string" || typeof r === "number" ? r : a;
+  var body = target.parentNode.parentNode;
+  var fallback = body.innerHTML;
+  body.innerHTML =
+      '<h3><center class="loading"><i class="icon ion-ios-refresh"></i></center></h3>';
+
+  var url = '/api/v1/comments/<%= video.id %>?format=html&continuation=' +
+      continuation;
+  var xhr = new XMLHttpRequest();
+  xhr.responseType = 'json';
+  xhr.timeout = 20000;
+  xhr.open('GET', url, true);
+  xhr.send();
+
+  xhr.onreadystatechange = function() {
+    if (xhr.readyState == 4) {
+      if (xhr.status == 200) {
+        body.innerHTML = ' \
+            <p><a href="javascript:void(0)" \
+                onclick="hide_youtube_replies(this)">Hide replies \
+            </a></p> \
+            <div>{contentHtml}</div>'.supplant({
+            contentHtml: xhr.response.contentHtml,
         });
-};
+      } else {
+        body.innerHTML = fallback;
+      }
+    }
+  };
+
+  xhr.ontimeout = function() {
+    console.log('Pulling comments timed out.');
+
+    body.innerHTML = fallback;
+  };
+}
 
 <% if preferences %>
     <% if preferences.comments[0] == "youtube" %>