use bash script for some customizations instead of Git patches

README.md

@@ -36,49 +36,51 @@ cd /home/mastodon/utils
 
 ## Scripts
 
-- [`init.sh`](init.sh): A small helper that runs at the very beginning of each script below to initialize `rbenv` and set consistent environment variables.
+- [`init.sh`](init.sh): A small helper that runs at the very beginning of each script below to initialize `nvm`/`rbenv` and set consistent environment variables.
   - **Optional:** To make your life easier, you can also source this script from the `.bashrc` of the `mastodon` user and/or whichever user you regularly SSH in as:
 
 ```sh
 [ -s /home/mastodon/utils/init.sh ] && \. /home/mastodon/utils/init.sh >/dev/null 2>&1
 ```
 
-- [`version.sh`](scripts/version.sh): Tests `init.sh` by printing version of Mastodon, rbenv, nvm, Ruby, Node, and Yarn.
+- [`version.sh`](scripts/version.sh): Tests `init.sh` by printing the versions of Mastodon, rbenv, nvm, Ruby, Node, and Yarn.
 
 #### Periodic tasks
 
 - [`backup.sh`](scripts/backup.sh): Backs up Postgres, Redis, and `.env.production` secrets to a `.tar.gz` file in `/home/mastodon/backups` — useful for a [periodic cronjob](https://github.com/jakejarvis/mastodon-utils/wiki/Cron-jobs#backups).
 - [`weekly_cleanup.sh`](scripts/weekly_cleanup.sh): Runs Mastodon's built-in [cleanup commands](https://docs.joinmastodon.org/admin/setup/#cleanup), designed for a [weekly cronjob](https://github.com/jakejarvis/mastodon-utils/wiki/Cron-jobs#media-cleanup).
-  - Keeps 7 days of media (in object storage)
+  - Keeps 14 days of media
-  - Keeps 90 days of generated preview cards
+  - Keeps 90 days of profile avatars, headers, and link preview cards
 
 #### Dangerous
 
 **The following scripts are highly opinionated, catastrophically destructive, and very specific to me.** Check them out line-by-line instead of running them.
 
 - [`install.sh`](scripts/install.sh): Assumes an absolutely clean install of Ubuntu and installs Mastodon ***with all of the quirks from this repo.*** Configure `MASTODON_USER` and other paths in [`init.sh`](init.sh) first if necessary. [Get the far less dangerous version of `install.sh` here instead.](https://github.com/jakejarvis/mastodon-installer/blob/main/install.sh)
-- [`upgrade.sh`](scripts/upgrade.sh): Upgrades Mastodon server (latest version if vanilla Mastodon, latest commit if `glitch-soc`) and ***re-applies every patch*** listed below. [Get the far less dangerous version of `upgrade.sh` here instead.](https://github.com/jakejarvis/mastodon-installer/blob/main/upgrade.sh)
+- [`upgrade.sh`](scripts/upgrade.sh): Upgrades Mastodon server (latest version if vanilla Mastodon, latest commit if `glitch-soc`) and ***re-applies all customizations***. [Get the far less dangerous version of `upgrade.sh` here instead.](https://github.com/jakejarvis/mastodon-installer/blob/main/upgrade.sh)
+- [`customize.sh`](scripts/customize.sh): Applies ***every Git patch below***, sets defaults (mostly for logged-out visitors) and removes unused files.
 
 ## Patches
 
+#### Vanilla only:
+
+- [`increase-sidekiq-timeout.patch`](patches/increase-sidekiq-timeout.patch): Small bump in Sidekiq's timeout before it decides a remote instance isn't available. **Use this one very carefully!**
+
 #### Vanilla and `glitch-soc`:
 
-- [`robots.patch`](patches/robots.patch): Disallow search engines for all of Mastodon
-- [`increase-sidekiq-timeout.patch`](patches/increase-sidekiq-timeout.patch): Small bump in Sidekiq's timeout before it decides a remote instance isn't available. **Use this one very carefully!**
 - [`system-font.patch`](patches/system-font.patch): Use the system's default sans-serif font stack instead of Roboto
   - [`glitch/system-font.patch`](patches/glitch/system-font.patch)
 - [`hide-contact-email.patch`](patches/hide-contact-email.patch): Hides the `mailto:` link on the About page
   - [`glitch/hide-contact-email.patch`](patches/glitch/hide-contact-email.patch)
-- [`hide-rules.patch`](patches/hide-rules.patch): Applies just to homepage, meant only for single-user instances
+- [`hide-rules.patch`](patches/hide-rules.patch): Hides the list of rules on the About page (meant only for single-user instances)
   - [`glitch/hide-rules.patch`](patches/glitch/hide-rules.patch)
 - [`hide-signup.patch`](patches/hide-signup.patch): Hide the "create account" button (for aesthetics, **not security!**)
   - [`glitch/hide-signup.patch`](patches/glitch/hide-signup.patch)
 
 #### `glitch-soc` only:
-  - [`custom-glitch-defaults.patch`](patches/glitch/custom-glitch-defaults.patch): Sets default Glitch appearance settings for logged-out users
+
+- [`mastodon-logo.patch`](patches/glitch/sidebar-logo.patch): Restore the Mastodon logo in the non-advanced sidebar
 - [`settings-sidebar-cleanup.patch`](patches/glitch/settings-sidebar-cleanup.patch): Why is the most frequently used admin page listed under a link that takes you to another page to open a submenu in the sidebar to finally be able to click on it to go to the page?!?
-  - [`remove-glitch-cruft.patch`](patches/glitch/remove-glitch-cruft.patch): Removes a bunch of junk no longer used by `glitch-soc`
-  - [`sidebar-logo.patch`](patches/glitch/sidebar-logo.patch): Restore Mastodon logo in logged-out sidebar
 
 ## License
 

init.sh

@@ -12,6 +12,10 @@ export LOGS_ROOT="$MASTODON_ROOT/logs"        # logs destintation
 export RBENV_ROOT="$MASTODON_ROOT/.rbenv"     # rbenv (w/ ruby-build plugin) directory
 export NVM_DIR="$MASTODON_ROOT/.nvm"          # nvm directory
 
+# automatically detect glitch-soc
+# shellcheck disable=SC2155
+export MASTODON_IS_GLITCH=$(test -d "$APP_ROOT/app/javascript/flavours/glitch" && echo true || echo false)
+
 # ---
 
 # initialize rbenv
@@ -23,7 +27,8 @@ fi
 
 # initialize nvm
 if [ -s "$NVM_DIR/nvm.sh" ]; then
-  . "$NVM_DIR/nvm.sh"
+  # shellcheck disable=SC1091
+  source "$NVM_DIR/nvm.sh"
 else
   echo "⚠️ Couldn't find nvm.sh in '$NVM_DIR', double check the paths set in '$UTILS_ROOT/init.sh'..."
 fi
@@ -72,3 +77,8 @@ as_mastodon() {
 tootctl() {
   ( cd "$APP_ROOT" && as_mastodon RAILS_ENV=production ruby ./bin/tootctl "$@" )
 }
+
+# ---
+
+# keep track of whether this file has already been run
+export MASTODON_INIT_RUN=true

patches/glitch/custom-glitch-defaults.patch

@@ -1,53 +0,0 @@
-diff --git a/app/javascript/flavours/glitch/reducers/local_settings.js b/app/javascript/flavours/glitch/reducers/local_settings.js
-index 81ab1cb0d..8aa07d4c3 100644
---- a/app/javascript/flavours/glitch/reducers/local_settings.js
-+++ b/app/javascript/flavours/glitch/reducers/local_settings.js
-@@ -11,7 +11,7 @@ const initialState = ImmutableMap({
-   navbar_under : false,
-   side_arm  : 'none',
-   side_arm_reply_mode : 'keep',
--  show_reply_count : false,
-+  show_reply_count : true,
-   always_show_spoilers_field: false,
-   confirm_missing_media_description: false,
-   confirm_boost_missing_media_description: false,
-@@ -19,10 +19,10 @@ const initialState = ImmutableMap({
-   prepend_cw_re: true,
-   preselect_on_reply: true,
-   inline_preview_cards: true,
--  hicolor_privacy_icons: false,
-+  hicolor_privacy_icons: true,
-   show_content_type_choice: false,
-   tag_misleading_links: true,
--  rewrite_mentions: 'no',
-+  rewrite_mentions: 'acct',
-   content_warnings : ImmutableMap({
-     filter       : null,
-     media_outside: false,
-@@ -33,7 +33,7 @@ const initialState = ImmutableMap({
-     auto        : ImmutableMap({
-       all              : false,
-       notifications    : true,
--      lengthy          : true,
-+      lengthy          : false,
-       reblogs          : false,
-       replies          : false,
-       media            : false,
-@@ -45,7 +45,7 @@ const initialState = ImmutableMap({
-     show_action_bar : true,
-   }),
-   media     : ImmutableMap({
--    letterbox        : true,
-+    letterbox        : false,
-     fullwidth        : true,
-     reveal_behind_cw : false,
-     pop_in_player    : true,
-@@ -56,7 +56,7 @@ const initialState = ImmutableMap({
-     tab_badge     : true,
-   }),
-   status_icons : ImmutableMap({
--    language:   true,
-+    language:   false,
-     reply:      true,
-     local_only: true,
-     media:      true,

patches/robots.patch

@@ -1,17 +0,0 @@
-diff --git a/public/robots.txt b/public/robots.txt
-index 771bf2160b..af16ddb0ce 100644
---- a/public/robots.txt
-+++ b/public/robots.txt
-@@ -1,5 +1,8 @@
--# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
--
-+# block everything except About page
- User-agent: *
--Disallow: /media_proxy/
--Disallow: /interact/
-+Allow: /about
-+Disallow: /
-+
-+# sorry, Elon
-+User-agent: Twitterbot
-+Disallow: /

scripts/backup.sh

@@ -23,10 +23,6 @@ if [ ! -d "$BACKUPS_ROOT" ]; then
   as_mastodon mkdir -p "$BACKUPS_ROOT"
 fi
 
-if [ ! -d "$LOGS_ROOT" ]; then
-  as_mastodon mkdir -p "$LOGS_ROOT"
-fi
-
 TEMP_DIR=$(as_mastodon mktemp -d)
 
 echo "Backing up Postgres..."

scripts/customize.sh

@@ -0,0 +1,94 @@
+#!/bin/bash
+
+# exit when any step fails
+set -euo pipefail
+
+# initialize path (only if needed)
+if [ "${MASTODON_INIT_RUN:=}" != true ]; then
+  . "$(dirname "$(realpath "$0")")"/../init.sh
+fi
+
+# re-detect glitch-soc
+MASTODON_IS_GLITCH="$(test -d "$APP_ROOT/app/javascript/flavours/glitch" && echo true || echo false)"
+
+# ---
+
+# apply custom patches (skips errors)
+for f in "$UTILS_ROOT"/patches/*.patch; do
+  as_mastodon git apply --reject --allow-binary-replacement "$f" || true
+done
+
+# apply additional glitch-only patches if applicable
+if [ "$MASTODON_IS_GLITCH" = true ]; then
+  for f in "$UTILS_ROOT"/patches/glitch/*.patch; do
+    as_mastodon git apply --reject --allow-binary-replacement "$f" || true
+  done
+fi
+
+# ---
+
+# remove this list of unused glitch cruft
+if [ "$MASTODON_IS_GLITCH" = true ]; then
+  removePaths=(
+    "app/javascript/images/clippy_frame.png"
+    "app/javascript/images/clippy_wave.gif"
+    "app/javascript/images/icon_*.png"
+    "app/javascript/images/start.png"
+    "app/javascript/skins/vanilla/win95/"
+    "app/javascript/styles/win95.scss"
+    "public/background-cybre.png"
+    "public/clock.js"
+    "public/logo-cybre-glitch.gif"
+    "public/riot-glitch.png"
+  )
+
+  for f in "${removePaths[@]}"; do
+    as_mastodon rm -rf --preserve-root "$APP_ROOT/$f"
+  done
+fi
+
+# ---
+
+# apply a more restrictive robots.txt
+as_mastodon tee "$APP_ROOT/public/robots.txt" > /dev/null <<EOT
+# block everything except About page
+User-agent: *
+Allow: /about
+Disallow: /
+
+# sorry, Elon :)
+User-agent: Twitterbot
+Disallow: /
+EOT
+
+# ---
+
+# change default settings, mostly for logged-out visitors
+# requires yq (`snap install yq` - it's like jq but for yaml: https://github.com/mikefarah/yq/#install)
+if command -v yq >/dev/null 2>&1; then
+  as_mastodon yq -i '.defaults.site_title = "Mastodon"' "$APP_ROOT/config/settings.yml"
+  as_mastodon yq -i '.defaults.show_application = true' "$APP_ROOT/config/settings.yml"
+fi
+
+# ---
+
+# change glitch-only settings located in a JS file, also mostly for logged-out visitors
+if [ "$MASTODON_IS_GLITCH" = true ]; then
+  set_default() {
+    as_mastodon sed \
+      -e "s/$1\s*:\s*.*/$1: $2, \/\/ updated by customize.sh/g" \
+      -i "$APP_ROOT/app/javascript/flavours/glitch/reducers/local_settings.js"
+  }
+
+  set_default "show_reply_count" "true"
+  set_default "hicolor_privacy_icons" "true"
+  set_default "rewrite_mentions" "'acct'"
+  set_default "lengthy" "false"
+  set_default "letterbox" "false"
+  set_default "language" "false"
+fi
+
+# ---
+
+# create a blank 'custom.css' to save a request to the backend, assuming it's not being used
+as_mastodon touch "$APP_ROOT/public/custom.css"

scripts/install.sh

@@ -23,10 +23,10 @@ if [ -d "$APP_ROOT" ]; then
 fi
 
 # ask for required info up-front
-read -p "Server FQDN? " MASTODON_DOMAIN
+read -rp "Server FQDN? " MASTODON_DOMAIN
-read -p "Public domain? (the second part of usernames, usually the same as FQDN) " MASTODON_USERNAME_DOMAIN
+read -rp "Public domain? (the second part of usernames, usually the same as FQDN) " MASTODON_USERNAME_DOMAIN
-read -p "Admin username? " MASTODON_ADMIN_USERNAME
+read -rp "Admin username? " MASTODON_ADMIN_USERNAME
-read -p "Admin email? " MASTODON_ADMIN_EMAIL
+read -rp "Admin email? " MASTODON_ADMIN_EMAIL
 
 # leave our mark
 INSTALLER_WUZ_HERE="# Generated by mastodon-installer @ $(date)"
@@ -44,7 +44,7 @@ if ! id -u "$MASTODON_USER" >/dev/null 2>&1; then
   sudo chown -R "$MASTODON_USER":"$MASTODON_USER" "$MASTODON_ROOT"
 fi
 
-# install latest ubuntu updates
+# install latest ubuntu updates & basic prerequisites
 sudo DEBIAN_FRONTEND=noninteractive apt-get update
 sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
   curl \
@@ -52,6 +52,7 @@ sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
   gnupg \
   apt-transport-https \
   lsb-release \
+  git \
   ca-certificates
 
 # add official postgresql apt repository
@@ -70,39 +71,39 @@ echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/nginx
 # https://docs.joinmastodon.org/admin/install/#system-packages
 sudo DEBIAN_FRONTEND=noninteractive apt-get update
 sudo DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-  git-core \
-  g++ \
-  libpq-dev \
-  libxml2-dev \
-  libxslt1-dev \
-  imagemagick \
-  redis-server \
-  redis-tools \
-  postgresql \
-  postgresql-contrib \
-  libidn11-dev \
-  libicu-dev \
-  libreadline6-dev \
   autoconf \
   bison \
   build-essential \
   ffmpeg \
   file \
+  g++ \
   gcc \
+  imagemagick \
+  libaugeas-dev \
   libffi-dev \
   libgdbm-dev \
+  libicu-dev \
+  libidn11-dev \
   libjemalloc-dev \
-  libncurses5-dev \
+  libncurses-dev \
+  libpq-dev \
   libprotobuf-dev \
+  libreadline-dev \
   libssl-dev \
+  libxml2-dev \
+  libxslt1-dev \
   libyaml-dev \
-  pkg-config \
-  protobuf-compiler \
-  zlib1g-dev \
   nginx \
+  pkg-config \
+  postgresql \
+  postgresql-contrib \
+  protobuf-compiler \
   python3 \
   python3-venv \
-  libaugeas0
+  redis-server \
+  redis-tools \
+  shared-mime-info \
+  zlib1g-dev
 
 # install rbenv & ruby-build
 # https://github.com/rbenv/rbenv#basic-git-checkout
@@ -118,21 +119,13 @@ as_mastodon git clone https://github.com/nvm-sh/nvm.git "$NVM_DIR"
 as_mastodon git clone https://github.com/mastodon/mastodon.git "$APP_ROOT" && cd "$APP_ROOT"
 as_mastodon git config --global --add safe.directory "$APP_ROOT"
 as_mastodon git checkout "$(as_mastodon git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)"
-# uncomment to install glitch-soc fork:
+# uncomment (and keep above lines) to install glitch-soc fork:
 # as_mastodon git remote add glitch-soc https://github.com/glitch-soc/mastodon
 # as_mastodon git fetch --all
 # as_mastodon git checkout glitch-soc/main
 
-# apply custom patches (skips errors):
+# apply customizations
-for PATCH in "$UTILS_ROOT"/patches/*.patch; do
+. "$UTILS_ROOT"/scripts/customize.sh
-  as_mastodon git apply --reject --allow-binary-replacement "$PATCH" || true
-done
-# apply additional glitch-only patches if applicable:
-if [ -d "$APP_ROOT/app/javascript/flavours/glitch" ]; then
-  for PATCH in "$UTILS_ROOT"/patches/glitch/*.patch; do
-    as_mastodon git apply --reject --allow-binary-replacement "$PATCH" || true
-  done
-fi
 
 # install ruby
 as_mastodon RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install --skip-existing
@@ -215,7 +208,7 @@ STREAMING_CLUSTER_NUM=1
 # STATSD_ADDR=localhost:9125" | as_mastodon tee "$APP_ROOT/.env.production" >/dev/null
 
 # manually setup db
-as_mastodon RAILS_ENV=production bundle exec rails db:setup
+as_mastodon RAILS_ENV=production SAFETY_ASSURED=1 bundle exec rails db:setup
 
 # precompile assets
 as_mastodon RAILS_ENV=production bundle exec rails assets:precompile
@@ -276,10 +269,15 @@ tootctl accounts create \
   --role Owner \
   --confirmed
 
-# set cleanup tasks to run weekly
+# create directory for cron logdrain
+as_mastodon mkdir -p "$LOGS_ROOT"
+as_mastodon touch "$LOGS_ROOT"/cron.log
+
+# set cleanup & backup tasks to run weekly
 # https://docs.joinmastodon.org/admin/setup/#cleanup
 (sudo crontab -l; echo -e "\n$INSTALLER_WUZ_HERE
 @weekly  bash -c \"$UTILS_ROOT/scripts/weekly_cleanup.sh >> $LOGS_ROOT/cron.log 2>&1\"
+@weekly  bash -c \"$UTILS_ROOT/scripts/backup.sh >> $LOGS_ROOT/cron.log 2>&1\"
 ") | sudo crontab -
 
 echo "🎉 done! don't forget to fill in .env.production with optional credentials"

scripts/upgrade.sh

@@ -19,7 +19,7 @@ fi
 # pull latest mastodon source
 cd "$APP_ROOT"
 as_mastodon git fetch --all
-as_mastodon git stash push --message "pre-upgrade changes"
+as_mastodon git stash push --include-untracked --message "pre-upgrade changes"
 if [ -d "$APP_ROOT/app/javascript/flavours/glitch" ]; then
   # glitch-soc (uses latest commits)
   echo "Pulling latest glitch-soc commits..."
@@ -30,16 +30,8 @@ else
   as_mastodon git checkout "$(as_mastodon git tag -l | grep -v 'rc[0-9]*$' | sort -V | tail -n 1)"
 fi
 
-# apply custom patches (skips errors):
+# re-apply customizations
-for PATCH in "$UTILS_ROOT"/patches/*.patch; do
+. "$UTILS_ROOT"/scripts/customize.sh
-  as_mastodon git apply --reject --allow-binary-replacement "$PATCH" || true
-done
-# apply additional glitch-only patches if applicable:
-if [ -d "$APP_ROOT/app/javascript/flavours/glitch" ]; then
-  for PATCH in "$UTILS_ROOT"/patches/glitch/*.patch; do
-    as_mastodon git apply --reject --allow-binary-replacement "$PATCH" || true
-  done
-fi
 
 # set new ruby version
 as_mastodon RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install --skip-existing

scripts/weekly_cleanup.sh

@@ -5,18 +5,15 @@
 #   0 3 * * 0  bash -c "/home/mastodon/utils/scripts/weekly_cleanup.sh >> /home/mastodon/logs/cron.log 2>&1"
 
 # exit when any step fails
-set -euo pipefail
+set -o pipefail
 
 echo -e "\n===== weekly_cleanup.sh: started at $(date '+%Y-%m-%d %H:%M:%S') =====\n"
 
 # initialize path
 . "$(dirname "$(realpath "$0")")"/../init.sh
 
-if [ ! -d "$LOGS_ROOT" ]; then
+tootctl media remove --days 14
-  as_mastodon mkdir -p "$LOGS_ROOT"
+tootctl media remove --prune-profiles --days 90
-fi
-
-tootctl media remove --days 7
 tootctl preview_cards remove --days 90
 
 echo -e "\n===== weekly_cleanup.sh: finished at $(date '+%Y-%m-%d %H:%M:%S') =====\n"