jake/mastodon-utils
1
Fork 0
mirror of https://github.com/jakejarvis/mastodon-utils.git synced 2025-04-25 22:45:22 -04:00
Code Issues Releases Wiki Activity

move notes to GitHub wiki

Browse Source
This commit is contained in:
Jake Jarvis 2022-12-11 10:27:20 -05:00
parent 89ccd7fd2a
commit 479ce2c774
Signed by: jake
GPG Key ID: 2B0C9CF251E69A39
4 changed files with 9 additions and 509 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@ -2,9 +2,17 @@
Random opinionated helper scripts & front-end customizations for my [personal Mastodon instance](https://fediverse.jarv.is/about) (running on [`glitch-soc`](https://github.com/glitch-soc/mastodon)). You definitely don't want to use any of this as-is — check out my more general purpose [mastodon-installer](https://github.com/jakejarvis/mastodon-installer) scripts instead.
## Notes
The [wiki of this repo](https://github.com/jakejarvis/mastodon-scripts/wiki) and the [`/etc` folder](etc/) are simply my way of not forgetting how I did something, which I do quite a bit. Refer there for random notes on PgBouncer, Grafana, etc. but **DO NOT BLINDLY COPY & PASTE** anything there without doing your own research!
- [Grafana & Prometheus](https://github.com/jakejarvis/mastodon-scripts/wiki/Prometheus-&-Grafana)
- [PgBouncer](https://github.com/jakejarvis/mastodon-scripts/wiki/Postgres-&-PgBouncer)
- [Brotli compression](https://github.com/jakejarvis/mastodon-scripts/wiki/Brotli-compression-for-nginx)
## Usage
**AGAIN, DEFINITELY DO NOT JUST RUN THIS IF YOU'RE NOT ME!!! 😊**
***AGAIN, DEFINITELY DO NOT JUST RUN THIS IF YOU'RE NOT ME!!! 😊***
```sh
git clone https://github.com/jakejarvis/mastodon-scripts.git /home/mastodon/scripts

70
notes/brotli.md
View File

@ -1,70 +0,0 @@
# Brotli compression for nginx
- https://github.com/google/ngx_brotli
- https://www.atlantic.net/dedicated-server-hosting/how-to-install-brotli-module-for-nginx-on-ubuntu-20-04/
- https://linuxhint.com/enable-brotli-compression-nginx/
- https://www.bowsercache.com/blog/enable-brotli-for-nginx-on-ubuntu-20-04/#install-the-brotli-module-for-nginx
---
/etc/apt/sources.list.d/nginx.list:
```
deb [arch=amd64 signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/ubuntu/ focal nginx
deb-src [arch=amd64 signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/ubuntu/ focal nginx
```
---
```bash
cd /usr/local/src
apt-get source nginx
apt-get build-dep nginx -y
git clone --recursive https://github.com/google/ngx_brotli
cd nginx-1.22.1/
./configure --with-compat --add-dynamic-module=../ngx_brotli
make modules
cp ./objs/ngx_http_brotli_*.so /usr/lib/nginx/modules/
```
---
/etc/nginx/nginx.conf:
```
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;
```
---
nginx site config: ([ref](https://github.com/google/ngx_brotli#sample-configuration))
```
server {
# ...
brotli on;
brotli_comp_level 4;
brotli_static on;
brotli_types application/atom+xml application/javascript application/json application/rss+xml
application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype
application/x-font-ttf application/x-javascript application/xhtml+xml application/xml
font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon
image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml;
brotli_min_length 256;
# ...
}
```
---
```bash
nginx -t
nginx -s reload
```

99
notes/postgres.md
View File

@ -1,99 +0,0 @@
# Postgres
## Optimization
- https://pgtune.leopard.in.ua/#/
### PgBouncer
- https://docs.joinmastodon.org/admin/scaling/#pgbouncer
- https://masto.host/mastodon-pgbouncer-guide/
#### Installation
creating the pgbouncer admin user:
```bash
DB_PASSWORD=$(< /dev/urandom tr -dc A-Za-z0-9 | head -c32; echo)
echo "pgbouncer password (save this securely): $DB_PASSWORD"
echo "CREATE USER pgbouncer WITH PASSWORD '$DB_PASSWORD' CREATEDB" | sudo -u postgres psql -f -
```
#### Running database migrations
Mastodon `db:migrate`s should be pointed directly at Postgres (default port: 5432), ***not through PgBouncer***, by overriding `DB_PORT` env variable.
```bash
RAILS_ENV=production DB_PORT=5432 bundle exec rails db:migrate
```
#### Config
.env.production:
```sh
DB_HOST=localhost
DB_USER=mastodon
DB_NAME=mastodon_production
DB_PASS=
# change from postgres port (default: 5432) to pgbouncer (default: 6432)
DB_PORT=6432
# add this:
PREPARED_STATEMENTS=false
```
---
/etc/pgbouncer/pgbouncer.ini:
```ini
[databases]
mastodon_production = host=127.0.0.1 port=5432 dbname=mastodon_production user=mastodon password=
[pgbouncer]
listen_addr = localhost
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = pgbouncer
pool_mode = transaction
max_client_conn = 100
default_pool_size = 20
```
---
/etc/pgbouncer/userlist.txt:
generate md5 hash of postgres passwords with `echo -n "pass" | md5sum`
```
"mastodon" "md5xxxxxxxx"
"pgbouncer" "md5xxxxxxxx"
```
## Connecting from TablePlus.app via Tailscale
Connect directly to Postgres (default port: 5432), ***not via PgBouncer!***
---
/etc/postgresql/15/main/postgres.conf
```
listen_addresses = '*'
```
---
/etc/postgresql/15/main/pg_hba.conf:
```
# tailscale
host all all 100.64.0.0/10 md5
```
---
![](https://user-images.githubusercontent.com/1703673/206910912-1dea1173-7090-47db-b964-1b4bbe0d197e.png)

339
notes/prometheus.md
View File

@ -1,339 +0,0 @@
# Prometheus & Grafana notes
- https://grafana.pipe.fail/public-dashboards/b5ca7a7c8e844f90b0973d2ab02bad0a
- https://ipng.ch/s/articles/2022/11/27/mastodon-3.html
- https://ourcodeworld.com/articles/read/1686/how-to-install-prometheus-node-exporter-on-ubuntu-2004
## Exporters
- https://github.com/prometheus-community/postgres_exporter
- https://github.com/prometheus/statsd_exporter
- https://github.com/oliver006/redis_exporter
- https://github.com/prometheus/node_exporter
- https://github.com/nginxinc/nginx-prometheus-exporter
- https://github.com/prometheus-community/json_exporter
## Installation
repeat for each exporter:
```bash
wget https://github.com/oliver006/redis_exporter/releases/download/v1.45.0/redis_exporter-v1.45.0.linux-amd64.tar.gz
tar xvf redis_exporter-v1.45.0.linux-amd64.tar.gz
cp redis_exporter-v1.45.0.linux-amd64/redis_exporter /usr/local/bin/
useradd --no-create-home --shell /bin/false redis_exporter
chown redis_exporter:redis_exporter /usr/local/bin/redis_exporter
nano /etc/system/systemd/redis-exporter.service # see below
systemctl daemon-reload
systemctl enable --now redis-exporter.service
systemctl status redis-exporter.service
```
## Config
/home/mastodon/live/.env.production:
```sh
STATSD_ADDR=localhost:9125
```
---
/etc/prometheus/prometheus.yml:
```yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node_exporter_metrics"
static_configs:
- targets: ["localhost:9100"]
- job_name: "redis_exporter_targets"
static_configs:
- targets: ["redis://localhost:6379"]
metrics_path: /scrape
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9121
- job_name: "redis_exporter_metrics"
static_configs:
- targets: ["localhost:9121"]
- job_name: "postgres_exporter_metrics"
static_configs:
- targets: ["localhost:9187"]
- job_name: "nginx_exporter_metrics"
static_configs:
- targets: ["localhost:9113"]
- job_name: "statsd_exporter_metrics"
static_configs:
- targets: ["localhost:9102"]
- job_name: "elasticsearch_exporter_metrics"
static_configs:
- targets: ["localhost:9114"]
- job_name: "json_exporter_metrics"
static_configs:
- targets: ["localhost:9079"]
- job_name: "json_exporter_targets"
metrics_path: /probe
scrape_interval: 30s
params:
module: [linode_bucket]
static_configs:
- targets:
- https://api.linode.com/v4/object-storage/buckets/us-east-1/jarvis-mastodon
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9079
- job_name: "json_exporter_targets"
metrics_path: /probe
scrape_interval: 30s
params:
module: [linode_transfer]
static_configs:
- targets:
- https://api.linode.com/v4/account/transfer
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9079
```
---
/etc/prometheus/json-config.yml:
```yml
modules:
linode_bucket:
headers:
# https://cloud.linode.com/profile/tokens
Authorization: "Bearer XXXXXX"
metrics:
- name: json_linode_size
path: "{.size}"
labels:
bucket: "{.label}"
zone: "{.cluster}"
hostname: "{.hostname}"
- name: json_linode_objects
path: "{.objects}"
labels:
bucket: "{.label}"
zone: "{.cluster}"
hostname: "{.hostname}"
linode_transfer:
headers:
# https://cloud.linode.com/profile/tokens
Authorization: "Bearer XXXXXX"
metrics:
- name: json_linode_transfer_used
path: "{.used}"
- name: json_linode_transfer_quota
path: "{.quota}"
```
---
/etc/prometheus/statsd-mapping.yml:
```yml
## Prometheus Statsd Exporter mapping for Mastodon 4.0+
##
## Version 1.0, November 2022
##
## Documentation: https://ipng.ch/s/articles/2022/11/27/mastodon-3.html
mappings:
## Web collector
- match: Mastodon\.production\.web\.(.+)\.(.+)\.(.+)\.status\.(.+)
match_type: regex
name: "mastodon_controller_status"
labels:
controller: $1
action: $2
format: $3
status: $4
mastodon: "web"
- match: Mastodon\.production\.web\.(.+)\.(.+)\.(.+)\.db_time
match_type: regex
name: "mastodon_controller_db_time"
labels:
controller: $1
action: $2
format: $3
mastodon: "web"
- match: Mastodon\.production\.web\.(.+)\.(.+)\.(.+)\.view_time
match_type: regex
name: "mastodon_controller_view_time"
labels:
controller: $1
action: $2
format: $3
mastodon: "web"
- match: Mastodon\.production\.web\.(.+)\.(.+)\.(.+)\.total_duration
match_type: regex
name: "mastodon_controller_duration"
labels:
controller: $1
action: $2
format: $3
mastodon: "web"
## Database collector
- match: Mastodon\.production\.db\.tables\.(.+)\.queries\.(.+)\.duration
match_type: regex
name: "mastodon_db_operation"
labels:
table: "$1"
operation: "$2"
mastodon: "db"
## Cache collector
- match: Mastodon\.production\.cache\.(.+)\.duration
match_type: regex
name: "mastodon_cache_duration"
labels:
operation: "$1"
mastodon: "cache"
## Sidekiq collector
- match: Mastodon\.production\.sidekiq\.(.+)\.processing_time
match_type: regex
name: "mastodon_sidekiq_worker_processing_time"
labels:
worker: "$1"
mastodon: "sidekiq"
- match: Mastodon\.production\.sidekiq\.(.+)\.success
match_type: regex
name: "mastodon_sidekiq_worker_success_total"
labels:
worker: "$1"
mastodon: "sidekiq"
- match: Mastodon\.production\.sidekiq\.(.+)\.failure
match_type: regex
name: "mastodon_sidekiq_worker_failure_total"
labels:
worker: "$1"
mastodon: "sidekiq"
- match: Mastodon\.production\.sidekiq\.queues\.(.+)\.enqueued
match_type: regex
name: "mastodon_sidekiq_queue_enqueued"
labels:
queue: "$1"
mastodon: "sidekiq"
- match: Mastodon\.production\.sidekiq\.queues\.(.+)\.latency
match_type: regex
name: "mastodon_sidekiq_queue_latency"
labels:
queue: "$1"
mastodon: "sidekiq"
- match: Mastodon\.production\.sidekiq\.(.+)
match_type: regex
name: "mastodon_sidekiq_$1"
labels:
mastodon: "sidekiq"
```
---
(example) /etc/systemd/system/redis-exporter.service:
```
[Unit]
Description=Redis Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=redis_exporter
Group=redis_exporter
Type=simple
ExecStart=/usr/local/bin/redis_exporter
Restart=always
[Install]
WantedBy=multi-user.target
```
---
/etc/grafana/grafana.ini:
```ini
[server]
http_addr =
http_port = 3003
root_url = https://grafana.pipe.fail
[analytics]
reporting_enabled = false
check_for_updates = false
check_for_plugin_updates = false
feedback_links_enabled = false
[security]
disable_initial_admin_creation = true
disable_gravatar = true
cookie_secure = true
[snapshots]
external_enabled = false
[dashboards]
versions_to_keep = 100
[users]
allow_sign_up = false
default_theme = dark
[auth]
disable_login = true
disable_login_form = true
[auth.grafana_com]
enabled = true
allow_sign_up = false
client_id =
client_secret =
scopes = user:email
allowed_organizations =
[metrics]
enabled = false
[live]
max_connections = 10
[feature_toggles]
publicDashboards = true
```