You've already forked domainstack.io
mirror of
https://github.com/jakejarvis/domainstack.io.git
synced 2025-12-02 19:33:48 -05:00
refactor: remove Redis integration and related rate limiting logic (for now)
This commit is contained in:
@@ -4,13 +4,9 @@ NEXT_PUBLIC_POSTHOG_KEY=
|
|||||||
POSTHOG_API_KEY=
|
POSTHOG_API_KEY=
|
||||||
POSTHOG_ENV_ID=
|
POSTHOG_ENV_ID=
|
||||||
|
|
||||||
# Postgres credentials (set by PlanetScale integration)
|
# Postgres connection string (with credentials)
|
||||||
DATABASE_URL=
|
DATABASE_URL=
|
||||||
|
|
||||||
# Redis credentials (set by Upstash integration)
|
|
||||||
KV_REST_API_TOKEN=
|
|
||||||
KV_REST_API_URL=
|
|
||||||
|
|
||||||
# Inngest credentials (set by Inngest integration)
|
# Inngest credentials (set by Inngest integration)
|
||||||
INNGEST_EVENT_KEY=
|
INNGEST_EVENT_KEY=
|
||||||
INNGEST_SIGNING_KEY=
|
INNGEST_SIGNING_KEY=
|
||||||
@@ -28,8 +24,5 @@ BLOB_READ_WRITE_TOKEN=
|
|||||||
# Blob key hashing secret (required outside development)
|
# Blob key hashing secret (required outside development)
|
||||||
BLOB_SIGNING_SECRET=
|
BLOB_SIGNING_SECRET=
|
||||||
|
|
||||||
# Vercel Cron secret for authenticating scheduled tasks
|
|
||||||
CRON_SECRET=
|
|
||||||
|
|
||||||
# Optional: override user agent sent with upstream requests
|
# Optional: override user agent sent with upstream requests
|
||||||
EXTERNAL_USER_AGENT=
|
EXTERNAL_USER_AGENT=
|
||||||
|
|||||||
@@ -16,7 +16,7 @@
|
|||||||
- `trpc/` tRPC client setup, query client, and error handling.
|
- `trpc/` tRPC client setup, query client, and error handling.
|
||||||
|
|
||||||
## Build, Test, and Development Commands
|
## Build, Test, and Development Commands
|
||||||
- `pnpm dev` — start all local services (Postgres, Redis, SRH, Inngest) and Next.js dev server at http://localhost:3000 using `concurrently`.
|
- `pnpm dev` — start all local services (Postgres, Inngest, etc.) and Next.js dev server at http://localhost:3000 using `concurrently`.
|
||||||
- `pnpm build` — compile production bundle.
|
- `pnpm build` — compile production bundle.
|
||||||
- `pnpm start` — serve compiled output for smoke tests.
|
- `pnpm start` — serve compiled output for smoke tests.
|
||||||
- `pnpm lint` — run Biome lint + type-aware checks (`--write` to fix).
|
- `pnpm lint` — run Biome lint + type-aware checks (`--write` to fix).
|
||||||
@@ -71,17 +71,15 @@
|
|||||||
- Keep secrets in `.env.local`. See `.env.example` for required variables.
|
- Keep secrets in `.env.local`. See `.env.example` for required variables.
|
||||||
- Vercel Edge Config provides dynamic, low-latency configuration without redeployment:
|
- Vercel Edge Config provides dynamic, low-latency configuration without redeployment:
|
||||||
- `domain_suggestions` (array): Homepage domain suggestions; fails gracefully to empty array
|
- `domain_suggestions` (array): Homepage domain suggestions; fails gracefully to empty array
|
||||||
- `rate_limits` (object): Service rate limits; **fails open** (no limits) if unavailable for maximum availability
|
|
||||||
- Edge Config is completely optional and gracefully degrades when not configured
|
- Edge Config is completely optional and gracefully degrades when not configured
|
||||||
- Uses Next.js 16 `"use cache"` directive with `@vercel/edge-config` SDK for SSR compatibility
|
- Uses Next.js 16 `"use cache"` directive with `@vercel/edge-config` SDK for SSR compatibility
|
||||||
- Vercel Blob backs favicon/screenshot storage with automatic public URLs; metadata cached in Postgres.
|
- Vercel Blob backs favicon/screenshot storage with automatic public URLs; metadata cached in Postgres.
|
||||||
- Screenshots (Puppeteer): prefer `puppeteer-core` + `@sparticuz/chromium` on Vercel.
|
- Screenshots (Puppeteer): prefer `puppeteer-core` + `@sparticuz/chromium` on Vercel.
|
||||||
- Persist domain data in Postgres via Drizzle with per-table TTL columns (`expiresAt`).
|
- Persist domain data in Postgres via Drizzle with per-table TTL columns (`expiresAt`).
|
||||||
- **Redis usage**: ONLY for IP-based rate limiting via `@upstash/ratelimit`. All other caching uses Next.js Data Cache (`unstable_cache`) or Postgres.
|
- All caching uses Next.js Data Cache (`unstable_cache`) or Postgres.
|
||||||
- Database connections: Use Vercel's Postgres connection pooling (`@vercel/postgres`) for optimal performance.
|
- Database connections: Use Vercel's Postgres connection pooling (`@vercel/postgres`) for optimal performance.
|
||||||
- Background revalidation: Event-driven via Inngest functions in `lib/inngest/functions/` with built-in concurrency control.
|
- Background revalidation: Event-driven via Inngest functions in `lib/inngest/functions/` with built-in concurrency control.
|
||||||
- Use Next.js 16 `after()` for fire-and-forget background operations (analytics, domain access tracking) with graceful degradation.
|
- Use Next.js 16 `after()` for fire-and-forget background operations (analytics, domain access tracking) with graceful degradation.
|
||||||
- Cron jobs trigger Inngest events via `app/api/cron/` endpoints secured with `CRON_SECRET`.
|
|
||||||
- Review `trpc/init.ts` when extending procedures to ensure auth/context remain intact.
|
- Review `trpc/init.ts` when extending procedures to ensure auth/context remain intact.
|
||||||
|
|
||||||
## Analytics & Observability
|
## Analytics & Observability
|
||||||
|
|||||||
@@ -10,8 +10,8 @@
|
|||||||
- **SEO insights**: Extract titles, meta tags, social previews, canonical data, and `robots.txt` signals.
|
- **SEO insights**: Extract titles, meta tags, social previews, canonical data, and `robots.txt` signals.
|
||||||
- **Screenshots & favicons**: Server-side screenshots and favicon extraction, cached in Postgres with Vercel Blob storage.
|
- **Screenshots & favicons**: Server-side screenshots and favicon extraction, cached in Postgres with Vercel Blob storage.
|
||||||
- **Fast, private, no sign-up**: Live fetches with intelligent multi-layer caching.
|
- **Fast, private, no sign-up**: Live fetches with intelligent multi-layer caching.
|
||||||
- **Reliable data pipeline**: Postgres persistence with per-table TTLs (Drizzle), event-driven background revalidation (Inngest), and Redis for short-lived rate limiting.
|
- **Reliable data pipeline**: Postgres persistence with per-table TTLs (Drizzle) and event-driven background revalidation (Inngest).
|
||||||
- **Dynamic configuration**: Vercel Edge Config for runtime-adjustable rate limits and domain suggestions without redeployment.
|
- **Dynamic configuration**: Vercel Edge Config for runtime-adjustable domain suggestions without redeployment.
|
||||||
|
|
||||||
## 🛠️ Tech Stack
|
## 🛠️ Tech Stack
|
||||||
|
|
||||||
@@ -20,8 +20,7 @@
|
|||||||
- **tRPC** API
|
- **tRPC** API
|
||||||
- **PlanetScale Postgres** + **Drizzle ORM** with connection pooling
|
- **PlanetScale Postgres** + **Drizzle ORM** with connection pooling
|
||||||
- **Inngest** for event-driven background revalidation with built-in concurrency control
|
- **Inngest** for event-driven background revalidation with built-in concurrency control
|
||||||
- **Upstash Redis** for IP-based rate limiting
|
- **Vercel Edge Config** for runtime configuration (domain suggestions)
|
||||||
- **Vercel Edge Config** for runtime configuration (domain suggestions, service rate limits)
|
|
||||||
- **Vercel Blob** for favicon/screenshot storage with Postgres metadata caching
|
- **Vercel Blob** for favicon/screenshot storage with Postgres metadata caching
|
||||||
- [**rdapper**](https://github.com/jakejarvis/rdapper) for RDAP lookups with WHOIS fallback
|
- [**rdapper**](https://github.com/jakejarvis/rdapper) for RDAP lookups with WHOIS fallback
|
||||||
- **Puppeteer** (with `@sparticuz/chromium` on Vercel) for server-side screenshots
|
- **Puppeteer** (with `@sparticuz/chromium` on Vercel) for server-side screenshots
|
||||||
@@ -66,8 +65,6 @@ pnpm dev
|
|||||||
|
|
||||||
This single command boots:
|
This single command boots:
|
||||||
- **Postgres** on `localhost:5432`
|
- **Postgres** on `localhost:5432`
|
||||||
- **Redis** on `localhost:6379`
|
|
||||||
- **Serverless Redis HTTP (SRH)** on `http://localhost:8079` (Upstash-compatible REST proxy)
|
|
||||||
- **Inngest dev server** on `http://localhost:8288`
|
- **Inngest dev server** on `http://localhost:8288`
|
||||||
- **Next.js dev server** on `http://localhost:3000`
|
- **Next.js dev server** on `http://localhost:3000`
|
||||||
|
|
||||||
|
|||||||
@@ -8,28 +8,6 @@ const handler = (req: Request) =>
|
|||||||
req,
|
req,
|
||||||
router: appRouter,
|
router: appRouter,
|
||||||
createContext: () => createContext({ req }),
|
createContext: () => createContext({ req }),
|
||||||
responseMeta: ({ errors }) => {
|
|
||||||
const err = errors.find((e) => e.code === "TOO_MANY_REQUESTS");
|
|
||||||
if (!err) return {};
|
|
||||||
|
|
||||||
// Prefer formatted data from errorFormatter for consistent headers
|
|
||||||
const data = (
|
|
||||||
err as {
|
|
||||||
data?: { retryAfter?: number; limit?: number; remaining?: number };
|
|
||||||
}
|
|
||||||
).data;
|
|
||||||
const retryAfter = Math.max(1, Math.round(Number(data?.retryAfter ?? 1)));
|
|
||||||
const headers: Record<string, string> = {
|
|
||||||
"Retry-After": String(retryAfter),
|
|
||||||
"Cache-Control": "no-cache, no-store",
|
|
||||||
};
|
|
||||||
if (typeof data?.limit === "number")
|
|
||||||
headers["X-RateLimit-Limit"] = String(data.limit);
|
|
||||||
if (typeof data?.remaining === "number")
|
|
||||||
headers["X-RateLimit-Remaining"] = String(data.remaining);
|
|
||||||
|
|
||||||
return { headers, status: 429 };
|
|
||||||
},
|
|
||||||
onError: ({ path, error }) => {
|
onError: ({ path, error }) => {
|
||||||
console.error(`[trpc] unhandled error ${path}`, error);
|
console.error(`[trpc] unhandled error ${path}`, error);
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -16,25 +16,6 @@ services:
|
|||||||
timeout: 5s
|
timeout: 5s
|
||||||
retries: 5
|
retries: 5
|
||||||
|
|
||||||
# Local Redis (binary protocol on 6379)
|
|
||||||
redis:
|
|
||||||
image: redis:7-alpine
|
|
||||||
ports:
|
|
||||||
- "6379:6379"
|
|
||||||
|
|
||||||
# Serverless Redis HTTP (SRH) — Upstash-compatible REST proxy on 8079
|
|
||||||
# Set KV_REST_API_URL to http://localhost:8079
|
|
||||||
srh:
|
|
||||||
image: hiett/serverless-redis-http:latest
|
|
||||||
environment:
|
|
||||||
SRH_MODE: "env"
|
|
||||||
SRH_TOKEN: "dev-token"
|
|
||||||
SRH_CONNECTION_STRING: "redis://redis:6379"
|
|
||||||
ports:
|
|
||||||
- "8079:80"
|
|
||||||
depends_on:
|
|
||||||
- redis
|
|
||||||
|
|
||||||
# Inngest Dev Server on 8288
|
# Inngest Dev Server on 8288
|
||||||
# Next.js runs on the HOST at :3000; the dev server reaches it via host.docker.internal
|
# Next.js runs on the HOST at :3000; the dev server reaches it via host.docker.internal
|
||||||
inngest:
|
inngest:
|
||||||
|
|||||||
@@ -1,55 +1,6 @@
|
|||||||
"use cache";
|
"use cache";
|
||||||
|
|
||||||
import { get } from "@vercel/edge-config";
|
import { get } from "@vercel/edge-config";
|
||||||
import type { ServiceLimits } from "@/lib/ratelimit";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Fetches rate limits from Vercel Edge Config.
|
|
||||||
*
|
|
||||||
* FAILS OPEN: Returns null if Edge Config is not configured or fails,
|
|
||||||
* completely disabling rate limiting for maximum availability.
|
|
||||||
*
|
|
||||||
* Edge Config key: `rate_limits`
|
|
||||||
*
|
|
||||||
* Expected schema:
|
|
||||||
* ```json
|
|
||||||
* {
|
|
||||||
* "rate_limits": {
|
|
||||||
* "dns": { "points": 60, "window": "1 m" },
|
|
||||||
* "headers": { "points": 60, "window": "1 m" },
|
|
||||||
* "certs": { "points": 30, "window": "1 m" },
|
|
||||||
* "registration": { "points": 10, "window": "1 m" },
|
|
||||||
* "screenshot": { "points": 30, "window": "1 h" },
|
|
||||||
* "favicon": { "points": 100, "window": "1 m" },
|
|
||||||
* "seo": { "points": 30, "window": "1 m" },
|
|
||||||
* "hosting": { "points": 30, "window": "1 m" },
|
|
||||||
* "pricing": { "points": 30, "window": "1 m" }
|
|
||||||
* }
|
|
||||||
* }
|
|
||||||
* ```
|
|
||||||
*
|
|
||||||
* @returns Service rate limits or null if unavailable (fail open)
|
|
||||||
*/
|
|
||||||
export async function getRateLimits(): Promise<ServiceLimits | null> {
|
|
||||||
// If EDGE_CONFIG is not set, fail open
|
|
||||||
if (!process.env.EDGE_CONFIG) {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
try {
|
|
||||||
const limits = await get<ServiceLimits>("rate_limits");
|
|
||||||
|
|
||||||
// Return limits if they exist, otherwise null (fail open)
|
|
||||||
return limits ?? null;
|
|
||||||
} catch (error) {
|
|
||||||
// Log the error but fail open (no limits enforced)
|
|
||||||
console.warn(
|
|
||||||
"[edge-config] failed to fetch rate limits, failing open (no limits)",
|
|
||||||
error instanceof Error ? error.message : String(error),
|
|
||||||
);
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Fetches the default domain suggestions from Vercel Edge Config.
|
* Fetches the default domain suggestions from Vercel Edge Config.
|
||||||
|
|||||||
110
lib/ratelimit.ts
110
lib/ratelimit.ts
@@ -1,110 +0,0 @@
|
|||||||
import "server-only";
|
|
||||||
|
|
||||||
import { TRPCError } from "@trpc/server";
|
|
||||||
import { Ratelimit } from "@upstash/ratelimit";
|
|
||||||
import { Redis } from "@upstash/redis";
|
|
||||||
import { after } from "next/server";
|
|
||||||
import { getRateLimits } from "@/lib/edge-config";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Rate limit configuration for each service.
|
|
||||||
*/
|
|
||||||
export type RateLimitConfig = {
|
|
||||||
points: number;
|
|
||||||
window: `${number} ${"s" | "m" | "h"}`;
|
|
||||||
};
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Service rate limits configuration.
|
|
||||||
*
|
|
||||||
* Each service has a points budget and time window for rate limiting.
|
|
||||||
* Example: { points: 60, window: "1 m" } = 60 requests per minute
|
|
||||||
*/
|
|
||||||
export type ServiceLimits = {
|
|
||||||
dns: RateLimitConfig;
|
|
||||||
headers: RateLimitConfig;
|
|
||||||
certs: RateLimitConfig;
|
|
||||||
registration: RateLimitConfig;
|
|
||||||
screenshot: RateLimitConfig;
|
|
||||||
favicon: RateLimitConfig;
|
|
||||||
seo: RateLimitConfig;
|
|
||||||
hosting: RateLimitConfig;
|
|
||||||
pricing: RateLimitConfig;
|
|
||||||
};
|
|
||||||
export type ServiceName = keyof ServiceLimits;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Redis client for rate limiting only.
|
|
||||||
*
|
|
||||||
* Uses KV_REST_API_URL and KV_REST_API_TOKEN from Vercel KV integration.
|
|
||||||
* All other caching uses Next.js Data Cache or Postgres.
|
|
||||||
*/
|
|
||||||
export const redis = Redis.fromEnv();
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Assert that a rate limit is not exceeded for a given service and IP address.
|
|
||||||
* @param service - The service name
|
|
||||||
* @param ip - The IP address
|
|
||||||
* @returns The rate limit result
|
|
||||||
* @throws TRPCError if the rate limit is exceeded
|
|
||||||
*/
|
|
||||||
export async function assertRateLimit(
|
|
||||||
service: keyof ServiceLimits,
|
|
||||||
ip: string,
|
|
||||||
): Promise<{ limit: number; remaining: number; reset: number }> {
|
|
||||||
const limits = await getRateLimits();
|
|
||||||
|
|
||||||
// Fail open: if no limits configured or Edge Config fails, skip rate limiting
|
|
||||||
if (limits === null) {
|
|
||||||
console.info(`[ratelimit] rate limiting disabled (fail open)`);
|
|
||||||
return { limit: 0, remaining: 0, reset: 0 };
|
|
||||||
}
|
|
||||||
|
|
||||||
const cfg = limits[service];
|
|
||||||
if (!cfg) {
|
|
||||||
console.warn(`[ratelimit] no config for service ${service}, skipping`);
|
|
||||||
return { limit: 0, remaining: 0, reset: 0 };
|
|
||||||
}
|
|
||||||
|
|
||||||
const limiter = new Ratelimit({
|
|
||||||
redis,
|
|
||||||
limiter: Ratelimit.slidingWindow(cfg.points, cfg.window),
|
|
||||||
analytics: true,
|
|
||||||
});
|
|
||||||
|
|
||||||
const res = await limiter.limit(`${service}:${ip}`);
|
|
||||||
|
|
||||||
if (!res.success) {
|
|
||||||
const retryAfterSec = Math.max(
|
|
||||||
1,
|
|
||||||
Math.ceil((res.reset - Date.now()) / 1000),
|
|
||||||
);
|
|
||||||
|
|
||||||
console.warn(
|
|
||||||
`[ratelimit] blocked ${service} for ${ip} (limit=${res.limit}, retry in ${retryAfterSec}s)`,
|
|
||||||
);
|
|
||||||
|
|
||||||
throw new TRPCError({
|
|
||||||
code: "TOO_MANY_REQUESTS",
|
|
||||||
message: `Rate limit exceeded for ${service}. Try again in ${retryAfterSec}s.`,
|
|
||||||
cause: {
|
|
||||||
retryAfter: retryAfterSec,
|
|
||||||
service,
|
|
||||||
limit: res.limit,
|
|
||||||
remaining: res.remaining,
|
|
||||||
reset: res.reset,
|
|
||||||
},
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
// allow ratelimit analytics to be sent in background
|
|
||||||
try {
|
|
||||||
after(async () => {
|
|
||||||
await res.pending;
|
|
||||||
});
|
|
||||||
} catch {
|
|
||||||
// no-op
|
|
||||||
}
|
|
||||||
|
|
||||||
return { limit: res.limit, remaining: res.remaining, reset: res.reset };
|
|
||||||
}
|
|
||||||
@@ -40,8 +40,6 @@
|
|||||||
"@trpc/client": "^11.7.1",
|
"@trpc/client": "^11.7.1",
|
||||||
"@trpc/server": "^11.7.1",
|
"@trpc/server": "^11.7.1",
|
||||||
"@trpc/tanstack-react-query": "^11.7.1",
|
"@trpc/tanstack-react-query": "^11.7.1",
|
||||||
"@upstash/ratelimit": "^2.0.7",
|
|
||||||
"@upstash/redis": "^1.35.6",
|
|
||||||
"@vercel/analytics": "^1.5.0",
|
"@vercel/analytics": "^1.5.0",
|
||||||
"@vercel/blob": "^2.0.0",
|
"@vercel/blob": "^2.0.0",
|
||||||
"@vercel/edge-config": "^1.4.3",
|
"@vercel/edge-config": "^1.4.3",
|
||||||
@@ -64,7 +62,7 @@
|
|||||||
"ipaddr.js": "^2.2.0",
|
"ipaddr.js": "^2.2.0",
|
||||||
"lucide-react": "^0.554.0",
|
"lucide-react": "^0.554.0",
|
||||||
"mapbox-gl": "^3.16.0",
|
"mapbox-gl": "^3.16.0",
|
||||||
"media-chrome": "^4.15.1",
|
"media-chrome": "^4.16.0",
|
||||||
"motion": "^12.23.24",
|
"motion": "^12.23.24",
|
||||||
"ms": "3.0.0-canary.202508261828",
|
"ms": "3.0.0-canary.202508261828",
|
||||||
"next": "16.0.3",
|
"next": "16.0.3",
|
||||||
|
|||||||
46
pnpm-lock.yaml
generated
46
pnpm-lock.yaml
generated
@@ -41,12 +41,6 @@ importers:
|
|||||||
'@trpc/tanstack-react-query':
|
'@trpc/tanstack-react-query':
|
||||||
specifier: ^11.7.1
|
specifier: ^11.7.1
|
||||||
version: 11.7.1(@tanstack/react-query@5.90.10(react@19.2.0))(@trpc/client@11.7.1(@trpc/server@11.7.1(typescript@5.9.3))(typescript@5.9.3))(@trpc/server@11.7.1(typescript@5.9.3))(react-dom@19.2.0(react@19.2.0))(react@19.2.0)(typescript@5.9.3)
|
version: 11.7.1(@tanstack/react-query@5.90.10(react@19.2.0))(@trpc/client@11.7.1(@trpc/server@11.7.1(typescript@5.9.3))(typescript@5.9.3))(@trpc/server@11.7.1(typescript@5.9.3))(react-dom@19.2.0(react@19.2.0))(react@19.2.0)(typescript@5.9.3)
|
||||||
'@upstash/ratelimit':
|
|
||||||
specifier: ^2.0.7
|
|
||||||
version: 2.0.7(@upstash/redis@1.35.6)
|
|
||||||
'@upstash/redis':
|
|
||||||
specifier: ^1.35.6
|
|
||||||
version: 1.35.6
|
|
||||||
'@vercel/analytics':
|
'@vercel/analytics':
|
||||||
specifier: ^1.5.0
|
specifier: ^1.5.0
|
||||||
version: 1.5.0(next@16.0.3(@babel/core@7.28.5)(@opentelemetry/api@1.9.0)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.2.0(react@19.2.0))(react@19.2.0))(react@19.2.0)
|
version: 1.5.0(next@16.0.3(@babel/core@7.28.5)(@opentelemetry/api@1.9.0)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.2.0(react@19.2.0))(react@19.2.0))(react@19.2.0)
|
||||||
@@ -114,8 +108,8 @@ importers:
|
|||||||
specifier: ^3.16.0
|
specifier: ^3.16.0
|
||||||
version: 3.16.0
|
version: 3.16.0
|
||||||
media-chrome:
|
media-chrome:
|
||||||
specifier: ^4.15.1
|
specifier: ^4.16.0
|
||||||
version: 4.15.1(react@19.2.0)
|
version: 4.16.0(react@19.2.0)
|
||||||
motion:
|
motion:
|
||||||
specifier: ^12.23.24
|
specifier: ^12.23.24
|
||||||
version: 12.23.24(react-dom@19.2.0(react@19.2.0))(react@19.2.0)
|
version: 12.23.24(react-dom@19.2.0(react@19.2.0))(react@19.2.0)
|
||||||
@@ -2773,15 +2767,6 @@ packages:
|
|||||||
'@types/yauzl@2.10.3':
|
'@types/yauzl@2.10.3':
|
||||||
resolution: {integrity: sha512-oJoftv0LSuaDZE3Le4DbKX+KS9G36NzOeSap90UIK0yMA/NhKJhqlSGtNDORNRaIbQfzjXDrQa0ytJ6mNRGz/Q==}
|
resolution: {integrity: sha512-oJoftv0LSuaDZE3Le4DbKX+KS9G36NzOeSap90UIK0yMA/NhKJhqlSGtNDORNRaIbQfzjXDrQa0ytJ6mNRGz/Q==}
|
||||||
|
|
||||||
'@upstash/core-analytics@0.0.10':
|
|
||||||
resolution: {integrity: sha512-7qJHGxpQgQr9/vmeS1PktEwvNAF7TI4iJDi8Pu2CFZ9YUGHZH4fOP5TfYlZ4aVxfopnELiE4BS4FBjyK7V1/xQ==}
|
|
||||||
engines: {node: '>=16.0.0'}
|
|
||||||
|
|
||||||
'@upstash/ratelimit@2.0.7':
|
|
||||||
resolution: {integrity: sha512-qNQW4uBPKVk8c4wFGj2S/vfKKQxXx1taSJoSGBN36FeiVBBKHQgsjPbKUijZ9Xu5FyVK+pfiXWKIsQGyoje8Fw==}
|
|
||||||
peerDependencies:
|
|
||||||
'@upstash/redis': ^1.34.3
|
|
||||||
|
|
||||||
'@upstash/redis@1.35.6':
|
'@upstash/redis@1.35.6':
|
||||||
resolution: {integrity: sha512-aSEIGJgJ7XUfTYvhQcQbq835re7e/BXjs8Janq6Pvr6LlmTZnyqwT97RziZLO/8AVUL037RLXqqiQC6kCt+5pA==}
|
resolution: {integrity: sha512-aSEIGJgJ7XUfTYvhQcQbq835re7e/BXjs8Janq6Pvr6LlmTZnyqwT97RziZLO/8AVUL037RLXqqiQC6kCt+5pA==}
|
||||||
|
|
||||||
@@ -3127,8 +3112,8 @@ packages:
|
|||||||
canonicalize@1.0.8:
|
canonicalize@1.0.8:
|
||||||
resolution: {integrity: sha512-0CNTVCLZggSh7bc5VkX5WWPWO+cyZbNd07IHIsSXLia/eAq+r836hgk+8BKoEh7949Mda87VUOitx5OddVj64A==}
|
resolution: {integrity: sha512-0CNTVCLZggSh7bc5VkX5WWPWO+cyZbNd07IHIsSXLia/eAq+r836hgk+8BKoEh7949Mda87VUOitx5OddVj64A==}
|
||||||
|
|
||||||
ce-la-react@0.3.1:
|
ce-la-react@0.3.2:
|
||||||
resolution: {integrity: sha512-g0YwpZDPIwTwFumGTzNHcgJA6VhFfFCJkSNdUdC04br2UfU+56JDrJrJva3FZ7MToB4NDHAFBiPE/PZdNl1mQA==}
|
resolution: {integrity: sha512-QJ6k4lOD/btI08xG8jBPxRCGXvCnusGGkTsiXk0u3NqUu/W+BXRnFD4PYjwtqh8AWmGa5LDbGk0fLQsqr0nSMA==}
|
||||||
peerDependencies:
|
peerDependencies:
|
||||||
react: '>=17.0.0'
|
react: '>=17.0.0'
|
||||||
|
|
||||||
@@ -4085,8 +4070,8 @@ packages:
|
|||||||
mdn-data@2.12.2:
|
mdn-data@2.12.2:
|
||||||
resolution: {integrity: sha512-IEn+pegP1aManZuckezWCO+XZQDplx1366JoVhTpMpBB1sPey/SbveZQUosKiKiGYjg1wH4pMlNgXbCiYgihQA==}
|
resolution: {integrity: sha512-IEn+pegP1aManZuckezWCO+XZQDplx1366JoVhTpMpBB1sPey/SbveZQUosKiKiGYjg1wH4pMlNgXbCiYgihQA==}
|
||||||
|
|
||||||
media-chrome@4.15.1:
|
media-chrome@4.16.0:
|
||||||
resolution: {integrity: sha512-Hxqr0qQ67ewmRaLJBqe5ayu53txFX+DODb9xBSHgTbw7j+gITGZ4llbPPEmqMlDnatw7IsF+AUh9rJYbpnn4ZQ==}
|
resolution: {integrity: sha512-c5xpTYcYo9nYsC/G/C1PyOcPXEL6iIaSR9MH3GncVuj4S90aHqvGbsyUWFDPPBKx5sCwWLxDnbszE/24eMT54g==}
|
||||||
|
|
||||||
mime-db@1.52.0:
|
mime-db@1.52.0:
|
||||||
resolution: {integrity: sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==}
|
resolution: {integrity: sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==}
|
||||||
@@ -7720,18 +7705,10 @@ snapshots:
|
|||||||
'@types/node': 24.10.1
|
'@types/node': 24.10.1
|
||||||
optional: true
|
optional: true
|
||||||
|
|
||||||
'@upstash/core-analytics@0.0.10':
|
|
||||||
dependencies:
|
|
||||||
'@upstash/redis': 1.35.6
|
|
||||||
|
|
||||||
'@upstash/ratelimit@2.0.7(@upstash/redis@1.35.6)':
|
|
||||||
dependencies:
|
|
||||||
'@upstash/core-analytics': 0.0.10
|
|
||||||
'@upstash/redis': 1.35.6
|
|
||||||
|
|
||||||
'@upstash/redis@1.35.6':
|
'@upstash/redis@1.35.6':
|
||||||
dependencies:
|
dependencies:
|
||||||
uncrypto: 0.1.3
|
uncrypto: 0.1.3
|
||||||
|
optional: true
|
||||||
|
|
||||||
'@vercel/analytics@1.5.0(next@16.0.3(@babel/core@7.28.5)(@opentelemetry/api@1.9.0)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.2.0(react@19.2.0))(react@19.2.0))(react@19.2.0)':
|
'@vercel/analytics@1.5.0(next@16.0.3(@babel/core@7.28.5)(@opentelemetry/api@1.9.0)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.2.0(react@19.2.0))(react@19.2.0))(react@19.2.0)':
|
||||||
optionalDependencies:
|
optionalDependencies:
|
||||||
@@ -8042,7 +8019,7 @@ snapshots:
|
|||||||
|
|
||||||
canonicalize@1.0.8: {}
|
canonicalize@1.0.8: {}
|
||||||
|
|
||||||
ce-la-react@0.3.1(react@19.2.0):
|
ce-la-react@0.3.2(react@19.2.0):
|
||||||
dependencies:
|
dependencies:
|
||||||
react: 19.2.0
|
react: 19.2.0
|
||||||
|
|
||||||
@@ -8986,9 +8963,9 @@ snapshots:
|
|||||||
|
|
||||||
mdn-data@2.12.2: {}
|
mdn-data@2.12.2: {}
|
||||||
|
|
||||||
media-chrome@4.15.1(react@19.2.0):
|
media-chrome@4.16.0(react@19.2.0):
|
||||||
dependencies:
|
dependencies:
|
||||||
ce-la-react: 0.3.1(react@19.2.0)
|
ce-la-react: 0.3.2(react@19.2.0)
|
||||||
transitivePeerDependencies:
|
transitivePeerDependencies:
|
||||||
- react
|
- react
|
||||||
|
|
||||||
@@ -9871,7 +9848,8 @@ snapshots:
|
|||||||
|
|
||||||
ulid@2.4.0: {}
|
ulid@2.4.0: {}
|
||||||
|
|
||||||
uncrypto@0.1.3: {}
|
uncrypto@0.1.3:
|
||||||
|
optional: true
|
||||||
|
|
||||||
undici-types@6.21.0: {}
|
undici-types@6.21.0: {}
|
||||||
|
|
||||||
|
|||||||
@@ -61,10 +61,6 @@ wait_for_port() {
|
|||||||
# --- Wait for exposed services ----------------------------------------------
|
# --- Wait for exposed services ----------------------------------------------
|
||||||
# Postgres TCP
|
# Postgres TCP
|
||||||
wait_for_port "127.0.0.1" 5432 "Postgres"
|
wait_for_port "127.0.0.1" 5432 "Postgres"
|
||||||
# Redis TCP
|
|
||||||
wait_for_port "127.0.0.1" 6379 "Redis"
|
|
||||||
# Serverless Redis HTTP (SRH)
|
|
||||||
wait_for_port "127.0.0.1" 8079 "SRH (Upstash-compatible HTTP)"
|
|
||||||
# Inngest Dev Server
|
# Inngest Dev Server
|
||||||
wait_for_port "127.0.0.1" 8288 "Inngest Dev Server"
|
wait_for_port "127.0.0.1" 8288 "Inngest Dev Server"
|
||||||
|
|
||||||
@@ -72,8 +68,6 @@ wait_for_port "127.0.0.1" 8288 "Inngest Dev Server"
|
|||||||
echo
|
echo
|
||||||
echo "🎉 Local infra is ready!"
|
echo "🎉 Local infra is ready!"
|
||||||
echo " * Postgres: postgres://postgres:postgres@localhost:5432/main"
|
echo " * Postgres: postgres://postgres:postgres@localhost:5432/main"
|
||||||
echo " * Redis: redis://localhost:6379"
|
|
||||||
echo " * SRH: http://localhost:8079"
|
|
||||||
echo " * Inngest: http://localhost:8288"
|
echo " * Inngest: http://localhost:8288"
|
||||||
echo
|
echo
|
||||||
|
|
||||||
|
|||||||
@@ -20,7 +20,11 @@ import { getPricingForTld } from "@/server/services/pricing";
|
|||||||
import { getRegistration } from "@/server/services/registration";
|
import { getRegistration } from "@/server/services/registration";
|
||||||
import { getOrCreateScreenshotBlobUrl } from "@/server/services/screenshot";
|
import { getOrCreateScreenshotBlobUrl } from "@/server/services/screenshot";
|
||||||
import { getSeo } from "@/server/services/seo";
|
import { getSeo } from "@/server/services/seo";
|
||||||
import { createTRPCRouter, domainProcedure } from "@/trpc/init";
|
import {
|
||||||
|
createTRPCRouter,
|
||||||
|
domainProcedure,
|
||||||
|
publicProcedure,
|
||||||
|
} from "@/trpc/init";
|
||||||
|
|
||||||
const DomainInputSchema = z
|
const DomainInputSchema = z
|
||||||
.object({ domain: z.string().min(1) })
|
.object({ domain: z.string().min(1) })
|
||||||
@@ -32,47 +36,38 @@ const DomainInputSchema = z
|
|||||||
|
|
||||||
export const domainRouter = createTRPCRouter({
|
export const domainRouter = createTRPCRouter({
|
||||||
registration: domainProcedure
|
registration: domainProcedure
|
||||||
.meta({ service: "registration" })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(RegistrationSchema)
|
.output(RegistrationSchema)
|
||||||
.query(({ input }) => getRegistration(input.domain)),
|
.query(({ input }) => getRegistration(input.domain)),
|
||||||
dns: domainProcedure
|
dns: domainProcedure
|
||||||
.meta({ service: "dns" })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(DnsResolveResultSchema)
|
.output(DnsResolveResultSchema)
|
||||||
.query(({ input }) => resolveAll(input.domain)),
|
.query(({ input }) => resolveAll(input.domain)),
|
||||||
hosting: domainProcedure
|
hosting: domainProcedure
|
||||||
.meta({ service: "hosting" })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(HostingSchema)
|
.output(HostingSchema)
|
||||||
.query(({ input }) => detectHosting(input.domain)),
|
.query(({ input }) => detectHosting(input.domain)),
|
||||||
certificates: domainProcedure
|
certificates: domainProcedure
|
||||||
.meta({ service: "certs" })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(CertificatesSchema)
|
.output(CertificatesSchema)
|
||||||
.query(({ input }) => getCertificates(input.domain)),
|
.query(({ input }) => getCertificates(input.domain)),
|
||||||
headers: domainProcedure
|
headers: domainProcedure
|
||||||
.meta({ service: "headers" })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(HttpHeadersResponseSchema)
|
.output(HttpHeadersResponseSchema)
|
||||||
.query(({ input }) => probeHeaders(input.domain)),
|
.query(({ input }) => probeHeaders(input.domain)),
|
||||||
seo: domainProcedure
|
seo: domainProcedure
|
||||||
.meta({ service: "seo" })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(SeoResponseSchema)
|
.output(SeoResponseSchema)
|
||||||
.query(({ input }) => getSeo(input.domain)),
|
.query(({ input }) => getSeo(input.domain)),
|
||||||
favicon: domainProcedure
|
favicon: publicProcedure
|
||||||
.meta({ service: "favicon", recordAccess: false })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(StorageUrlSchema)
|
.output(StorageUrlSchema)
|
||||||
.query(({ input }) => getOrCreateFaviconBlobUrl(input.domain)),
|
.query(({ input }) => getOrCreateFaviconBlobUrl(input.domain)),
|
||||||
screenshot: domainProcedure
|
screenshot: publicProcedure
|
||||||
.meta({ service: "screenshot", recordAccess: false })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(StorageUrlSchema)
|
.output(StorageUrlSchema)
|
||||||
.query(({ input }) => getOrCreateScreenshotBlobUrl(input.domain)),
|
.query(({ input }) => getOrCreateScreenshotBlobUrl(input.domain)),
|
||||||
pricing: domainProcedure
|
pricing: publicProcedure
|
||||||
.meta({ service: "pricing", recordAccess: false })
|
|
||||||
.input(DomainInputSchema)
|
.input(DomainInputSchema)
|
||||||
.output(PricingSchema)
|
.output(PricingSchema)
|
||||||
.query(({ input }) => getPricingForTld(input.domain)),
|
.query(({ input }) => getPricingForTld(input.domain)),
|
||||||
|
|||||||
@@ -11,7 +11,6 @@ import { useState } from "react";
|
|||||||
import superjson from "superjson";
|
import superjson from "superjson";
|
||||||
import { TRPCProvider as Provider } from "@/lib/trpc/client";
|
import { TRPCProvider as Provider } from "@/lib/trpc/client";
|
||||||
import type { AppRouter } from "@/server/routers/_app";
|
import type { AppRouter } from "@/server/routers/_app";
|
||||||
import { errorToastLink } from "@/trpc/error-toast-link";
|
|
||||||
import { makeQueryClient } from "@/trpc/query-client";
|
import { makeQueryClient } from "@/trpc/query-client";
|
||||||
|
|
||||||
let browserQueryClient: ReturnType<typeof makeQueryClient> | undefined;
|
let browserQueryClient: ReturnType<typeof makeQueryClient> | undefined;
|
||||||
@@ -43,7 +42,6 @@ export function TRPCProvider({ children }: { children: React.ReactNode }) {
|
|||||||
process.env.NODE_ENV === "development" ||
|
process.env.NODE_ENV === "development" ||
|
||||||
(opts.direction === "down" && opts.result instanceof Error),
|
(opts.direction === "down" && opts.result instanceof Error),
|
||||||
}),
|
}),
|
||||||
errorToastLink(),
|
|
||||||
httpBatchStreamLink({
|
httpBatchStreamLink({
|
||||||
url: `${getBaseUrl()}/api/trpc`,
|
url: `${getBaseUrl()}/api/trpc`,
|
||||||
transformer: superjson,
|
transformer: superjson,
|
||||||
|
|||||||
@@ -1,53 +0,0 @@
|
|||||||
"use client";
|
|
||||||
|
|
||||||
import { TRPCClientError, type TRPCLink } from "@trpc/client";
|
|
||||||
import type { AnyRouter } from "@trpc/server";
|
|
||||||
import { observable } from "@trpc/server/observable";
|
|
||||||
import { formatDistanceToNowStrict } from "date-fns";
|
|
||||||
import { Gauge } from "lucide-react";
|
|
||||||
import { createElement } from "react";
|
|
||||||
import { toast } from "sonner";
|
|
||||||
|
|
||||||
export function errorToastLink<
|
|
||||||
TRouter extends AnyRouter = AnyRouter,
|
|
||||||
>(): TRPCLink<TRouter> {
|
|
||||||
return () =>
|
|
||||||
({ next, op }) =>
|
|
||||||
observable((observer) => {
|
|
||||||
const sub = next(op).subscribe({
|
|
||||||
next(value) {
|
|
||||||
observer.next(value);
|
|
||||||
},
|
|
||||||
error(err) {
|
|
||||||
if (err instanceof TRPCClientError) {
|
|
||||||
const code = err.data?.code;
|
|
||||||
if (code === "TOO_MANY_REQUESTS") {
|
|
||||||
const retryAfterSec = Math.max(
|
|
||||||
1,
|
|
||||||
Math.round(Number(err.data?.retryAfter ?? 1)),
|
|
||||||
);
|
|
||||||
const service = err.data?.service as string | undefined;
|
|
||||||
const retryUntil = new Date(Date.now() + retryAfterSec * 1000);
|
|
||||||
const friendly = formatDistanceToNowStrict(retryUntil, {
|
|
||||||
addSuffix: true,
|
|
||||||
});
|
|
||||||
const title = service
|
|
||||||
? `Too many ${service} requests`
|
|
||||||
: "You're doing that too much";
|
|
||||||
toast.error(title, {
|
|
||||||
id: "rate-limit",
|
|
||||||
description: `Try again ${friendly}.`,
|
|
||||||
icon: createElement(Gauge, { className: "h-4 w-4" }),
|
|
||||||
position: "top-center",
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
observer.error(err);
|
|
||||||
},
|
|
||||||
complete() {
|
|
||||||
observer.complete();
|
|
||||||
},
|
|
||||||
});
|
|
||||||
return () => sub.unsubscribe();
|
|
||||||
});
|
|
||||||
}
|
|
||||||
56
trpc/init.ts
56
trpc/init.ts
@@ -4,7 +4,6 @@ import { after } from "next/server";
|
|||||||
import superjson from "superjson";
|
import superjson from "superjson";
|
||||||
import { updateLastAccessed } from "@/lib/db/repos/domains";
|
import { updateLastAccessed } from "@/lib/db/repos/domains";
|
||||||
import { toRegistrableDomain } from "@/lib/domain-server";
|
import { toRegistrableDomain } from "@/lib/domain-server";
|
||||||
import { assertRateLimit, type ServiceName } from "@/lib/ratelimit";
|
|
||||||
|
|
||||||
const IP_HEADERS = ["x-real-ip", "x-forwarded-for", "cf-connecting-ip"];
|
const IP_HEADERS = ["x-real-ip", "x-forwarded-for", "cf-connecting-ip"];
|
||||||
|
|
||||||
@@ -42,33 +41,6 @@ export const t = initTRPC
|
|||||||
.meta<Record<string, unknown>>()
|
.meta<Record<string, unknown>>()
|
||||||
.create({
|
.create({
|
||||||
transformer: superjson,
|
transformer: superjson,
|
||||||
errorFormatter({ shape, error }) {
|
|
||||||
const cause = (
|
|
||||||
error as unknown as {
|
|
||||||
cause?: {
|
|
||||||
retryAfter?: number;
|
|
||||||
service?: string;
|
|
||||||
limit?: number;
|
|
||||||
remaining?: number;
|
|
||||||
};
|
|
||||||
}
|
|
||||||
).cause;
|
|
||||||
return {
|
|
||||||
...shape,
|
|
||||||
data: {
|
|
||||||
...shape.data,
|
|
||||||
retryAfter:
|
|
||||||
typeof cause?.retryAfter === "number"
|
|
||||||
? cause.retryAfter
|
|
||||||
: undefined,
|
|
||||||
service:
|
|
||||||
typeof cause?.service === "string" ? cause.service : undefined,
|
|
||||||
limit: typeof cause?.limit === "number" ? cause.limit : undefined,
|
|
||||||
remaining:
|
|
||||||
typeof cause?.remaining === "number" ? cause.remaining : undefined,
|
|
||||||
},
|
|
||||||
};
|
|
||||||
},
|
|
||||||
});
|
});
|
||||||
|
|
||||||
export const createTRPCRouter = t.router;
|
export const createTRPCRouter = t.router;
|
||||||
@@ -110,30 +82,12 @@ const withLogging = t.middleware(async ({ path, type, next }) => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
/**
|
|
||||||
* Middleware to rate limit requests.
|
|
||||||
* - Expects meta to have a `service` field containing the service name.
|
|
||||||
* - Expects ctx to have an `ip` field containing the IP address.
|
|
||||||
* - Throws a TRPCError if the rate limit is exceeded.
|
|
||||||
*/
|
|
||||||
const withRatelimit = t.middleware(async ({ ctx, next, meta }) => {
|
|
||||||
if (meta?.service && ctx.ip) {
|
|
||||||
await assertRateLimit(meta.service as ServiceName, ctx.ip);
|
|
||||||
}
|
|
||||||
return next();
|
|
||||||
});
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Middleware to record that a domain was accessed by a user (for decay calculation).
|
* Middleware to record that a domain was accessed by a user (for decay calculation).
|
||||||
* - Expects input to have a `domain` field.
|
* Expects input to have a `domain` field.
|
||||||
* - Can be disabled by setting `meta.recordAccess = false`.
|
|
||||||
* Schedules the write to happen after the response is sent using Next.js after().
|
* Schedules the write to happen after the response is sent using Next.js after().
|
||||||
*/
|
*/
|
||||||
const withDomainAccessUpdate = t.middleware(async ({ input, meta, next }) => {
|
const withDomainAccessUpdate = t.middleware(async ({ input, next }) => {
|
||||||
// Allow procedures to opt-out of access tracking
|
|
||||||
if (meta?.recordAccess === false) {
|
|
||||||
return next();
|
|
||||||
}
|
|
||||||
// Check if input is a valid object with a domain property
|
// Check if input is a valid object with a domain property
|
||||||
if (
|
if (
|
||||||
input &&
|
input &&
|
||||||
@@ -157,9 +111,7 @@ const withDomainAccessUpdate = t.middleware(async ({ input, meta, next }) => {
|
|||||||
export const publicProcedure = t.procedure.use(withLogging);
|
export const publicProcedure = t.procedure.use(withLogging);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Domain-specific procedure with rate limiting and access tracking.
|
* Domain-specific procedure with "last accessed at" tracking.
|
||||||
* Use this for all domain data endpoints (dns, hosting, seo, etc).
|
* Use this for all domain data endpoints (dns, hosting, seo, etc).
|
||||||
*/
|
*/
|
||||||
export const domainProcedure = publicProcedure
|
export const domainProcedure = publicProcedure.use(withDomainAccessUpdate);
|
||||||
.use(withRatelimit)
|
|
||||||
.use(withDomainAccessUpdate);
|
|
||||||
|
|||||||
Reference in New Issue
Block a user