jake/hoot
1
Fork 0
mirror of https://github.com/jakejarvis/hoot.git synced 2025-10-18 20:14:25 -04:00
Code Issues Activity

Migrate to uploadthing for file storage (#74)

Browse Source 
Co-authored-by: jake <jake@jarv.is>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This commit is contained in:
Jake Jarvis
2025-10-06 15:07:18 -04:00
committed by GitHub
parent 93bc8cf3d6
commit 6960c0b4a8
24 changed files with 487 additions and 584 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
.env.example
View File

@@ -6,11 +6,13 @@ NEXT_PUBLIC_POSTHOG_HOST=
KV_REST_API_URL=
KV_REST_API_TOKEN=
# Vercel Blob (add integration on Vercel; token for local/dev)
BLOB_READ_WRITE_TOKEN=
# UploadThing credentials
UPLOADTHING_SECRET=
UPLOADTHING_APP_ID=
# Secret used to derive unpredictable blob paths for generated images (favicons and screenshots)
BLOB_SIGNING_SECRET=
# Optional TTLs for generated assets (seconds)
FAVICON_TTL_SECONDS=
SCREENSHOT_TTL_SECONDS=
# Mapbox access token for react-map-gl
NEXT_PUBLIC_MAPBOX_TOKEN=

19
.github/copilot-instructions.md vendored
View File

@@ -30,7 +30,7 @@ Hoot is an all-in-one app for exploring domain names, providing instant insights
- **Tailwind CSS v4**
- **tRPC** for API endpoints
- **Upstash Redis** for caching
- **Vercel Blob** for favicon and screenshot storage
- **UploadThing** for favicon and screenshot storage
- **Puppeteer Core + @sparticuz/chromium** for server screenshots (fallback to `puppeteer` locally)
- **Biome** for linting and formatting
@@ -63,24 +63,23 @@ Hoot is an all-in-one app for exploring domain names, providing instant insights
- **TLS/SSL Certificates** - Chain analysis
- **HTTP Headers** - Security headers and tech detection
- **Geolocation** - IP → map
- **Favicon & Screenshot Storage** - Vercel Blob with time-bucketed HMAC paths
- **Favicon & Screenshot Storage** - UploadThing with Redis index
### Screenshots (New)
- Service: `server/services/screenshot.ts` using Puppeteer.
- Storage: `lib/blob.ts` computes `screenshots/{bucket}/{digest}/{width}x{height}.png` with HMAC and rotating buckets governed by `SCREENSHOT_TTL_SECONDS`.
- Storage: `lib/storage.ts` uploads via UploadThing and returns `{ url, key }` stored in Redis with TTLs.
- Client UI: `components/domain/screenshot.tsx` and `components/domain/screenshot-tooltip.tsx` with optimized loading state and one-time fetch gating.
- Router: `server/routers/domain.ts` exposes `domain.screenshot` via `createDomainProcedure`.
- Config: prefer `puppeteer-core` + `@sparticuz/chromium` on Vercel; fallback to `puppeteer` locally. In `next.config.ts`, `serverExternalPackages: ["@sparticuz/chromium", "puppeteer-core"]`.
- Env:
- `BLOB_SIGNING_SECRET` (prod required)
- `BLOB_READ_WRITE_TOKEN`
- `UPLOADTHING_SECRET` (required)
- `UPLOADTHING_APP_ID` (recommended)
- `SCREENSHOT_TTL_SECONDS` (optional, default 7 days)
- `HOOT_USER_AGENT` (optional UA override)
- `PUPPETEER_SKIP_DOWNLOAD=1` on Vercel to skip full puppeteer install
### Cron Blob Pruning
### Cron Upload Pruning
- Route: `app/api/cron/blob-prune/route.ts` (GET only, Bearer auth via `CRON_SECRET`).
- Deletes old buckets under `favicons/` and `screenshots/`; keep window via `BLOB_KEEP_BUCKETS`.
- Deletes expired UploadThing files by key from `purge:favicon` and `purge:screenshot` ZSETs; batch size configurable via `PURGE_BATCH`.
## Testing Guidelines
- Runner: **Vitest**; environment `jsdom` by default; Node tests with `/* @vitest-environment node */`.
@@ -88,8 +87,8 @@ Hoot is an all-in-one app for exploring domain names, providing instant insights
- UI tests: prefer behavior tests; mock Radix primitives and TRPC/React Query as needed.
- Server tests: hoisted ESM mocks; unique cache domains; reset Redis between tests.
- New tests:
- `server/services/screenshot.test.ts` mocks puppeteer and blob.
- `lib/blob.test.ts` includes screenshot path/put/head tests with bucket fallback.
- `server/services/screenshot.test.ts` mocks puppeteer and UploadThing storage.
- `lib/storage.test.ts` covers upload helpers.
## Security & Configuration
- Keep secrets in `.env.local`

2
AGENTS.md
View File

@@ -45,7 +45,7 @@
## Security & Configuration Tips
- Keep secrets in `.env.local`.
- Blob: `BLOB_SIGNING_SECRET`, `BLOB_READ_WRITE_TOKEN`, `FAVICON_TTL_SECONDS`, `SCREENSHOT_TTL_SECONDS`.
- UploadThing: `UPLOADTHING_SECRET` (required), optional `UPLOADTHING_APP_ID`; TTLs `FAVICON_TTL_SECONDS`, `SCREENSHOT_TTL_SECONDS`.
- Screenshots (Puppeteer): prefer `puppeteer-core` + `@sparticuz/chromium` on Vercel; optional `PUPPETEER_SKIP_DOWNLOAD=1` to avoid full download; `HOOT_USER_AGENT` to override UA; optional `PUPPETEER_EXECUTABLE_PATH` locally.
- Cache Cloudflare DoH, RDAP, TLS, and header probes via `lib/cache`; apply retry backoff to respect provider limits.
- Review `server/trpc.ts` when extending procedures to ensure auth/context remain intact.

10
README.md
View File

@@ -12,7 +12,7 @@
- **Comprehensive Reports:** See registration info, hosting & email, DNS records, SSL certificates, and HTTP headers.
- **Interactive UI:** Expand/collapse sections, copy data, and enjoy beautiful dark mode.
- **Fast & Private:** Data is fetched live, with caching for speed—no sign-up required.
- **Favicons & Screenshots:** Extract favicons and capture homepage screenshots, cached on Vercel Blob for quick reuse.
- **Favicons & Screenshots:** Extract favicons and capture homepage screenshots, cached on UploadThing for quick reuse.
---
@@ -24,7 +24,7 @@
- **Tailwind CSS v4**
- **tRPC** API endpoints
- **Upstash Redis** for caching
- **Vercel Blob** for favicon & screenshot storage
- **UploadThing** for favicon & screenshot storage
- **rdapper** for RDAP registration lookups with WHOIS fallback
- **Puppeteer** for server-side screenshots
- **Mapbox** for embedded IP geolocation maps
@@ -49,11 +49,11 @@
Open [http://localhost:3000](http://localhost:3000) in your browser.
3. **(Optional) Configure `.env.local`:**
See `.env.example` for Redis and Vercel Blob credentials (needed for caching and favicon/screenshot features).
See `.env.example` for Redis and UploadThing credentials (needed for caching and favicon/screenshot features).
Useful keys:
- `BLOB_SIGNING_SECRET` (required in production)
- `BLOB_READ_WRITE_TOKEN`
- `UPLOADTHING_SECRET` (required)
- `UPLOADTHING_APP_ID` (recommended)
- `FAVICON_TTL_SECONDS`, `SCREENSHOT_TTL_SECONDS` (optional TTLs)
- `HOOT_USER_AGENT` (optional UA override)
- `PUPPETEER_SKIP_DOWNLOAD=1` on Vercel to skip full `puppeteer` download

16
app/api/cron/blob-prune/route.test.ts
View File

@@ -1,16 +1,19 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("@vercel/blob", () => ({
del: vi.fn(async (_url: string) => undefined),
const utMock = vi.hoisted(() => ({
deleteFiles: vi.fn(async (_keys: string[]) => undefined),
}));
vi.mock("uploadthing/server", () => ({
UTApi: vi.fn().mockImplementation(() => utMock),
}));
// Use global redis mock; seed with URLs instead of pathnames
beforeEach(() => {
global.__redisTestHelper.reset();
const set = global.__redisTestHelper.zsets;
set.set("purge:favicon", new Map([["https://blob/f1", Date.now()]]));
set.set("purge:screenshot", new Map([["https://blob/s1", Date.now()]]));
set.set("purge:favicon", new Map([["ut-key-f1", Date.now()]]));
set.set("purge:screenshot", new Map([["ut-key-s1", Date.now()]]));
});
import { GET } from "./route";
@@ -18,6 +21,7 @@ import { GET } from "./route";
describe("/api/cron/blob-prune", () => {
afterEach(() => {
vi.restoreAllMocks();
utMock.deleteFiles.mockClear();
});
it("requires secret and prunes old buckets (GET)", async () => {
@@ -31,6 +35,10 @@ describe("/api/cron/blob-prune", () => {
expect(res.status).toBe(200);
const json = await res.json();
expect(json.deletedCount).toBeGreaterThan(0);
// Each kind is deleted in its own batch call
expect(utMock.deleteFiles).toHaveBeenCalledTimes(2);
expect(utMock.deleteFiles).toHaveBeenNthCalledWith(1, ["ut-key-f1"]);
expect(utMock.deleteFiles).toHaveBeenNthCalledWith(2, ["ut-key-s1"]);
});
it("rejects when secret missing or invalid (GET)", async () => {

21
app/api/cron/blob-prune/route.ts
View File

@@ -1,5 +1,5 @@
import { del } from "@vercel/blob";
import { NextResponse } from "next/server";
import { UTApi } from "uploadthing/server";
import { ns, redis } from "@/lib/redis";
export const runtime = "nodejs";
@@ -13,11 +13,12 @@ export async function GET(req: Request) {
const deleted: string[] = [];
const errors: Array<{ path: string; error: string }> = [];
const utapi = new UTApi();
const batch = process.env.BLOB_PURGE_BATCH
? parseInt(process.env.BLOB_PURGE_BATCH, 10)
: 500;
// Fixed batch size to avoid env coupling
const batch = 500;
const now = Date.now();
for (const kind of ["favicon", "screenshot"]) {
// Drain due items in batches
// Upstash supports zrange with byScore parameter; the SDK exposes zrange with options
@@ -29,12 +30,12 @@ export async function GET(req: Request) {
});
if (!due.length) break;
const succeeded: string[] = [];
for (const path of due) {
try {
await del(path, { token: process.env.BLOB_READ_WRITE_TOKEN });
deleted.push(path);
succeeded.push(path);
} catch (err) {
try {
await utapi.deleteFiles(due);
deleted.push(...due);
succeeded.push(...due);
} catch (err) {
for (const path of due) {
errors.push({ path, error: (err as Error)?.message || "unknown" });
}
}

21
components/domain/domain-search.test.tsx
View File

@@ -33,18 +33,16 @@ describe("DomainSearch (form variant)", () => {
it("submits valid domain and navigates", async () => {
render(<DomainSearch variant="lg" />);
await userEvent.type(
screen.getByLabelText(/Search any domain/i),
"example.com",
);
await userEvent.click(screen.getByRole("button", { name: /analyze/i }));
const input = screen.getByLabelText(/Search any domain/i);
await userEvent.type(input, "example.com{Enter}");
expect(nav.push).toHaveBeenCalledWith("/example.com");
// Input and button should be disabled while loading/submitting
expect(
(screen.getByLabelText(/Search any domain/i) as HTMLInputElement)
.disabled,
).toBe(true);
expect(screen.getByRole("button", { name: /analyze/i })).toBeDisabled();
// Submit button shows a loading spinner with accessible name "Loading"
expect(screen.getByRole("button", { name: /loading/i })).toBeDisabled();
});
it("shows error toast for invalid domain", async () => {
@@ -52,11 +50,8 @@ describe("DomainSearch (form variant)", () => {
toast: { error: (msg: string) => void };
};
render(<DomainSearch variant="lg" />);
await userEvent.type(
screen.getByLabelText(/Search any domain/i),
"not a domain",
);
await userEvent.click(screen.getByRole("button", { name: /analyze/i }));
const input = screen.getByLabelText(/Search any domain/i);
await userEvent.type(input, "not a domain{Enter}");
expect(toast.error).toHaveBeenCalled();
});
@@ -73,8 +68,8 @@ describe("DomainSearch (form variant)", () => {
expect(input.value).toBe("example.com");
// Navigation should have been triggered
expect(nav.push).toHaveBeenCalledWith("/example.com");
// Analyze button should be disabled (loading state)
expect(screen.getByRole("button", { name: /analyze/i })).toBeDisabled();
// Submit button shows a loading spinner and is disabled while navigating
expect(screen.getByRole("button", { name: /loading/i })).toBeDisabled();
// Input should be disabled while loading
expect(
(screen.getByLabelText(/Search any domain/i) as HTMLInputElement)

4
components/domain/favicon.test.tsx
View File

@@ -78,11 +78,11 @@ describe("Favicon", () => {
it("renders Image when url present", () => {
(useQuery as unknown as Mock).mockReturnValue({
data: { url: "https://x/y.png" },
data: { url: "https://app.ufs.sh/f/x.png" },
isLoading: false,
});
render(<Favicon domain="example.com" size={16} />);
const img = screen.getByRole("img", { name: /icon/i });
expect(img).toHaveAttribute("src", "https://x/y.png");
expect(img).toHaveAttribute("src", "https://app.ufs.sh/f/x.png");
});
});

9
components/domain/favicon.tsx
View File

@@ -5,7 +5,6 @@ import { Globe } from "lucide-react";
import Image from "next/image";
import { useEffect, useState } from "react";
import { Skeleton } from "@/components/ui/skeleton";
import { useImageRetry } from "@/hooks/use-image-retry";
import { useTRPC } from "@/lib/trpc/client";
import { cn } from "@/lib/utils";
@@ -36,10 +35,6 @@ export function Favicon({
);
const url = data?.url ?? null;
const { imageKey, showFallback, handleError } = useImageRetry(url, {
maxRetries: 2,
delayMs: 300,
});
if (!isHydrated || isPending) {
return (
@@ -50,7 +45,7 @@ export function Favicon({
);
}
if (!url || showFallback) {
if (!url) {
return (
<Globe
className={cn("text-muted-foreground", className)}
@@ -62,7 +57,6 @@ export function Favicon({
return (
<Image
key={imageKey}
src={url}
alt={`${domain} icon`}
width={size}
@@ -70,7 +64,6 @@ export function Favicon({
className={className}
loading="lazy"
unoptimized
onError={handleError}
/>
);
}

4
components/domain/screenshot-tooltip.test.tsx
View File

@@ -74,7 +74,7 @@ describe("ScreenshotTooltip", () => {
it("renders image when loaded", () => {
(useQuery as unknown as Mock).mockReturnValue({
data: { url: "https://blob/url.png" },
data: { url: "https://app.ufs.sh/f/url.png" },
isLoading: false,
isFetching: false,
});
@@ -87,6 +87,6 @@ describe("ScreenshotTooltip", () => {
const img = screen.getByRole("img", {
name: /homepage preview of example.com/i,
});
expect(img).toHaveAttribute("src", "https://blob/url.png");
expect(img).toHaveAttribute("src", "https://app.ufs.sh/f/url.png");
});
});

4
components/domain/screenshot.test.tsx
View File

@@ -68,7 +68,7 @@ describe("Screenshot", () => {
it("renders image when url present", () => {
(useQuery as unknown as Mock).mockReturnValue({
data: { url: "https://blob/url.png" },
data: { url: "https://app.ufs.sh/f/url.png" },
isLoading: false,
isFetching: false,
});
@@ -76,7 +76,7 @@ describe("Screenshot", () => {
const img = screen.getByRole("img", {
name: /homepage preview of example.com/i,
});
expect(img).toHaveAttribute("src", "https://blob/url.png");
expect(img).toHaveAttribute("src", "https://app.ufs.sh/f/url.png");
});
it("shows fallback when no url and not loading", () => {

9
components/domain/screenshot.tsx
View File

@@ -4,7 +4,6 @@ import { useQuery } from "@tanstack/react-query";
import { CircleX } from "lucide-react";
import Image from "next/image";
import { Spinner } from "@/components/ui/spinner";
import { useImageRetry } from "@/hooks/use-image-retry";
import { useTRPC } from "@/lib/trpc/client";
import { cn } from "@/lib/utils";
@@ -39,16 +38,11 @@ export function Screenshot({
const url = data?.url ?? null;
const loading = isLoading || isFetching;
const { imageKey, showFallback, handleError } = useImageRetry(url, {
maxRetries: 2,
delayMs: 300,
});
return (
<div className={className}>
{url && !showFallback ? (
{url ? (
<Image
key={imageKey}
src={url}
alt={`Homepage preview of ${domain}`}
width={width}
@@ -61,7 +55,6 @@ export function Screenshot({
unoptimized
priority={false}
draggable={false}
onError={handleError}
/>
) : (
<div

44
hooks/use-image-retry.ts
View File

@@ -1,44 +0,0 @@
"use client";
import { useEffect, useMemo, useState } from "react";
type UseImageRetryOptions = {
maxRetries?: number;
delayMs?: number;
};
export function useImageRetry(
src: string | null,
{ maxRetries = 2, delayMs = 300 }: UseImageRetryOptions = {},
) {
const [retryCount, setRetryCount] = useState(0);
const [fallback, setFallback] = useState(false);
// Reset state when src changes
useEffect(() => {
// Reference src so linters recognize the dependency is necessary
if (src !== undefined) {
setRetryCount(0);
setFallback(false);
}
}, [src]);
const imageKey = useMemo(() => retryCount, [retryCount]);
const handleError = () => {
setTimeout(() => {
if (retryCount < maxRetries) {
setRetryCount((c) => c + 1);
} else {
setFallback(true);
}
}, delayMs);
};
return {
retryCount,
imageKey,
showFallback: fallback || !src,
handleError,
} as const;
}

109
lib/blob.test.ts
View File

@@ -1,109 +0,0 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("@vercel/blob", () => ({
put: vi.fn(async (_path: string, _buf: unknown, _opts: unknown) => ({
url: "https://blob/put.png",
})),
}));
import {
computeFaviconBlobPath,
computeScreenshotBlobPath,
putFaviconBlob,
putScreenshotBlob,
} from "./blob";
const originalEnv = { ...process.env };
beforeEach(() => {
vi.resetModules();
});
afterEach(() => {
process.env = { ...originalEnv };
});
describe("blob utils", () => {
it("computeFaviconBlobPath is deterministic and secret-dependent", () => {
process.env.BLOB_SIGNING_SECRET = "secret-a";
const a1 = computeFaviconBlobPath("example.com", 32);
const a2 = computeFaviconBlobPath("example.com", 32);
expect(a1).toBe(a2);
// Different size yields different path
const a3 = computeFaviconBlobPath("example.com", 64);
expect(a3).not.toBe(a1);
process.env.BLOB_SIGNING_SECRET = "secret-b";
const b1 = computeFaviconBlobPath("example.com", 32);
expect(b1).not.toBe(a1);
expect(a1).toMatch(/^favicons\//);
});
it("computeScreenshotBlobPath is deterministic and secret-dependent", () => {
process.env.BLOB_SIGNING_SECRET = "secret-a";
const s1 = computeScreenshotBlobPath("example.com", 1200, 630);
const s2 = computeScreenshotBlobPath("example.com", 1200, 630);
expect(s1).toBe(s2);
process.env.BLOB_SIGNING_SECRET = "secret-b";
const s3 = computeScreenshotBlobPath("example.com", 1200, 630);
expect(s3).not.toBe(s1);
expect(s1).toMatch(/^screenshots\//);
});
it("stable paths do not change over time", () => {
process.env.FAVICON_TTL_SECONDS = "10";
process.env.SCREENSHOT_TTL_SECONDS = "10";
const base = 1_000_000_000_000;
const realNow = Date.now;
Date.now = () => base;
const f1 = computeFaviconBlobPath("example.com", 32);
const s1 = computeScreenshotBlobPath("example.com", 1200, 630);
Date.now = () => base + 11_000;
const f2 = computeFaviconBlobPath("example.com", 32);
const s2 = computeScreenshotBlobPath("example.com", 1200, 630);
expect(f1).toBe(f2);
expect(s1).toBe(s2);
Date.now = realNow;
});
// head* helpers removed in favor of Redis index; put* still tested
it("putFaviconBlob uploads with expected options and returns URL", async () => {
const { put } = await import("@vercel/blob");
const url = await putFaviconBlob("example.com", 32, Buffer.from([1]));
expect(url).toBe("https://blob/put.png");
expect(
(put as unknown as import("vitest").Mock).mock.calls[0]?.[2],
).toMatchObject({
access: "public",
contentType: "image/png",
});
});
it("putScreenshotBlob uploads with expected options and returns URL", async () => {
process.env.SCREENSHOT_TTL_SECONDS = "10";
const { put } = await import("@vercel/blob");
(put as unknown as import("vitest").Mock).mockClear();
const url = await putScreenshotBlob(
"example.com",
1200,
630,
Buffer.from([1]),
);
expect(url).toBe("https://blob/put.png");
const calls = (put as unknown as import("vitest").Mock).mock.calls;
const call = calls[calls.length - 1];
expect(call?.[0]).toMatch(/^screenshots\//);
expect(call?.[0]).toMatch(/\/1200x630\.png$/);
expect(call?.[2]).toMatchObject({
access: "public",
contentType: "image/png",
});
});
// head* helpers removed in favor of Redis index
});

92
lib/blob.ts
View File

@@ -1,92 +0,0 @@
import "server-only";
import { createHmac } from "node:crypto";
import { put } from "@vercel/blob";
const ONE_WEEK_SECONDS = 7 * 24 * 60 * 60;
function toPositiveInt(value: unknown, fallback: number): number {
const n = Number(value);
return Number.isFinite(n) && n > 0 ? Math.floor(n) : fallback;
}
export function getFaviconTtlSeconds(): number {
return toPositiveInt(process.env.FAVICON_TTL_SECONDS, ONE_WEEK_SECONDS);
}
export function getScreenshotTtlSeconds(): number {
return toPositiveInt(process.env.SCREENSHOT_TTL_SECONDS, ONE_WEEK_SECONDS);
}
function getSigningSecret(): string {
if (
process.env.NODE_ENV === "production" &&
!process.env.BLOB_SIGNING_SECRET
) {
throw new Error("BLOB_SIGNING_SECRET required in production");
}
const secret =
process.env.BLOB_SIGNING_SECRET ||
process.env.BLOB_READ_WRITE_TOKEN ||
"dev-favicon-secret";
return secret;
}
export function computeFaviconBlobPath(domain: string, size: number): string {
const input = `${domain}:${size}`;
const secret = getSigningSecret();
const digest = createHmac("sha256", secret).update(input).digest("hex");
// Avoid leaking domain; path is deterministic but unpredictable without secret
return `favicons/${digest}/${size}.png`;
}
export async function putFaviconBlob(
domain: string,
size: number,
png: Buffer,
): Promise<string> {
const input = `${domain}:${size}`;
const secret = getSigningSecret();
const digest = createHmac("sha256", secret).update(input).digest("hex");
const pathname = `favicons/${digest}/${size}.png`;
const res = await put(pathname, png, {
access: "public",
contentType: "image/png",
cacheControlMaxAge: getFaviconTtlSeconds(),
token: process.env.BLOB_READ_WRITE_TOKEN,
});
return res.url;
}
// Screenshot blob helpers (same HMAC hashing approach as favicons)
export function computeScreenshotBlobPath(
domain: string,
width: number,
height: number,
): string {
const input = `${domain}:${width}x${height}`;
const secret = getSigningSecret();
const digest = createHmac("sha256", secret).update(input).digest("hex");
return `screenshots/${digest}/${width}x${height}.png`;
}
export async function putScreenshotBlob(
domain: string,
width: number,
height: number,
png: Buffer,
): Promise<string> {
const input = `${domain}:${width}x${height}`;
const secret = getSigningSecret();
const digest = createHmac("sha256", secret).update(input).digest("hex");
const pathname = `screenshots/${digest}/${width}x${height}.png`;
const res = await put(pathname, png, {
access: "public",
contentType: "image/png",
cacheControlMaxAge: getScreenshotTtlSeconds(),
token: process.env.BLOB_READ_WRITE_TOKEN,
});
return res.url;
}

59
lib/storage.test.ts Normal file
View File

@@ -0,0 +1,59 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
const utMock = vi.hoisted(() => ({
uploadFiles: vi.fn(async () => ({
data: { ufsUrl: "https://app.ufs.sh/f/mock-key", key: "mock-key" },
error: null,
})),
}));
vi.mock("uploadthing/server", async () => {
const actual =
await vi.importActual<typeof import("uploadthing/server")>(
"uploadthing/server",
);
return {
...actual,
UTApi: vi.fn().mockImplementation(() => utMock),
};
});
import { uploadImage } from "./storage";
afterEach(() => {
vi.restoreAllMocks();
utMock.uploadFiles.mockClear();
});
describe("storage uploads", () => {
it("uploadImage (favicon) returns ufsUrl and key and calls UTApi", async () => {
const res = await uploadImage({
kind: "favicon",
domain: "example.com",
width: 32,
height: 32,
png: Buffer.from([1, 2, 3]),
});
expect(res.url).toBe("https://app.ufs.sh/f/mock-key");
expect(res.key).toBe("mock-key");
const callArg = (utMock.uploadFiles as unknown as import("vitest").Mock)
.mock.calls[0]?.[0];
expect(callArg).toBeInstanceOf(Blob);
});
it("uploadImage (screenshot) returns ufsUrl and key and calls UTApi", async () => {
const res = await uploadImage({
kind: "screenshot",
domain: "example.com",
width: 1200,
height: 630,
png: Buffer.from([4, 5, 6]),
});
expect(res.url).toBe("https://app.ufs.sh/f/mock-key");
expect(res.key).toBe("mock-key");
const callArg = (utMock.uploadFiles as unknown as import("vitest").Mock)
.mock.calls[0]?.[0];
expect(callArg).toBeInstanceOf(Blob);
});
});

60
lib/storage.ts Normal file
View File

@@ -0,0 +1,60 @@
import "server-only";
import { UTApi, UTFile } from "uploadthing/server";
const ONE_WEEK_SECONDS = 7 * 24 * 60 * 60;
function toPositiveInt(value: unknown, fallback: number): number {
const n = Number(value);
return Number.isFinite(n) && n > 0 ? Math.floor(n) : fallback;
}
export function getFaviconTtlSeconds(): number {
return toPositiveInt(process.env.FAVICON_TTL_SECONDS, ONE_WEEK_SECONDS);
}
export function getScreenshotTtlSeconds(): number {
return toPositiveInt(process.env.SCREENSHOT_TTL_SECONDS, ONE_WEEK_SECONDS);
}
const utapi = new UTApi();
type UploadThingResult =
| {
data: { key?: string; ufsUrl?: string; url?: string } | null;
error: unknown | null;
}
| Array<{
data: { key?: string; ufsUrl?: string; url?: string } | null;
error: unknown | null;
}>;
function extractUploadResult(result: UploadThingResult) {
const entry = (Array.isArray(result) ? result[0] : result) as {
data: { key?: string; ufsUrl?: string; url?: string } | null;
error: unknown | null;
};
const key = entry?.data?.key;
const url = entry?.data?.ufsUrl;
if (typeof key === "string" && typeof url === "string") return { url, key };
throw new Error("Upload failed: missing url/key in response");
}
export async function uploadImage(options: {
kind: "favicon" | "screenshot";
domain: string;
width: number;
height: number;
png: Buffer;
}): Promise<{ url: string; key: string }> {
const { kind, domain, width, height, png } = options;
const safeDomain = domain.replace(/[^a-zA-Z0-9]/g, "-");
const fileName = `${kind}_${safeDomain}_${width}x${height}.png`;
const customId = fileName; // deterministic id to prevent duplicate uploads
const file = new UTFile([new Uint8Array(png)], fileName, {
type: "image/png",
customId,
});
const result = await utapi.uploadFiles(file);
return extractUploadResult(result);
}

2
package.json
View File

@@ -34,7 +34,6 @@
"@trpc/tanstack-react-query": "^11.6.0",
"@upstash/redis": "^1.35.4",
"@vercel/analytics": "^1.5.0",
"@vercel/blob": "^2.0.0",
"@vercel/functions": "^3.1.1",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
@@ -62,6 +61,7 @@
"superjson": "^2.2.2",
"tailwind-merge": "^3.3.1",
"tldts": "^7.0.16",
"uploadthing": "^7.7.4",
"uuid": "^13.0.0",
"vaul": "^1.1.2",
"zod": "^4.1.11"

253
pnpm-lock.yaml generated
View File

@@ -38,9 +38,6 @@ importers:
'@vercel/analytics':
specifier: ^1.5.0
version: 1.5.0(next@15.6.0-canary.39(@babel/core@7.28.4)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.1.1(react@19.1.1))(react@19.1.1))(react@19.1.1)
'@vercel/blob':
specifier: ^2.0.0
version: 2.0.0
'@vercel/functions':
specifier: ^3.1.1
version: 3.1.1
@@ -122,6 +119,9 @@ importers:
tldts:
specifier: ^7.0.16
version: 7.0.16
uploadthing:
specifier: ^7.7.4
version: 7.7.4(next@15.6.0-canary.39(@babel/core@7.28.4)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.1.1(react@19.1.1))(react@19.1.1))(tailwindcss@4.1.14)
uuid:
specifier: ^13.0.0
version: 13.0.0
@@ -399,6 +399,11 @@ packages:
'@date-fns/utc@2.1.1':
resolution: {integrity: sha512-SlJDfG6RPeEX8wEVv6ZB3kak4MmbtyiI2qX/5zuKdordbrhB/iaJ58GVMZgJ6P1sJaM1gMgENFYYeg1JWrCFrA==}
'@effect/platform@0.90.3':
resolution: {integrity: sha512-XvQ37yzWQKih4Du2CYladd1i/MzqtgkTPNCaN6Ku6No4CK83hDtXIV/rP03nEoBg2R3Pqgz6gGWmE2id2G81HA==}
peerDependencies:
effect: ^3.17.7
'@emnapi/runtime@1.5.0':
resolution: {integrity: sha512-97/BJ3iXHww3djw6hYIfErCZFee7qCtrneuLa20UXFCOTCfBM2cvQHjWJ2EG0s0MtdNwInarqCTz35i4wWXHsQ==}
@@ -558,10 +563,6 @@ packages:
cpu: [x64]
os: [win32]
'@fastify/busboy@2.1.1':
resolution: {integrity: sha512-vBZP4NlzfOlerQTnba4aqZoMhE/a9HY7HRqoOPaETQcSQuWEIyZMHGfVu6w9wGtGK5fED5qRs2DteVCjOH60sA==}
engines: {node: '>=14'}
'@floating-ui/core@1.7.3':
resolution: {integrity: sha512-sGnvb5dmrJaKEZ+LDIpguvdX3bDlEllmv4/ClQ9awcmCZrlx5jQyyMWFM5kBI+EyNOCDDiKk8il0zeuX3Zlg/w==}
@@ -780,6 +781,36 @@ packages:
resolution: {integrity: sha512-cOZZOVhDSulgK0meTsTkmNXb1ahVvmTmWmfx9gRBwc6hq98wS9JP35ESIoNq3xqEan+UN+gn8187Z6E4NKhLsw==}
hasBin: true
'@msgpackr-extract/msgpackr-extract-darwin-arm64@3.0.3':
resolution: {integrity: sha512-QZHtlVgbAdy2zAqNA9Gu1UpIuI8Xvsd1v8ic6B2pZmeFnFcMWiPLfWXh7TVw4eGEZ/C9TH281KwhVoeQUKbyjw==}
cpu: [arm64]
os: [darwin]
'@msgpackr-extract/msgpackr-extract-darwin-x64@3.0.3':
resolution: {integrity: sha512-mdzd3AVzYKuUmiWOQ8GNhl64/IoFGol569zNRdkLReh6LRLHOXxU4U8eq0JwaD8iFHdVGqSy4IjFL4reoWCDFw==}
cpu: [x64]
os: [darwin]
'@msgpackr-extract/msgpackr-extract-linux-arm64@3.0.3':
resolution: {integrity: sha512-YxQL+ax0XqBJDZiKimS2XQaf+2wDGVa1enVRGzEvLLVFeqa5kx2bWbtcSXgsxjQB7nRqqIGFIcLteF/sHeVtQg==}
cpu: [arm64]
os: [linux]
'@msgpackr-extract/msgpackr-extract-linux-arm@3.0.3':
resolution: {integrity: sha512-fg0uy/dG/nZEXfYilKoRe7yALaNmHoYeIoJuJ7KJ+YyU2bvY8vPv27f7UKhGRpY6euFYqEVhxCFZgAUNQBM3nw==}
cpu: [arm]
os: [linux]
'@msgpackr-extract/msgpackr-extract-linux-x64@3.0.3':
resolution: {integrity: sha512-cvwNfbP07pKUfq1uH+S6KJ7dT9K8WOE4ZiAcsrSes+UY55E/0jLYc+vq+DO7jlmqRb5zAggExKm0H7O/CBaesg==}
cpu: [x64]
os: [linux]
'@msgpackr-extract/msgpackr-extract-win32-x64@3.0.3':
resolution: {integrity: sha512-x0fWaQtYp4E6sktbsdAqnehxDgEc/VwM7uLsRCYWaiGu0ykYdZPiS8zCWdnjHwyiumousxfBm4SO31eXqwEZhQ==}
cpu: [x64]
os: [win32]
'@next/env@15.6.0-canary.39':
resolution: {integrity: sha512-WvJxtTel5Yt+z1QmCfgdFTOwk3edEzvhh6G1AuV45g2JJfx/8PljYEGVApJlUe2pjfKpW3K3K56qmeaaa+h7pw==}
@@ -831,6 +862,10 @@ packages:
cpu: [x64]
os: [win32]
'@opentelemetry/semantic-conventions@1.37.0':
resolution: {integrity: sha512-JD6DerIKdJGmRp4jQyX5FlrQjA4tjOw1cvfsPAZXfOOEErMUHjPcPSICS+6WnM0nB0efSFARh0KAZss+bvExOA==}
engines: {node: '>=14'}
'@pkgjs/parseargs@0.11.0':
resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==}
engines: {node: '>=14'}
@@ -1667,6 +1702,12 @@ packages:
resolution: {integrity: sha512-pDyHiSp+buakpUq23b74JPC9T5M58665y6ULlh8uSuIDK0vxVGyLzjTTigQL202c6+0+NNp1Po5rgWcT7JSO5g==}
engines: {node: '>=20.11.0'}
'@standard-schema/spec@1.0.0':
resolution: {integrity: sha512-m2bOd0f2RT9k8QJx1JN85cZYyH1RqFBdlwtkSlf4tBDYLCiiZnv1fIIwacK6cqwXavOydf0NPToMQgpKq+dVlA==}
'@standard-schema/spec@1.0.0-beta.4':
resolution: {integrity: sha512-d3IxtzLo7P1oZ8s8YNvxzBUXRXojSut8pbPrTYtzsc5sn4+53jVqbk66pQerSZbZSJZQux6LkclB/+8IDordHg==}
'@swc/helpers@0.5.15':
resolution: {integrity: sha512-JQ5TuMi45Owi4/BIMAJBoSQoOJu12oOk/gADqlcUL9JEdHB8vyjUSsxqeNXnmXHjYKMi2WcYtezGEEhqUI/E2g==}
@@ -1884,6 +1925,12 @@ packages:
'@types/yauzl@2.10.3':
resolution: {integrity: sha512-oJoftv0LSuaDZE3Le4DbKX+KS9G36NzOeSap90UIK0yMA/NhKJhqlSGtNDORNRaIbQfzjXDrQa0ytJ6mNRGz/Q==}
'@uploadthing/mime-types@0.3.6':
resolution: {integrity: sha512-t3tTzgwFV9+1D7lNDYc7Lr7kBwotHaX0ZsvoCGe7xGnXKo9z0jG2Sjl/msll12FeoLj77nyhsxevXyGpQDBvLg==}
'@uploadthing/shared@7.1.10':
resolution: {integrity: sha512-R/XSA3SfCVnLIzFpXyGaKPfbwlYlWYSTuGjTFHuJhdAomuBuhopAHLh2Ois5fJibAHzi02uP1QCKbgTAdmArqg==}
'@upstash/redis@1.35.4':
resolution: {integrity: sha512-WE1ZnhFyBiIjTDW13GbO6JjkiMVVjw5VsvS8ENmvvJsze/caMQ5paxVD44+U68IUVmkXcbsLSoE+VIYsHtbQEw==}
@@ -1913,10 +1960,6 @@ packages:
vue-router:
optional: true
'@vercel/blob@2.0.0':
resolution: {integrity: sha512-oAj7Pdy83YKSwIaMFoM7zFeLYWRc+qUpW3PiDSblxQMnGFb43qs4bmfq7dr/+JIfwhs6PTwe1o2YBwKhyjWxXw==}
engines: {node: '>=20.0.0'}
'@vercel/functions@3.1.1':
resolution: {integrity: sha512-pMum41m6DtMGWZuCPy99mCE40shLvoxawOhetkTGq/HslY7J7weMOV+TeWcfWO6acDabDqDcb6bapWBQ+tNA3w==}
engines: {node: '>= 20'}
@@ -2060,9 +2103,6 @@ packages:
ast-v8-to-istanbul@0.3.5:
resolution: {integrity: sha512-9SdXjNheSiE8bALAQCQQuT6fgQaoxJh7IRYrRGZ8/9nv8WhJeC1aXAwN8TbaOssGOukUvyvnkgD9+Yuykvl1aA==}
async-retry@1.3.3:
resolution: {integrity: sha512-wfr/jstw9xNi/0teMHrRW7dsz3Lt5ARhYNZ2ewpadnhaIp5mbALhOAP+EAdsC7t4Z6wqsDVv9+W6gm1Dk9mEyw==}
asynckit@0.4.0:
resolution: {integrity: sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==}
@@ -2339,6 +2379,9 @@ packages:
easy-table@1.1.0:
resolution: {integrity: sha512-oq33hWOSSnl2Hoh00tZWaIPi1ievrD9aFG82/IgjlycAnW9hHx5PkJiXpxPsgEE+H7BsbVQXFVFST8TEXS6/pA==}
effect@3.17.7:
resolution: {integrity: sha512-dpt0ONUn3zzAuul6k4nC/coTTw27AL5nhkORXgTi6NfMPzqWYa1M05oKmOMTxpVSTKepqXVcW9vIwkuaaqx9zA==}
electron-to-chromium@1.5.230:
resolution: {integrity: sha512-A6A6Fd3+gMdaed9wX83CvHYJb4UuapPD5X5SLq72VZJzxHSY0/LUweGXRWmQlh2ln7KV7iw7jnwXK7dlPoOnHQ==}
@@ -2439,6 +2482,10 @@ packages:
engines: {node: '>= 10.17.0'}
hasBin: true
fast-check@3.23.2:
resolution: {integrity: sha512-h5+1OzzfCC3Ef7VbtKdcv7zsstUQwUDlYpUTvjeUsJAssPgLn7QzbboPtL5ro04Mq0rPOsMzl7q5hIbRs2wD1A==}
engines: {node: '>=8.0.0'}
fast-fifo@1.3.2:
resolution: {integrity: sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==}
@@ -2468,6 +2515,9 @@ packages:
resolution: {integrity: sha512-VZR5I7k5wkD0HgFnMsq5hOsSc710MJMu5Nc5QYsbe38NN5iPV/XTObYLc/cpttRTf6lX538+5uO1ZQRhYibiZQ==}
engines: {node: '>=18'}
find-my-way-ts@0.1.6:
resolution: {integrity: sha512-a85L9ZoXtNAey3Y6Z+eBWW658kO/MwR7zIafkIUPUMf3isZG0NCs2pjW2wtjxAKuJPxMAsHUIP4ZPGv0o5gyTA==}
flatted@3.3.3:
resolution: {integrity: sha512-GX+ysw4PBCz0PzosHDepZGANEuFCMLrnRTiEy9McGjmkCQYwRq4A/X786G/fjM/+OjsWSU1ZrY5qyARZmO/uwg==}
@@ -2633,10 +2683,6 @@ packages:
resolution: {integrity: sha512-Ag3wB2o37wslZS19hZqorUnrnzSkpOVy+IiiDEiTqNubEYpYuHWIf6K4psgN2ZWKExS4xhVCrRVfb/wfW8fWJA==}
engines: {node: '>= 10'}
is-buffer@2.0.5:
resolution: {integrity: sha512-i2R6zNFDwgEHJyQUtJEk0XFi1i0dPFn/oqjK3/vPCcDeJvW5NQ83V8QbicfF1SupOaB0h8ntgBC2YiE7dfyctQ==}
engines: {node: '>=4'}
is-extendable@0.1.1:
resolution: {integrity: sha512-5BMULNob1vgFX6EjQw5izWDxrecWK9AM72rugNr0TFldMOi0fj6Jk+zeKIt0xGj4cEfQIJth4w3OKWOJ4f+AFw==}
engines: {node: '>=0.10.0'}
@@ -2649,9 +2695,6 @@ packages:
resolution: {integrity: sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==}
engines: {node: '>=8'}
is-node-process@1.2.0:
resolution: {integrity: sha512-Vg4o6/fqPxIjtxgUH5QLJhwZ7gW5diGCVlXpuUfELC62CuxM1iHcRe51f2W1FDy04Ai4KJkagKjx3XaqyfRKXw==}
is-plain-object@2.0.4:
resolution: {integrity: sha512-h5PpgXkWitc38BBMYawTYMWJHFZJVnBquFE57xFpjB8pJFiF6gZ+bU+WyI/yqXiFR5mdLsgYNaPe8uao6Uv9Og==}
engines: {node: '>=0.10.0'}
@@ -2912,6 +2955,16 @@ packages:
ms@2.1.3:
resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==}
msgpackr-extract@3.0.3:
resolution: {integrity: sha512-P0efT1C9jIdVRefqjzOQ9Xml57zpOXnIuS+csaB4MdZbTdmGDLo8XhzBG1N7aO11gKDDkJvBLULeFTo46wwreA==}
hasBin: true
msgpackr@1.11.5:
resolution: {integrity: sha512-UjkUHN0yqp9RWKy0Lplhh+wlpdt9oQBYgULZOiFhV3VclSF1JnSQWZ5r9gORQlNYaUKQoR8itv7g7z1xDDuACA==}
multipasta@0.2.7:
resolution: {integrity: sha512-KPA58d68KgGil15oDqXjkUBEBYc00XvbPj5/X+dyzeo/lWm9Nc25pQRlf1D+gv4OpK7NM0J1odrbu9JNNGvynA==}
murmurhash-js@1.0.0:
resolution: {integrity: sha512-TvmkNhkv8yct0SVBSy+o8wYzXjE4Zz3PCesbfs8HiCXXdcTuocApFv11UWlNFWKYsP2okqrhb7JNlSm9InBhIw==}
@@ -2960,6 +3013,10 @@ packages:
encoding:
optional: true
node-gyp-build-optional-packages@5.2.2:
resolution: {integrity: sha512-s+w+rBWnpTMwSFbaE0UXsRlg7hU4FjekKU4eyAih5T8nJuNZT1nNsskXpxmeqSK9UzkBl6UgRlnKc8hz8IEqOw==}
hasBin: true
node-releases@2.0.23:
resolution: {integrity: sha512-cCmFDMSm26S6tQSDpBCg/NR8NENrVPhAJSf+XbxBG4rPFaaonlEoE9wHQmun+cls499TQGSb7ZyPBRlzgKfpeg==}
@@ -3098,6 +3155,9 @@ packages:
resolution: {integrity: sha512-M/Jhg4PWRANSbL/C9im//Yb55wsWBS5wdp+h59iwM+EPicVQQCNs56iC5aEAO7avfDPRfxs4MM16wHjOYHNJEw==}
engines: {node: '>=18'}
pure-rand@6.1.0:
resolution: {integrity: sha512-bVWawvoZoBYpp6yIoQtQXHZjmz35RSVHnUOTefl8Vcjr8snTPY1wnpSPMWekcFwbxI6gtmT7rSYPFvz71ldiOA==}
quickselect@3.0.0:
resolution: {integrity: sha512-XdjUArbK4Bm5fLLvlm5KpTFOiOThgfWWI4axAZDWg4E/0mKdZyI9tNEfds27qCi1ze/vwTR16kvmmGhRra3c2g==}
@@ -3203,10 +3263,6 @@ packages:
resolve-protobuf-schema@2.1.0:
resolution: {integrity: sha512-kI5ffTiZWmJaS/huM8wZfEMer1eRd7oJQhDuxeCLe3t7N7mX3z94CN0xPxBQxFYQTSNz9T0i+v6inKqSdK8xrQ==}
retry@0.13.1:
resolution: {integrity: sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg==}
engines: {node: '>= 4'}
rimraf@6.0.1:
resolution: {integrity: sha512-9dkvaxAsk/xNXSJzMgFqqMCuFgt2+KsOFek3TMLfo8NCPfWpBmqwyNn5Y+NX56QUYfCtsyhF3ayiboEoUmJk/A==}
engines: {node: 20 || >=22}
@@ -3327,6 +3383,9 @@ packages:
resolution: {integrity: sha512-NzNVhJDYpwceVVii8/Hu6DKfD2G+NrQHlS/V/qgv763EYudVwEcMQNxd2lh+0VrUByXN/oJkl5grOhYWvQUYiw==}
engines: {node: '>=0.10.0'}
sqids@0.3.0:
resolution: {integrity: sha512-lOQK1ucVg+W6n3FhRwwSeUijxe93b51Bfz5PMRMihVf1iVkl82ePQG7V5vwrhzB11v0NtsR25PSZRGiSomJaJw==}
stackback@0.0.2:
resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==}
@@ -3424,10 +3483,6 @@ packages:
text-decoder@1.2.3:
resolution: {integrity: sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA==}
throttleit@2.1.0:
resolution: {integrity: sha512-nt6AMGKW1p/70DF/hGBdJB57B8Tspmbp5gfJ8ilhLnt7kkr2ye7hzD6NVG8GGErk2HWF34igrL2CXmNIkzKqKw==}
engines: {node: '>=18'}
tinybench@2.9.0:
resolution: {integrity: sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==}
@@ -3540,10 +3595,6 @@ packages:
undici-types@7.13.0:
resolution: {integrity: sha512-Ov2Rr9Sx+fRgagJ5AX0qvItZG/JKKoBRAVITs1zk7IqZGTJUwgUr7qoYBpWwakpWilTZFM98rG/AFRocu10iIQ==}
undici@5.29.0:
resolution: {integrity: sha512-raqeBD6NQK4SkWhQzeYKd1KmIG6dllBOTt55Rmkt4HtI9mwdWtJljnrXjAFUBLTSN67HWrOIZ3EPF4kjUw80Bg==}
engines: {node: '>=14.0'}
union-value@1.0.1:
resolution: {integrity: sha512-tJfXmxMeWYnczCVs7XAEvIV7ieppALdyepWMkHkwciRpZraG/xwT+s2JN8+pr1+8jCRf80FFzvr+MpQeeoF4Xg==}
engines: {node: '>=0.10.0'}
@@ -3554,6 +3605,27 @@ packages:
peerDependencies:
browserslist: '>= 4.21.0'
uploadthing@7.7.4:
resolution: {integrity: sha512-rlK/4JWHW5jP30syzWGBFDDXv3WJDdT8gn9OoxRJmXLoXi94hBmyyjxihGlNrKhBc81czyv8TkzMioe/OuKGfA==}
engines: {node: '>=18.13.0'}
peerDependencies:
express: '*'
fastify: '*'
h3: '*'
next: '*'
tailwindcss: ^3.0.0 || ^4.0.0-beta.0
peerDependenciesMeta:
express:
optional: true
fastify:
optional: true
h3:
optional: true
next:
optional: true
tailwindcss:
optional: true
use-callback-ref@1.3.3:
resolution: {integrity: sha512-jQL3lRnocaFtu3V00JToYz/4QkNWswxijDaCVNZRiRTO3HQDLsdu1ZtmIUvV4yPp+rvWm5j0y0TG/S61cuijTg==}
engines: {node: '>=10'}
@@ -3986,6 +4058,14 @@ snapshots:
'@date-fns/utc@2.1.1': {}
'@effect/platform@0.90.3(effect@3.17.7)':
dependencies:
'@opentelemetry/semantic-conventions': 1.37.0
effect: 3.17.7
find-my-way-ts: 0.1.6
msgpackr: 1.11.5
multipasta: 0.2.7
'@emnapi/runtime@1.5.0':
dependencies:
tslib: 2.8.1
@@ -4069,8 +4149,6 @@ snapshots:
'@esbuild/win32-x64@0.25.10':
optional: true
'@fastify/busboy@2.1.1': {}
'@floating-ui/core@1.7.3':
dependencies:
'@floating-ui/utils': 0.2.10
@@ -4272,6 +4350,24 @@ snapshots:
rw: 1.3.3
sort-object: 3.0.3
'@msgpackr-extract/msgpackr-extract-darwin-arm64@3.0.3':
optional: true
'@msgpackr-extract/msgpackr-extract-darwin-x64@3.0.3':
optional: true
'@msgpackr-extract/msgpackr-extract-linux-arm64@3.0.3':
optional: true
'@msgpackr-extract/msgpackr-extract-linux-arm@3.0.3':
optional: true
'@msgpackr-extract/msgpackr-extract-linux-x64@3.0.3':
optional: true
'@msgpackr-extract/msgpackr-extract-win32-x64@3.0.3':
optional: true
'@next/env@15.6.0-canary.39': {}
'@next/swc-darwin-arm64@15.6.0-canary.39':
@@ -4298,6 +4394,8 @@ snapshots:
'@next/swc-win32-x64-msvc@15.6.0-canary.39':
optional: true
'@opentelemetry/semantic-conventions@1.37.0': {}
'@pkgjs/parseargs@0.11.0':
optional: true
@@ -5163,6 +5261,10 @@ snapshots:
- debug
- react-native-b4a
'@standard-schema/spec@1.0.0': {}
'@standard-schema/spec@1.0.0-beta.4': {}
'@swc/helpers@0.5.15':
dependencies:
tslib: 2.8.1
@@ -5372,6 +5474,14 @@ snapshots:
'@types/node': 24.6.2
optional: true
'@uploadthing/mime-types@0.3.6': {}
'@uploadthing/shared@7.1.10':
dependencies:
'@uploadthing/mime-types': 0.3.6
effect: 3.17.7
sqids: 0.3.0
'@upstash/redis@1.35.4':
dependencies:
uncrypto: 0.1.3
@@ -5381,14 +5491,6 @@ snapshots:
next: 15.6.0-canary.39(@babel/core@7.28.4)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.1.1(react@19.1.1))(react@19.1.1)
react: 19.1.1
'@vercel/blob@2.0.0':
dependencies:
async-retry: 1.3.3
is-buffer: 2.0.5
is-node-process: 1.2.0
throttleit: 2.1.0
undici: 5.29.0
'@vercel/functions@3.1.1':
dependencies:
'@vercel/oidc': 3.0.1
@@ -5538,10 +5640,6 @@ snapshots:
estree-walker: 3.0.3
js-tokens: 9.0.1
async-retry@1.3.3:
dependencies:
retry: 0.13.1
asynckit@0.4.0: {}
axios-proxy-builder@0.1.2:
@@ -5816,6 +5914,11 @@ snapshots:
optionalDependencies:
wcwidth: 1.0.1
effect@3.17.7:
dependencies:
'@standard-schema/spec': 1.0.0
fast-check: 3.23.2
electron-to-chromium@1.5.230: {}
emoji-regex@8.0.0: {}
@@ -5930,6 +6033,10 @@ snapshots:
transitivePeerDependencies:
- supports-color
fast-check@3.23.2:
dependencies:
pure-rand: 6.1.0
fast-fifo@1.3.2: {}
fd-slicer@1.1.0:
@@ -5957,6 +6064,8 @@ snapshots:
token-types: 6.1.1
uint8array-extras: 1.5.0
find-my-way-ts@0.1.6: {}
flatted@3.3.3: {}
follow-redirects@1.15.11: {}
@@ -6120,8 +6229,6 @@ snapshots:
ipaddr.js@2.2.0: {}
is-buffer@2.0.5: {}
is-extendable@0.1.1: {}
is-extendable@1.0.1:
@@ -6130,8 +6237,6 @@ snapshots:
is-fullwidth-code-point@3.0.0: {}
is-node-process@1.2.0: {}
is-plain-object@2.0.4:
dependencies:
isobject: 3.0.1
@@ -6389,6 +6494,24 @@ snapshots:
ms@2.1.3: {}
msgpackr-extract@3.0.3:
dependencies:
node-gyp-build-optional-packages: 5.2.2
optionalDependencies:
'@msgpackr-extract/msgpackr-extract-darwin-arm64': 3.0.3
'@msgpackr-extract/msgpackr-extract-darwin-x64': 3.0.3
'@msgpackr-extract/msgpackr-extract-linux-arm': 3.0.3
'@msgpackr-extract/msgpackr-extract-linux-arm64': 3.0.3
'@msgpackr-extract/msgpackr-extract-linux-x64': 3.0.3
'@msgpackr-extract/msgpackr-extract-win32-x64': 3.0.3
optional: true
msgpackr@1.11.5:
optionalDependencies:
msgpackr-extract: 3.0.3
multipasta@0.2.7: {}
murmurhash-js@1.0.0: {}
nanoid@3.3.11: {}
@@ -6428,6 +6551,11 @@ snapshots:
dependencies:
whatwg-url: 5.0.0
node-gyp-build-optional-packages@5.2.2:
dependencies:
detect-libc: 2.1.2
optional: true
node-releases@2.0.23: {}
once@1.4.0:
@@ -6578,6 +6706,8 @@ snapshots:
- supports-color
- utf-8-validate
pure-rand@6.1.0: {}
quickselect@3.0.0: {}
radix-ui@1.4.3(@types/react-dom@19.1.9(@types/react@19.1.16))(@types/react@19.1.16)(react-dom@19.1.1(react@19.1.1))(react@19.1.1):
@@ -6721,8 +6851,6 @@ snapshots:
dependencies:
protocol-buffers-schema: 3.6.0
retry@0.13.1: {}
rimraf@6.0.1:
dependencies:
glob: 11.0.3
@@ -6876,6 +7004,8 @@ snapshots:
dependencies:
extend-shallow: 3.0.2
sqids@0.3.0: {}
stackback@0.0.2: {}
std-env@3.9.0: {}
@@ -6996,8 +7126,6 @@ snapshots:
transitivePeerDependencies:
- react-native-b4a
throttleit@2.1.0: {}
tinybench@2.9.0: {}
tinycolor2@1.6.0: {}
@@ -7078,10 +7206,6 @@ snapshots:
undici-types@7.13.0: {}
undici@5.29.0:
dependencies:
'@fastify/busboy': 2.1.1
union-value@1.0.1:
dependencies:
arr-union: 3.1.0
@@ -7095,6 +7219,17 @@ snapshots:
escalade: 3.2.0
picocolors: 1.1.1
uploadthing@7.7.4(next@15.6.0-canary.39(@babel/core@7.28.4)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.1.1(react@19.1.1))(react@19.1.1))(tailwindcss@4.1.14):
dependencies:
'@effect/platform': 0.90.3(effect@3.17.7)
'@standard-schema/spec': 1.0.0-beta.4
'@uploadthing/mime-types': 0.3.6
'@uploadthing/shared': 7.1.10
effect: 3.17.7
optionalDependencies:
next: 15.6.0-canary.39(@babel/core@7.28.4)(babel-plugin-react-compiler@19.1.0-rc.3)(react-dom@19.1.1(react@19.1.1))(react@19.1.1)
tailwindcss: 4.1.14
use-callback-ref@1.3.3(@types/react@19.1.16)(react@19.1.1):
dependencies:
react: 19.1.1

1
pnpm-workspace.yaml
View File

@@ -3,5 +3,6 @@ onlyBuiltDependencies:
- '@tailwindcss/oxide'
- core-js
- esbuild
- msgpackr-extract
- puppeteer
- sharp

17
server/services/favicon.test.ts
View File

@@ -1,11 +1,14 @@
/* @vitest-environment node */
import { afterEach, describe, expect, it, vi } from "vitest";
const blobMock = vi.hoisted(() => ({
putFaviconBlob: vi.fn(async () => "blob://stored-url"),
const storageMock = vi.hoisted(() => ({
uploadImage: vi.fn(async () => ({
url: "https://app.ufs.sh/f/stored-url",
key: "ut-key",
})),
}));
vi.mock("@/lib/blob", () => blobMock);
vi.mock("@/lib/storage", () => storageMock);
// Mock sharp to return a pipeline that resolves a buffer
vi.mock("sharp", () => ({
@@ -23,7 +26,7 @@ import { getOrCreateFaviconBlobUrl } from "./favicon";
afterEach(() => {
vi.restoreAllMocks();
blobMock.putFaviconBlob.mockReset();
storageMock.uploadImage.mockReset();
global.__redisTestHelper.reset();
});
@@ -36,7 +39,7 @@ describe("getOrCreateFaviconBlobUrl", () => {
});
const out = await getOrCreateFaviconBlobUrl("example.com");
expect(out.url).toBe("blob://existing-url");
expect(blobMock.putFaviconBlob).not.toHaveBeenCalled();
expect(storageMock.uploadImage).not.toHaveBeenCalled();
});
it("reads object values from redis index", async () => {
@@ -69,8 +72,8 @@ describe("getOrCreateFaviconBlobUrl", () => {
const fetchSpy = vi.spyOn(global, "fetch").mockResolvedValue(resp);
const out = await getOrCreateFaviconBlobUrl("example.com");
expect(out.url).toBe("blob://stored-url");
expect(blobMock.putFaviconBlob).toHaveBeenCalled();
expect(out.url).toBe("https://app.ufs.sh/f/stored-url");
expect(storageMock.uploadImage).toHaveBeenCalled();
fetchSpy.mockRestore();
});

100
server/services/favicon.ts
View File

@@ -1,18 +1,12 @@
import { captureServer } from "@/lib/analytics/server";
import {
computeFaviconBlobPath,
getFaviconTtlSeconds,
putFaviconBlob,
} from "@/lib/blob";
import { USER_AGENT } from "@/lib/constants";
import { convertBufferToSquarePng } from "@/lib/image";
import { ns, redis } from "@/lib/redis";
import { getFaviconTtlSeconds, uploadImage } from "@/lib/storage";
const DEFAULT_SIZE = 32;
const REQUEST_TIMEOUT_MS = 1500; // per each method
const LOCK_TTL_SECONDS = 15;
const LOCK_WAIT_ATTEMPTS = 6;
const LOCK_WAIT_DELAY_MS = 250;
const LOCK_TTL_SECONDS = 10; // minimal barrier to avoid duplicate concurrent uploads
// Legacy Redis-based caching removed; Blob is now the canonical store
@@ -57,9 +51,9 @@ export async function getOrCreateFaviconBlobUrl(
console.debug("[favicon] start", { domain, size: DEFAULT_SIZE });
// 1) Check Redis index first
try {
const key = ns("favicon:url", `${domain}:${DEFAULT_SIZE}`);
console.debug("[favicon] redis get", { key });
const raw = (await redis.get(key)) as { url?: unknown } | null;
const indexKey = ns("favicon:url", `${domain}:${DEFAULT_SIZE}`);
console.debug("[favicon] redis get", { key: indexKey });
const raw = (await redis.get(indexKey)) as { url?: unknown } | null;
if (raw && typeof raw === "object" && typeof raw.url === "string") {
console.info("[favicon] cache hit", {
domain,
@@ -82,63 +76,18 @@ export async function getOrCreateFaviconBlobUrl(
}
// 2) Acquire short-lived lock to avoid duplicate concurrent uploads
async function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
async function waitForPublicUrl(url: string): Promise<void> {
if (process.env.NODE_ENV === "test") return;
for (let i = 0; i < 4; i++) {
try {
const res = await fetch(url, {
method: "HEAD",
cache: "no-store" as RequestCache,
});
if (res.ok) return;
} catch {}
await sleep(200);
}
}
// Minimal single-writer barrier (no waiting): if another worker is uploading, bail out
const lockKey = ns("lock", `favicon:${domain}:${DEFAULT_SIZE}`);
let acquiredLock = false;
try {
console.debug("[favicon] lock attempt", { lockKey });
const setRes = await redis.set(lockKey, "1", {
nx: true,
ex: LOCK_TTL_SECONDS,
});
// Upstash returns "OK" on success; our test mock returns undefined
acquiredLock = setRes === "OK" || setRes === undefined;
console.info("[favicon] lock status", { acquiredLock });
} catch {
// If the client doesn't support NX in some env, proceed without blocking
acquiredLock = true;
console.warn("[favicon] lock unsupported; proceeding without lock");
}
const acquired = setRes === "OK" || setRes === undefined;
if (!acquired) return { url: null };
} catch {}
if (!acquiredLock) {
// Another worker is producing the blob; wait briefly for index to be populated
for (let i = 0; i < LOCK_WAIT_ATTEMPTS; i++) {
try {
const key = ns("favicon:url", `${domain}:${DEFAULT_SIZE}`);
console.debug("[favicon] waiting for index", { attempt: i + 1, key });
const raw = (await redis.get(key)) as { url?: unknown } | null;
if (raw && typeof raw === "object" && typeof raw.url === "string") {
console.info("[favicon] index appeared while waiting", {
url: raw.url,
});
return { url: raw.url };
}
} catch {}
await sleep(LOCK_WAIT_DELAY_MS);
}
// Give up to avoid duplicate work; a subsequent request will retry
console.warn("[favicon] gave up waiting for lock; returning null", {
domain,
});
return { url: null };
}
// Removed Redis wait loops; we only guard with a short NX barrier
// 3) Fetch/convert via existing pipeline, then upload
try {
@@ -177,34 +126,38 @@ export async function getOrCreateFaviconBlobUrl(
return "unknown";
})();
const blobPath = computeFaviconBlobPath(domain, DEFAULT_SIZE);
console.info("[favicon] uploading to blob", { blobPath });
const url = await putFaviconBlob(domain, DEFAULT_SIZE, png);
console.info("[favicon] uploaded", { url });
await waitForPublicUrl(url);
console.debug("[favicon] public url ready", { url });
console.info("[favicon] uploading via uploadthing");
const { url, key } = await uploadImage({
kind: "favicon",
domain,
width: DEFAULT_SIZE,
height: DEFAULT_SIZE,
png,
});
console.info("[favicon] uploaded", { url, key });
// No need to wait for public URL
// 3) Write Redis index and schedule purge
try {
const ttl = getFaviconTtlSeconds();
const expiresAtMs = Date.now() + ttl * 1000;
const key = ns("favicon:url", `${domain}:${DEFAULT_SIZE}`);
const indexKey = ns("favicon:url", `${domain}:${DEFAULT_SIZE}`);
console.debug("[favicon] redis set index", {
key,
key: indexKey,
ttlSeconds: ttl,
expiresAtMs,
});
await redis.set(
key,
{ url, expiresAtMs },
indexKey,
{ url, key, expiresAtMs },
{
ex: ttl,
},
);
console.debug("[favicon] redis zadd purge", { url, expiresAtMs });
console.debug("[favicon] redis zadd purge", { key, expiresAtMs });
await redis.zadd(ns("purge", "favicon"), {
score: expiresAtMs,
member: url, // store full URL for deletion API
member: key, // store UploadThing file key for deletion API
});
} catch {
// best effort
@@ -243,7 +196,6 @@ export async function getOrCreateFaviconBlobUrl(
} finally {
try {
await redis.del(lockKey);
console.debug("[favicon] lock released", { lockKey });
} catch {}
}
}

21
server/services/screenshot.test.ts
View File

@@ -1,11 +1,14 @@
/* @vitest-environment node */
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const blobMock = vi.hoisted(() => ({
putScreenshotBlob: vi.fn(async () => "blob://stored-screenshot"),
const storageMock = vi.hoisted(() => ({
uploadImage: vi.fn(async () => ({
url: "https://app.ufs.sh/f/stored-screenshot",
key: "ut-key",
})),
}));
vi.mock("@/lib/blob", () => blobMock);
vi.mock("@/lib/storage", () => storageMock);
// Mock puppeteer environments
const pageMock = {
@@ -44,7 +47,7 @@ beforeEach(() => {
afterEach(() => {
vi.restoreAllMocks();
blobMock.putScreenshotBlob.mockReset();
storageMock.uploadImage.mockReset();
global.__redisTestHelper.reset();
pageMock.goto.mockReset();
pageMock.waitForNetworkIdle.mockReset();
@@ -60,15 +63,15 @@ describe("getOrCreateScreenshotBlobUrl", () => {
});
const out = await getOrCreateScreenshotBlobUrl("example.com");
expect(out.url).toBe("blob://existing");
expect(blobMock.putScreenshotBlob).not.toHaveBeenCalled();
expect(storageMock.uploadImage).not.toHaveBeenCalled();
});
// Drop string JSON case now that we assume automatic deserialization
it("captures, uploads and returns url when not cached", async () => {
const out = await getOrCreateScreenshotBlobUrl("example.com");
expect(out.url).toBe("blob://stored-screenshot");
expect(blobMock.putScreenshotBlob).toHaveBeenCalled();
expect(out.url).toBe("https://app.ufs.sh/f/stored-screenshot");
expect(storageMock.uploadImage).toHaveBeenCalled();
});
it("retries navigation failure and succeeds on second attempt", async () => {
@@ -85,7 +88,7 @@ describe("getOrCreateScreenshotBlobUrl", () => {
backoffMaxMs: 2,
});
Math.random = originalRandom;
expect(out.url).toBe("blob://stored-screenshot");
expect(out.url).toBe("https://app.ufs.sh/f/stored-screenshot");
expect(pageMock.goto).toHaveBeenCalledTimes(2);
});
@@ -105,7 +108,7 @@ describe("getOrCreateScreenshotBlobUrl", () => {
backoffMaxMs: 2,
});
Math.random = originalRandom;
expect(out.url).toBe("blob://stored-screenshot");
expect(out.url).toBe("https://app.ufs.sh/f/stored-screenshot");
expect(pageMock.screenshot).toHaveBeenCalledTimes(2);
});

184
server/services/screenshot.ts
View File

@@ -1,14 +1,10 @@
import type { Browser } from "puppeteer-core";
import { captureServer } from "@/lib/analytics/server";
import {
computeScreenshotBlobPath,
getScreenshotTtlSeconds,
putScreenshotBlob,
} from "@/lib/blob";
import { USER_AGENT } from "@/lib/constants";
import { addWatermarkToScreenshot, optimizePngCover } from "@/lib/image";
import { launchChromium } from "@/lib/puppeteer";
import { ns, redis } from "@/lib/redis";
import { getScreenshotTtlSeconds, uploadImage } from "@/lib/storage";
const VIEWPORT_WIDTH = 1200;
const VIEWPORT_HEIGHT = 630;
@@ -18,9 +14,8 @@ const IDLE_TIMEOUT_MS = 3000;
const CAPTURE_MAX_ATTEMPTS_DEFAULT = 3;
const CAPTURE_BACKOFF_BASE_MS_DEFAULT = 200;
const CAPTURE_BACKOFF_MAX_MS_DEFAULT = 1200;
const LOCK_TTL_SECONDS = 15;
const LOCK_WAIT_ATTEMPTS = 6;
const LOCK_WAIT_DELAY_MS = 250;
const LOCK_TTL_SECONDS = 10; // minimal barrier to avoid duplicate concurrent uploads
// Removed legacy URL propagation waits
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
@@ -98,83 +93,30 @@ export async function getOrCreateScreenshotBlobUrl(
}
// 2) Acquire short-lived lock to avoid duplicate concurrent captures/uploads
async function waitForPublicUrl(url: string): Promise<void> {
if (process.env.NODE_ENV === "test") return;
for (let i = 0; i < 4; i++) {
try {
const res = await fetch(url, {
method: "HEAD",
cache: "no-store" as RequestCache,
});
if (res.ok) return;
} catch {}
await sleep(200);
}
}
// URL propagation wait removed
// Removed Redis lock acquisition and wait loops
// Minimal single-writer barrier (no waiting): if another worker is uploading, bail out
const lockKey = ns(
"lock",
`screenshot:${domain}:${VIEWPORT_WIDTH}x${VIEWPORT_HEIGHT}`,
);
let acquiredLock = false;
try {
console.debug("[screenshot] lock attempt", { lockKey });
const setRes = await redis.set(lockKey, "1", {
nx: true,
ex: LOCK_TTL_SECONDS,
});
acquiredLock = setRes === "OK" || setRes === undefined;
console.info("[screenshot] lock status", { acquiredLock });
} catch {
acquiredLock = true;
console.warn("[screenshot] lock unsupported; proceeding without lock");
}
if (!acquiredLock) {
// Another worker is producing the blob; wait briefly for index to be populated
for (let i = 0; i < LOCK_WAIT_ATTEMPTS; i++) {
try {
const key = ns(
"screenshot:url",
`${domain}:${VIEWPORT_WIDTH}x${VIEWPORT_HEIGHT}`,
);
console.debug("[screenshot] waiting for index", {
attempt: i + 1,
key,
});
const raw = (await redis.get(key)) as { url?: unknown } | null;
if (raw && typeof raw === "object" && typeof raw.url === "string") {
console.info("[screenshot] index appeared while waiting", {
url: raw.url,
});
return { url: raw.url };
}
} catch {}
await sleep(LOCK_WAIT_DELAY_MS);
}
console.warn("[screenshot] gave up waiting for lock; returning null", {
domain,
});
return { url: null };
}
const acquired = setRes === "OK" || setRes === undefined;
if (!acquired) return { url: null };
} catch {}
// 3) Attempt to capture (wrapped to ensure lock release)
try {
let browser: Browser | null = null;
try {
// Re-check index after acquiring lock in case another writer finished
try {
const key = ns(
"screenshot:url",
`${domain}:${VIEWPORT_WIDTH}x${VIEWPORT_HEIGHT}`,
);
console.debug("[screenshot] redis recheck after lock", { key });
const raw = (await redis.get(key)) as { url?: unknown } | null;
if (raw && typeof raw === "object" && typeof raw.url === "string") {
console.info("[screenshot] found index after lock", { url: raw.url });
return { url: raw.url };
}
} catch {}
// Skip index recheck
browser = await launchChromium();
console.debug("[screenshot] browser launched", { mode: "chromium" });
@@ -185,39 +127,46 @@ export async function getOrCreateScreenshotBlobUrl(
for (let attemptIndex = 0; attemptIndex < attempts; attemptIndex++) {
try {
const page = await browser.newPage();
await page.setViewport({
width: VIEWPORT_WIDTH,
height: VIEWPORT_HEIGHT,
deviceScaleFactor: 1,
});
await page.setUserAgent(USER_AGENT);
console.debug("[screenshot] navigating", {
url,
attempt: attemptIndex + 1,
});
await page.goto(url, {
waitUntil: "domcontentloaded",
timeout: NAV_TIMEOUT_MS,
});
// Give chatty pages/CDNs a brief chance to settle without hanging
let rawPng: Buffer;
try {
await page.waitForNetworkIdle({
idleTime: IDLE_TIME_MS,
timeout: IDLE_TIMEOUT_MS,
await page.setViewport({
width: VIEWPORT_WIDTH,
height: VIEWPORT_HEIGHT,
deviceScaleFactor: 1,
});
} catch {}
await page.setUserAgent(USER_AGENT);
console.debug("[screenshot] navigated", {
url,
attempt: attemptIndex + 1,
});
console.debug("[screenshot] navigating", {
url,
attempt: attemptIndex + 1,
});
await page.goto(url, {
waitUntil: "domcontentloaded",
timeout: NAV_TIMEOUT_MS,
});
const rawPng: Buffer = (await page.screenshot({
type: "png",
fullPage: false,
})) as Buffer;
// Give chatty pages/CDNs a brief chance to settle without hanging
try {
await page.waitForNetworkIdle({
idleTime: IDLE_TIME_MS,
timeout: IDLE_TIMEOUT_MS,
});
} catch {}
console.debug("[screenshot] navigated", {
url,
attempt: attemptIndex + 1,
});
rawPng = (await page.screenshot({
type: "png",
fullPage: false,
})) as Buffer;
} finally {
try {
await page.close();
} catch {}
}
console.debug("[screenshot] raw screenshot bytes", {
bytes: rawPng.length,
});
@@ -239,51 +188,47 @@ export async function getOrCreateScreenshotBlobUrl(
console.debug("[screenshot] watermarked png bytes", {
bytes: pngWithWatermark.length,
});
const blobPath = computeScreenshotBlobPath(
console.info("[screenshot] uploading via uploadthing");
const { url: storedUrl, key: fileKey } = await uploadImage({
kind: "screenshot",
domain,
VIEWPORT_WIDTH,
VIEWPORT_HEIGHT,
);
console.info("[screenshot] uploading to blob", { blobPath });
const storedUrl = await putScreenshotBlob(
domain,
VIEWPORT_WIDTH,
VIEWPORT_HEIGHT,
pngWithWatermark,
);
console.info("[screenshot] uploaded", { url: storedUrl });
await waitForPublicUrl(storedUrl);
console.debug("[screenshot] public url ready", {
url: storedUrl,
width: VIEWPORT_WIDTH,
height: VIEWPORT_HEIGHT,
png: pngWithWatermark,
});
console.info("[screenshot] uploaded", {
url: storedUrl,
key: fileKey,
});
// No need to wait for public URL
// Write Redis index and schedule purge
try {
const ttl = getScreenshotTtlSeconds();
const expiresAtMs = Date.now() + ttl * 1000;
const key = ns(
const indexKey = ns(
"screenshot:url",
`${domain}:${VIEWPORT_WIDTH}x${VIEWPORT_HEIGHT}`,
);
console.debug("[screenshot] redis set index", {
key,
key: indexKey,
ttlSeconds: ttl,
expiresAtMs,
});
await redis.set(
key,
{ url: storedUrl, expiresAtMs },
indexKey,
{ url: storedUrl, key: fileKey, expiresAtMs },
{
ex: ttl,
},
);
console.debug("[screenshot] redis zadd purge", {
url: storedUrl,
key: fileKey,
expiresAtMs,
});
await redis.zadd(ns("purge", "screenshot"), {
score: expiresAtMs,
member: storedUrl, // store full URL for deletion API
member: fileKey, // store UploadThing file key for deletion API
});
} catch {
// best effort
@@ -360,7 +305,6 @@ export async function getOrCreateScreenshotBlobUrl(
} finally {
try {
await redis.del(lockKey);
console.debug("[screenshot] lock released", { lockKey });
} catch {}
}
}