bingran.you
← Skills
Other

social-scraping-policy

social-scraping-policy

Description

REQUIRED before reading, scraping, navigating, or extracting metadata from X / Twitter, Xiaohongshu / RedNote, YouTube, Bilibili, or LinkedIn, including the bingran.ai /posts pipeline. Trigger for social URLs, "add this to /posts", "go look at X", "grab those posts", mention summaries, social heartbeats, or any fetch/browser automation touching those domains. Defines safe account, tool, pacing, fingerprint, abort-signal, data-handling, and post-card rules.

SKILL.md

Social Scraping Policy & Operating Manual

How to read / scrape X (Twitter) and Xiaohongshu (RedNote) without damaging Bingran's accounts, and how to turn a post URL into a card on bingran.ai/posts. Three parts:

  1. Why these platforms are different — the threat model, so the rules below make sense.
  2. The operating manual — concrete, numeric defaults: account, tool, pacing, fingerprint, signals, backoff.
  3. The /posts pipeline — concrete recipes that take a post URL and land a card on bingran.ai (data layout, per-platform extractors, thumbnail strategy, "paste-a-link" flow).

This skill is the gate AND the playbook. If you read it and still don't know what to do, stop and ask Bingran.

Canonical index — code & docs that belong to this skill

Everything you'd reach for when handling these tasks. The skill is the single source of truth; AGENTS.md just points here.

Asset Where it lives Why it isn't inside this skill dir
add-post.mjs — URL → metadata → JSON-append script personal-site/scripts/add-post.mjs Wired into personal-site/package.json as npm run post:add; moving it would break that workflow. Treat this SKILL.md as the docs of record; --help in the script is intentionally terse.
posts.json — the data personal-site/content/posts/posts.json It's the personal-site's content, not a skill artifact.
post-card.tsx — render component personal-site/components/post-card.tsx Same — site code.
Thumbnail bucket personal-site/public/posts-thumbs/xiaohongshu/<note-id>.jpg Public site assets.
Account / pacing / threat-model rules this file, §§ Part 1–2 —
Per-platform extractor recipes & XHS fallback this file, §§ 3.3–3.4 —
Trigger phrases & decision rule this file, § 3 preamble + § 3.8 —
Process discipline ("ship complete, not half-done") this file, § 3.10 below —

If you find scraping-related know-how anywhere else in the workspace (random memory/*.md, an outdated CLAUDE.md note, a stray script in social-media/), the right move is: lift it into this skill, drop a one-line pointer at the original location, and tell Bingran in your status update. The skill stays canonical.

When this skill applies

Trigger if the task involves any of:

  • Opening / navigating / clicking on x.com, twitter.com, mobile.twitter.com, xiaohongshu.com, xhslink.com, rednote.com, or their app variants, while a Bingran-owned account is logged in.
  • Asking for "summary / recent posts / inbox / mentions / DMs / search results / a user's feed / comments under a note" from those platforms.
  • A heartbeat / cron / /loop that touches one of those platforms periodically.
  • Plans that would use claude-in-chrome, computer-use, Playwright, Puppeteer, or scripted HTTP against those domains.

Not needed for: reading a public news article that quotes a tweet, looking at a screenshot of a post Bingran shared, reading the official X API docs.

Part 1 — Threat model (why platforms react)

What platforms see

Every page load and click ships them a signal vector roughly like this:

  • Account graph: account ID, age, follower/following ratio, prior interaction history.
  • IP / network: IP, ASN, geo, residential vs datacenter, IPv4/IPv6.
  • Device fingerprint: User-Agent + header order, Canvas hash, WebGL renderer, font list, screen size, timezone, language, hardware concurrency, audio context.
  • Behavior: time between actions, scroll velocity, dwell distribution, click coordinates, mouse path geometry, time-of-day pattern.
  • Session shape: ratio of read / search / profile-view / reply / like / DM, request burstiness, total session length.

Anti-bot ML rolls all of that into an account-level risk score. Crossing a threshold gets you throttled, captcha-walled, "limited" mode, shadow-banned, or banned outright.

What "safe" means

Safe = looks like Bingran himself opening the app to check something. Not "bot wearing a human costume." The two are different:

  • ❌ Bot costume: rotating proxies, anti-detect browser, randomized fingerprint, headless Chromium with webdriver=false, scripted scroll-and-grab loops.
  • ✅ Bingran-himself: his real Chrome profile, his real cookies, MCP-driven DOM reads at human cadence, < 10 minutes per session, < 50 items per session, ends naturally.

The "bot costume" approach scales but invites detection. We don't scale. We do small, infrequent, human-paced reads.

Per-platform reality

X / Twitter

  • ToS: X's published automation rules ban non-API automation of the website. Repeated triggers can permanently suspend.
  • Read ceilings (post-2023 cap): ~1000 posts/day unverified, ~10000/day Premium.
  • Detection: very behavior-aware. Rapid scrolling through 100 tweets in a row, opening 20 profiles in 60 s, repeat-searching the same query — all flagged.
  • Punishment ladder: rate-limit (HTTP 429 / "you're going too fast" toast) → temporary read-only / "limited" mode (12 h – 1 wk) → shadow-ban → permanent suspension.
  • Bingran's account: @bingran_bry. Treat as expensive — don't risk it for casual reads.

Xiaohongshu / RedNote

  • Highest anti-bot stack in the workspace. Multi-dim fingerprint (Canvas + WebGL + fonts + UA-header order), x-s signature rotates ~5 min, cookies ~10 min, signing algo reshuffled quarterly.
  • Account-side: "read-only with zero interaction" looks like a zombie account; sudden interaction bursts also flag. ~60 % of bans come from same-IP multi-account linking.
  • Legal red line (China): criminal case law exists for scraping XHS data commercially — convictions under 《非法获取计算机信息系统数据罪》. Personal lookup is grey; bulk collection / resale / external product use is the red line.
  • Bingran's account: never use the personal one for any scraping. Use a dedicated observer account that's warmed up.

Part 2 — The operating manual

2.1 Pre-flight checklist (before any scrape)

Run through this every time. Skip = ask Bingran.

  1. What is the actual deliverable? "Look at one thing" vs "build a dataset" vs "monitor over time" — these have different rules.
  2. Which account am I using? Confirm allowed (see § 2.2).
  3. Which tool am I using? Confirm preferred for this platform (see § 2.3).
  4. What's my budget? Items, time, navigations — set a cap before starting (see § 2.4).
  5. What are my stop signals? Decide what abort looks like (see § 2.6).
  6. Am I scheduled / recurring? If yes, must be Bingran-approved with a stop condition.
  7. Where will the data go? No commits / no external LLM calls / no cross-agent forwarding without a § 2.7 review.

2.2 Account policy

Platform Allowed for read Forbidden Notes
X / Twitter @bingran_bry (Bingran's main) at low volume Any newly-created throwaway X account driven from this workspace X bans throwaway accounts faster than reads damage the main; the main is paradoxically safer at low rate
Xiaohongshu Dedicated observer account (separate from Bingran's personal XHS) Bingran's personal XHS account, period Observer must be warmed up: real phone register, ≥ 7 days normal browsing, occasional like/follow before any scrape work

If no observer account exists yet for XHS, stop and tell Bingran. Do not register one autonomously — account creation is on the explicit-permission list.

2.3 Tool policy (in preference order)

X / Twitter:

  1. Paid X API (if Bingran authorizes). Cleanest, no ToS issue. First choice for anything recurring or > 50 items.
  2. claude-in-chrome MCP with Bingran's real Chrome profile. DOM-aware, ships the real fingerprint, looks like normal browsing. Default for one-off lookups.
  3. computer-use pixel control. Slow, more obvious as a bot, only when 1 and 2 are blocked.
  4. ❌ Avoid: headless Playwright/Puppeteer with default fingerprint, anti-detect browsers, proxy-pool tooling.

Xiaohongshu:

  1. Manual browsing while Bingran watches for one-off lookups. This is fine and lowest risk.
  2. MediaCrawler-style flow (saved-login Playwright context, JS-extracted x-s signature) for sustained read tasks on the observer account.
  3. claude-in-chrome MCP with the observer-account cookie for low-volume.
  4. ❌ Avoid: pure HTTP / requests without signature handling (will fail and burn the account fast), pixel-driven computer-use (high fingerprint risk, slow), main account anything.

2.4 Pacing budget

Hard caps per session. Stop on cap, do not "round up."

Knob X / Twitter Xiaohongshu (observer)
Items read per session ≤ 50 ≤ 30
Session length ≤ 10 min ≤ 8 min
Page navigations / min ≤ 6 ≤ 4
Dwell per item 5–15 s, jittered 8–20 s, jittered
Scroll bursts 2–4 wheel ticks then pause 2–5 s 2–3 ticks then pause 3–6 s
Sessions per day ≤ 3 ≤ 2
Min gap between sessions ≥ 60 min ≥ 90 min
Daily ceiling (% of platform cap) ≤ 10 % of 1000/10000 n/a (XHS doesn't publish, treat conservatively)

Quick mental model: a curious human checking the platform, not a worker pulling a list.

2.5 Fingerprint and behavior hygiene

Do:

  • Use Bingran's real, unmodified Chrome profile (or the observer's real Chrome profile).
  • Let claude-in-chrome use the real cookies, headers, and timezone.
  • Move through the UI like a person: open one thing, dwell, scroll a bit, sometimes go back without clicking through.
  • Vary inter-action delays — never an exact 2.0 s loop. Use 2–8 s randomized.
  • Keep the same IP for the whole session. If on Wi-Fi, don't switch mid-session.
  • Match time-of-day to Bingran's normal use. 3 am scraping looks like a bot.

Don't:

  • Don't strip or spoof User-Agent / headers. The real one is the safe one.
  • Don't flip navigator.webdriver or any "stealth" plugin tricks. Detection vendors fingerprint the evasion.
  • Don't rotate proxies. Don't VPN-hop mid-session.
  • Don't scroll faster than ~1 viewport per 2 s.
  • Don't open profile after profile after profile rapidly. Mix in scrolls and back-navigations.
  • Don't repeat the same search query 5 times in 10 minutes — that pattern is uniquely bot-shaped.
  • Don't trigger any action button "just to verify" — no like, no follow, no reply, no DM, no save, no share. Read-only is read-only.

2.6 Signals to watch — and how to back off

Watch for these in the page / network / UI:

Signal Severity Action
Slow loading, spinners that linger > 5 s on previously-fast pages 🟡 yellow slow pacing 2x, finish current item, end session early
"You're going too fast" toast / banner 🟡 yellow end session immediately, 30+ min cooldown
HTTP 429 in network panel 🟠 orange end session, 2 hr cooldown, halve next session's budget
Captcha appears 🔴 red stop. Do not solve. Tell Bingran. No more scraping today.
"Limited" / read-only mode banner 🔴 red stop. Tell Bingran. No scraping for 24 h minimum.
Login wall when already logged in 🔴 red stop. Tell Bingran. Don't re-login from this session — re-login under suspicion is a strong bot signal.
Email / SMS verification request 🔴 red stop. Don't satisfy. Tell Bingran.
Suddenly-empty search results / timeline 🟠 orange possible shadow-ban. End session, tell Bingran.
403 / "your account has been flagged" 🔴 red stop. Don't retry. Tell Bingran with full context.

After any red signal: do not touch that platform from this workspace for 24 h. After any yellow signal: at least 30 min cooldown, halve the next session's budget.

Never solve a captcha on Bingran's behalf. That's a separate platform rule and a strong bot-confirmation signal.

2.7 Data handling (after the read)

What you pulled is Bingran's data viewed through his account. Treat it accordingly:

  • Default storage: in-memory or memory/YYYY-MM-DD.md only.
  • Default sharing scope: just back to Bingran in this conversation.
  • Forbidden without explicit per-task approval:
    • Committing scraped content to git (any repo).
    • Pasting into PRs / issues / public docs.
    • Forwarding to other agents (Codex, sub-agents, external LLM APIs).
    • Posting to Discord / Slack / email.
    • Storing in Notion / Google Drive / external SaaS.
  • PII check: if the content contains DMs, private replies, draft posts, or third-party personal info, treat as confidential — do not log even to memory/.
  • XHS-specific: do not commit XHS-scraped content to any public-visible location. Anything that looks like a dataset (> 30 items, structured) needs explicit Bingran approval before being saved anywhere durable.

2.8 Reporting back

After every scrape, report to Bingran in this shape:

  • Account used: which one.
  • Tool used: claude-in-chrome / computer-use / API / manual.
  • Volume + time: e.g. "27 items, 6 min, 4 page navs."
  • Risk signals seen: none / yellow toast at item 18 / etc.
  • Where the data lives: in this turn only / memory/2026-05-07.md / etc.
  • Sharing recommendation: default no, justify any yes.

Decision tree (quick reference)

incoming task touches X / XHS?
├── no  → skill not needed
└── yes
    ├── one-off lookup, < 5 items, < 5 min?
    │   ├── X        → claude-in-chrome MCP, Bingran's profile, human pace ✅
    │   └── XHS      → manual + Bingran watching, OR observer account ✅
    ├── lookup, 5–50 items, < 10 min?
    │   ├── X        → claude-in-chrome with §2.4 budget ✅
    │   └── XHS      → observer account only, MediaCrawler-style or claude-in-chrome ✅
    ├── recurring / scheduled / heartbeat-driven?
    │   └── STOP. Confirm cadence + stop condition with Bingran 🟡
    ├── building a dataset / > 50 items / sustained > 10 min?
    │   └── STOP. Confirm scope, account, storage, and use case with Bingran 🟡
    ├── commercial / external use of the data?
    │   └── STOP. Hard escalation — XHS legal + X ToS exposure 🔴
    └── any §2.6 red signal hit mid-run?
        └── STOP immediately. 24 h cooldown. Report to Bingran 🔴

What NEVER works (don't try)

  • Rotating proxy pools to "look like different users." Detection ML treats fast IP-switching as itself a bot signature.
  • Anti-detect browsers (Multilogin, Nstbrowser, GoLogin) on Bingran's real account. The fingerprint mismatch from the user's history is the giveaway.
  • Solving captcha to "keep going." Captcha-solved-then-resumed is a strong bot pattern.
  • Bypassing X read limits via "scroll harder." The cap is enforced server-side; client-side tricks just spend rate budget faster.
  • Multiple XHS accounts on the same IP. ~60 % of XHS bans come from this exact pattern.
  • Headless Chromium with --disable-blink-features=AutomationControlled and other "stealth" flags. Detection vendors fingerprint the stealth flags.
  • Treating "I'm just reading" as a free pass. Read-only sessions still ship every signal in § Part 1.

When in doubt

Stop and ask Bingran. The cost of a 60-second pause to confirm is much lower than the cost of a banned account.

Part 3 — The /posts pipeline (bingran.ai/posts)

Bingran's site has a /posts page that aggregates original posts from X, Xiaohongshu, YouTube and (eventually) Bilibili as cards in a masonry grid. When the user pastes a post URL with intent like "add this to /posts" / "grab these posts" / "把这条加进去" — this is what to do.

🛑 Decision rule before you write a single line of code

For any "add this to /posts" task, the SOP is:

  1. First run cd personal-site && npm run post:add -- "<url>" and read what it printed. Don't open a browser yet.
  2. If the printed entry is poisoned (XHS canary — see §3.3 "Reality update"), the only fallback is claude-in-chrome MCP against Bingran's already-logged-in Chrome profile. That is the tool. Not Playwright. Not Puppeteer. Not headless Chromium. Not WebFetch. Not "let me try one more curl with a different UA."
  3. Then download the cover URL with curl -H 'Referer: https://www.xiaohongshu.com/' … and sips-convert webp→jpg. Hand-edit posts.json (do not re-run the script) to keep the diff at +9 / -0.
  4. Then commit (just posts.json and the one new thumbnail), push, PR.

Why so prescriptive: the only path that consistently lands an XHS thumbnail is Bingran's real Chrome session, where he's already logged in. Every "headless browser with a clever UA" detour we tried (Node Playwright with chromium-headless-shell, anti-detect tricks, mobile UA, fresh xsec_token) lands on the same login wall and wastes 5–10 minutes. Document for future-you below in §3.4.

Trigger phrases (don't second-guess these)

These all mean "do §3.5 / §3.8":

  • "add this to /posts" / "把这条加进来" / "更新到个人网站"
  • A bare URL pasted in chat from x.com, xhslink.com, xiaohongshu.com, youtube.com, bilibili.com (any subdomain), with no other instructions.
  • "我刚发了这个X / 小红书 / YouTube" / "I just posted this"
  • Bilingual share-text patterns: "...【小红书】里有精彩内容，快去瞧瞧！" + a URL.

If you got here because the user pasted such a phrase, do not improvise. Follow §3.5 or §3.8 line by line.

3.1 Data layout (one source of truth)

personal-site/
├── content/posts/posts.json          # the data — single JSON array, sorted desc by date
├── lib/posts.ts                       # types + loader (Post, Platform, getAllPosts)
├── components/post-card.tsx    # card rendering — image OR text-as-visual fallback
├── app/(personal)/posts/page.tsx      # the page (CSS-columns masonry)
└── scripts/add-post.mjs        # the URL → metadata → JSON-append script

Schema of one entry in posts.json:

{
  "id": "x-2052477417240031355",            // <platform>-<native-id>; primary key for dedupe
  "platform": "x",                          // "x" | "xiaohongshu" | "bilibili" | "youtube" | "linkedin" | "other"
  "url": "https://x.com/.../status/...",    // canonical post URL (with xsec_token for XHS)
  "title": "...",                           // optional preview text
  "description": "...",                     // optional, longer body
  "thumbnail": "/posts-thumbs/...jpg",      // optional; LOCAL path for XHS, remote https for YouTube/Bilibili, absent for X
  "date": "2026-05-07",                     // YYYY-MM-DD; drives sort order
  "addedVia": "manual" | "auto"             // metadata only
}

Per-platform shape (what's actually present after the script runs):

Platform id url date title description thumbnail
x ✅ x-<digits> ✅ ✅ — — —
youtube ✅ youtube-<vidId> ✅ ✅ ✅ ✅ ✅ remote i.ytimg.com
xiaohongshu ✅ xhs-<noteId> ✅ ✅ ✅ ✅ ✅ local /posts-thumbs/xiaohongshu/<id>.jpg
bilibili ✅ bilibili-BV... ✅ ✅ ✅ ✅ ✅ remote i0.hdslb.com

X is intentionally minimal — react-tweet fetches everything (text, author, avatar, media, tweet date) from the syndication API at render time, so the only field we record locally is the tweet id.

3.2 The script: npm run post:add -- <url>

scripts/add-post.mjs is the one entry point. Detects platform from URL, runs the right extractor, dedupes by id / url, appends to posts.json, re-sorts.

Override flags (when extraction misses something or you want to curate):

  • --title "Custom" / --description "..." — override extracted text
  • --date 2026-05-07 — override extracted date
  • --thumbnail https://... — override extracted image
  • --platform NAME — force platform bucket

3.3 Per-platform extractor recipes

Per-platform reality, what works server-side, what needs the browser:

Platform Server-side fetch Auth needed Best metadata source
YouTube ✅ works no https://www.youtube.com/oembed?url=...&format=json (title, author, thumbnail)
Bilibili ✅ works no https://api.bilibili.com/x/web-interface/view?bvid=BV... (full JSON incl. pubdate)
Xiaohongshu ✅ works with xsec_token no (token IS the auth) the share URL responds to plain curl and serves og:image / og:title / og:description
X / Twitter ✅ id-only no parse /status/(\d+) from the URL — that's all the data we need
LinkedIn ⚠️ partial yes generic OG-tag scrape

Concrete per-platform notes:

YouTube — oembed is the cleanest path. No anti-bot risk, no auth. Date isn't in oembed; either parse the watch-page HTML <meta itemprop="datePublished"> or pull from the channel RSS feed (youtube.com/feeds/videos.xml?channel_id=UC... returns last ~15 entries with full timestamps).

Bilibili — api.bilibili.com/x/web-interface/view?bvid=BV... returns {title, desc, pic, pubdate} directly. pic is already https. pubdate is unix seconds. For b23.tv shortlinks, fetch with redirect: 'follow' to resolve to the long URL first.

Xiaohongshu — the big trick. Bare /explore/<id> → 404. Share URLs with xsec_token used to answer to anonymous curl reliably; the historical recipe was:

curl -s "https://www.xiaohongshu.com/explore/<id>?xsec_token=<token>&xsec_source=pc_user" \
  -A "Mozilla/5.0 ... Chrome/124.0.0.0 ..." \
  | grep -oE '<meta[^>]*og:(image|title|description)[^>]*'

The xsec_token is what XHS calls "share-link auth" — tied to the note ID, doesn't expire on a session timer, doesn't burn the account. Get it from the profile page DOM (the <a> href on each note tile carries it) or from a real "复制链接" share action on mobile.

🚨 Reality update (observed 2026-05-07): fresh desktop-share xsec_token URLs now redirect server-side fetches (Node/curl/Claude WebFetch) to /404?errorCode=-510001 even with a proper Chrome UA. The page exists — opening the same URL in the user's logged-in Chrome works fine. So add-post.mjs falls through to the OG tags of the 404 page and writes a poisoned entry: id: "xiaohongshu-aHR0cHM6Ly93" (base64 fallback because the /404 path doesn't match the /explore/<id> regex), title: "小红书 - 你访问的页面不见了", plus a 4 KB cached "page not found" graphic in public/posts-thumbs/xiaohongshu/. Treat that title and that id prefix as the canary.

When the script returns the canary, fall back to the logged-in browser path — this is the canonical recipe, last verified 2026-05-09:

# 0. Get a tab in Bingran's already-logged-in Chrome
mcp__Claude_in_Chrome__list_connected_browsers   # pick the deviceId
mcp__Claude_in_Chrome__select_browser deviceId=<...>
mcp__Claude_in_Chrome__tabs_context_mcp createIfEmpty=true   # → tabId

# 1. Open the post (use one browser_batch — navigate + wait + screenshot)
browser_batch [
  { navigate, url: "https://www.xiaohongshu.com/explore/<id>?xsec_token=<token>&xsec_source=pc_user", tabId },
  { computer, action: wait, duration: 4, tabId },
  { computer, action: screenshot, tabId },
]
# If the screenshot shows "马上登录即可" you're in the wrong Chrome profile.
# Stop and ask Bingran which browser to select.

# 2. Pull the cover URL via JS (return wrapped in JSON.stringify to dodge the safety filter)
mcp__Claude_in_Chrome__javascript_tool tabId=<tabId> text='
  JSON.stringify({
    imgs: Array.from(document.querySelectorAll("img"))
      .map(i => i.src)
      .filter(s => s && s.includes("xhscdn") && !s.includes("avatar")),
    title: document.title.replace(/\s*-\s*小红书\s*$/, "")
  })
'
# imgs[0] is the cover. Title for sanity-check / fallback.

# 3. Close the MCP tab BEFORE downloading — keeps the §2.4 budget tight
mcp__Claude_in_Chrome__tabs_close_mcp tabId=<tabId>

# 4. Download with Referer (CDN 403s without it), convert webp→jpg, save in place
curl -s -o /tmp/cover.webp \
  -H "Referer: https://www.xiaohongshu.com/" \
  -H "User-Agent: Mozilla/5.0 ... Chrome/124.0.0.0 ..." \
  "<imgs[0]>"
sips -s format jpeg /tmp/cover.webp \
  --out personal-site/public/posts-thumbs/xiaohongshu/<note-id>.jpg

# 5. Hand-edit posts.json to insert the new entry at the top (never re-run the script —
#    its date-equal resort re-shuffles every same-day sibling and bloats the diff).
#    Target diff: +8/-0 for the entry, +1 binary file for the thumbnail.

# 6. If a canary entry was previously written, drop it from posts.json and trash its bad thumb.

Hard rules:

  • Do not invoke Playwright / Puppeteer / headless Chromium / chromium-headless-shell. They hit the XHS login wall (proven repeatedly) and burn time. Chrome MCP against Bingran's signed-in profile is the only thing that works.
  • Don't read og:image from the DOM — XHS's SPA doesn't set it, you'll just get the picasso placeholder.
  • Wrap MCP javascript_tool returns in JSON.stringify({...}) so the filter doesn't strip CDN URLs that contain query strings.
  • One navigate + one JS read = one ~10-second session. Don't pile on more nav-and-reads in the same session for unrelated posts; if you need three posts done, do them as three separate Chrome MCP sessions to stay obviously human-paced.

XHS bonus: the first 8 hex chars of the note ID are a unix timestamp — Date(parseInt(noteId.slice(0,8), 16) * 1000) gives you the post date without any extra request.

XHS thumbnail caveat: the og:image URL is signed with a timestamp (/2026MMDDHHMM/...) and expires within hours. The add-post.mjs script handles this — it downloads the image immediately into public/posts-thumbs/xiaohongshu/<note-id>.jpg and rewrites thumbnail to the local path. See § 3.4 for why and how.

X / Twitter — the simple one. No metadata fetch. The extractX function in the script just parses the tweet id from status/(\d+) and writes a 4-field stub: {platform, id, url, date}. Everything visible (text, author, avatar, embedded media, like/repost counts, original post date) is fetched at render time by <Tweet id={id}> from the react-tweet library hitting https://cdn.syndication.twimg.com/tweet-result?id=<id>. Do not restore the old publish.twitter.com/oembed path — it added 80 lines of HTML-entity decoding and footer-regex parsing for data that's now redundant. If --date isn't passed and the tweet was back-dated, sort order will be off until you edit the JSON.

3.4 Thumbnail strategy — never empty, never broken

Goal: every card has visual content, and that content survives forever (no broken images six months later).

The card component (components/post-card.tsx) routes by platform:

  1. platform === "x" → render <Tweet id={tweetId}> from react-tweet. Server component, fetches from cdn.syndication.twimg.com at build/request time. No posts.json thumbnail involved. Deleted tweets degrade to a built-in <TweetNotFound> tombstone.
  2. Has thumbnail (other platforms) → <img src={post.thumbnail}> inside an aspect-ratio-locked container, with title + date below.
  3. Has no thumbnail and not X (rare; only manual --platform other entries) → render text as a serif quote that fills the image slot, with platform icon as watermark.

Where the thumbnail comes from:

  • YouTube → oembed thumbnail_url (e.g., i.ytimg.com/vi/<id>/hqdefault.jpg). Stable forever, leave remote.
  • Bilibili → API pic field (e.g., i0.hdslb.com/...). Stable, leave remote.
  • Xiaohongshu → 🚨 download to local. XHS CDN URLs are signed with a timestamp embedded in the path (/2026MMDDHHMM/...) and expire — leaving a ? placeholder where the cover used to be. The add-post.mjs script downloads the og:image immediately to public/posts-thumbs/xiaohongshu/<note-id>.jpg (HTTP fetch with Referer: https://www.xiaohongshu.com/ header — required, otherwise the CDN 403s) and rewrites thumbnail to /posts-thumbs/xiaohongshu/<note-id>.jpg so the page references the local copy. Total cost is ~135 KB per card. When the script's server-side path is blocked (see § 3.3 reality update) and you fall back to the DOM-extracted cover, the image is usually a webp despite the .jpg filename — convert with sips -s format jpeg <id>.jpg --out /tmp/<id>.jpg && mv /tmp/<id>.jpg <id>.jpg to keep the bucket JPEG-only and avoid extension/MIME mismatch.
  • X → no thumbnail; <Tweet> handles all media inline.

Why react-tweet for X (not the older text-as-visual path):

  • Tweets that were "Post on X" stubs (script-extracted title was empty) now render the real tweet content.
  • Embedded media (images / videos / quoted tweets) shows up automatically.
  • Card heights become consistent within the masonry — no more alternating tall-text-card / short-tweet-stub.
  • Zero data to maintain in posts.json.

What we tried and rejected (don't waste budget retrying):

  • ❌ claude-in-chrome screenshot / zoom of each XHS or X post — the in-page MCP overlay (a "Stop Claude" button) gets baked into the saved image. Cropping it out adds fragility. Use server-side fetch of the underlying CDN URL instead.
  • ⚠️ JS-based extraction of CDN URLs via Chrome MCP javascript_tool — the safety filter sometimes blocks the return with [BLOCKED: Cookie/query string data], especially when the page URL itself (location.href) contains a token and gets included in the return shape. Workaround that worked for XHS cover extraction: omit location.href, return only the specific image src strings, and wrap in JSON.stringify({...}). If still blocked, fall through to find / read_page / DOM rect coordinates, or fetch the URL server-side.
  • ❌ Headless Playwright / Puppeteer / chromium-headless-shell hitting XHS at all — observed multiple times (2026-05-07, 2026-05-09): even with a real desktop Chrome UA, real xsec_token, zh-CN locale, and a viewport, the page renders the unauthenticated login wall (document.title === "小红书 - 你的生活兴趣社区", body text starts with 创作中心 ... 马上登录即可). It's not a UA / fingerprint problem you can fix; XHS gates note rendering on logged-in cookies. The only fallback that works is claude-in-chrome MCP against Bingran's already-authenticated Chrome profile. Don't burn 5 minutes re-installing browser binaries — go straight to Chrome MCP.
  • ❌ Spawning a fresh Node process to install/import playwright from personal-site — personal-site/package.json doesn't list it as a dep; reaching into ~/.local/lib/node_modules/@playwright/cli/node_modules/playwright works mechanically but pulls a CommonJS entry that breaks ESM import { chromium }, and the headless-shell binary may need npx playwright install. Both are signs you're on the wrong path — see the bullet above.
  • ❌ Headless Playwright / Puppeteer hitting X while logged in — login wall + § Part 1 anti-bot risk on Bingran's account.
  • ❌ Restoring the publish.twitter.com/oembed extractor for X — react-tweet supersedes it. The 80 lines of footer-regex + entity decoding bought us no rendered value.

3.4.5 Cross-browser layout stability — the masonry trap

CSS columns + variable-height children + async-loading images = different layout in Chrome / Safari / Firefox. Three rules pin it down:

  1. Aspect-ratio per platform, set on the image's container, before the image loads:
    const THUMB_ASPECT = { youtube: "16 / 9", xiaohongshu: "4 / 5", bilibili: "16 / 10" };
    Reserves vertical space → no reflow when the image network arrives. object-fit: cover on the inner <img> handles cropping.
  2. Triple-property break-inside avoidance on every grid child:
    .post-grid > * {
  break-inside: avoid;
  page-break-inside: avoid;
  -webkit-column-break-inside: avoid;
}
    Old Safari needs the -webkit- prefix; old Firefox respects the page-break- legacy name better than the modern one.
  3. react-tweet CSS overrides (kill the lib's default 550px max-width and outer margin so the embed fills its column):
    .post-tweet > div { margin: 0 !important; }
.post-tweet .react-tweet-theme { max-width: 100% !important; }

Together these three things turn a jagged masonry that reflows visibly on first paint into a stable grid that's identical in all three browsers.

3.5 The "paste a link" flow

This is the default path when the user pastes a single post URL.

1. cd personal-site
2. npm run post:add -- "<paste url here>"           # auto-detects platform from domain
3. Read the printed JSON entry. Sanity check:
   - Is the platform right?
   - Is the date plausible? (script defaults to today if extractor can't find one)
   - For YouTube/Bilibili: does the remote thumbnail URL load?
   - For Xiaohongshu: does `public/posts-thumbs/xiaohongshu/<id>.jpg` exist on disk?
     (the script logs `[ok] cached thumbnail (<bytes>B) -> /posts-thumbs/...`)
   - For X: only id+url+date — no other content; react-tweet renders the rest.
4. If date is wrong → re-run with --date YYYY-MM-DD. Other --title / --description /
   --thumbnail overrides are available but rarely needed for the four primary platforms.
5. npm run lint && npm run build                    # verify; /posts is statically generated
6. Open localhost:3000/posts to spot-check the card across viewport sizes
7. git add personal-site/content/posts/posts.json
   git add personal-site/public/posts-thumbs/xiaohongshu/<note-id>.jpg   # only for XHS
8. git commit -m "personal-site: add post — <one-line summary>"
9. git push + gh pr create + (if Bingran says ship) gh pr merge --squash

Step 7 is intentionally narrow:

  • posts.json always
  • public/posts-thumbs/xiaohongshu/<note-id>.jpg for XHS only
  • Do NOT bulk-add unrelated regenerated files. skills.generated.json and public/skill-files/*.md auto-regenerate during npm run dev and belong to a different change.

3.6 The bulk-harvest flow (initial backfill / catch-up)

When the task is "pull everything from X / XHS / YouTube" rather than one URL — the recipes that landed the original 69 posts on 2026-05-07. Apply § Part 2 budgets at all times.

X originals — use the search URL, not profile scroll.

https://x.com/search?q=from%3A<HANDLE>%20-filter%3Areplies&f=live

Why search > profile: from:HANDLE excludes retweets natively (no DOM filtering needed); -filter:replies keeps only main-thread tweets; the live feed loads denser than the profile timeline (no "Who to follow" interruptions). Then in Chrome MCP:

  1. navigate to the search URL
  2. find with query "links to HANDLE status (e.g. /HANDLE/status/...)" — returns up to 20 status URLs per call
  3. Append IDs to a tracker file, dedupe
  4. Scroll ~25 wheel-ticks, wait ~3 s, find again
  5. When find stops returning new IDs, paginate older with until:YYYY-MM-DD filter on the search URL

Why find over javascript_exec: Chrome MCP's safety filter blocks JS that extracts URLs (cookie/query-string heuristic). find queries the accessibility tree and returns matched links — same data, no block.

XHS originals — profile + End key.

https://www.xiaohongshu.com/user/profile/<USER-ID>
  1. navigate, wait, press End to scroll to bottom (XHS profiles fully load on End in ~3 s)
  2. find for /explore/<id> paths — returns up to 20 per call with xsec_token in adjacent links
  3. Read accessibility tree (read_page filter=all) to get card titles
  4. Date from parseInt(id.slice(0,8), 16) * 1000
  5. For thumbnail, hit the share URL with curl as in § 3.3 — server-side, no browser needed

YouTube — RSS, not browser.

https://www.youtube.com/feeds/videos.xml?channel_id=UC...

Returns the last ~15 videos with <yt:videoId>, <title>, <published>, <media:thumbnail url>, <media:description>. No auth, no anti-bot, no Chrome MCP needed. To resolve a @handle to a channel ID, fetch youtube.com/@handle and grep "channelId":"(UC[^"]+)" from the HTML.

Older videos beyond the RSS window: must be added via npm run post:add -- <watch-url> one at a time.

3.7 Common gotchas (real ones we hit)

  • XHS bare /explore/<id> 404s — always use the share URL with xsec_token. Profile-page hrefs include the token; copy them whole.
  • XHS share URL still 404s server-side (observed 2026-05-07) — even with token + UA, anonymous fetches from Node/curl/Claude WebFetch hit /404?errorCode=-510001. The script does not detect this; it scrapes the 404 page's OG tags and writes a poisoned entry (id xiaohongshu-aHR0cHM6Ly93, title 小红书 - 你访问的页面不见了, 4 KB "page not found" graphic in posts-thumbs/xiaohongshu/). Always sanity-check the script's printed JSON before committing. When you see the canary, follow the logged-in browser fallback in § 3.3 (Chrome MCP for title + cover URL, curl with Referer for the image, sips to convert webp→jpg, hand-edit posts.json).
  • add-post.mjs resort-on-add inflates the diff — the script calls arr.sort((a,b)=> a.date<b.date?1:-1) after appending. Sort is unstable for equal-date entries, so adding one post can shuffle every same-day sibling and produce a hundred-line diff for one logical change. For one-off adds (especially the manual XHS path above), use git show HEAD:personal-site/content/posts/posts.json + a small node unshift-and-write-back snippet so the diff stays at +9 / -0.
  • XHS thumbnail 403 cross-origin — even when the URL is fresh, fetching it from a non-XHS origin (or without a Referer header) returns 403. The script sets Referer: https://www.xiaohongshu.com/. If you need to download by hand, the same header makes curl work.
  • XHS DOM cover is webp, not jpeg — the first <img> you read out of the SPA via Chrome MCP returns a *_webp_3 URL. Saving it as .jpg works in browsers (they sniff magic bytes) but breaks the JPEG-only convention of existing thumbs and may serve with the wrong Content-Type. Convert with sips -s format jpeg before committing.
  • XHS thumbnail expiry — signed timestamp in the path; valid for hours, not days. Always cache locally on add. If you spot a ? placeholder on /posts, the URL has rotted — re-add the post or copy a fresh share URL.
  • Date defaults to today — if extractor can't find a date (X always; XHS sometimes), the script writes today. Pass --date YYYY-MM-DD for back-dated entries, or sort order will be wrong.
  • xsec_token URL encoding — the token contains = and +. Use encodeURIComponent when building the URL programmatically. XHS server accepts both encoded and raw, but consistent encoding makes the JSON cleaner.
  • npm run dev regenerates skills data — don't git add -A or you'll commit lib/skills.generated.json and public/skill-files/*.md along with your post change. Stage explicit paths only.
  • YouTube RSS only returns ~15 entries — fine for steady-state monitoring, misses anything older. Use npm run post:add per URL for backfill.
  • react-tweet build-time fetch — <Tweet> calls the syndication API during next build. If the build server has no internet (rare in CI but possible), tweets fail to render. Vercel build env has internet; local builds offline will get tombstones.
  • react-tweet light theme by default — looks fine on the cream Berkeley palette, slightly off in dark mode. Wrap in <div data-theme="dark"> based on prefers-color-scheme if it matters.
  • Chrome MCP [BLOCKED: Cookie/query string data] — the safety filter sometimes strips JS-tool returns when the value mixes a tokened location.href with other strings. Workaround: omit location.href, return only the specific values you need, wrap in JSON.stringify({...}). If still blocked, server-side fetch + og:image parse, or Chrome MCP find for DOM hrefs.
  • Account isolation for the script — add-post.mjs only does anonymous server-side fetch (YouTube oembed, Bilibili API, XHS share URL, XHS image CDN). Doesn't touch Bingran's account cookies on any platform. § Part 2 risk applies only when a workflow uses claude-in-chrome against the live X / XHS UI.

3.8 Decision tree for "add this to /posts"

incoming URL
├── youtube.com / youtu.be       → npm run post:add -- <url>           ✅ 1 step, remote thumb
├── bilibili.com / b23.tv        → npm run post:add -- <url>           ✅ 1 step, remote thumb
├── xhslink.com / xiaohongshu.com
│    ├── has xsec_token in URL   → npm run post:add -- <url>           ⚠️ try first; sanity-check the
│    │                                                                    printed entry — see § 3.3
│    │                                                                    canary. If poisoned, fall back
│    │                                                                    to logged-in browser path
│    │                                                                    (Chrome MCP + curl + sips +
│    │                                                                    hand-edit JSON).
│    └── bare /explore/<id>      → ask Bingran for the share URL,
│                                  OR open profile in claude-in-chrome and copy
│                                  the token-bearing href                ⚠️ needs browser
├── x.com / twitter.com          → npm run post:add -- <url>           ✅ 1 step (id-only; react-tweet renders the rest)
│                                  Pass --date YYYY-MM-DD for back-dated tweets.
├── linkedin.com                 → npm run post:add -- <url>           ⚠️ generic OG, often thin
└── anything else                → npm run post:add -- <url>           ✅ falls through to generic OG scrape

For YouTube / Bilibili / X, the happy path is one command + --date if needed. For XHS, attempt the script first but be ready to fall back to the logged-in browser path (§ 3.3) — as of 2026-05-07 the server-side share-URL fetch is blocked and the script silently writes a poisoned entry.

3.9 Anti-detection considerations specific to /posts work

The /posts pipeline mostly stays out of § Part 2 risk because:

  • Server-side fetch from add-post.mjs uses no Bingran-account cookies. Doesn't count against any account's risk score. Free.
  • YouTube oembed, Bilibili JSON, XHS image CDN, Twitter syndication (via react-tweet) are unauthenticated public endpoints. ToS-fine for personal use. (XHS share-URL HTML was one too; as of 2026-05-07 it's blocking anonymous fetches — see § 3.3.)
  • The browser-driven steps are the bulk harvest (§ 3.6) and now the per-post XHS fallback when the script's server-side path 404s. Both touch Bingran's logged-in Chrome session, so apply § Part 2 budgets: stay under 50 items / 10 min / 6 navs/min on X, under 30 / 8 / 4 on XHS. A single fallback add (one navigate, one JS read, one image download) is well under any budget; don't queue ten of them in a tight loop.
  • Don't rebuild the harvest just to "refresh" data. New posts come in trickle; use the paste-a-link flow per post. Re-running a full harvest is the kind of pattern that flips a yellow signal.

3.10 Process discipline — ship the whole thing, not half

Bingran's standing rule for these tasks: one PR contains the complete change. No "I'll add the thumbnail in a follow-up", no "let me ship the JSON now and update the skill later." If you can't finish all of it, don't push yet.

A complete /posts add is:

  1. ✅ The post entry in personal-site/content/posts/posts.json (correct id, url, title, date, addedVia: "manual").
  2. ✅ The thumbnail file present at personal-site/public/posts-thumbs/xiaohongshu/<note-id>.jpg (for XHS) or the remote URL verified loadable (YouTube/Bilibili).
  3. ✅ The thumbnail field set in the JSON entry, pointing at #2.
  4. ✅ Diff = +8/-0 in posts.json, +1 binary thumbnail file. No regenerated skills.generated.json, no other unrelated files.
  5. ✅ Any new pitfall observed during this add → folded back into this SKILL.md in the same PR, not as a follow-up.
  6. ✅ Trigger surfaces still pointing at this skill — workspace AGENTS.md and personal-site/AGENTS.md need at most one short sentence each, and they should still match what this file says.
  7. ✅ PR description names what shipped + what was verified. Squash-merge after Vercel preview goes green.

If step 5 produced changes (new fallback recipe, new gotcha), those changes go in this file — not in memory/, not in MEMORY.md, not in a workspace doc. The skill is canonical; everything else is a pointer.

The reason: shipping in halves means future-you reads the half that landed and assumes the job is done. Bingran will read posts.json, see no thumbnail, and fix it himself. That's the failure mode this rule prevents.

References