bingran.you
← Skills
Other

social-scraping-policy

social-scraping-policy

Description

Use before any browser-driven (computer-use, Playwright, claude-in-chrome, MCP, scripted fetch) read or scrape against social platforms — especially X / Twitter and Xiaohongshu / RedNote. Defines (1) SAFE scraping rules — account choice, tool choice, pacing, fingerprint hygiene, session length, abort signals, the legal / ToS boundaries Bingran's accounts must stay inside; (2) the bingran.ai /posts pipeline — paste a post URL → extract metadata → land a card with a thumbnail, including the recipes that actually work for X, Xiaohongshu, YouTube and Bilibili. Read this BEFORE clicking, navigating, or pulling data from these sites — even when the user only says "go look at X", "grab those posts", or "add this link to /posts."

SKILL.md

Social Scraping Policy & Operating Manual

How to read / scrape X (Twitter) and Xiaohongshu (RedNote) without damaging Bingran's accounts, and how to turn a post URL into a card on bingran.ai/posts. Three parts:

  1. Why these platforms are different — the threat model, so the rules below make sense.
  2. The operating manual — concrete, numeric defaults: account, tool, pacing, fingerprint, signals, backoff.
  3. The /posts pipeline — concrete recipes that take a post URL and land a card on bingran.ai (data layout, per-platform extractors, thumbnail strategy, "paste-a-link" flow).

This skill is the gate AND the playbook. If you read it and still don't know what to do, stop and ask Bingran.

When this skill applies

Trigger if the task involves any of:

  • Opening / navigating / clicking on x.com, twitter.com, mobile.twitter.com, xiaohongshu.com, xhslink.com, rednote.com, or their app variants, while a Bingran-owned account is logged in.
  • Asking for "summary / recent posts / inbox / mentions / DMs / search results / a user's feed / comments under a note" from those platforms.
  • A heartbeat / cron / /loop that touches one of those platforms periodically.
  • Plans that would use claude-in-chrome, computer-use, Playwright, Puppeteer, or scripted HTTP against those domains.

Not needed for: reading a public news article that quotes a tweet, looking at a screenshot of a post Bingran shared, reading the official X API docs.

Part 1 — Threat model (why platforms react)

What platforms see

Every page load and click ships them a signal vector roughly like this:

  • Account graph: account ID, age, follower/following ratio, prior interaction history.
  • IP / network: IP, ASN, geo, residential vs datacenter, IPv4/IPv6.
  • Device fingerprint: User-Agent + header order, Canvas hash, WebGL renderer, font list, screen size, timezone, language, hardware concurrency, audio context.
  • Behavior: time between actions, scroll velocity, dwell distribution, click coordinates, mouse path geometry, time-of-day pattern.
  • Session shape: ratio of read / search / profile-view / reply / like / DM, request burstiness, total session length.

Anti-bot ML rolls all of that into an account-level risk score. Crossing a threshold gets you throttled, captcha-walled, "limited" mode, shadow-banned, or banned outright.

What "safe" means

Safe = looks like Bingran himself opening the app to check something. Not "bot wearing a human costume." The two are different:

  • ❌ Bot costume: rotating proxies, anti-detect browser, randomized fingerprint, headless Chromium with webdriver=false, scripted scroll-and-grab loops.
  • ✅ Bingran-himself: his real Chrome profile, his real cookies, MCP-driven DOM reads at human cadence, < 10 minutes per session, < 50 items per session, ends naturally.

The "bot costume" approach scales but invites detection. We don't scale. We do small, infrequent, human-paced reads.

Per-platform reality

X / Twitter

  • ToS: X's published automation rules ban non-API automation of the website. Repeated triggers can permanently suspend.
  • Read ceilings (post-2023 cap): ~1000 posts/day unverified, ~10000/day Premium.
  • Detection: very behavior-aware. Rapid scrolling through 100 tweets in a row, opening 20 profiles in 60 s, repeat-searching the same query — all flagged.
  • Punishment ladder: rate-limit (HTTP 429 / "you're going too fast" toast) → temporary read-only / "limited" mode (12 h – 1 wk) → shadow-ban → permanent suspension.
  • Bingran's account: @bingran_bry. Treat as expensive — don't risk it for casual reads.

Xiaohongshu / RedNote

  • Highest anti-bot stack in the workspace. Multi-dim fingerprint (Canvas + WebGL + fonts + UA-header order), x-s signature rotates ~5 min, cookies ~10 min, signing algo reshuffled quarterly.
  • Account-side: "read-only with zero interaction" looks like a zombie account; sudden interaction bursts also flag. ~60 % of bans come from same-IP multi-account linking.
  • Legal red line (China): criminal case law exists for scraping XHS data commercially — convictions under 《非法获取计算机信息系统数据罪》. Personal lookup is grey; bulk collection / resale / external product use is the red line.
  • Bingran's account: never use the personal one for any scraping. Use a dedicated observer account that's warmed up.

Part 2 — The operating manual

2.1 Pre-flight checklist (before any scrape)

Run through this every time. Skip = ask Bingran.

  1. What is the actual deliverable? "Look at one thing" vs "build a dataset" vs "monitor over time" — these have different rules.
  2. Which account am I using? Confirm allowed (see § 2.2).
  3. Which tool am I using? Confirm preferred for this platform (see § 2.3).
  4. What's my budget? Items, time, navigations — set a cap before starting (see § 2.4).
  5. What are my stop signals? Decide what abort looks like (see § 2.6).
  6. Am I scheduled / recurring? If yes, must be Bingran-approved with a stop condition.
  7. Where will the data go? No commits / no external LLM calls / no cross-agent forwarding without a § 2.7 review.

2.2 Account policy

Platform Allowed for read Forbidden Notes
X / Twitter @bingran_bry (Bingran's main) at low volume Any newly-created throwaway X account driven from this workspace X bans throwaway accounts faster than reads damage the main; the main is paradoxically safer at low rate
Xiaohongshu Dedicated observer account (separate from Bingran's personal XHS) Bingran's personal XHS account, period Observer must be warmed up: real phone register, ≥ 7 days normal browsing, occasional like/follow before any scrape work

If no observer account exists yet for XHS, stop and tell Bingran. Do not register one autonomously — account creation is on the explicit-permission list.

2.3 Tool policy (in preference order)

X / Twitter:

  1. Paid X API (if Bingran authorizes). Cleanest, no ToS issue. First choice for anything recurring or > 50 items.
  2. claude-in-chrome MCP with Bingran's real Chrome profile. DOM-aware, ships the real fingerprint, looks like normal browsing. Default for one-off lookups.
  3. computer-use pixel control. Slow, more obvious as a bot, only when 1 and 2 are blocked.
  4. ❌ Avoid: headless Playwright/Puppeteer with default fingerprint, anti-detect browsers, proxy-pool tooling.

Xiaohongshu:

  1. Manual browsing while Bingran watches for one-off lookups. This is fine and lowest risk.
  2. MediaCrawler-style flow (saved-login Playwright context, JS-extracted x-s signature) for sustained read tasks on the observer account.
  3. claude-in-chrome MCP with the observer-account cookie for low-volume.
  4. ❌ Avoid: pure HTTP / requests without signature handling (will fail and burn the account fast), pixel-driven computer-use (high fingerprint risk, slow), main account anything.

2.4 Pacing budget

Hard caps per session. Stop on cap, do not "round up."

Knob X / Twitter Xiaohongshu (observer)
Items read per session ≤ 50 ≤ 30
Session length ≤ 10 min ≤ 8 min
Page navigations / min ≤ 6 ≤ 4
Dwell per item 5–15 s, jittered 8–20 s, jittered
Scroll bursts 2–4 wheel ticks then pause 2–5 s 2–3 ticks then pause 3–6 s
Sessions per day ≤ 3 ≤ 2
Min gap between sessions ≥ 60 min ≥ 90 min
Daily ceiling (% of platform cap) ≤ 10 % of 1000/10000 n/a (XHS doesn't publish, treat conservatively)

Quick mental model: a curious human checking the platform, not a worker pulling a list.

2.5 Fingerprint and behavior hygiene

Do:

  • Use Bingran's real, unmodified Chrome profile (or the observer's real Chrome profile).
  • Let claude-in-chrome use the real cookies, headers, and timezone.
  • Move through the UI like a person: open one thing, dwell, scroll a bit, sometimes go back without clicking through.
  • Vary inter-action delays — never an exact 2.0 s loop. Use 2–8 s randomized.
  • Keep the same IP for the whole session. If on Wi-Fi, don't switch mid-session.
  • Match time-of-day to Bingran's normal use. 3 am scraping looks like a bot.

Don't:

  • Don't strip or spoof User-Agent / headers. The real one is the safe one.
  • Don't flip navigator.webdriver or any "stealth" plugin tricks. Detection vendors fingerprint the evasion.
  • Don't rotate proxies. Don't VPN-hop mid-session.
  • Don't scroll faster than ~1 viewport per 2 s.
  • Don't open profile after profile after profile rapidly. Mix in scrolls and back-navigations.
  • Don't repeat the same search query 5 times in 10 minutes — that pattern is uniquely bot-shaped.
  • Don't trigger any action button "just to verify" — no like, no follow, no reply, no DM, no save, no share. Read-only is read-only.

2.6 Signals to watch — and how to back off

Watch for these in the page / network / UI:

Signal Severity Action
Slow loading, spinners that linger > 5 s on previously-fast pages 🟡 yellow slow pacing 2x, finish current item, end session early
"You're going too fast" toast / banner 🟡 yellow end session immediately, 30+ min cooldown
HTTP 429 in network panel 🟠 orange end session, 2 hr cooldown, halve next session's budget
Captcha appears 🔴 red stop. Do not solve. Tell Bingran. No more scraping today.
"Limited" / read-only mode banner 🔴 red stop. Tell Bingran. No scraping for 24 h minimum.
Login wall when already logged in 🔴 red stop. Tell Bingran. Don't re-login from this session — re-login under suspicion is a strong bot signal.
Email / SMS verification request 🔴 red stop. Don't satisfy. Tell Bingran.
Suddenly-empty search results / timeline 🟠 orange possible shadow-ban. End session, tell Bingran.
403 / "your account has been flagged" 🔴 red stop. Don't retry. Tell Bingran with full context.

After any red signal: do not touch that platform from this workspace for 24 h. After any yellow signal: at least 30 min cooldown, halve the next session's budget.

Never solve a captcha on Bingran's behalf. That's a separate platform rule and a strong bot-confirmation signal.

2.7 Data handling (after the read)

What you pulled is Bingran's data viewed through his account. Treat it accordingly:

  • Default storage: in-memory or memory/YYYY-MM-DD.md only.
  • Default sharing scope: just back to Bingran in this conversation.
  • Forbidden without explicit per-task approval:
    • Committing scraped content to git (any repo).
    • Pasting into PRs / issues / public docs.
    • Forwarding to other agents (Codex, sub-agents, external LLM APIs).
    • Posting to Discord / Slack / email.
    • Storing in Notion / Google Drive / external SaaS.
  • PII check: if the content contains DMs, private replies, draft posts, or third-party personal info, treat as confidential — do not log even to memory/.
  • XHS-specific: do not commit XHS-scraped content to any public-visible location. Anything that looks like a dataset (> 30 items, structured) needs explicit Bingran approval before being saved anywhere durable.

2.8 Reporting back

After every scrape, report to Bingran in this shape:

  • Account used: which one.
  • Tool used: claude-in-chrome / computer-use / API / manual.
  • Volume + time: e.g. "27 items, 6 min, 4 page navs."
  • Risk signals seen: none / yellow toast at item 18 / etc.
  • Where the data lives: in this turn only / memory/2026-05-07.md / etc.
  • Sharing recommendation: default no, justify any yes.

Decision tree (quick reference)

incoming task touches X / XHS?
├── no  → skill not needed
└── yes
    ├── one-off lookup, < 5 items, < 5 min?
    │   ├── X        → claude-in-chrome MCP, Bingran's profile, human pace ✅
    │   └── XHS      → manual + Bingran watching, OR observer account ✅
    ├── lookup, 5–50 items, < 10 min?
    │   ├── X        → claude-in-chrome with §2.4 budget ✅
    │   └── XHS      → observer account only, MediaCrawler-style or claude-in-chrome ✅
    ├── recurring / scheduled / heartbeat-driven?
    │   └── STOP. Confirm cadence + stop condition with Bingran 🟡
    ├── building a dataset / > 50 items / sustained > 10 min?
    │   └── STOP. Confirm scope, account, storage, and use case with Bingran 🟡
    ├── commercial / external use of the data?
    │   └── STOP. Hard escalation — XHS legal + X ToS exposure 🔴
    └── any §2.6 red signal hit mid-run?
        └── STOP immediately. 24 h cooldown. Report to Bingran 🔴

What NEVER works (don't try)

  • Rotating proxy pools to "look like different users." Detection ML treats fast IP-switching as itself a bot signature.
  • Anti-detect browsers (Multilogin, Nstbrowser, GoLogin) on Bingran's real account. The fingerprint mismatch from the user's history is the giveaway.
  • Solving captcha to "keep going." Captcha-solved-then-resumed is a strong bot pattern.
  • Bypassing X read limits via "scroll harder." The cap is enforced server-side; client-side tricks just spend rate budget faster.
  • Multiple XHS accounts on the same IP. ~60 % of XHS bans come from this exact pattern.
  • Headless Chromium with --disable-blink-features=AutomationControlled and other "stealth" flags. Detection vendors fingerprint the stealth flags.
  • Treating "I'm just reading" as a free pass. Read-only sessions still ship every signal in § Part 1.

When in doubt

Stop and ask Bingran. The cost of a 60-second pause to confirm is much lower than the cost of a banned account.

Part 3 — The /posts pipeline (bingran.ai/posts)

Bingran's site has a /posts page that aggregates original posts from X, Xiaohongshu, YouTube and (eventually) Bilibili as cards in a masonry grid. When the user pastes a post URL with intent like "add this to /posts" / "grab these posts" / "把这条加进去" — this is what to do.

3.1 Data layout (one source of truth)

personal-site/
├── content/social/posts.json          # the data — single JSON array, sorted desc by date
├── lib/social-posts.ts                # types + loader (SocialPost, SocialPlatform, getAllSocialPosts)
├── components/social-post-card.tsx    # card rendering — image OR text-as-visual fallback
├── app/(personal)/posts/page.tsx      # the page (CSS-columns masonry)
└── scripts/add-social-post.mjs        # the URL → metadata → JSON-append script

Schema of one entry in posts.json:

{
  "id": "x-2052477417240031355",            // <platform>-<native-id>; primary key for dedupe
  "platform": "x",                          // "x" | "xiaohongshu" | "bilibili" | "youtube" | "linkedin" | "other"
  "url": "https://x.com/.../status/...",    // canonical post URL (with xsec_token for XHS)
  "title": "...",                           // optional preview text
  "description": "...",                     // optional, longer body
  "thumbnail": "/posts-thumbs/...jpg",      // optional; LOCAL path for XHS, remote https for YouTube/Bilibili, absent for X
  "date": "2026-05-07",                     // YYYY-MM-DD; drives sort order
  "addedVia": "manual" | "auto"             // metadata only
}

Per-platform shape (what's actually present after the script runs):

Platform id url date title description thumbnail
x ✅ x-<digits> ✅ ✅ — — —
youtube ✅ youtube-<vidId> ✅ ✅ ✅ ✅ ✅ remote i.ytimg.com
xiaohongshu ✅ xhs-<noteId> ✅ ✅ ✅ ✅ ✅ local /posts-thumbs/xiaohongshu/<id>.jpg
bilibili ✅ bilibili-BV... ✅ ✅ ✅ ✅ ✅ remote i0.hdslb.com

X is intentionally minimal — react-tweet fetches everything (text, author, avatar, media, tweet date) from the syndication API at render time, so the only field we record locally is the tweet id.

3.2 The script: npm run post:add -- <url>

scripts/add-social-post.mjs is the one entry point. Detects platform from URL, runs the right extractor, dedupes by id / url, appends to posts.json, re-sorts.

Override flags (when extraction misses something or you want to curate):

  • --title "Custom" / --description "..." — override extracted text
  • --date 2026-05-07 — override extracted date
  • --thumbnail https://... — override extracted image
  • --platform NAME — force platform bucket

3.3 Per-platform extractor recipes

Per-platform reality, what works server-side, what needs the browser:

Platform Server-side fetch Auth needed Best metadata source
YouTube ✅ works no https://www.youtube.com/oembed?url=...&format=json (title, author, thumbnail)
Bilibili ✅ works no https://api.bilibili.com/x/web-interface/view?bvid=BV... (full JSON incl. pubdate)
Xiaohongshu ✅ works with xsec_token no (token IS the auth) the share URL responds to plain curl and serves og:image / og:title / og:description
X / Twitter ✅ id-only no parse /status/(\d+) from the URL — that's all the data we need
LinkedIn ⚠️ partial yes generic OG-tag scrape

Concrete per-platform notes:

YouTube — oembed is the cleanest path. No anti-bot risk, no auth. Date isn't in oembed; either parse the watch-page HTML <meta itemprop="datePublished"> or pull from the channel RSS feed (youtube.com/feeds/videos.xml?channel_id=UC... returns last ~15 entries with full timestamps).

Bilibili — api.bilibili.com/x/web-interface/view?bvid=BV... returns {title, desc, pic, pubdate} directly. pic is already https. pubdate is unix seconds. For b23.tv shortlinks, fetch with redirect: 'follow' to resolve to the long URL first.

Xiaohongshu — the big trick. Bare /explore/<id> → 404. But share URLs with xsec_token work without auth, even from curl with no cookies. The recipe:

curl -s "https://www.xiaohongshu.com/explore/<id>?xsec_token=<token>&xsec_source=pc_user" \
  -A "Mozilla/5.0 ... Chrome/124.0.0.0 ..." \
  | grep -oE '<meta[^>]*og:(image|title|description)[^>]*'

The xsec_token is what XHS calls "share-link auth" — it's tied to the note ID, doesn't expire on a session timer, doesn't burn the account. Get it from the profile page DOM (the <a> href on each note tile carries it) or from a real "复制链接" share action on mobile.

XHS bonus: the first 8 hex chars of the note ID are a unix timestamp — Date(parseInt(noteId.slice(0,8), 16) * 1000) gives you the post date without any extra request.

XHS thumbnail caveat: the og:image URL is signed with a timestamp (/2026MMDDHHMM/...) and expires within hours. The add-social-post.mjs script handles this — it downloads the image immediately into public/posts-thumbs/xiaohongshu/<note-id>.jpg and rewrites thumbnail to the local path. See § 3.4 for why and how.

X / Twitter — the simple one. No metadata fetch. The extractX function in the script just parses the tweet id from status/(\d+) and writes a 4-field stub: {platform, id, url, date}. Everything visible (text, author, avatar, embedded media, like/repost counts, original post date) is fetched at render time by <Tweet id={id}> from the react-tweet library hitting https://cdn.syndication.twimg.com/tweet-result?id=<id>. Do not restore the old publish.twitter.com/oembed path — it added 80 lines of HTML-entity decoding and footer-regex parsing for data that's now redundant. If --date isn't passed and the tweet was back-dated, sort order will be off until you edit the JSON.

3.4 Thumbnail strategy — never empty, never broken

Goal: every card has visual content, and that content survives forever (no broken images six months later).

The card component (components/social-post-card.tsx) routes by platform:

  1. platform === "x" → render <Tweet id={tweetId}> from react-tweet. Server component, fetches from cdn.syndication.twimg.com at build/request time. No posts.json thumbnail involved. Deleted tweets degrade to a built-in <TweetNotFound> tombstone.
  2. Has thumbnail (other platforms) → <img src={post.thumbnail}> inside an aspect-ratio-locked container, with title + date below.
  3. Has no thumbnail and not X (rare; only manual --platform other entries) → render text as a serif quote that fills the image slot, with platform icon as watermark.

Where the thumbnail comes from:

  • YouTube → oembed thumbnail_url (e.g., i.ytimg.com/vi/<id>/hqdefault.jpg). Stable forever, leave remote.
  • Bilibili → API pic field (e.g., i0.hdslb.com/...). Stable, leave remote.
  • Xiaohongshu → 🚨 download to local. XHS CDN URLs are signed with a timestamp embedded in the path (/2026MMDDHHMM/...) and expire — leaving a ? placeholder where the cover used to be. The add-social-post.mjs script downloads the og:image immediately to public/posts-thumbs/xiaohongshu/<note-id>.jpg (HTTP fetch with Referer: https://www.xiaohongshu.com/ header — required, otherwise the CDN 403s) and rewrites thumbnail to /posts-thumbs/xiaohongshu/<note-id>.jpg so the page references the local copy. Total cost is ~135 KB per card.
  • X → no thumbnail; <Tweet> handles all media inline.

Why react-tweet for X (not the older text-as-visual path):

  • Tweets that were "Post on X" stubs (script-extracted title was empty) now render the real tweet content.
  • Embedded media (images / videos / quoted tweets) shows up automatically.
  • Card heights become consistent within the masonry — no more alternating tall-text-card / short-tweet-stub.
  • Zero data to maintain in posts.json.

What we tried and rejected (don't waste budget retrying):

  • ❌ claude-in-chrome screenshot / zoom of each XHS or X post — the in-page MCP overlay (a "Stop Claude" button) gets baked into the saved image. Cropping it out adds fragility. Use server-side fetch of the underlying CDN URL instead.
  • ❌ JS-based extraction of CDN URLs via Chrome MCP javascript_tool — its safety filter blocks any string that looks like a signed URL ([BLOCKED: Cookie/query string data]). Use find / read_page / DOM rect coordinates instead, or fetch the URL server-side.
  • ❌ Headless Playwright / Puppeteer hitting X or XHS while logged in — login wall + § Part 1 anti-bot risk on Bingran's account.
  • ❌ Restoring the publish.twitter.com/oembed extractor for X — react-tweet supersedes it. The 80 lines of footer-regex + entity decoding bought us no rendered value.

3.4.5 Cross-browser layout stability — the masonry trap

CSS columns + variable-height children + async-loading images = different layout in Chrome / Safari / Firefox. Three rules pin it down:

  1. Aspect-ratio per platform, set on the image's container, before the image loads:
    const THUMB_ASPECT = { youtube: "16 / 9", xiaohongshu: "4 / 5", bilibili: "16 / 10" };
    Reserves vertical space → no reflow when the image network arrives. object-fit: cover on the inner <img> handles cropping.
  2. Triple-property break-inside avoidance on every grid child:
    .social-post-grid > * {
  break-inside: avoid;
  page-break-inside: avoid;
  -webkit-column-break-inside: avoid;
}
    Old Safari needs the -webkit- prefix; old Firefox respects the page-break- legacy name better than the modern one.
  3. react-tweet CSS overrides (kill the lib's default 550px max-width and outer margin so the embed fills its column):
    .social-tweet > div { margin: 0 !important; }
.social-tweet .react-tweet-theme { max-width: 100% !important; }

Together these three things turn a jagged masonry that reflows visibly on first paint into a stable grid that's identical in all three browsers.

3.5 The "paste a link" flow

This is the default path when the user pastes a single post URL.

1. cd personal-site
2. npm run post:add -- "<paste url here>"           # auto-detects platform from domain
3. Read the printed JSON entry. Sanity check:
   - Is the platform right?
   - Is the date plausible? (script defaults to today if extractor can't find one)
   - For YouTube/Bilibili: does the remote thumbnail URL load?
   - For Xiaohongshu: does `public/posts-thumbs/xiaohongshu/<id>.jpg` exist on disk?
     (the script logs `[ok] cached thumbnail (<bytes>B) -> /posts-thumbs/...`)
   - For X: only id+url+date — no other content; react-tweet renders the rest.
4. If date is wrong → re-run with --date YYYY-MM-DD. Other --title / --description /
   --thumbnail overrides are available but rarely needed for the four primary platforms.
5. npm run lint && npm run build                    # verify; /posts is statically generated
6. Open localhost:3000/posts to spot-check the card across viewport sizes
7. git add personal-site/content/social/posts.json
   git add personal-site/public/posts-thumbs/xiaohongshu/<note-id>.jpg   # only for XHS
8. git commit -m "personal-site: add post — <one-line summary>"
9. git push + gh pr create + (if Bingran says ship) gh pr merge --squash

Step 7 is intentionally narrow:

  • posts.json always
  • public/posts-thumbs/xiaohongshu/<note-id>.jpg for XHS only
  • Do NOT bulk-add unrelated regenerated files. skills.generated.json and public/skill-files/*.md auto-regenerate during npm run dev and belong to a different change.

3.6 The bulk-harvest flow (initial backfill / catch-up)

When the task is "pull everything from X / XHS / YouTube" rather than one URL — the recipes that landed the original 69 posts on 2026-05-07. Apply § Part 2 budgets at all times.

X originals — use the search URL, not profile scroll.

https://x.com/search?q=from%3A<HANDLE>%20-filter%3Areplies&f=live

Why search > profile: from:HANDLE excludes retweets natively (no DOM filtering needed); -filter:replies keeps only main-thread tweets; the live feed loads denser than the profile timeline (no "Who to follow" interruptions). Then in Chrome MCP:

  1. navigate to the search URL
  2. find with query "links to HANDLE status (e.g. /HANDLE/status/...)" — returns up to 20 status URLs per call
  3. Append IDs to a tracker file, dedupe
  4. Scroll ~25 wheel-ticks, wait ~3 s, find again
  5. When find stops returning new IDs, paginate older with until:YYYY-MM-DD filter on the search URL

Why find over javascript_exec: Chrome MCP's safety filter blocks JS that extracts URLs (cookie/query-string heuristic). find queries the accessibility tree and returns matched links — same data, no block.

XHS originals — profile + End key.

https://www.xiaohongshu.com/user/profile/<USER-ID>
  1. navigate, wait, press End to scroll to bottom (XHS profiles fully load on End in ~3 s)
  2. find for /explore/<id> paths — returns up to 20 per call with xsec_token in adjacent links
  3. Read accessibility tree (read_page filter=all) to get card titles
  4. Date from parseInt(id.slice(0,8), 16) * 1000
  5. For thumbnail, hit the share URL with curl as in § 3.3 — server-side, no browser needed

YouTube — RSS, not browser.

https://www.youtube.com/feeds/videos.xml?channel_id=UC...

Returns the last ~15 videos with <yt:videoId>, <title>, <published>, <media:thumbnail url>, <media:description>. No auth, no anti-bot, no Chrome MCP needed. To resolve a @handle to a channel ID, fetch youtube.com/@handle and grep "channelId":"(UC[^"]+)" from the HTML.

Older videos beyond the RSS window: must be added via npm run post:add -- <watch-url> one at a time.

3.7 Common gotchas (real ones we hit)

  • XHS bare /explore/<id> 404s — always use the share URL with xsec_token. Profile-page hrefs include the token; copy them whole.
  • XHS thumbnail 403 cross-origin — even when the URL is fresh, fetching it from a non-XHS origin (or without a Referer header) returns 403. The script sets Referer: https://www.xiaohongshu.com/. If you need to download by hand, the same header makes curl work.
  • XHS thumbnail expiry — signed timestamp in the path; valid for hours, not days. Always cache locally on add. If you spot a ? placeholder on /posts, the URL has rotted — re-add the post or copy a fresh share URL.
  • Date defaults to today — if extractor can't find a date (X always; XHS sometimes), the script writes today. Pass --date YYYY-MM-DD for back-dated entries, or sort order will be wrong.
  • xsec_token URL encoding — the token contains = and +. Use encodeURIComponent when building the URL programmatically. XHS server accepts both encoded and raw, but consistent encoding makes the JSON cleaner.
  • npm run dev regenerates skills data — don't git add -A or you'll commit lib/skills.generated.json and public/skill-files/*.md along with your post change. Stage explicit paths only.
  • YouTube RSS only returns ~15 entries — fine for steady-state monitoring, misses anything older. Use npm run post:add per URL for backfill.
  • react-tweet build-time fetch — <Tweet> calls the syndication API during next build. If the build server has no internet (rare in CI but possible), tweets fail to render. Vercel build env has internet; local builds offline will get tombstones.
  • react-tweet light theme by default — looks fine on the cream Berkeley palette, slightly off in dark mode. Wrap in <div data-theme="dark"> based on prefers-color-scheme if it matters.
  • Chrome MCP [BLOCKED: Cookie/query string data] — the safety filter strips signed-URL strings from JS-tool returns. If you need a CDN URL, fetch the page server-side and parse og:image from <meta>, or use Chrome MCP find to get DOM hrefs (those return through a different path that isn't filtered).
  • Account isolation for the script — add-social-post.mjs only does anonymous server-side fetch (YouTube oembed, Bilibili API, XHS share URL, XHS image CDN). Doesn't touch Bingran's account cookies on any platform. § Part 2 risk applies only when a workflow uses claude-in-chrome against the live X / XHS UI.

3.8 Decision tree for "add this to /posts"

incoming URL
├── youtube.com / youtu.be       → npm run post:add -- <url>           ✅ 1 step, remote thumb
├── bilibili.com / b23.tv        → npm run post:add -- <url>           ✅ 1 step, remote thumb
├── xhslink.com / xiaohongshu.com
│    ├── has xsec_token in URL   → npm run post:add -- <url>           ✅ 1 step, thumb auto-cached locally
│    └── bare /explore/<id>      → ask Bingran for the share URL,
│                                  OR open profile in claude-in-chrome and copy
│                                  the token-bearing href                ⚠️ needs browser
├── x.com / twitter.com          → npm run post:add -- <url>           ✅ 1 step (id-only; react-tweet renders the rest)
│                                  Pass --date YYYY-MM-DD for back-dated tweets.
├── linkedin.com                 → npm run post:add -- <url>           ⚠️ generic OG, often thin
└── anything else                → npm run post:add -- <url>           ✅ falls through to generic OG scrape

For all four primary platforms (YouTube / Bilibili / XHS / X), the happy path is one command + --date if needed.

3.9 Anti-detection considerations specific to /posts work

The /posts pipeline mostly stays out of § Part 2 risk because:

  • Server-side fetch from add-social-post.mjs uses no Bingran-account cookies. Doesn't count against any account's risk score. Free.
  • YouTube oembed, Bilibili JSON, XHS share URL, XHS image CDN, Twitter syndication (via react-tweet) are unauthenticated public endpoints. ToS-fine for personal use.
  • The only browser-driven steps are the bulk harvest (§ 3.6). Apply § Part 2 budgets there: stay under 50 items / 10 min / 6 navs/min on X, under 30 / 8 / 4 on XHS. Use the search URL pattern (denser, fewer navs) rather than scrolling the whole profile feed.
  • Don't rebuild the harvest just to "refresh" data. New posts come in trickle; use the paste-a-link flow per post. Re-running a full harvest is the kind of pattern that flips a yellow signal.

References