Every publisher that joins Nyhilism gets scanned on day one and re-scanned monthly. The output is a single visible grade — A, B, or C — that every advertiser on the network can see before they spend a dollar on that publisher’s inventory. Here’s what’s actually under the hood.
The pipeline in eight bullets
- BullMQ enqueues a scan job when a publisher registers a domain or when the monthly cron fires.
- A Playwright worker dials Chrome in headed mode with a real user agent, sets a desktop viewport (1440×900), and navigates to the publisher’s homepage.
- The worker captures the page HTML, every script src, every third-party iframe, the <meta> tags, and a full-page screenshot.
- Each script src and iframe URL gets compared against our fingerprint library — currently 42 patterns covering adult networks (JuicyAds, ExoClick, TrafficJunky, etc.), mainstream networks (AdSense, Taboola, Outbrain), and red-flag scripts (cryptojacking, fake-CAPTCHA pop-under loaders).
- A compliance pass checks for the 18+ gate (interstitial, age-confirmation form, or geo-redirect), the DMCA footer, and a 2257 statement link.
- Performance signals — TTFB, first-paint, total transfer size — get recorded against the same scan ID.
- Findings serialize into
site_scan_findingsrows, one per detection, with severity (info / warn / fail) and the fingerprint that matched. - A scoring pass aggregates findings into the public grade and writes the screenshot + scan metadata to
site_scores, replacing the previous monthly row.
Why 42 fingerprints (and not 4,200)
Most published “script detection” libraries — wappalyzer being the obvious one — sit in the thousands. We deliberately bounded the catalog. The questions we want SiteScan to answer are narrow: is this publisher already monetizing, is anything on this page actively hostile, and does the page comply with adult-industry requirements. We don’t need to know whether they’re running Cloudflare or Squarespace.
The 42 patterns break down roughly:
- 18 adult ad-network fingerprints. The big three plus the long tail — JuicyAds, ExoClick, TrafficJunky, TrafficStars, EroAdvertising, AdsterraAdult, ClickAdu, AdMaven, PopAds, etc. Detecting these is informational, not punitive — running another network on a different zone is allowed under the AUP. We just want to know.
- 10 mainstream fingerprints. AdSense, GAM / DFP, Taboola, Outbrain, Sharethrough, Index Exchange, Magnite, PubMatic, Criteo, MediaMath. Their presence on an adult site is interesting because most of them disallow adult inventory by TOS, so a detection is usually a tell that the site has policy-arbitrage going on. Worth a flag, not an instant fail.
- 8 red-flag fingerprints. Coinhive successors (cryptominer scripts), fake-CAPTCHA loaders, browser-locker pop-unders, history-injection redirectors, malicious tracking-pixel networks. These are severity: fail — single detection drops the site to a C grade and triggers a manual moderator review.
- 6 compliance signals. 18+ gate present, DMCA footer link, 2257 statement link, terms link, privacy link, contact page. Missing any of the top three is structurally suspicious.
How a scan rolls up into a grade
The grade isn’t a regression model. It’s a small deterministic rubric so publishers can audit their own grade and contest specific findings rather than appealing a black box. Roughly:
- Any
severity: failfinding caps the grade at C regardless of everything else. - Two or more
severity: warnfindings cap the grade at B. - All three of (18+ gate, DMCA, 2257) present + zero warns + zero fails = A.
Re-scans on the same domain compare against the previous run, so a publisher who fixes a compliance issue between scans rolls into the next grade window without waiting for the human moderator queue.
What the scan does NOT do
We don’t look inside the iframes. We don’t crawl deeper than the homepage. We don’t fingerprint the page’s users or visitors. We don’t take a long session recording. The headless browser is on for ~6 seconds per scan, captures one screenshot, and tears down.
Inside-the-page content moderation lives in a separate process — PhotoDNA on creatives at upload, and a human review queue for landing pages on a sampled basis. SiteScan’s job is to answer “is this site a serious publisher running a compliant operation,” not “is every video on the site licensed.”
The audit trail
Every individual finding survives in the database with a timestamp, the fingerprint that matched, and the URL evidence for what triggered it. The publisher sees this list, the advertiser sees the rolled-up grade, and we keep the raw rows for 13 months. A grade isn’t something we ever “just give” — there’s always a list of inputs you can challenge per row.
That’s the public-transparency angle people ask about most. The reason it’s safe to publish the same number on the advertiser side that the publisher sees on their dashboard is precisely because the number is reproducible from the rows we kept.
Where this goes next
Big nearby unlocks we’re sequencing:
- Geo-aware compliance.US states keep passing age-verification laws (TX, UT, LA, MS, FL, plus several more on the wire). We’ll start flagging publishers whose 18+ gate doesn’t implement state-specific age-verification for visitors geo-located to those states. The flag’s a warn at first while we calibrate against the implementations that have settled in the wild.
- Real-user performance correlation.Right now our performance pass is synthetic — Playwright in a Hetzner DC. Adding RUM beacons from the tag will give us the publisher’s actual visitor performance, which is a much better signal for ad-loading regressions.
- Public scan diff.Side-by-side view of last scan vs current scan with the changed findings highlighted. The DB rows are already there; it’s a UI build.
The methodology page at /methodology is the canonical reference and gets updated when the rubric changes. Drop a note at [email protected] if you spot a script we should be fingerprinting and aren’t.