LinkedIn Comment Scraper

A comment section is the cheapest buyer signal on LinkedIn: people self-select by responding to a specific claim. Turning that into leads is mostly a cost problem. The scrape is cheap per row and enormous in volume; the enrichment that makes a row useful is expensive per row and strictly limited. Everything here is about spending the expensive step on the right people.

The failure mode that costs the most is not a crash. It is a run that looks like it worked.

Onboarding — start here

1. What this skill does

Takes a set of LinkedIn posts (or creators), pulls their commenters, and walks them down a funnel where each stage costs more per row than the one above it — so each stage's job is to shrink the input to the next. It ends with commenters scored on whether they can buy (role), whether the company is real (headcount), whether they can afford it (revenue), and whether they'd want it (industry vs the creator's ICP).

It also makes you answer three questions before writing any code — partial failure, storage, duplicates — because all three are cheap to decide up front and expensive to retrofit after 10,000 rows have landed.

2. What it can't do — say this before promising anything

Reach for icp-research when the creator's ICP isn't known yet — in_icp cannot be computed against an ICP that doesn't exist.

3. Setup

set -a && . ./local.env && set +a

Needs APIFY_API_KEY (scraping), OPENROUTER_API_KEY (classification), and whichever enrichment key the company step uses (BLITZ_API_KEY, CLAY_API_KEY). Fail loudly on a missing key rather than half-running a paid job.

Verify — this reports the spend ceiling that actually binds the run:

curl -s -H "Authorization: Bearer $APIFY_API_KEY" \
  "https://api.apify.com/v2/users/me/limits" | python -m json.tool

Read maxMonthlyUsageUsd against current.monthlyUsageUsd. A comment scrape that hits the cap mid-run is the single most damaging failure in this skill, for reasons in step 5.

4. How to invoke

"Scrape the comments on these posts", "who's commenting on X's posts", "find buyers in this creator's comment section", "build a lead list from post engagement", "which posts pull buyers".

5. Cost per run

Real money, per row, non-refundable. Order of magnitude on a 10,000-comment job:

Classification is ~0.1% of the total. This is the most important number in the skill: it means you should never economise on classification, and should always economise on who reaches the two scrape stages. Tell the user the projected cost before running, and say which number is a projection.

6. Rest of the skill

• references/failure-modes.md — the observed ways this goes wrong, with the symptom each one presents as. Read before debugging anything that "ran fine".
• references/storage-and-schema.md — table shapes, dedupe keys, provenance columns, and the resume query.

Before anything: two conversations, in this order

Step 0 — route to the model picker

Classification here runs over every row, so it looks like the place to optimise. It isn't — see the cost table. But the right model still matters, because a model that disagrees with itself between runs makes every downstream count unreproducible.

Invoke `openrouter-cost-optimizer` before choosing any model. Pass the task as Classification and the measured token profile, not an estimate. Picking from memory skips the open-weight and Chinese labs that are routinely 5–20× cheaper, and skips the per-task board, which regularly disagrees with the headline ranking.

Then, because price won't decide it at these volumes, check stability: run the same 100 rows through the top two candidates 3× each and count rows that change label. A model that flips 20% of rows between identical runs cannot support a threshold-based gate, however cheap it is.

Batch composition is itself a source of drift. Sending 40 headlines in one prompt means each is judged in the context of the other 39; reorder the input and answers change even at temperature: 0. If stability matters more than throughput, shrink the batch. Measure it rather than assuming — the difference between "the provider is nondeterministic" and "our batching is" leads to opposite fixes.

Step 1 — ask the three questions

Use AskUserQuestion. These are genuinely the user's call: each has a defensible answer in both directions, and each is painful to change once rows exist.

Q1 — When a run dies halfway, what should happen?

• Resume from checkpoint (recommended) — rows are written as they arrive and a re-run skips what's already stored. Costs a little complexity, makes failure free.
• Fail the whole run and start over — simpler, but these APIs are not idempotent, so the restart pays for every row a second time.
• Keep partial data and mark it — acceptable only if the partial-ness is recorded per post, because of the bias described below.

Explain the stakes: a comment scrape that stops partway loses rows non-randomly — biased toward whichever creators or posts the actor hadn't reached. If the run's output is a rate (buyers per post, peers per magnet), that missing tail silently moves the number, and nothing in the output looks wrong.

Q2 — Where does the data live?

• Database table (recommended for anything repeated) — makes resume a query and dedupe an index.
• CSV / JSON on disk — fine for one-offs; dedupe becomes manual.
• Existing project schema — check the schema first and match its keys.

Whatever the answer: cache the raw actor output to disk as well, keyed by post. Re-parsing should be free; only re-fetching should cost. A real incident in this repo: a parsing bug invalidated 22 extractions, and because raw output hadn't been kept, re-running cost ~$11 to recover data that was already paid for.

Q3 — How should duplicates be handled?

• Dedupe by person, keep one row per person — cheapest for lead generation.
• Keep one row per (person, post) — correct for research, since the same person commenting on two posts is two observations.
• Both — one row per (person, post) for analysis, deduped to unique people before any paid enrichment.

The third is usually right, and worth explaining: the analysis denominator and the enrichment denominator are different numbers. Paying to enrich the same person twice is pure waste; counting them once in a per-post rate is a bug.

Also dedupe companies, not just people. Several commenters share an employer, and company enrichment is normally the scarcest budget in the run.

The actor — don't write a scraper

The scraping is solved; what you write is the orchestration around it (resume, provenance, dedupe, storage). Use `harvestapi/linkedin-post-comments`:

{
  "posts": ["https://www.linkedin.com/posts/…"],
  "maxItems": 150,
  "scrapeReplies": false,
  "profileScraperMode": "short"
}

• $0.002 per comment (BRONZE; $0.0015 at GOLD+). Profile enrichment inline: main +$0.002, full +$0.004, +email $0.01.
• No cookies or account. That matters more than price: an actor wanting a session cookie puts the account itself at risk, and losing it costs far more than any per-row saving. Same reason to prefer this publisher's linkedin-profile-scraper for stage 3.
• maxItems is per post, so it is the sampling cap — record it.
• Measured: 25 comments in ~6 seconds.

Two things the output actually gives you, both worth knowing before you design around it:

`actor.type` distinguishes `profile` from `company`. Company pages comment too — roughly 1 in 6 in a real sample — and they arrive with a headline like "1,050 followers". Filter on actor.type == "profile" first: it is free, and it stops you paying to classify and hand-label rows that were never people.

`actor.position` is the headline, not a structured title and employer. It is the same string the classifier already reads, so the base result does not settle founder-vs-employee. Getting the employer needs profileScraperMode or a separate profile scrape.

Prefer the separate profile scrape when the funnel filters: inline enrichment bills per comment, so a person commenting on four posts is enriched four times — which defeats the dedupe that makes the funnel cheap. Inline only wins when you intend to enrich nearly everyone anyway.

The funnel

Each arrow is a filter. Cost per row rises going down, so every row you drop early is money kept.

1  SCRAPE COMMENTS        all posts        cheap per row, huge volume
       ↓  drop actor.type == 'company'  (free, ~1 in 6 rows)
       ↓  dedupe → unique person_key
2  CLASSIFY HEADLINE      every person     ~free; a triage filter, not a score
       ↓  drop only confident non-buyers
3  SCRAPE PROFILE         survivors        title, full experience, AND the
       ↓                                   company's LinkedIn URL
       ↓  keep founder/owner/C-level
4  ENRICH COMPANY         unique companies headcount + revenue + industry,
       ↓                                   ONE call, keyed on that exact URL
5  SCORE                  role × size × affordability × ICP fit

Scoring a lead needs two LinkedIn objects, not one: the person's profile (who they are, what they do, how long) and their company's page (how real it is, how big, what it sells). The profile is what links them — it names the company page, so you never have to guess which company a name refers to.

Stage 2 is a filter, not a verdict, and that changes its metric. A real buyer wrongly dropped here is never enriched, never scored, and never noticed — the error leaves no trace. A non-buyer wrongly kept costs one cheap lookup and gets killed at stage 4. Those costs are wildly asymmetric, so tune stage 2 for recall, not balanced accuracy. Drop only confident non-buyers and let the later stages do the discriminating. An F1 gate is the wrong instrument for a stage whose precision is repaired downstream for free.

This also defuses most model-instability worry: label flips within the kept set are harmless. Only flips across the keep/drop line matter, and that is a much smaller number than overall drift.

Stage 4 takes one call per company, not one per field. Headcount, revenue and industry come back together. Splitting them into stages multiplies the scarcest budget in the run by three or four for identical data.

Never resolve a company by name

This is the single most damaging shortcut available here, because it produces a confident answer that no downstream check can catch.

A commenter's employer arrives as a display string ("Luxury Digital"), and company enrichment wants an identifier. Searching a data provider by name always returns something, and nothing in the response says whether it is the same company. Measured on 10 real rows, roughly 1 in 5 matched a different company entirely — and string similarity cannot separate the good from the bad:

"Arizona Fire & Water" -> "Arizona Water And Fire Restoration"   0.95  correct
"Luxury Digital"       -> "DLG (Digital Luxury Group)"           0.95  WRONG
"The fools"            -> "The Fools Imersão Em Línguas"         0.95  WRONG

A fuller legal name and an unrelated company sharing a word score identically. There is no threshold to tune, so don't try to build a guard — remove the guess instead.

The person's own profile carries the exact identifier. A profile scrape returns currentPosition.companyLinkedinUrl (plus companyId and companyUniversalName) alongside the title, so the chain is:

comment -> profile URL -> profile scrape -> currentPosition.companyLinkedinUrl
                                         -> company enrichment (exact, 1 credit)

The same measured row, both ways:

A 60x headcount error, and it inverts the label: two employees plus "CEO/Founder" is fractional under rule 2, not the buyer the wrong company implied. Errors of this kind do not fail safe — a bigger wrong company looks like a better prospect.

So the profile scrape is not an optional refinement for ambiguous rows. It is what makes the company step correct at all, and it pays for itself by halving the per-company credit cost.

Two things worth taking from the same scrape while you have it: the full experience history (the employment check, without a second call) and each position's description, which is often where someone states who they sell to far more precisely than their headline does.

Getting to the company's website — profile first, Exa second, Firecrawl to read

Company enrichment gives you firmographics. It does not tell you what the business actually does, which is what ICP fit turns on. For that you need their site, and there is a strict order for getting to it.

1 · From the LinkedIn profile, always, when it is there. currentPosition.companyLinkedinUrl came back with the profile scrape, and the company page carries the website. This is an identifier, not a guess, and it is the whole reason the profile scrape happens before the company step.

2 · Exa, only when the profile has no company URL. Some profiles list an employer as free text with no linked page. That is a hard-to-specify target — you know the name and not the address — which is exactly what semantic search is for. Firecrawl is the wrong tool here; it is precise once you know where to look, and you don't yet.

Guard the Exa result the same way you guard a name lookup, because it is one: confirm the site belongs to the company you meant before reading it. A resolver returns a real, well-formed, confident URL for a different business without complaint — that is how a creator in this project ended up with a complete confidence 1.0 ICP record for a company he had never worked at. Cross-check the domain against the person's own headline, their post text, or the company page before spending anything on it.

3 · Firecrawl to read it, and read more than the homepage. firecrawl_map enumerates the site's URLs cheaply; then scrape the pages that actually state the business, not the one that states the brand:

Never read only the root domain. Measured across 10 companies: a homepage sells the brand and frequently the wrong business — coldiq.com's root now sells an API product while the agency that is the actual business lives at /agency. A root-domain read returned the wrong business at 0.6 confidence.

Route the whole sequence through icp-research rather than reimplementing it. It already encodes the two-level client-roster pass, the ad-library read for who they target rather than who they won, and the per-value evidence rule.

4 · No website at all → fall back to the data providers. Blitz first.

Plenty of real businesses have no site worth reading, or none at all — a one-person consultancy operating entirely on LinkedIn, a company whose site is a single splash page. That is a legitimate finding about the lead, not a failure. Record it: no readable web presence is itself a signal about size and formality, and it belongs in the row rather than being silently blank.

Then enrich from the providers instead:

Check the providers are actually configured before you get here, not when the funnel stalls. The scrape and classification stages need neither, so a missing Blitz key surfaces only after money has already been spent on stages 1–3.

set -a && . ./local.env && set +a
[ -n "$BLITZ_API_KEY" ] || echo "BLITZ_API_KEY missing"
[ -n "$CLAY_API_KEY" ]  || echo "CLAY_API_KEY missing (fallback unavailable)"

If BLITZ_API_KEY is unset, say so plainly and tell the user what to do — do not silently skip the company step or quietly fall through to Clay:

The company enrichment step needs BLITZ_API_KEY in local.env (gitignored). Add it as BLITZ_API_KEY=... and reload with set -a && . ./local.env && set +a. .env.example documents the expected names. Without it I can resolve companies but not size or qualify them.

If Clay is the intended fallback and neither its MCP server nor CLAY_API_KEY is configured, say that too — a waterfall with one working provider is a single point of failure, and the moment to learn that is before the run, not during it. The Clay MCP is configured in .cursor/mcp.json / the Claude MCP settings; the REST key is separate and uses a non-standard auth header. See clay-api.

Never hardcode either key in a script or a skill. Read from the environment and fail loudly when unset — a paid pipeline that half-runs because a key was missing is worse than one that refuses to start.

Blitz first — /v2/enrichment/company off the company LinkedIn URL returns size, exact employees_on_linkedin, industry and HQ, plus department and country distribution. Department distribution is the genuinely useful one for "could they use this service": a company with no marketing function is a different prospect from one with twelve.

Clay only on genuine failure. And be precise about what failure means: a 429 or a 5xx is a retry, not a fallback. Falling through on a transient error pays Clay for data Blitz would have returned free, and at 5 RPS you will hit rate limits. Fall through only when Blitz returns a definitive "not found".

Order matters for cost as well as accuracy — check the key's remaining balance and rate limit before running at volume, and say what a run will cost before starting it. None of these APIs are idempotent: a retried write is a second charge. See clay-api and the Blitz skills for the call shapes.

The provider path gives firmographics but not business context — it tells you how big they are, never who they sell to. A lead enriched this way should carry a lower confidence on ICP fit than one whose site was actually read, and the row should record which of the two produced it.

Cost discipline: this is per company, not per lead, so dedupe on the company URL before crawling. Several commenters share an employer, and paying to read the same site four times is the same mistake as enriching a person once per post.

Fit is a relationship, not an attribute. Whether someone is "in ICP" or "a peer" is only defined relative to a creator — the same person can be squarely in one creator's ICP and irrelevant to another's. Store fit keyed on (person, creator) and record which ICP version produced it; a fit score whose ICP generation is unknown can't be audited later. Enrich the company once globally, score it N times per creator — the expensive half is per-company, the per-pair half is arithmetic.

Keep "in ICP" and "is a peer" as independent booleans. In GTM the peer often is the buyer — an agency owner can be both — and collapsing them into one label throws away the distinction the research usually cares about.

The sampling trap

Comment scrapers cap per post. That cap is a sampling design, and it is the most under-recorded thing in this kind of work.

Two consequences, both easy to miss:

1. First-N is not random. Early commenters differ systematically from late ones — they're closer to the creator, more likely to be peers.
2. *The sampling fraction correlates with post popularity.* A 150-comment cap is 100% of a 40-comment post and 7.5% of a 2,000-comment one. If popularity is also your independent variable, the sampling rate is entangled with it.

So record, per post: total comments available, how many were captured, and by what rule. Without those three, a per-post rate cannot be interpreted later, and whoever inherits the data will assume completeness.

Before trusting a cap, check whether it binds at all. In one corpus only 8.4% of posts exceeded 150 comments and the median post had 28 — the cap looked frightening and barely mattered. Compute sum(min(comments, cap)) over the real selection rather than posts × cap; the projection can overstate volume by a third or more, and the whole budget is built on that number.

Robustness that actually matters

Write the scraper so these are true. They cost little up front and are painful to add later:

• Write rows as they arrive, per post — not one bulk insert at the end. A bulk insert turns any mid-run failure into total loss.
• Make re-runs free and safe: skip inputs already stored. These APIs have no idempotency key, so a retried write is a second charge, not a no-op.
• Record attempted-and-failed distinctly from not-yet-attempted. Otherwise a permanent failure is retried forever, paying each time.
• Never let a partial run exit silently. Compare rows expected against rows written and say the difference out loud. Silent partial success is the bug that corrupts denominators, and it presents as a clean exit code.
• Verify writes by reading back, especially into spreadsheets and databases that return 200 on a no-op. Trust the read, not the status code.
• Cap the batch and dry-run at small N before the full volume.

A cache is only as good as what it stores. If a cached record is missing a field the code later reads, the cache key can't tell — every run returns the same empty value and nothing errors. When adding a field to a cached record, treat records lacking it as misses, or the fix silently does nothing.

Verify the employer before spending, not before sending

This is a stage in the funnel, not a courtesy check handed to whoever consumes the list. It runs before contact-level enrichment, because enrichment spent on someone who left the company is money spent on a row that was never going to work, and because emailing someone about a company they left is the most visible possible signal that outreach is automated and unchecked — it burns the domain as well as the lead.

Contact data decays at roughly 2%/month. A list that was clean a year ago is about a quarter wrong now.

The three checks, in order

1 · The company resolves, and is the company you think it is. A plausible name is not confirmation. Resolve to a LinkedIn company URL from the person's profile, never by searching the name — see "Never resolve a company by name" above.

2 · Read the current position from their LinkedIn experience. Not an enrichment provider's employer field, which returns a confident answer for every row whether or not it is true.

Two traps in reading experience, and both produce false confidence:

• A missing end date does not mean current. People leave "Founder, Acme" open for years after leaving. Treating "no end date" as "still there" is the single most common way this check passes when it should fail.
• Between roles looks like employed. Someone open to work often still lists a last position with no end date, or lists a placeholder. A between-roles phrase in the headline ("open to work", "looking for my next role") overrides the position entirely — no employer means no budget.

3 · Read their last 5 posts. The check most pipelines skip, and the one that catches what the other two miss.

People forget to update their profile but they do announce moves: "excited to share I've joined X", "after four years I'm leaving Y". The profile is stale data; the posts are fresh data. When a post contradicts the profile, the post wins.

A cheap and surprisingly decisive form of check 3, when you already hold the person's posts: does the assumed employer's name appear in their own writing at all? Someone who works somewhere mentions it. Measured on this project's 36 creators, this separated 4 broken mappings from 32 correct ones where a profile-page check had no signal whatsoever.

Two things to get right or it produces noise instead of findings:

• Match on word boundaries, not substrings. "HP" matches inside "sharp".
• Tokenise the company name on whitespace as well as punctuation, and drop generic words. "AFAS Software" searched as one string finds nothing; searched as afas it matches immediately. Both of these produced false positives on first run here, and a false positive on an integrity check is worse than none — it trains you to ignore the check.

Record which source established employment, so a stale row can be re-checked later rather than silently trusted.

The creator's own company must be verified too — with more care than the leads

Everything above applies to commenters. It applies harder to the creator, because the creator's ICP is what every commenter on their posts gets scored against. A wrong company here does not corrupt one row; it corrupts every score derived from that creator.

Observed on this project, 2026-08-14: a creator was mapped to a company name before any page was fetched. The domain resolver then worked perfectly, read that company's site, and produced a complete, internally consistent ICP record at confidence 1.0 — for a company the person has no connection to. His actual employer was the single most frequent brand word in his own posts.

Nothing downstream can catch this. The record is coherent; it simply describes someone else. Validation passes, quote-coverage passes, confidence is maximal.

So verify the creator → company mapping at the point it is created, and treat these as separate failure modes:

A fabricated company is not a small problem either: there is no website that states that entity's ICP, so whatever gets extracted is assembled from a personal site or a newsletter and is not an ICP at all.

Volume is a projection until the scrape runs

Say which numbers are measured and which are estimated, every time. A cost estimate quoted without that distinction gets remembered as a commitment.