This matters for RAG because retrieval quality depends on chunk quality. A table that gets flattened into "1 2 3 4 5 6 7 8 9" instead of a proper Markdown table is useless to both your chunker and the embedding model. A section header that disappears means your chunk boundaries stop lining up with the document's actual structure.

This post is a practical, tested walkthrough of converting documents to Markdown for LLM ingestion: what the output actually looks like, where OCR fits in, and where the limits are.

Why Markdown specifically

Markdown is the sweet spot between "plain text" and "full HTML/XML" for LLM pipelines:

• Headings (#, ##) preserve document structure so you can chunk by section instead of by arbitrary character count
• Tables render as pipe tables, which both humans and LLMs parse correctly
• Lists stay lists instead of collapsing into runs of text
• It's token-efficient compared to HTML, with no tag overhead

That's why most modern RAG tooling (LangChain loaders, LlamaIndex parsers, and dedicated services) converge on Markdown or Markdown-like intermediate formats rather than plain text or raw HTML.

Testing a document-to-Markdown Actor on a real PDF

Rather than describe this abstractly, I ran a real file through our PDF to Markdown Converter (full disclosure: we built it) and I'm showing the unedited output below. It's an Apify Actor (a serverless program with a REST API), so the whole interaction is a single HTTP call.

I pointed it at the first page of a well-known arXiv paper (public, stable URL, good test case because it has a title block, author table, and body text):

export APIFY_TOKEN="your-apify-token"

curl -s "https://api.apify.com/v2/acts/magenta_waterwheel~document-to-markdown/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://arxiv.org/pdf/1706.03762"}],
    "pageRange": "1",
    "maxPagesPerDocument": 1,
    "outputMode": "document"
  }'

The real, unedited response (truncated for length; the full markdown field is longer):

[{
  "url": "https://arxiv.org/pdf/1706.03762",
  "status": "success",
  "fileName": "1706.03762v7.pdf",
  "title": "Attention Is All You Need",
  "pageCount": 15,
  "pagesExtracted": 1,
  "ocrPages": 0,
  "tableCount": 2,
  "wordCount": 420,
  "markdown": "# Attention Is All You Need\n\n| Ashish Vaswani * | Noam Shazeer * | Niki Parmar * | Jakob Uszkoreit * |\n| --- | --- | --- | --- |\n| Google Brain | Google Brain | Google Research | Google Research |\n\n## Abstract\n\nThe dominant sequence transduction models are based on complex recurrent or convolutional neural networks..."
}]

Two things worth noting in that output: the author block, which is laid out as a table in the PDF, came back as an actual Markdown table (tableCount: 2), and the heading structure (# Attention Is All You Need, ## Abstract) survived instead of collapsing into one text blob. ocrPages: 0 confirms this went through text extraction, not OCR, since arXiv PDFs have a real text layer.

Getting per-page output for chunking

For RAG, you usually don't want one giant Markdown blob per document; you want chunks, and page boundaries are a reasonable default chunking unit before you apply your own semantic splitter. Setting outputMode to "page" gives you one dataset item per page instead of per document. I tested this too, with a plain Python script using requests:

import os
import requests

TOKEN = os.environ["APIFY_TOKEN"]
URL = "https://api.apify.com/v2/acts/magenta_waterwheel~document-to-markdown/run-sync-get-dataset-items"

payload = {
    "startUrls": [{"url": "https://arxiv.org/pdf/1706.03762"}],
    "pageRange": "1",
    "maxPagesPerDocument": 1,
    "outputMode": "page",
}

resp = requests.post(URL, params={"token": TOKEN}, json=payload, timeout=180)
resp.raise_for_status()
items = resp.json()

for item in items:
    print(item["url"], "page", item["pageNumber"],
          "words:", item["wordCount"], "tables:", item["tableCount"])

Output from the actual run:

https://arxiv.org/pdf/1706.03762 page 1 words: 420 tables: 2

Each item in "page" mode carries its own pageNumber, markdown, and metadata, which maps directly onto a loop that upserts into Pinecone, Weaviate, pgvector, or Chroma: one page, one (pre-chunking) unit, with the document's own structure already intact rather than an arbitrary character-count split.

Where OCR fits in

Not every PDF has a usable text layer: scanned contracts, faxed forms, and image-only exports are common in real document corpora (especially anything from legal, finance, or older archival systems). The Actor exposes an ocrMode input with three settings:

• auto: runs OCR only on pages that have no usable text layer (the sensible default; it doesn't waste OCR cost on pages that already have real text)
• always: runs OCR on every page, useful if you don't trust the embedded text layer (some PDFs have garbled or wrong text layers from bad scans with a OCR pass baked in)
• never: skip OCR entirely

OCR pages are billed at a higher per-page rate than text-layer pages, which is a fair trade: OCR is more compute-intensive, and auto mode means you're not paying that rate on documents that don't need it. I didn't burn budget re-testing OCR specifically for this post since the pricing and auto-detection logic are documented on the Actor's product page, but the behavior (skip OCR when a text layer exists) matches what I'd expect from checking the ocrPages: 0 field on the arXiv test above: a native-text PDF correctly didn't trigger OCR.

Honest limitations

• Table extraction is good, not infallible. Simple grid tables convert cleanly, as shown above. Deeply nested or merged-cell tables, or tables that are really just visually aligned text without actual table structure in the PDF, can still come out imperfect. Spot-check anything with complex tables before trusting it in production.
• This is priced per page, including OCR pages at a higher rate. For a large backlog (tens of thousands of pages), do the math up front rather than discovering the bill after a bulk run. Failed documents aren't charged, which helps with messy crawls that include corrupt or password-protected files, but successful pages are billed even if the content turns out to be low-value (e.g., a mostly-blank cover page).
• It converts structure, it doesn't deduplicate or summarize. You still own the chunking strategy, embedding choice, and dedup logic downstream; this tool solves "get clean structured text out of a binary file," not the rest of the RAG pipeline.
• Max file size and page caps exist (configurable, defaults around 50MB and 1,000 pages per document) to prevent a single huge file from blowing up a batch job's cost or runtime unexpectedly. Good to set deliberately rather than leave on defaults if your corpus includes unusually large files.

Where this fits in a bigger pipeline

If you're already running document ingestion as part of a content or competitive-intelligence workflow, this slots in naturally alongside other scraping-style data collection. For instance, if you're pulling competitor whitepapers or decks to analyze alongside search visibility data (something like Magenta Waterwheel's AI Overview tracker for checking whether those same competitors show up in Google's AI answers), the pattern is the same: call an Actor's REST endpoint, get structured JSON back, feed it into whatever you're already building.

For the full input schema, pricing table, and more worked examples, see the PDF to Markdown Converter product page, or the Apify Store listing directly if you want to read the raw schema before writing any code.

If you wired curly/keyword-search-volume-scraper into a script, cron, webhook, or no-code flow, you have a few days to pick a path that still returns Google search volume after the Actor goes dark. This post is a migration checklist — not a full SEO stack review.

How Curly's volume API worked

According to the Apify listing, each run took:

• One seed keyword (string, up to 200 characters)
• Optional language (two-letter code, default en)
• Optional region (two-letter code, default US, about 30 markets)

You got back a single rich report: average search volume, monthly volume, CPC, competition, SEO difficulty, search intent, clickstream and audience splits, related keyword ideas, questions and comparisons, content ideas, and ranking pages.

Pricing was pay-per-report (on the order of $1.50–$2.00 per 1,000 successful reports depending on plan, plus a small run-start fee). One keyword per run. That made it convenient for "give me everything about this seed" workflows — and awkward if you only needed three fields.

What actually breaks on Oct 28

Anything that still POSTs to curly~keyword-search-volume-scraper (or the Store slug curly/keyword-search-volume-scraper) will stop delivering those reports after shutdown. Schedules keep firing; datasets go empty or fail.

The painful part is not "pick any SEO tool." It is matching input shape and being honest about output coverage.

Fields that usually have a replacement

• Search volume (Keyword Planner average monthly searches)
• Monthly volume history
• CPC and competition (often with bid ranges)

Fields that usually do not

• SEO difficulty
• Search intent labels
• Related ideas / questions / comparisons generated from the seed
• Content ideas and ranking pages
• Clickstream / audience splits

If your pipeline only logged volume, CPC, and competition into a sheet or warehouse, a Keyword Planner–backed API is enough. If you treated Curly as a mini Ahrefs, you need a second tool for difficulty and ideas.

Migration checklist (Planner-style Actors)

Use this when moving to any Google Keyword Planner–shaped Actor or Ads API wrapper:

1. Change keyword → keywords (array). Even one term becomes ["your keyword"]. Batch when you can; many tools charge a per-run lookup fee.
2. Keep language. Same two-letter codes (en, de, …).
3. Change region codes to location names. US → United States, GB → United Kingdom. Some tools accept cities like London,England,United Kingdom.
4. Do not assume SEO difficulty or ideas. If the tool says it is Keyword Planner data, it will not invent difficulty scores or a related-keyword tree the way Curly's report did.
5. Re-test one sync call with run-sync-get-dataset-items (or your client) before you flip production schedules.
6. Update docs and secrets hygiene. Same rule as before: never put an Apify token in client-side code.

Before / after example

Curly:

{
  "keyword": "content marketing automation",
  "language": "en",
  "region": "US"
}

Planner-style:

{
  "keywords": ["content marketing automation"],
  "language": "en",
  "location": "United States"
}

Same endpoint pattern on Apify (run-sync-get-dataset-items); different Actor id and input schema.

Options after Curly (honest comparison)

1. Google Keyword Planner (UI or Ads API)

Best if you already have a Google Ads account and mostly need spot checks or Ads-native workflows. Exact Planner numbers, official source, no third-party Actor. Downsides: account and spend history quirks, API access is heavier than a single Apify POST, and many people used Curly specifically to avoid standing up Ads.

2. Free or freemium volume checkers

Fine for a handful of keywords. Usually rate-limited, UI-only, or rounded. Poor fit for scheduled pipelines and large lists.

3. Full SEO suites (Ahrefs, Semrush, Similarweb, etc.)

Right answer if you relied on Curly for difficulty, intent, ideas, and SERP context — not only volume. Higher cost; different data model; overkill if you only need Planner metrics.

4. Apify Store pay-per-event keyword Actors

Closest operationally if you already lived in Apify schedules and webhooks. You keep the same platform, swap the Actor, and adjust the JSON.

One option in that bucket is Magenta Waterwheel's Keyword Planner API (magenta_waterwheel/keyword-search-volume) and Bulk Keyword Planner (magenta_waterwheel/bulk-keyword-planner). They return Google Keyword Planner volume, monthly history, CPC, bids, and competition for the keywords you send. They do not replace SEO difficulty, search intent, or keyword ideas. Input mapping and field table: Curly alternative guide. Free keyword checkers and other Planner alternatives also live on magentawaterwheel.com if you want to compare before committing a pipeline.

Price note from that guide (live PPE as of 2026-10-24; always re-check the Store): Keyword Planner API charges per keyword with data plus a per-run lookup fee, so one keyword per run is more expensive than Curly's old per-report model unless you batch. Bulk is a flat fee per batch of up to 1,000 keywords. Keywords with no Google data are free on the per-keyword path.

What I would do this week

• Inventory every call site for curly/keyword-search-volume-scraper (and the SERP/News Actors if you used those).
• Decide which fields you actually consume in code or sheets.
• If the answer is volume + CPC + competition only: migrate to Keyword Planner (Ads) or a Planner Actor with the array / language / location checklist above.
• If you need difficulty or ideas: pick a suite or SERP tool in parallel; do not expect a volume Actor to fake those fields.
• Cut over before October 28, 2026, then delete or disable the Curly schedules so failed runs do not pile up.

That is the whole job: match the fields you use, fix the input shape, and do not pretend a Planner API is a full SEO suite.

Disclosure: I work on Magenta Waterwheel's keyword tools on Apify. The migration table and pricing above match our public Curly alternative page; re-check Apify Store prices before you budget.