Skip to content

Instantly share code, notes, and snippets.

@ethanabrooks

ethanabrooks/scrapers.ipynb

Last active December 30, 2025 15:01
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ethanabrooks/c19ee76c4dc7c96ff9a4a27f3a5deedc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ethanabrooks/c19ee76c4dc7c96ff9a4a27f3a5deedc to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
scrapers.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "555a0ef8",
"metadata": {
"papermill": {
"duration": 0.008237,
"end_time": "2025-12-30T14:58:40.121620",
"exception": false,
"start_time": "2025-12-30T14:58:40.113383",
"status": "completed"
},
"tags": []
},
"source": [
"# Web Content Extraction: Tool Comparison Report\n",
"\n",
"**Goal**: Evaluate tools for converting arbitrary URLs into readable text/markdown.\n",
"\n",
"## Tools Compared\n",
"\n",
"| Tool | Type | Cost | Strengths |\n",
"|------|------|------|-----------|\n",
"| **Trafilatura** | Open Source | Free | Best pure-Python extractor, native markdown |\n",
"| **Parallel.ai** | API | Paid | Handles PDFs, JS-heavy pages, high quality |\n",
"| **Exa** | API | Paid | Fast, good for indexed content |\n",
"\n",
"## Test Cases\n",
"\n",
"We selected URLs that stress-test different extraction challenges:\n",
"\n",
"| Test Case | Challenge |\n",
"|-----------|-----------|\n",
"| arXiv PDF | Binary PDF parsing |\n",
"| Amazon homepage | JavaScript-rendered, bot detection |\n",
"| GitHub blog | Clean article extraction |\n",
"| Wikipedia table | Structured data preservation |"
]
},
{
"cell_type": "markdown",
"id": "2bc50481",
"metadata": {
"papermill": {
"duration": 0.003805,
"end_time": "2025-12-30T14:58:40.130569",
"exception": false,
"start_time": "2025-12-30T14:58:40.126764",
"status": "completed"
},
"tags": []
},
"source": [
"---\n",
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "990a0177",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:40.139056Z",
"iopub.status.busy": "2025-12-30T14:58:40.138873Z",
"iopub.status.idle": "2025-12-30T14:58:40.526439Z",
"shell.execute_reply": "2025-12-30T14:58:40.525935Z"
},
"papermill": {
"duration": 0.392224,
"end_time": "2025-12-30T14:58:40.527066",
"exception": false,
"start_time": "2025-12-30T14:58:40.134842",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"import os\n",
"import trafilatura\n",
"from parallel import Parallel\n",
"from exa_py import Exa"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f5f3ec35",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:40.531303Z",
"iopub.status.busy": "2025-12-30T14:58:40.531160Z",
"iopub.status.idle": "2025-12-30T14:58:40.532832Z",
"shell.execute_reply": "2025-12-30T14:58:40.532450Z"
},
"papermill": {
"duration": 0.004272,
"end_time": "2025-12-30T14:58:40.533315",
"exception": false,
"start_time": "2025-12-30T14:58:40.529043",
"status": "completed"
},
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"MAX_CHARS = 2000"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "65464d5f",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:40.537495Z",
"iopub.status.busy": "2025-12-30T14:58:40.537401Z",
"iopub.status.idle": "2025-12-30T14:58:40.570137Z",
"shell.execute_reply": "2025-12-30T14:58:40.569770Z"
},
"papermill": {
"duration": 0.035672,
"end_time": "2025-12-30T14:58:40.570727",
"exception": false,
"start_time": "2025-12-30T14:58:40.535055",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"# API clients\n",
"parallel_client = None\n",
"exa_client = None\n",
"\n",
"if os.getenv(\"PARALLEL_API_KEY\"):\n",
" parallel_client = Parallel(api_key=os.getenv(\"PARALLEL_API_KEY\"))\n",
"\n",
"if os.getenv(\"EXA_API_KEY\"):\n",
" exa_client = Exa(os.getenv(\"EXA_API_KEY\"))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "178e6b7e",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:40.578550Z",
"iopub.status.busy": "2025-12-30T14:58:40.578461Z",
"iopub.status.idle": "2025-12-30T14:58:40.581059Z",
"shell.execute_reply": "2025-12-30T14:58:40.580751Z"
},
"papermill": {
"duration": 0.007688,
"end_time": "2025-12-30T14:58:40.581558",
"exception": false,
"start_time": "2025-12-30T14:58:40.573870",
"status": "completed"
},
"tags": []
},
"outputs": [],
"source": [
"def extract_with_trafilatura(url: str) -> str | None:\n",
" \"\"\"Trafilatura with its own fetcher (handles some JS).\"\"\"\n",
" downloaded = trafilatura.fetch_url(url)\n",
" if downloaded:\n",
" return trafilatura.extract(\n",
" downloaded,\n",
" output_format=\"markdown\",\n",
" include_tables=True,\n",
" include_links=True,\n",
" )\n",
" return None\n",
"\n",
"\n",
"def extract_with_parallel(url: str) -> str | None:\n",
" \"\"\"Parallel.ai API extraction.\"\"\"\n",
" if not parallel_client:\n",
" return None\n",
" result = parallel_client.beta.extract(\n",
" urls=[url],\n",
" objective=\"Extract the main content\",\n",
" full_content=True,\n",
" )\n",
" if result.results and result.results[0].full_content:\n",
" return result.results[0].full_content\n",
" return None\n",
"\n",
"\n",
"def extract_with_exa(url: str) -> str | None:\n",
" \"\"\"Exa API extraction with livecrawl fallback.\"\"\"\n",
" if not exa_client:\n",
" return None\n",
" result = exa_client.get_contents(\n",
" urls=[url],\n",
" text=True,\n",
" livecrawl=\"fallback\",\n",
" )\n",
" if result.results and result.results[0].text:\n",
" return result.results[0].text\n",
" return None\n",
"\n",
"\n",
"def compare(url: str, description: str) -> dict:\n",
" \"\"\"Run all extractors on a URL and return results.\"\"\"\n",
" print(f\"Extracting: {url}\")\n",
" return {\n",
" \"url\": url,\n",
" \"description\": description,\n",
" \"trafilatura\": extract_with_trafilatura(url),\n",
" \"parallel\": extract_with_parallel(url),\n",
" \"exa\": extract_with_exa(url),\n",
" }"
]
},
{
"cell_type": "markdown",
"id": "eb086f4a",
"metadata": {
"papermill": {
"duration": 0.001907,
"end_time": "2025-12-30T14:58:40.585362",
"exception": false,
"start_time": "2025-12-30T14:58:40.583455",
"status": "completed"
},
"tags": []
},
"source": [
"---\n",
"## Test 1: PDF Document (arXiv Paper)\n",
"\n",
"**Challenge**: Parse binary PDF content, extract structured text.\n",
"\n",
"This is the hardest test - most HTML extractors fail completely on PDFs."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "09ab0848",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:40.589757Z",
"iopub.status.busy": "2025-12-30T14:58:40.589668Z",
"iopub.status.idle": "2025-12-30T14:58:42.551939Z",
"shell.execute_reply": "2025-12-30T14:58:42.550010Z"
},
"papermill": {
"duration": 1.966248,
"end_time": "2025-12-30T14:58:42.553689",
"exception": false,
"start_time": "2025-12-30T14:58:40.587441",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Extracting: https://arxiv.org/pdf/2307.06435.pdf\n"
]
}
],
"source": [
"pdf_results = compare(\n",
" \"https://arxiv.org/pdf/2307.06435.pdf\",\n",
" \"arXiv PDF - A Comprehensive Overview of Large Language Models\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "37f9c83e",
"metadata": {
"papermill": {
"duration": 0.004457,
"end_time": "2025-12-30T14:58:42.564738",
"exception": false,
"start_time": "2025-12-30T14:58:42.560281",
"status": "completed"
},
"tags": []
},
"source": [
"### Trafilatura (Open Source)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "625b39d4",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:42.573576Z",
"iopub.status.busy": "2025-12-30T14:58:42.573411Z",
"iopub.status.idle": "2025-12-30T14:58:42.576182Z",
"shell.execute_reply": "2025-12-30T14:58:42.575380Z"
},
"papermill": {
"duration": 0.008073,
"end_time": "2025-12-30T14:58:42.576818",
"exception": false,
"start_time": "2025-12-30T14:58:42.568745",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"❌ No content extracted\n"
]
}
],
"source": [
"if pdf_results[\"trafilatura\"]:\n",
" print(pdf_results[\"trafilatura\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "8115ac89",
"metadata": {
"papermill": {
"duration": 0.002817,
"end_time": "2025-12-30T14:58:42.583086",
"exception": false,
"start_time": "2025-12-30T14:58:42.580269",
"status": "completed"
},
"tags": []
},
"source": [
"### Parallel.ai (API)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "248487f8",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:42.589544Z",
"iopub.status.busy": "2025-12-30T14:58:42.589415Z",
"iopub.status.idle": "2025-12-30T14:58:42.591558Z",
"shell.execute_reply": "2025-12-30T14:58:42.591081Z"
},
"papermill": {
"duration": 0.005968,
"end_time": "2025-12-30T14:58:42.592072",
"exception": false,
"start_time": "2025-12-30T14:58:42.586104",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"A Comprehensive Overview of Large Language Models\n",
"\n",
"Humza Naveed <sup>a</sup> , Asad Ullah Khan <sup>b,</sup> <sup>∗</sup> , Shi Qiu <sup>c,</sup> <sup>∗</sup> , Muhammad Saqib <sup>d,e,</sup> <sup>∗</sup> , Saeed Anwar <sup>f,g</sup> , Muhammad Usman <sup>f,g</sup> , Naveed Akhtar <sup>h,j</sup> ,\n",
"\n",
"Nick Barnes <sup>i</sup> , Ajmal Mian <sup>j</sup>\n",
"\n",
"_a_ _The University of Sydney, Sydney, Australia_\n",
"\n",
"_b_ _University of Engineering and Technology (UET), Lahore, Pakistan_\n",
"\n",
"_c_ _The Chinese University of Hong Kong (CUHK), HKSAR, China_\n",
"\n",
"_d_ _University of Technology Sydney (UTS), Sydney, Australia_\n",
"\n",
"_e_ _Commonwealth Scientific and Industrial Research Organisation (CSIRO), Sydney, Australia_\n",
"\n",
"_f_ _King Fahd University of Petroleum and Minerals (KFUPM), Dhahran, Saudi Arabia_\n",
"\n",
"_g_ _SDAIA-KFUPM Joint Research Center for Artificial Intelligence (JRCAI), Dhahran, Saudi Arabia_\n",
"\n",
"_h_ _The University of Melbourne (UoM), Melbourne, Australia_\n",
"\n",
"_i_ _Australian National University (ANU), Canberra, Australia_\n",
"\n",
"_j_ _The University of Western Australia (UWA), Perth, Australia_\n",
"\n",
"**Abstract**\n",
"\n",
"Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and\n",
"\n",
"beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse\n",
"\n",
"topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs,\n",
"\n",
"robotics, datasets, benchmarking, e ffi ciency, and more. With the rapid development of techniques and regular breakthroughs in\n",
"\n",
"LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering\n",
"\n",
"the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise\n",
"\n",
"yet comprehensive overview of the recent developments in this field. This article provides an overview of the literature on a broa\n"
]
}
],
"source": [
"if pdf_results[\"parallel\"]:\n",
" print(pdf_results[\"parallel\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted (API key not set?)\")"
]
},
{
"cell_type": "markdown",
"id": "ad18ec4c",
"metadata": {
"papermill": {
"duration": 0.002403,
"end_time": "2025-12-30T14:58:42.597056",
"exception": false,
"start_time": "2025-12-30T14:58:42.594653",
"status": "completed"
},
"tags": []
},
"source": [
"### Exa (API)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "d3be31fe",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:42.602672Z",
"iopub.status.busy": "2025-12-30T14:58:42.602549Z",
"iopub.status.idle": "2025-12-30T14:58:42.604363Z",
"shell.execute_reply": "2025-12-30T14:58:42.604058Z"
},
"papermill": {
"duration": 0.005345,
"end_time": "2025-12-30T14:58:42.604953",
"exception": false,
"start_time": "2025-12-30T14:58:42.599608",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[We gratefully acknowledge support from\\\n",
"the Simons Foundation and member institutions.](https://confluence.cornell.edu/x/ALlRF)\n"
]
}
],
"source": [
"if pdf_results[\"exa\"]:\n",
" print(pdf_results[\"exa\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted (API key not set?)\")"
]
},
{
"cell_type": "markdown",
"id": "c60eae50",
"metadata": {
"papermill": {
"duration": 0.002298,
"end_time": "2025-12-30T14:58:42.609724",
"exception": false,
"start_time": "2025-12-30T14:58:42.607426",
"status": "completed"
},
"tags": []
},
"source": [
"### PDF Test Summary\n",
"\n",
"| Tool | Result |\n",
"|------|--------|\n",
"| Trafilatura | ❌ Cannot parse PDF binary |\n",
"| Parallel.ai | ✅ Full paper extraction with structure |\n",
"| Exa | ❌ Returns only footer text (~129 chars) |"
]
},
{
"cell_type": "markdown",
"id": "73f8e5f3",
"metadata": {
"papermill": {
"duration": 0.002229,
"end_time": "2025-12-30T14:58:42.614274",
"exception": false,
"start_time": "2025-12-30T14:58:42.612045",
"status": "completed"
},
"tags": []
},
"source": [
"---\n",
"## Test 2: JavaScript-Heavy Page (Amazon)\n",
"\n",
"**Challenge**: Render JavaScript, bypass bot detection."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7acdd540",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:42.619561Z",
"iopub.status.busy": "2025-12-30T14:58:42.619449Z",
"iopub.status.idle": "2025-12-30T14:58:44.140934Z",
"shell.execute_reply": "2025-12-30T14:58:44.140296Z"
},
"papermill": {
"duration": 1.525563,
"end_time": "2025-12-30T14:58:44.142078",
"exception": false,
"start_time": "2025-12-30T14:58:42.616515",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Extracting: https://amazon.com\n"
]
}
],
"source": [
"amazon_results = compare(\n",
" \"https://amazon.com\",\n",
" \"Amazon homepage - JS-rendered, bot detection\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "b8e4a5b5",
"metadata": {
"papermill": {
"duration": 0.002796,
"end_time": "2025-12-30T14:58:44.148314",
"exception": false,
"start_time": "2025-12-30T14:58:44.145518",
"status": "completed"
},
"tags": []
},
"source": [
"### Trafilatura"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "630be201",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:44.154782Z",
"iopub.status.busy": "2025-12-30T14:58:44.154649Z",
"iopub.status.idle": "2025-12-30T14:58:44.156963Z",
"shell.execute_reply": "2025-12-30T14:58:44.156348Z"
},
"papermill": {
"duration": 0.006132,
"end_time": "2025-12-30T14:58:44.157428",
"exception": false,
"start_time": "2025-12-30T14:58:44.151296",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Click the button below to continue shopping\n",
"Continue shopping\n",
"Conditions of Use\n",
"Privacy Policy\n",
"© 1996-2025, Amazon.com, Inc. or its affiliates\n"
]
}
],
"source": [
"if amazon_results[\"trafilatura\"]:\n",
" print(amazon_results[\"trafilatura\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "0b4e7812",
"metadata": {
"papermill": {
"duration": 0.002467,
"end_time": "2025-12-30T14:58:44.162868",
"exception": false,
"start_time": "2025-12-30T14:58:44.160401",
"status": "completed"
},
"tags": []
},
"source": [
"### Parallel.ai"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "ef02ee85",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:44.168469Z",
"iopub.status.busy": "2025-12-30T14:58:44.168363Z",
"iopub.status.idle": "2025-12-30T14:58:44.170358Z",
"shell.execute_reply": "2025-12-30T14:58:44.169953Z"
},
"papermill": {
"duration": 0.005497,
"end_time": "2025-12-30T14:58:44.170788",
"exception": false,
"start_time": "2025-12-30T14:58:44.165291",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"## Skip to\n",
"\n",
"* [Main content]()\n",
"\n",
"* * *\n",
"\n",
"## Keyboard shortcuts\n",
"\n",
"* [Search alt \\+ /](javascript:void\\(0\\))\n",
"* [Cart shift \\+ alt \\+ C](javascript:void\\(0\\))\n",
"* [Home shift \\+ alt \\+ H](javascript:void\\(0\\))\n",
"* [Orders shift \\+ alt \\+ O](javascript:void\\(0\\))\n",
"* Show/Hide shortcuts\n",
" \n",
" shift \\+ alt \\+ Z\n",
"\n",
"To move between items, use your keyboard's up or down arrows.\n",
"\n",
"[.us](/ref=nav_logo)\n",
"\n",
"Delivering to Secaucus 07094 Update location\n",
"\n",
"All\n",
"\n",
"Select the department you want to search in All Departments Alexa Skills Amazon Autos Amazon Devices Amazon Fresh Amazon Global Store Amazon Haul Amazon One Medical Amazon Pharmacy Amazon Resale Appliances Apps & Games Arts, Crafts & Sewing Audible Books & Originals Automotive Parts & Accessories Baby Beauty & Personal Care Books CDs & Vinyl Cell Phones & Accessories Clothing, Shoes & Jewelry Women's Clothing, Shoes & Jewelry Men's Clothing, Shoes & Jewelry Girl's Clothing, Shoes & Jewelry Boy's Clothing, Shoes & Jewelry Baby Clothing, Shoes & Jewelry Collectibles & Fine Art Computers Credit and Payment Cards Digital Music Electronics Garden & Outdoor Gift Cards Grocery & Gourmet Food Handmade Health, Household & Baby Care Home & Business Services Home & Kitchen Industrial & Scientific Just for Prime Kindle Store Luggage & Travel Gear Luxury Stores Magazine Subscriptions Movies & TV Musical Instruments Office Products Pet Supplies Premium Beauty Prime Video Same-Day Store Smart Home Software Sports & Outdoors Subscribe & Save Subscription Boxes Tools & Home Improvement Toys & Games Under $10 Video Games Whole Foods Market\n",
"\n",
"Search Amazon\n",
"\n",
"[EN](/customer-preferences/edit?ie=UTF8&preferencesReturnUrl=%2F&ref_=topnav_lang)\n",
"\n",
"[Hello, sign in Account & Lists](https://www.amazon.com/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2F%3Fref_%3Dnav_ya_signin&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.claimed_id=http%3A%2F\n"
]
}
],
"source": [
"if amazon_results[\"parallel\"]:\n",
" print(amazon_results[\"parallel\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "75d19acb",
"metadata": {
"papermill": {
"duration": 0.002195,
"end_time": "2025-12-30T14:58:44.175371",
"exception": false,
"start_time": "2025-12-30T14:58:44.173176",
"status": "completed"
},
"tags": []
},
"source": [
"### Exa"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "e9f8e81a",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:44.180449Z",
"iopub.status.busy": "2025-12-30T14:58:44.180349Z",
"iopub.status.idle": "2025-12-30T14:58:44.182106Z",
"shell.execute_reply": "2025-12-30T14:58:44.181806Z"
},
"papermill": {
"duration": 0.004957,
"end_time": "2025-12-30T14:58:44.182609",
"exception": false,
"start_time": "2025-12-30T14:58:44.177652",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Amazon.com. Spend less. Smile more.\n",
"![](http://fls-na.amazon.com/1/batch/1/OP/ATVPDKIKX0DER:131-1984468-8102452:86MZDBATS9FV26T5ZBKE$uedata=s:%2Frd%2Fuedata%3Fstaticb%26id%3D86MZDBATS9FV26T5ZBKE:0)![](https://m.media-amazon.com/images/G/01/gno/sprites/nav-sprite-global-1x-reorg-privacy._CB779528203_.png)\n",
"[.us](http://amazon.com/ref=nav_logo)\n",
"[\n",
"Delivering to Boardman 97818Update location\n",
"]()\n",
"All**\n",
"Select the department you want to search inAll DepartmentsAlexa SkillsAmazon AutosAmazon DevicesAmazon Global StoreAmazon HaulAmazon One MedicalAmazon PharmacyAmazon ResaleAppliancesApps & GamesArts, Crafts & SewingAudible Books & OriginalsAutomotive Parts & AccessoriesBabyBeauty & Personal CareBooksCDs & VinylCell Phones & AccessoriesClothing, Shoes & JewelryWomen's Clothing, Shoes & JewelryMen's Clothing, Shoes & JewelryGirl's Clothing, Shoes & JewelryBoy's Clothing, Shoes & JewelryBaby Clothing, Shoes & JewelryCollectibles & Fine ArtComputersCredit and Payment CardsDigital MusicElectronicsGarden & OutdoorGift CardsGrocery & Gourmet FoodHandmadeHealth, Household & Baby CareHome & Business ServicesHome & KitchenIndustrial & ScientificJust for PrimeKindle StoreLuggage & Travel GearLuxury StoresMagazine SubscriptionsMovies & TVMusical InstrumentsOffice ProductsPet SuppliesPremium BeautyPrime VideoSmart HomeSoftwareSports & OutdoorsSubscribe & SaveSubscription BoxesTools & Home ImprovementToys & GamesUnder $10Video Games\n",
"Search Amazon\n",
"[\n",
"EN\n",
"](http://amazon.com/customer-preferences/edit?ie=UTF8&preferencesReturnUrl=/&ref_=topnav_lang)\n",
"[\n",
"Hello, sign in\n",
"Account & Lists](https://www.amazon.com/ap/signin?openid.pape.max_auth_age=0&openid.return_to=https://www.amazon.com/?_encoding=UTF8&ref_=nav_ya_signin&openid.identity=http://specs.openid.net/auth/2.0/identifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.claimed_id=http://specs.openid.net/auth/2.0/identifier_select&openid.ns=http://specs.openid.net/auth/2.0)\n",
"[Returns& Orders](http://amazon.com/gp/css/order-\n"
]
}
],
"source": [
"if amazon_results[\"exa\"]:\n",
" print(amazon_results[\"exa\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "5740e340",
"metadata": {
"papermill": {
"duration": 0.002024,
"end_time": "2025-12-30T14:58:44.186802",
"exception": false,
"start_time": "2025-12-30T14:58:44.184778",
"status": "completed"
},
"tags": []
},
"source": [
"### Amazon Test Summary\n",
"\n",
"| Tool | Result |\n",
"|------|--------|\n",
"| Trafilatura | ❌ Blocked by JS challenge (\"Enable JavaScript\") |\n",
"| Parallel.ai | ✅ Full page content with navigation and categories |\n",
"| Exa | ✅ Indexed homepage content |\n",
"\n",
"**Insight**: Amazon's bot detection blocks simple HTTP fetchers. API services handle this via browser rendering or cached indexes."
]
},
{
"cell_type": "markdown",
"id": "f3adbc95",
"metadata": {
"papermill": {
"duration": 0.001965,
"end_time": "2025-12-30T14:58:44.190766",
"exception": false,
"start_time": "2025-12-30T14:58:44.188801",
"status": "completed"
},
"tags": []
},
"source": [
"---\n",
"## Test 3: Clean Article (GitHub Blog)\n",
"\n",
"**Challenge**: Standard blog post - all tools should handle this well."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "4467fd58",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:44.195583Z",
"iopub.status.busy": "2025-12-30T14:58:44.195488Z",
"iopub.status.idle": "2025-12-30T14:58:45.081959Z",
"shell.execute_reply": "2025-12-30T14:58:45.081060Z"
},
"papermill": {
"duration": 0.890547,
"end_time": "2025-12-30T14:58:45.083514",
"exception": false,
"start_time": "2025-12-30T14:58:44.192967",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Extracting: https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/\n"
]
}
],
"source": [
"blog_results = compare(\n",
" \"https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/\",\n",
" \"GitHub blog post - clean article\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "c5a92f78",
"metadata": {
"papermill": {
"duration": 0.004431,
"end_time": "2025-12-30T14:58:45.093819",
"exception": false,
"start_time": "2025-12-30T14:58:45.089388",
"status": "completed"
},
"tags": []
},
"source": [
"### Trafilatura"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "bbc995c9",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:45.105146Z",
"iopub.status.busy": "2025-12-30T14:58:45.105002Z",
"iopub.status.idle": "2025-12-30T14:58:45.107584Z",
"shell.execute_reply": "2025-12-30T14:58:45.106895Z"
},
"papermill": {
"duration": 0.010388,
"end_time": "2025-12-30T14:58:45.108212",
"exception": false,
"start_time": "2025-12-30T14:58:45.097824",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Leader spotlight: Erin Spiceland\n",
"\n",
"We’re spending Women’s History Month with women leaders who are making history every day in the tech community. Read more about Erin Spiceland: Software Engineer at SpaceX.\n",
"\n",
"*Every March we recognize the women who have shaped history—and now, we’re taking a look forward. From driving software development in large companies to maintaining thriving open source communities, we’re spending Women’s History Month with women leaders who are making history every day in the tech community. Erin Spiceland is a Software Engineer for SpaceX. Born and raised in rural south Georgia, she is a Choctaw and Chickasaw mother of two now living in downtown Los Angeles. Erin didn’t finish college—she’s a predominantly self-taught software engineer. In her spare time, she makes handmade Native American beadwork and regalia and attends powwows. *\n",
"\n",
"**How would you summarize your career (so far) in a single sentence?**\n",
"\n",
"**How would you summarize your career (so far) in a single sentence?**\n",
"\n",
"My career has been a winding road through periods of stimulation and health as well as periods of personal misery. During it all, I’ve learned a variety of programming languages and technologies while working on a diverse array of products and services. I’m a domestic abuse survivor and a Choctaw bisexual polyamorous woman. I’m so proud of myself that I made it this far considering where I came from.\n",
"\n",
"**What was your first job in tech like?**\n",
"\n",
"**What was your first job in tech like?**\n",
"\n",
"In 2007, I had a three-year-old daughter and I was trying to finish my computer science degree one class at a time, all while keeping my house and family running smoothly. I found the math classes exciting and quickly finished my math minor, leaving only computer science classes. I was looking at about five years before I would graduate. Then, my husband at the time recommended me for an entry software developer position at a telecom and digital communications company.\n",
"\n",
"When faced with th\n"
]
}
],
"source": [
"if blog_results[\"trafilatura\"]:\n",
" print(blog_results[\"trafilatura\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "ed5460e6",
"metadata": {
"papermill": {
"duration": 0.003058,
"end_time": "2025-12-30T14:58:45.114803",
"exception": false,
"start_time": "2025-12-30T14:58:45.111745",
"status": "completed"
},
"tags": []
},
"source": [
"### Parallel.ai"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "2ac45889",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:45.121179Z",
"iopub.status.busy": "2025-12-30T14:58:45.121078Z",
"iopub.status.idle": "2025-12-30T14:58:45.122964Z",
"shell.execute_reply": "2025-12-30T14:58:45.122582Z"
},
"papermill": {
"duration": 0.005725,
"end_time": "2025-12-30T14:58:45.123459",
"exception": false,
"start_time": "2025-12-30T14:58:45.117734",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Skip to content]() [Skip to sidebar]()\n",
"\n",
"[](https://github.com) / [Blog](https://github.blog/)\n",
"\n",
"* [Changelog](https://github.blog/changelog/)\n",
"* [Docs](https://docs.github.com/)\n",
"* [Customer stories](https://github.com/customer-stories)\n",
"\n",
"[Try GitHub Copilot](https://github.com/features/copilot?utm_source=blog-tap-nav&utm_medium=blog&utm_campaign=universe25) [See what's new](https://github.com/events/universe/recap?utm_source=k2k-blog-tap-nav&utm_medium=blog&utm_campaign=universe25)\n",
"\n",
"* [AI & ML](https://github.blog/ai-and-ml/)\n",
" \n",
" + [AI & ML](https://github.blog/ai-and-ml/)\n",
" \n",
" Learn about artificial intelligence and machine learning across the GitHub ecosystem and the wider industry.\n",
" \n",
" - [Generative AI](https://github.blog/ai-and-ml/generative-ai/)\n",
" \n",
" Learn how to build with generative AI.\n",
" - [GitHub Copilot](https://github.blog/ai-and-ml/github-copilot/)\n",
" \n",
" Change how you work with GitHub Copilot.\n",
" - [LLMs](https://github.blog/ai-and-ml/llms/)\n",
" \n",
" Everything developers need to know about LLMs.\n",
" - [Machine learning](https://github.blog/ai-and-ml/machine-learning/)\n",
" \n",
" Machine learning tips, tricks, and best practices.\n",
" + [How AI code generation works](https://github.blog/ai-and-ml/generative-ai/how-ai-code-generation-works/)\n",
" \n",
" Explore the capabilities and benefits of AI code generation and how it can improve your developer experience.\n",
" \n",
" Learn more\n",
"* [Developer skills](https://github.blog/developer-skills/)\n",
" \n",
" + [Developer skills](https://github.blog/developer-skills/)\n",
" \n",
" Resources for developers to grow in their skills and careers.\n",
" \n",
" - [Application development](https://github.blog/developer-skills/application-development/)\n",
" \n",
" Insights and best practices for building apps.\n",
" - [Care\n"
]
}
],
"source": [
"if blog_results[\"parallel\"]:\n",
" print(blog_results[\"parallel\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "2b70bfeb",
"metadata": {
"papermill": {
"duration": 0.002581,
"end_time": "2025-12-30T14:58:45.128844",
"exception": false,
"start_time": "2025-12-30T14:58:45.126263",
"status": "completed"
},
"tags": []
},
"source": [
"### Exa"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "86f0e55a",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:45.134466Z",
"iopub.status.busy": "2025-12-30T14:58:45.134382Z",
"iopub.status.idle": "2025-12-30T14:58:45.136107Z",
"shell.execute_reply": "2025-12-30T14:58:45.135767Z"
},
"papermill": {
"duration": 0.005137,
"end_time": "2025-12-30T14:58:45.136575",
"exception": false,
"start_time": "2025-12-30T14:58:45.131438",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Skip to content](https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/#start-of-content)\n",
"\n",
"- [Community](https://github.blog/category/community/)\n",
"\n",
"# Leader spotlight: Erin Spiceland\n",
"\n",
"We’re spending Women’s History Month with women leaders who are making history every day in the tech community. Read more about Erin Spiceland: Software Engineer at SpaceX.\n",
"\n",
"![Erin Spiceland: Software Engineer at SpaceX](https://github.blog/wp-content/uploads/2019/03/Erin_blog.png?resize=1600%2C850)\n",
"\n",
"Author\n",
"\n",
"[![Jessica Rudder](https://avatars3.githubusercontent.com/u/6540763?v=4&s=200)Jessica Rudder](https://github.blog/author/jessrudder/)\n",
"\n",
"March 29, 2019\n",
"\n",
"_Every March we recognize the women who have shaped history—and now, we’re taking a look forward. From driving software development in large companies to maintaining thriving open source communities, we’re spending Women’s History Month with women leaders who are making history every day in the tech community. Erin Spiceland is a Software Engineer for SpaceX. Born and raised in rural south Georgia, she is a Choctaw and Chickasaw mother of two now living in downtown Los Angeles. Erin didn’t finish college—she’s a predominantly self-taught software engineer. In her spare time, she makes handmade Native American beadwork and regalia and attends powwows._\n",
"\n",
"## **How would you summarize your career (so far) in a single sentence?**\n",
"\n",
"My career has been a winding road through periods of stimulation and health as well as periods of personal misery. During it all, I’ve learned a variety of programming languages and technologies while working on a diverse array of products and services. I’m a domestic abuse survivor and a Choctaw bisexual polyamorous woman. I’m so proud of myself that I made it this far considering where I came from.\n",
"\n",
"## **What was your first job in tech like?**\n",
"\n",
"In 2007, I had a three-year-old daughter and I was trying to finish my computer science degree one class at a time, all while keeping my house and family runni\n"
]
}
],
"source": [
"if blog_results[\"exa\"]:\n",
" print(blog_results[\"exa\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "44ea555b",
"metadata": {
"papermill": {
"duration": 0.002415,
"end_time": "2025-12-30T14:58:45.141623",
"exception": false,
"start_time": "2025-12-30T14:58:45.139208",
"status": "completed"
},
"tags": []
},
"source": [
"### Blog Test Summary\n",
"\n",
"| Tool | Result |\n",
"|------|--------|\n",
"| Trafilatura | ✅ Clean extraction with headings and links |\n",
"| Parallel.ai | ✅ Full content (includes some nav elements) |\n",
"| Exa | ✅ Good extraction from index |\n",
"\n",
"**Insight**: All tools handle clean blog posts well. Trafilatura produces the cleanest output by filtering navigation. API services include more context but also more noise."
]
},
{
"cell_type": "markdown",
"id": "0beeb9e0",
"metadata": {
"papermill": {
"duration": 0.002335,
"end_time": "2025-12-30T14:58:45.146423",
"exception": false,
"start_time": "2025-12-30T14:58:45.144088",
"status": "completed"
},
"tags": []
},
"source": [
"---\n",
"## Test 4: Structured Data (Wikipedia Table)\n",
"\n",
"**Challenge**: Preserve table structure in extracted text."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "54472684",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:45.151565Z",
"iopub.status.busy": "2025-12-30T14:58:45.151463Z",
"iopub.status.idle": "2025-12-30T14:58:46.298803Z",
"shell.execute_reply": "2025-12-30T14:58:46.297584Z"
},
"papermill": {
"duration": 1.151662,
"end_time": "2025-12-30T14:58:46.300374",
"exception": false,
"start_time": "2025-12-30T14:58:45.148712",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Extracting: https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)\n"
]
}
],
"source": [
"wiki_results = compare(\n",
" \"https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nominal)\",\n",
" \"Wikipedia - GDP table with complex structure\",\n",
")"
]
},
{
"cell_type": "markdown",
"id": "73287bae",
"metadata": {
"papermill": {
"duration": 0.005851,
"end_time": "2025-12-30T14:58:46.314278",
"exception": false,
"start_time": "2025-12-30T14:58:46.308427",
"status": "completed"
},
"tags": []
},
"source": [
"### Trafilatura"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "0b2cfbf1",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:46.326562Z",
"iopub.status.busy": "2025-12-30T14:58:46.326365Z",
"iopub.status.idle": "2025-12-30T14:58:46.329394Z",
"shell.execute_reply": "2025-12-30T14:58:46.328758Z"
},
"papermill": {
"duration": 0.010336,
"end_time": "2025-12-30T14:58:46.330052",
"exception": false,
"start_time": "2025-12-30T14:58:46.319716",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# List of countries by GDP (nominal)\n",
"\n",
"\n",
"\n",
"| Largest economies in the world by GDP (nominal) in 2026 according to\n",
"|\n",
"\n",
"\n",
"[[n 2]](#cite_note-3)| > $20 trillion $10–20 trillion $5–10 trillion $1–5 trillion $750 billion – $1 trillion $500–750 billion | $250–500 billion $100–250 billion $50–100 billion $25–50 billion $5–25 billion < $5 billion |\n",
"\n",
"[Gross domestic product](/wiki/Gross_domestic_product) (GDP) is the [market value](/wiki/Market_value) of all final goods and services from a nation in a given year. [2] Countries are sorted by nominal GDP estimates from financial and statistical institutions, which are calculated at market or government official\n",
"\n",
"[exchange rates](/wiki/Exchange_rate). Nominal GDP does not take into account differences in the\n",
"\n",
"[cost of living](/wiki/Cost_of_living)in different countries, and the results can vary greatly from one year to another based on fluctuations in the exchange rates of the country's\n",
"\n",
"[currency](/wiki/Currency).\n",
"\n",
"Such fluctuations may change a country's ranking from one year to the next, even though they often make little or no difference in the standard of living of its population.\n",
"\n",
"[[3]](#cite_note-5)\n",
"\n",
"[[4]](#cite_note-6)Comparisons of national wealth are also frequently made based on [purchasing power parity](/wiki/Purchasing_power_parity) (PPP), to adjust for differences in the cost of living in different countries. Other metrics, [nominal GDP per capita](/wiki/List_of_countries_by_GDP_(nominal)_per_capita) and a corresponding [GDP (PPP) per capita](/wiki/List_of_countries_by_GDP_(PPP)_per_capita), are used for comparing national [standard of living](/wiki/Standard_of_living). On the whole, PPP per capita figures are less spread than nominal GDP per capita figures.[[5]](#cite_note-7)\n",
"\n",
"The rankings of national economies have changed significantly over time. For instance, the United States overtook the British Empire around 1916; Japan rose rapidly in the post-World War II period to become the world’s second-largest economy by \n"
]
}
],
"source": [
"if wiki_results[\"trafilatura\"]:\n",
" print(wiki_results[\"trafilatura\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "22ca520c",
"metadata": {
"papermill": {
"duration": 0.003472,
"end_time": "2025-12-30T14:58:46.337894",
"exception": false,
"start_time": "2025-12-30T14:58:46.334422",
"status": "completed"
},
"tags": []
},
"source": [
"### Parallel.ai"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "a1dfb52a",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:46.346239Z",
"iopub.status.busy": "2025-12-30T14:58:46.346017Z",
"iopub.status.idle": "2025-12-30T14:58:46.348572Z",
"shell.execute_reply": "2025-12-30T14:58:46.348127Z"
},
"papermill": {
"duration": 0.007957,
"end_time": "2025-12-30T14:58:46.349450",
"exception": false,
"start_time": "2025-12-30T14:58:46.341493",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[Jump to content]()\n",
"\n",
"Main menu\n",
"\n",
"Main menu\n",
"\n",
"move to sidebar hide\n",
"\n",
"Navigation\n",
"\n",
"* [Main page](/wiki/Main_Page \"Visit the main page [z]\")\n",
"* [Contents](/wiki/Wikipedia:Contents \"Guides to browsing Wikipedia\")\n",
"* [Current events](/wiki/Portal:Current_events \"Articles related to current events\")\n",
"* [Random article](/wiki/Special:Random \"Visit a randomly selected article [x]\")\n",
"* [About Wikipedia](/wiki/Wikipedia:About \"Learn about Wikipedia and how it works\")\n",
"* [Contact us](//en.wikipedia.org/wiki/Wikipedia:Contact_us \"How to contact Wikipedia\")\n",
"\n",
"Contribute\n",
"\n",
"* [Help](/wiki/Help:Contents \"Guidance on how to use and edit Wikipedia\")\n",
"* [Learn to edit](/wiki/Help:Introduction \"Learn how to edit Wikipedia\")\n",
"* [Community portal](/wiki/Wikipedia:Community_portal \"The hub for editors\")\n",
"* [Recent changes](/wiki/Special:RecentChanges \"A list of recent changes to Wikipedia [r]\")\n",
"* [Upload file](/wiki/Wikipedia:File_upload_wizard \"Add images or other media for use on Wikipedia\")\n",
"* [Special pages](/wiki/Special:SpecialPages)\n",
"\n",
"[](/wiki/Main_Page)\n",
"\n",
"[Search](/wiki/Special:Search \"Search Wikipedia [f]\")\n",
"\n",
"Search\n",
"\n",
"Appearance\n",
"\n",
"* [Donate](https://donate.wikimedia.org/?wmf_source=donate&wmf_medium=sidebar&wmf_campaign=en.wikipedia.org&uselang=en)\n",
"* [Create account](/w/index.php?title=Special:CreateAccount&returnto=List+of+countries+by+GDP+%28nominal%29 \"You are encouraged to create an account and log in; however, it is not mandatory\")\n",
"* [Log in](/w/index.php?title=Special:UserLogin&returnto=List+of+countries+by+GDP+%28nominal%29 \"You're encouraged to log in; however, it's not mandatory. [o]\")\n",
"\n",
"Personal tools\n",
"\n",
"* [Donate](https://donate.wikimedia.org/?wmf_source=donate&wmf_medium=sidebar&wmf_campaign=en.wikipedia.org&uselang=en)\n",
"* [Create account](/w/index.php?title=Special:CreateAccount&returnto=List+of+countries+by+GDP+%28nominal%29 \"You are encouraged to create an account and log in; however, it is not mandatory\")\n",
"* [Log in](/w/index.php?title=Special:UserLogin&returnto=List+of+countries+by+GDP+%\n"
]
}
],
"source": [
"if wiki_results[\"parallel\"]:\n",
" print(wiki_results[\"parallel\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "33b624c1",
"metadata": {
"papermill": {
"duration": 0.003188,
"end_time": "2025-12-30T14:58:46.356981",
"exception": false,
"start_time": "2025-12-30T14:58:46.353793",
"status": "completed"
},
"tags": []
},
"source": [
"### Exa"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "1a5c830e",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:46.364270Z",
"iopub.status.busy": "2025-12-30T14:58:46.364133Z",
"iopub.status.idle": "2025-12-30T14:58:46.366323Z",
"shell.execute_reply": "2025-12-30T14:58:46.365756Z"
},
"papermill": {
"duration": 0.006669,
"end_time": "2025-12-30T14:58:46.367116",
"exception": false,
"start_time": "2025-12-30T14:58:46.360447",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"List of countries by GDP (nominal) - Wikipedia\n",
"[Jump to content](#bodyContent)\n",
"[![](https://en.wikipedia.org/static/images/icons/wikipedia.png)![Wikipedia](https://en.wikipedia.org/static/images/mobile/copyright/wikipedia-wordmark-en.svg)![The Free Encyclopedia](https://en.wikipedia.org/static/images/mobile/copyright/wikipedia-tagline-en.svg)](https://en.wikipedia.org/wiki/Main_Page)\n",
"[Search](https://en.wikipedia.org/wiki/Special:Search)\n",
"Search\n",
"# List of countries by GDP (nominal)\n",
"79 languages\n",
"* [Afrikaans](https://af.wikipedia.org/wiki/Lys_van_lande_volgens_BBP_(nominaal))\n",
"* [العربية](https://ar.wikipedia.org/wiki/قائمة_الدول_حسب_الناتج_المحلي_الإجمالي)\n",
"* [Azərbaycanca](https://az.wikipedia.org/wiki/Ümumi_daxili_məhsul_üzrə_ölkələrin_reytinqi)\n",
"* [تۆرکجه](https://azb.wikipedia.org/wiki/عومومی_داخیلی_محصول_اوزره_اؤلکه‌لرین_لیستی)\n",
"* [বাংলা](https://bn.wikipedia.org/wiki/জিডিপি_(মনোনীত)_অনুযায়ী_রাষ্ট্রের_তালিকা)\n",
"* [閩南語 / Bân-lâm-gí](https://zh-min-nan.wikipedia.org/wiki/Kok-ka_bêng-gī_GDP_lia̍t-toaⁿ)\n",
"* [Башҡортса](https://ba.wikipedia.org/wiki/ЭТП_буйынса_илдәр_исемлеге_(номинал))\n",
"* [Беларуская](https://be.wikipedia.org/wiki/Спіс_краін_паводле_ВУП_(намінал))\n",
"* [भोजपुरी](https://bh.wikipedia.org/wiki/नॉमिनल_जीडीपी_के_अनुसार_बिस्व_के_देस_सभ_के_लिस्ट)\n",
"* [Български](https://bg.wikipedia.org/wiki/Списък_на_страните_по_БВП_(по_номинална_стойност))\n",
"* [Català](https://ca.wikipedia.org/wiki/Llista_de_països_per_PIB_(nominal))\n",
"* [Čeština](https://cs.wikipedia.org/wiki/Seznam_států_světa_podle_HDP)\n",
"* [Cymraeg](https://cy.wikipedia.org/wiki/Rhestr_o_wledydd_yn_nhrefn_CMC_y_pen)\n",
"* [Dansk](https://da.wikipedia.org/wiki/Verdens_landes_BNP)\n",
"* [Deutsch](https://de.wikipedia.org/wiki/Liste_der_Länder_nach_Bruttoinlandsprodukt)\n",
"* [Ελληνικά](https://el.wikipedia.org/wiki/Κατάλογος_χωρών_ανά_ΑΕΠ_(ονομαστικό))\n",
"* [Español](https://es.wikipedia.org/wiki/Anexo:Países_por_PIB_(nominal))\n",
"* [Esperanto](https://eo.wikipedia.org/wiki/Listo_de_landoj_laŭ_MEP_(ĝenerale))\n",
"* [فارسی](https://fa.wikiped\n"
]
}
],
"source": [
"if wiki_results[\"exa\"]:\n",
" print(wiki_results[\"exa\"][:MAX_CHARS])\n",
"else:\n",
" print(\"❌ No content extracted\")"
]
},
{
"cell_type": "markdown",
"id": "fcdbeae2",
"metadata": {
"papermill": {
"duration": 0.002756,
"end_time": "2025-12-30T14:58:46.372921",
"exception": false,
"start_time": "2025-12-30T14:58:46.370165",
"status": "completed"
},
"tags": []
},
"source": [
"### Wikipedia Table Test Summary\n",
"\n",
"| Tool | Result |\n",
"|------|--------|\n",
"| Trafilatura | ✅ Preserves tables in markdown format |\n",
"| Parallel.ai | ⚠️ Includes nav menus, tables less structured |\n",
"| Exa | ✅ Good content but may include page chrome |\n",
"\n",
"**Insight**: Trafilatura excels at structured content - it preserves markdown tables cleanly. API services capture more content but with less filtering."
]
},
{
"cell_type": "markdown",
"id": "3d8b63e1",
"metadata": {
"papermill": {
"duration": 0.002807,
"end_time": "2025-12-30T14:58:46.380021",
"exception": false,
"start_time": "2025-12-30T14:58:46.377214",
"status": "completed"
},
"tags": []
},
"source": [
"---\n",
"## Summary"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "c2a2627a",
"metadata": {
"execution": {
"iopub.execute_input": "2025-12-30T14:58:46.386583Z",
"iopub.status.busy": "2025-12-30T14:58:46.386454Z",
"iopub.status.idle": "2025-12-30T14:58:46.388959Z",
"shell.execute_reply": "2025-12-30T14:58:46.388592Z"
},
"papermill": {
"duration": 0.00647,
"end_time": "2025-12-30T14:58:46.389472",
"exception": false,
"start_time": "2025-12-30T14:58:46.383002",
"status": "completed"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Results Summary (character counts)\n",
"======================================================================\n",
"Test Case Trafilatura Parallel Exa\n",
"----------------------------------------------------------------------\n",
"arXiv PDF - A Comprehensive Ov 0 100,000 128\n",
"Amazon homepage - JS-rendered, 142 93,063 82,822\n",
"GitHub blog post - clean artic 7,028 29,087 9,245\n",
"Wikipedia - GDP table with com 19,117 40,313 97,362\n"
]
}
],
"source": [
"all_results = [pdf_results, amazon_results, blog_results, wiki_results]\n",
"\n",
"print(\"Results Summary (character counts)\")\n",
"print(\"=\" * 70)\n",
"print(f\"{'Test Case':<30} {'Trafilatura':>12} {'Parallel':>12} {'Exa':>12}\")\n",
"print(\"-\" * 70)\n",
"\n",
"for r in all_results:\n",
" traf = len(r[\"trafilatura\"] or \"\")\n",
" para = len(r[\"parallel\"] or \"\")\n",
" exa = len(r[\"exa\"] or \"\")\n",
" print(f\"{r['description'][:30]:<30} {traf:>12,} {para:>12,} {exa:>12,}\")"
]
},
{
"cell_type": "markdown",
"id": "17802a41",
"metadata": {
"lines_to_next_cell": 2,
"papermill": {
"duration": 0.002844,
"end_time": "2025-12-30T14:58:46.395493",
"exception": false,
"start_time": "2025-12-30T14:58:46.392649",
"status": "completed"
},
"tags": []
},
"source": [
"## Conclusions\n",
"\n",
"### Recommendation\n",
"\n",
"**For general web extraction**: Use **Trafilatura** as the primary tool (free, fast, good quality).\n",
"\n",
"**For PDFs and difficult pages**: Fall back to **Parallel.ai** (handles PDFs, JS-heavy sites).\n",
"\n",
"**For indexed content at scale**: Consider **Exa** for batch operations on popular web pages.\n",
"\n",
"### Tool Characteristics\n",
"\n",
"| Aspect | Trafilatura | Parallel.ai | Exa |\n",
"|--------|-------------|-------------|-----|\n",
"| **Cost** | Free | $0.001/request | Varies |\n",
"| **PDF Support** | ❌ | ✅ | ❌ |\n",
"| **JS Rendering** | Limited | ✅ | Via index |\n",
"| **Table Preservation** | Good | Good | Varies |\n",
"| **Speed** | Fast | Medium | Fast |\n",
"| **Reliability** | High | High | Index-dependent |"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
},
"papermill": {
"default_parameters": {},
"duration": 7.462986,
"end_time": "2025-12-30T14:58:46.716806",
"environment_variables": {},
"exception": null,
"input_path": "report.ipynb",
"output_path": "report_output.ipynb",
"parameters": {},
"start_time": "2025-12-30T14:58:39.253820",
"version": "2.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment