Access internal web app APIs using Puppeteer to capture headers, then replicate in Python

headers.md

Access Internal APIs with Puppeteer + Python

Many internal web apps expose APIs that aren't publicly documented. This technique uses Puppeteer to intercept the browser's XHR requests, capture the exact headers being sent, then replicate them in Python to access the API directly — bypassing the UI entirely.

Steps

1. Intercept headers with Puppeteer

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

// Capture request headers on XHR calls
page.on('request', request => {
  if (request.resourceType() === 'xhr' || request.resourceType() === 'fetch') {
    console.log('URL:', request.url());
    console.log('Headers:', JSON.stringify(request.headers(), null, 2));
  }
});

await page.goto('https://internal-app.example.com');
// Perform the action in the UI that triggers the API call

2. Replicate in Python

The headers Puppeteer captures are the ones you need. Common required headers for internal apps:

import requests

headers = {
    'Connection': '...',
    'Origin': 'https://internal-app.example.com',
    'User-Agent': '...',          # Must match browser UA
    'Content-type': 'application/json',
    'Accept': 'application/json',
    'X-Prototype-Version': '...',
    'X-Requested-With': 'XMLHttpRequest',
    'DNT': '1',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Referer': 'https://internal-app.example.com/page',
}

response = requests.post('https://internal-app.example.com/api/endpoint',
                         headers=headers,
                         json={'key': 'value'})

Key insight

Internal apps often validate X-Requested-With, Origin, and Referer headers as a CSRF guard — without them, requests return 403. Puppeteer reveals exactly which headers are required.

headers.py

['Connection',
'Origin',
'User-Agent',
'Content-type',
'Accept',
'X-Prototype-Version',
'X-Requested-With',
'DNT',
'Sec-Fetch-Site',
'Sec-Fetch-Mode',
'Referer']