name: web-scraper description: > Build reliable, production-grade web scrapers through a two-phase approach: Discovery (finding the best data source on a site) then Scraping (generating a Python script for scheduled extraction). Use this skill whenever the user wants to scrape a website, extract data from web pages, build a crawler, set up recurring data collection, reverse-engineer a site's API, or find hidden data endpoints. Also trigger when the user mentions "scrape", "crawl", "extract from site", "web data", "pull data from URL", "scheduled scraping",
This guide documents how to use Factory's Droid CLI with your Claude Code Max subscription (OAuth authentication) instead of pay-per-token API keys. The solution leverages CLIProxyAPI as a transparent authentication proxy that converts API key requests from Factory CLI into OAuth-authenticated requests for Anthropic's API.
Factory CLI → [Anthropic Format + API Key] → CLIProxyAPI → [Anthropic Format + OAuth] → Anthropic API
↓
(Auth Header Swap)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| add_user() { | |
| getent passwd "$1" >>"$BUILDROOT/etc/passwd" | |
| getent shadow "$1" >>"$BUILDROOT/etc/shadow" | |
| getent group "$(id -Gn "$1")" >>"$BUILDROOT/etc/group" | |
| } | |
| build() { | |
| add_systemd_unit cryptsetup-pre.target |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # If Debian 11 is ran on a LXC container (Proxmox), SSH login and sudo actions can be slow | |
| # Check if in /var/log/auth.log the following messages | |
| Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms) | |
| -> Run systemctl mask systemd-logind | |
| -> Run pam-auth-update (and deselect Register user sessions in the systemd control group hierarchy) |
All packages, except for Tini have been added to termux-root. To install them, simply pkg install root-repo && pkg install docker. This will install the whole docker suite, left only Tini to be compiled manually.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Install Ubiquiti Unifi Controller on Ubuntu 20.04. | |
| # As tested on a fresh install of ubuntu-20.04.1-live-server, August 22nd 2020. | |
| # Thanks to https://gist.github.com/tmuncks for posting the updated install steps. | |
| sudo apt update | |
| sudo apt install --yes apt-transport-https | |
| echo 'deb https://www.ui.com/downloads/unifi/debian stable ubiquiti' | sudo tee /etc/apt/sources.list.d/100-ubnt-unifi.list | |
| sudo wget -O /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ui.com/unifi/unifi-repo.gpg |
This exemplary initial setup uses two devices /dev/sdb and /dev/sdc but can be applied to any amount of devices by following the steps with additional devices.
Create keyfile:
dd bs=64 count=1 if=/dev/urandom of=/etc/cryptkey iflag=fullblock
chmod 600 /etc/cryptkey
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # 1. put frpc and frpc.ini under /usr/local/frpc/ | |
| # 2. put this file (frpc.service) at /etc/systemd/system | |
| # 3. run `sudo systemctl daemon-reload && sudo systemctl enable frpc && sudo systemctl start frpc` | |
| # Then we can manage frpc with `sudo service frpc {start|stop|restart|status}` | |
| # See also: https://nosame.net/use-frp-to-reverse-proxy-your-nas/ | |
| # Alternative for server: | |
| # - Offical: https://github.com/fatedier/frp/blob/a4cfab6/conf/systemd/frpc%40.service | |
| [Unit] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # Reference guides: | |
| # https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF | |
| # https://pve.proxmox.com/wiki/Pci_passthrough | |
| # https://pve.proxmox.com/wiki/Nested_Virtualization | |
| # Remember to turn on SVM in BIOS and disable CSM | |
| # Update packages |
NewerOlder