App Store template
Crawl4AI icon

Crawl4AI

Open-source LLM-friendly web crawler and scraper with an API server, dashboard, and playground.

Launch in your Sealos workspace.

Crawl4AI template preview
Template previewFull screenshot
Template deployments
15 on Sealos
Deployment
Your own instance
Category
AI

About this template

Overview

Crawl4AI is an open-source LLM-friendly web crawler and scraper. This template deploys the official Crawl4AI API server with its dashboard and playground on Sealos Cloud.

From the documentation

Crawl4AI runs as a single API/Web service on port 11235. The deployment exposes the dashboard, playground, crawl APIs, and browser-backed scraping features through a public HTTPS URL protected by CRAWL4AI_API_TOKEN.

How to deploy Crawl4AI

  1. Choose Deploy now to start Crawl4AI in your Sealos workspace.
  2. Review api_token in the deployment form.
  3. Launch the template, then inspect the application status and resource cards in Canvas.

Resources to plan for

Start with the resources defined by the Crawl4AI template. Review CPU, memory, persistent storage, and network allocations for every service in Canvas. Capacity needs depend on your data and workload; monitor usage as they grow.

Template configuration and setup

Access after deployment

Use the application URL or connection details shown in Canvas. Follow the deployment guide for first-time account setup or client configuration, and keep generated credentials available for that step.

Hosting and billing

Sealos monthly plans include compute, memory, storage, and traffic. Size your plan for all deployed services. Software licenses and external AI or API services may have separate terms and charges. Confirm the applicable plan and optional charges in Cost Center.

Compare Sealos resource plans
Read full documentationCollapse documentation

For current cloud charges, refer to Sealos plan pricing. Upstream documentation may reference earlier billing models.

Deploy and Host Crawl4AI on Sealos

Crawl4AI is an open-source LLM-friendly web crawler and scraper. This template deploys the official Crawl4AI API server with its dashboard and playground on Sealos Cloud.

Crawl4AI Screenshot

About Hosting Crawl4AI

Crawl4AI runs as a single API/Web service on port 11235. The deployment exposes the dashboard, playground, crawl APIs, and browser-backed scraping features through a public HTTPS URL protected by CRAWL4AI_API_TOKEN.

The template provisions a 1 GiB persistent Playwright browser cache, 1 GiB of persistent browser shared memory at /dev/shm, and persistent screenshot and PDF artifact storage at /var/lib/crawl4ai/outputs. An init container installs the image-compatible Chromium revision into the cache, while Sealos handles TLS, ingress routing, restart management, and resource controls.

Common Use Cases

  • LLM-ready extraction: Crawl pages and produce markdown or structured content for AI workflows.
  • Scraping API: Run synchronous crawls through /crawl, or submit asynchronous jobs through /crawl/job and retrieve results through /crawl/job/{task_id}.
  • Interactive testing: Use /playground to test requests and generate example code.
  • Self-hosted crawling backend: Run Crawl4AI as a shared service for internal automation.

Dependencies for Crawl4AI Hosting

The Sealos template includes the Crawl4AI Docker server image, persistent output storage, browser shared memory, public HTTPS ingress, and the App shortcut.

Deployment Dependencies
Implementation Details

Architecture Components:

  • Crawl4AI StatefulSet: Runs unclecode/crawl4ai:0.9.1 on port 11235 with token authentication enabled.
  • Persistent Playwright cache: Installs and retains the exact Chromium revision required by the image.
  • Persistent /dev/shm volume: Provides 1 GiB of browser shared-memory space through a Sealos-compatible persistent volume.
  • Persistent output volume: Stores generated crawl artifacts at /var/lib/crawl4ai/outputs across restarts.
  • Ingress and App: Publish /playground/, /dashboard/, and API endpoints through HTTPS.

Configuration:

  • The App shortcut opens /playground/; requests to / redirect there.
  • API base URL is the root deployment URL.
  • Playground, Dashboard, and protected API endpoints use the deployment value of CRAWL4AI_API_TOKEN.
  • The public Ingress disables access logs so Dashboard WebSocket token query parameters stay out of ingress request logs.
  • A generated SECRET_KEY keeps issued session credentials valid across Pod restarts.
  • Health checks use the public /health endpoint.

License Information:

Crawl4AI is licensed under Apache-2.0. This Sealos template is provided under the repository license.

Why Deploy Crawl4AI on Sealos?

Sealos is an AI-assisted Cloud Operating System built on Kubernetes that unifies the entire application lifecycle, from development in cloud IDEs to production deployment and management. By deploying Crawl4AI on Sealos, you get:

  • One-Click Deployment: Deploy the Crawl4AI API server quickly.
  • Public HTTPS API: Use the crawl API from browsers, scripts, and AI tools.
  • Protected Public Access: Require the deployment API token for Playground, Dashboard, and crawl requests.
  • Persistent Crawl Outputs: Keep generated crawl artifacts across restarts.
  • Easy Resource Tuning: Increase CPU and memory from the Canvas for heavier crawl jobs.
  • Integrated Management: Restart, inspect logs, and adjust settings from Sealos.

Deployment Guide

  1. Open the Crawl4AI template and click Deploy Now.
  2. Keep the generated API token or replace it with your own strong value, then deploy. Save this value for the Playground, Dashboard, and API clients.
  3. Wait 2-3 minutes for deployment to complete. After deployment, you will be redirected to the Canvas, where the AI dialog and resource cards support later changes.
  4. Access the application via:
    • Playground: Open https://<your-app-url>/playground/ and enter the deployment API token in the API token field when prompted.
    • Dashboard: Open https://<your-app-url>/dashboard/ and enter the same API token in the API token field when prompted.
    • API: Send requests to https://<your-app-url>/crawl with Authorization: Bearer <CRAWL4AI_API_TOKEN>.
    • Health: Check the public endpoint at https://<your-app-url>/health.

Configuration

After deployment, configure Crawl4AI through:

  • Playground and Dashboard: Enter the exact CRAWL4AI_API_TOKEN value selected during deployment in each interface's API token field.
  • API clients: Send synchronous requests to /crawl, or create asynchronous jobs at /crawl/job and poll /crawl/job/{task_id} with the bearer token.
  • Persistent outputs: Retrieve generated artifacts from /var/lib/crawl4ai/outputs inside the StatefulSet.
  • Resource Cards: Increase CPU, memory, or storage for larger crawls.

Troubleshooting

Browser-backed crawls fail under load
  • Cause: The deployment needs more CPU, memory, or browser shared-memory space for the requested workload.
  • Solution: Increase StatefulSet CPU and memory from the Canvas, then retry with a smaller crawl batch.
Playground, Dashboard, or API returns an authorization error
  • Cause: The submitted token differs from the deployment value of CRAWL4AI_API_TOKEN.
  • Solution: Enter the deployment API token in the Playground or Dashboard API token field, or send it as a bearer token from API clients.
API request times out
  • Cause: The target site is slow or the crawl job is large.
  • Solution: Submit an asynchronous job through /crawl/job, poll /crawl/job/{task_id}, and tune the proxy read timeout for long-running internal workloads.

Additional Resources

License

This Sealos template is provided under the repository license. Crawl4AI itself is licensed under Apache-2.0.

From launch to everyday operations

Why deploy
on Sealos

A shorter path from an app you want to an app you can run. Sealos brings deployment and ongoing operations into one place.

  1. One-click deployment

    Start with a ready-made template. Review its configuration and launch from the Sealos console.

  2. Managed Kubernetes

    Run on managed infrastructure with built-in workload scheduling and recovery.

  3. Automatic HTTPS

    Give your application a public HTTPS endpoint with certificates managed for you.

  4. Persistent storage

    Keep application data on persistent volumes across container restarts.

  5. Room to grow

    Adjust CPU, memory, and replicas from the console as your workload changes.

One template. Connected resources.

You Get the Whole Stack

Sealos provisions the resources defined by your template and brings them together in your workspace.

  • App Service

    Container workloads with configurable CPU, memory, and replicas.

  • Public HTTPS URL

    An address for your app, with managed TLS certificates.

  • Database

    A database provisioned alongside your app when the template calls for one.

  • Persistent Volume

    Storage for the files and data your application needs to keep.

  • Environment Variables

    Application settings and secrets configured in one place.

  • Logs & Metrics

    Inspect container logs and resource usage from the console.

Resources and their configuration vary by template. Review the deployment form for this app’s exact setup.

Explore more production-ready templates.

More