Getting Started with Browserbeam's Python SDK

By the end of this guide, you'll have a working Python script that opens a browser session, navigates to a page, fills a form, clicks a button, and extracts structured data into JSON. All with five method calls and zero browser dependencies.

We'll start with pip install, configure the SDK, and build up from a basic session to a complete python web scraping workflow. Every example runs as-is. Copy, paste, run.

In this guide, we'll build:

• A basic browser session that navigates and observes pages
• A search form filler that types, clicks, and reads results
• An extraction pipeline that pulls structured JSON from any page
• A price monitoring script, lead list builder, and content change detector
• A migration path from Selenium, Playwright, or Scrapy to Browserbeam

TL;DR: The Browserbeam Python SDK gives you a clean interface for python browser automation without managing Chromium, Playwright, or Selenium. Install with pip install browserbeam, create a session with a URL, and use methods like click, fill, and extract to interact with pages and pull structured data. Supports both sync and async.

Installing and Configuring the Python SDK

The SDK requires Python 3.8+ and has minimal dependencies.

pip install browserbeam

That's it. No browser binary to download. No Chromium build to compile. Unlike traditional python headless browser libraries that bundle Chromium or require a WebDriver, Browserbeam runs the browser in the cloud, so your local environment stays clean.

Configuration

The SDK reads your API key from the BROWSERBEAM_API_KEY environment variable by default. Set it once and every script picks it up:

export BROWSERBEAM_API_KEY="your_api_key_here"

Then create a client:

from browserbeam import Browserbeam

client = Browserbeam()

You can also pass the key explicitly and configure request timeouts:

client = Browserbeam(
    api_key="your_api_key_here",
    timeout=120.0,
)

Creating Your First Browser Session

A session is an isolated browser tab in the cloud. It has its own cookies, local storage, and viewport. Let's create one and see what comes back.

from browserbeam import Browserbeam

client = Browserbeam()
session = client.sessions.create(url="https://books.toscrape.com")

print(f"Title: {session.page.title}")
print(f"Stable: {session.page.stable}")
print(f"Content preview: {session.page.markdown.content[:200]}")

That single call does three things: creates a browser context, navigates to the URL, and observes the page. The response includes:

• session.page.title -- the page title
• session.page.stable -- True when the page is fully loaded (network idle + DOM quiet)
• session.page.markdown.content -- the page content as clean markdown
• session.page.interactive_elements -- every clickable, fillable element with a ref
• session.page.forms -- detected forms with field associations
• session.page.map -- a lightweight outline of page sections (nav, main, footer)

Let's print the interactive elements to see what we can work with:

for el in session.page.interactive_elements:
    print(f"  {el['ref']}: [{el['tag']}] {el.get('label', 'no label')}")

# Output:
#   e1: [input] Search
#   e2: [button] Submit
#   e3: [a] More information

Each element has a ref like e1 or e2. These are stable identifiers you use to target elements in the next step. No CSS selectors. No XPath. Just e1.

Async Support

If your application uses asyncio, the SDK has a drop-in async client:

from browserbeam import AsyncBrowserbeam

client = AsyncBrowserbeam()
session = await client.sessions.create(url="https://books.toscrape.com")
print(session.page.title)
await session.close()

Every method on the sync client has an async equivalent. The API is identical, just add await.

Navigating, Interacting, and Extracting Data

Now that we have a session, let's do something with it. We'll fill a search form, click the submit button, and extract the results.

Filling and Clicking by Ref

Your agent (or your script) reads the refs from the session response and uses them to interact:

session.fill(ref="e1", value="python web scraping")
session.click(ref="e2")

print(f"New URL: {session.page.url}")
print(f"Content changed: {session.page.changes['content_changed']}")

After each action, the SDK auto-observes the page. The session.page object updates to reflect the current state. The changes object tells you what's different since the last observation: new content, added elements, removed elements.

You can also target elements by visible text or label instead of ref:

session.click(text="Submit")
session.fill(label="Email", value="user@example.com")

Navigating to a New URL

Use goto to navigate to a different page within the same session. Cookies and state carry over:

session.goto(url="https://books.toscrape.com/catalogue/page-2.html")
print(session.page.title)

For SPAs or pages that render with JavaScript, use wait_for or wait_until to wait for specific conditions:

session.goto(
    url="https://books.toscrape.com/catalogue/category/books/mystery_3/index.html",
    wait_until="document.querySelectorAll('article.product_pod').length > 0",
    wait_timeout=10000
)

Extracting Structured Data

This is where the SDK shines for python web scraping workflows. Define a schema and Browserbeam extracts structured JSON:

result = session.extract(
    page_title="h1 >> text",
    description="meta[name=description] >> content",
    links=["a >> href"]
)

print(result.extraction)
# {'page_title': 'Example Domain', 'description': '...', 'links': ['https://...']}

The >> text and >> href suffixes tell Browserbeam what to extract from each matched element. No regex. No BeautifulSoup. Just a declarative schema.

For repeating structures (product listings, search results, table rows), use the _parent pattern:

result = session.extract(
    products=[{
        "_parent": ".product-card",
        "name": "h3 >> text",
        "price": ".price >> text",
        "url": "a >> href",
        "in_stock": ".availability >> text"
    }]
)

for product in result.extraction["products"]:
    print(f"{product['name']}: {product['price']}")

Each _parent match becomes an object in the array. This replaces dozens of lines of BeautifulSoup or lxml code with a single method call.

Filling an Entire Form

For forms with multiple fields, fill_form handles everything in one call. It auto-detects field types (text inputs, selects, checkboxes) and uses the right interaction for each:

session.fill_form(
    fields={
        "Email": "user@example.com",
        "Password": "secure_password",
        "Country": "US",
        "Remember me": True
    },
    submit=True
)

Setting submit=True finds and clicks the submit button automatically.

Quick Reference: Core Methods

Next Steps: Advanced SDK Features

Once you're comfortable with the basics, these features handle the trickier parts of python browser automation.

Scraping Pages with Infinite Scroll

Pages that load content as you scroll (social feeds, product catalogs, search results) need the scroll_collect method. It scrolls through the entire page, waits for lazy-loaded content at each position, and returns everything in one observation:

session.goto(url="https://quotes.toscrape.com/scroll")
result = session.scroll_collect(max_scrolls=30, max_text_length=50000)

print(f"Collected {len(result.page.markdown.content)} characters")
print(result.page.markdown.content[:1000])

One method call replaces the scroll-wait-check loop you'd build with Selenium or Playwright.

Screenshots and PDFs

Capture visual state for debugging or reporting:

result = session.screenshot(full_page=True, format="png")
screenshot_data = result.media[0]["data"]  # base64-encoded PNG

pdf_result = session.pdf(format="A4", print_background=True)
pdf_data = pdf_result.media[0]["data"]  # base64-encoded PDF

Using Proxies

Route sessions through a proxy for geo-targeting or anonymity. Set it at session creation:

session = client.sessions.create(
    url="https://books.toscrape.com",
    proxy="http://user:pass@proxy.example.com:8080",
    locale="en-GB",
    timezone="Europe/London"
)

Combine with locale and timezone for a complete geo-targeted browsing profile.

Page Map and Full Mode

The first observe call includes a page.map, a structural outline of the page's sections:

for entry in session.page.map:
    print(f"{entry['section']}: {entry['hint']}")

# nav: Home · Docs · Pricing (3 links)
# main: Getting started with Browserbeam... (12 elements)
# aside: Related posts · Popular tags (5 links)

When you need content from all sections (sidebars, footers, nav), use full mode:

full = session.observe(mode="full", max_text_length=20_000)
print(full.page.markdown.content)
# ## [nav]
# Home · Docs · Pricing
# ## [main]
# ...article content...
# ## [aside]
# Related posts · ...

Error Handling

The SDK raises typed exceptions for common errors:

from browserbeam import Browserbeam, RateLimitError, SessionNotFoundError

try:
    session = client.sessions.create(url="https://books.toscrape.com")
    session.click(ref="e99")
except RateLimitError as e:
    print(f"Rate limited. Retry after {e.retry_after}s")
except SessionNotFoundError:
    print("Session expired. Create a new one.")

When a step fails (like clicking a ref that doesn't exist), the SDK still updates session.page with the current page state. Your code can inspect the page and decide what to do next.

Session Lifecycle

Always close sessions when done. Open sessions consume resources and keep the billing clock running:

session = client.sessions.create(url="https://books.toscrape.com")
try:
    result = session.extract(title="h1 >> text")
    print(result.extraction)
finally:
    session.close()

You can also list and manage sessions programmatically:

active = client.sessions.list(status="active")
for s in active:
    print(f"{s['session_id']}: {s['current_url']}")

client.sessions.destroy("ses_abc123")

Common Patterns and Recipes

Now that you know the core methods, let's put them to work. These three recipes cover the most common python automation scripts people build with Browserbeam.

Price Monitoring Script

Track prices across multiple product pages and flag changes. This script visits a list of URLs, extracts pricing data, and compares it against stored values.

from browserbeam import Browserbeam
import json
from pathlib import Path

client = Browserbeam()

products = [
    {"name": "A Light in the Attic", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"},
    {"name": "Tipping the Velvet", "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"},
]

baseline_file = Path("prices.json")
baseline = json.loads(baseline_file.read_text()) if baseline_file.exists() else {}

for product in products:
    session = client.sessions.create(url=product["url"])
    result = session.extract(price=".price_color >> text", in_stock=".instock.availability >> text")
    session.close()

    current_price = result.extraction["price"]
    previous_price = baseline.get(product["name"])

    if previous_price and current_price != previous_price:
        print(f"PRICE CHANGE: {product['name']} {previous_price} -> {current_price}")

    baseline[product["name"]] = current_price

baseline_file.write_text(json.dumps(baseline, indent=2))

Run this daily with cron or a task scheduler. Add Slack or email notifications for the price change alert, and you have a complete competitive monitoring system. For more agent architecture patterns like this, see our guide on building intelligent web agents.

Lead List Builder

Visit company "About" pages, extract team and contact information, and build a structured lead list.

sites = [
    "https://books.toscrape.com",
    "https://quotes.toscrape.com",
    "https://realpython.github.io/fake-jobs/",
]

site_info = []
for url in sites:
    session = client.sessions.create(url=url)
    result = session.extract(
        site_name="h1 >> text",
        description="meta[name=description] >> content",
        nav_links=[{
            "_parent": "nav a, .sidebar a",
            "_limit": 5,
            "text": "a >> text",
            "url": "a >> href",
        }]
    )
    session.close()
    site_info.append({"url": url, **result.extraction})

for site in site_info:
    print(f"{site['site_name']}: {len(site.get('nav_links', []))} nav links")

The comma-separated selectors (nav a, .sidebar a) handle different HTML structures across sites. Browserbeam tries each selector and returns the first match.

Content Change Detection

Monitor a page for content changes and get notified when something updates. This pattern is useful for tracking documentation updates, regulatory changes, or news coverage.

import hashlib

def check_for_changes(client, url, content_selector, previous_hash=None):
    session = client.sessions.create(url=url)
    result = session.extract(content=f"{content_selector} >> text")
    session.close()

    content = result.extraction.get("content", "")
    current_hash = hashlib.sha256(content.encode()).hexdigest()

    if previous_hash and current_hash != previous_hash:
        return {"changed": True, "content": content, "hash": current_hash}
    return {"changed": False, "content": content, "hash": current_hash}

# Usage
state = check_for_changes(client, "https://news.ycombinator.com", ".athing")
if state["changed"]:
    print("Documentation updated!")
    print(state["content"][:500])

Store the hash between runs (in a file, database, or Redis) and compare on each check.

Debugging and Troubleshooting

Things will break. Here's how to figure out what went wrong and fix it quickly.

Reading Error Responses

When a step fails, Browserbeam returns the error alongside the current page state. The error includes the step index, action type, error code, and a human-readable message:

result = session.act(steps=[
    {"click": {"ref": "e99"}}
])

if result.error:
    print(f"Step {result.error['step']} failed: {result.error['code']}")
    print(f"Message: {result.error['message']}")
    # Step 0 failed: element_not_found
    # Message: No visible element found matching ref "e99"

    # The page state is still available
    print(f"Current URL: {result.page.url}")
    print(f"Available elements: {len(result.page.interactive_elements)}")

Common error codes: element_not_found, navigation_timeout, captcha_detected, rate_limited, session_expired. Each one tells you exactly what happened and whether retrying makes sense.

Using Screenshots for Debugging

When extraction returns empty results or your agent clicks the wrong element, take a screenshot to see what the browser actually sees:

session = client.sessions.create(url="https://books.toscrape.com")

# Something isn't working? Take a screenshot
result = session.screenshot(full_page=True)

import base64
with open("debug.png", "wb") as f:
    f.write(base64.b64decode(result.media[0]["data"]))

print("Screenshot saved to debug.png")

Screenshots show you cookie banners that weren't dismissed, modals blocking content, or pages that loaded differently than expected. They're faster than guessing.

Handling Timeouts and Retries

Network issues, slow pages, and rate limits all cause transient failures. Build retry logic into your scripts from the start:

import time
from browserbeam import RateLimitError

def scrape_with_retry(client, url, schema, max_retries=3):
    for attempt in range(max_retries):
        try:
            session = client.sessions.create(url=url)
            result = session.extract(**schema)
            session.close()
            return result.extraction
        except RateLimitError as e:
            wait = e.retry_after or (2 ** attempt)
            print(f"Rate limited. Waiting {wait}s...")
            time.sleep(wait)
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt)
    return None

The RateLimitError includes a retry_after value from the API's X-RateLimit-Reset header. Use it instead of guessing.

Checking Session Status

If you're debugging session lifecycle issues (sessions expiring mid-workflow, hitting concurrency limits), list your active sessions:

active = client.sessions.list(status="active")
print(f"Active sessions: {len(active)}")

for s in active:
    print(f"  {s['session_id']}: {s['current_url']} (created {s['created_at']})")

If you see sessions that should have been closed, your cleanup code has a bug. Check your try/finally blocks. For more on secure session management, see the security best practices guide.

Migrating from Selenium, Playwright, or Scrapy

If you're already using a python web scraping library, switching to Browserbeam doesn't require rewriting everything. The patterns map directly. Here's how each migration looks.

Selenium to Browserbeam in 5 Minutes

The biggest change: no WebDriver, no browser binary, no ChromeOptions. Everything runs in the cloud.

The ref-based targeting (e1, e2) replaces CSS selectors. You get the available refs from session.page.interactive_elements after each observation. No more By.ID, By.CSS_SELECTOR, or By.XPATH.

Playwright to Browserbeam

Playwright's API is closer to Browserbeam's, so the migration is smoother. The main differences are infrastructure (cloud vs local) and output format (structured markdown vs raw HTML).

If you're using Playwright for testing, keep it. If you're using it for scraping or agent workflows, Browserbeam handles the infrastructure so you can focus on the data.

Scrapy to Browserbeam

Scrapy and Browserbeam solve different problems. Scrapy is a crawling framework for static HTML at scale. Browserbeam is a browser API for dynamic pages. The migration makes sense when your Scrapy spiders break on JavaScript-rendered content.

The biggest difference: Scrapy manages crawl queues, rate limiting, and parallel requests. With Browserbeam, you handle orchestration yourself (or use asyncio.gather for parallel sessions). For simple scraping jobs, that's less code. For large-scale crawls across thousands of pages, you may want to keep Scrapy's crawler and use Browserbeam as a download middleware for JavaScript-heavy pages.

Performance Tips

A few patterns make a measurable difference in speed and cost when building python automation scripts.

Async Batching for High-Throughput Scraping

When scraping multiple URLs, use asyncio.gather to run sessions in parallel. Each session is independent, so there's no shared state to worry about.

import asyncio
from browserbeam import AsyncBrowserbeam

client = AsyncBrowserbeam()

async def scrape_page(url, schema):
    session = await client.sessions.create(url=url)
    try:
        result = await session.extract(**schema)
        return {"url": url, "data": result.extraction}
    finally:
        await session.close()

async def batch_scrape(urls, schema, concurrency=5):
    semaphore = asyncio.Semaphore(concurrency)

    async def limited_scrape(url):
        async with semaphore:
            return await scrape_page(url, schema)

    return await asyncio.gather(*[limited_scrape(u) for u in urls])

urls = ["https://books.toscrape.com/catalogue/page-1.html", "https://books.toscrape.com/catalogue/page-2.html"]
results = asyncio.run(batch_scrape(urls, {"title": "h1 >> text"}))

The semaphore limits concurrency to avoid hitting your plan's session limit. Start with 5 concurrent sessions and increase based on your plan.

Minimizing Token Usage with Extraction Schemas

If you're feeding Browserbeam output to an LLM, the page markdown can still be larger than necessary. Use extract with a targeted schema to pull only the fields you need, instead of sending the full markdown to the LLM.

# Expensive: sending full page markdown to the LLM
page_content = session.page.markdown.content  # Could be 3,000+ tokens
llm_response = ask_llm("What is the price?", page_content)

# Cheaper: extract exactly what you need
result = session.extract(price=".price_color >> text", title="h3 a >> text")
# result.extraction is ~50 tokens

For AI agent workflows, extract first, then send the structured data to the LLM. This cuts token costs by 90% or more compared to sending raw page content.

Reusing Sessions for Multi-Page Flows

Creating a new session for every page is wasteful when you're navigating within the same site. Reuse the session with goto:

session = client.sessions.create(url="https://books.toscrape.com")

# Page 1: category listing from sidebar
categories = session.extract(
    cats=[{"_parent": ".side_categories ul li a", "_limit": 5, "name": "a >> text", "url": "a >> href"}]
)

# Page 2-N: visit each category
for cat in categories.extraction["cats"][:5]:
    session.goto(url=cat["url"])
    books = session.extract(
        items=[{"_parent": "article.product_pod", "title": "h3 a >> text", "price": ".price_color >> text"}]
    )
    print(f"{cat['name']}: {len(books.extraction['items'])} books")

session.close()

One session, multiple pages, shared cookies and state. This is faster than creating separate sessions and preserves login state if you're scraping authenticated content.

Frequently Asked Questions

How do I scrape a website with Python using Browserbeam?

Install the SDK with pip install browserbeam, create a session with your target URL, and use the extract method to pull structured data. Browserbeam handles JavaScript rendering, cookie banners, and page stability automatically. For a complete walkthrough, see the navigating and extracting section above.

Is Browserbeam a Python Selenium alternative?

Yes, for most browser automation and scraping use cases. Selenium requires you to install a browser driver, manage browser binaries, and write low-level interaction code. Browserbeam runs the browser in the cloud and gives you high-level methods like click(ref="e1") and extract(title="h1 >> text"). You don't manage infrastructure, and the responses are structured for programmatic use. For raw browser testing with full WebDriver protocol access, Selenium is still the right tool.

Does Browserbeam handle JavaScript-rendered pages?

Yes. Browserbeam runs real Chromium, so JavaScript executes exactly like it would in a regular browser. The SDK waits for stability (network idle + DOM quiet) before returning page state. For SPAs with custom loading logic, use wait_until on the goto call to define your own readiness condition.

What's the difference between fill and type?

fill clears the input and sets the value instantly. type enters text character by character with a configurable delay between keystrokes. Use type for autocomplete fields, search-as-you-type inputs, or any field that triggers events on individual key presses.

Can I use async/await with the SDK?

Yes. Import AsyncBrowserbeam instead of Browserbeam. Every method has an async equivalent. The API is identical, you just add await before each call. This works with asyncio, aiohttp, FastAPI, and any other async Python framework.

How does Browserbeam compare to BeautifulSoup or Scrapy for python web scraping?

BeautifulSoup and Scrapy are python web scraping libraries built for static HTML. They can't render JavaScript, interact with forms, or handle dynamic content. Browserbeam gives you a full browser that renders JavaScript, clicks buttons, fills forms, and waits for content to load. The extract method returns structured JSON directly, so you skip the HTML parsing step entirely. Think of Browserbeam as handling the browser layer while you focus on the data.

How does Python Requests compare to Selenium and Browserbeam?

Python Requests is an HTTP client. It sends HTTP requests and receives raw HTML responses. It can't render JavaScript, click buttons, or interact with forms. Selenium is a browser automation tool that controls a real browser but requires local browser management and driver downloads. Browserbeam combines the simplicity of an API call (like Requests) with the full browser capability (like Selenium), without requiring local browser management. Use Requests for static API calls, Selenium for browser testing, and Browserbeam for scraping and automation that needs JavaScript rendering.

I'm new to Python web scraping. Where should I start?

Start with this guide. Install the SDK (pip install browserbeam), set your API key, and run the first example in the "Creating Your First Browser Session" section. Once that works, try the extraction example on a real site you want to scrape. The SDK handles the hard parts (JavaScript rendering, page stability, cookie banners) so you can focus on defining what data you want. No HTML parsing knowledge required. For beginners, the Browserbeam API docs cover every parameter and option in detail.

Try It Yourself

You've got the SDK installed, a session created, and a working extraction pipeline. That covers most python web scraping workflows. Here's what to try next.

Point the SDK at a site you actually need to scrape. Change the extraction schema to match the data you want. Try scroll_collect on a page with infinite scroll. Use fill_form to log into a site and scrape authenticated content.

If you hit a wall, the API docs cover every parameter and edge case. The building AI browser agents guide shows how to wire the SDK into an LLM-powered agent loop. And the security best practices guide covers credential management and session isolation for production deployments.

pip install browserbeam

Start building.