Bulk Data API Reference
Parquet-first edge gateway for bulk financial data. All endpoints stream ZSTD-compressed Parquet files — load directly into DuckDB, Pandas, or Polars.
https://data.valuein.bizAuthentication
Authenticated endpoints require a Bearer token in the Authorization header. Tokens are provisioned automatically when you subscribe via Stripe. The /v1/sample/* endpoints are always public — no token required.
Unauthenticated (sample tier)
curl https://data.valuein.biz/v1/sample/entity \ --output entity.parquetAuthenticated (sp500 / pro / full tier)
curl https://data.valuein.biz/v1/sp500/fact \ -H "Authorization: Bearer $VALUEIN_TOKEN" \ --output fact.parquetSet the token once: export VALUEIN_TOKEN=your_token. The Python SDK reads VALUEIN_API_KEY instead.
# One company's rows only (10-digit CIK, .parquet suffix on partitions)curl https://data.valuein.biz/v1/pro/fact/0000320193.parquet \ -H "Authorization: Bearer $VALUEIN_TOKEN" \ --output aapl_fact.parquetA request to /v1/sp500, /v1/pro or /v1/full with no Authorization header is answered with a 302 to the matching /v1/sample/{table}. Whole tables use the bare name (no .parquet); a per-CIK partition ends in /{cik}.parquet.
Check your token plan
curl https://data.valuein.biz/v1/me \ -H "Authorization: Bearer $VALUEIN_TOKEN"{ "plan": "sp500", "status": "active", "email": "user@example.com", "createdAt": "2026-04-11T00:00:00.000Z"}Response Format
Data endpoints return raw Parquet bytes. The Content-Type is application/octet-stream. Files are ZSTD-compressed — DuckDB, Pandas, and Polars decompress automatically via read_parquet(). Non-data endpoints return application/json.
| Response Type | Content-Type | Endpoints |
|---|---|---|
| Parquet stream | application/octet-stream | /v1/sample/*, /v1/sp500/*, /v1/pro/*, /v1/full/* |
| JSON | application/json | /health, /v1/me, /v1/manifest, /v1/usage |
Plans
Your token's plan determines which bucket you can access. A higher plan grants access to all lower tiers as well.
| Plan | Auth Required | Bucket | Coverage |
|---|---|---|---|
sample | No | R2_SAMPLE | Public 5-year S&P 500 slice |
sp500 | Yes | R2_SP500 | Benchmark tier: the large-cap US benchmark universe incl. former members, fundamentals 1993–present |
pro | Yes | R2_PRO | Active + delisted US universe (28,000+ entities), 15-year history (2011→present) |
full | Yes | R2_FULL | Institutional tier: US + foreign issuers, 1993→present, intraday accepted_at, webhooks, internal-use license |
Endpoints
12 endpoints across the gateway. Expand a row for its query parameters and an example response.
GET/healthPublicSystem
/healthLiveness check. Returns 200 OK with service status.
{ "ok": true, "worker": "edge-gateway", "env": "production" }GET/v1/meBearer tokenAuth
/v1/meReturns token metadata: plan, status, email and createdAt.
{ "plan": "sp500", "status": "active", "email": "user@example.com", "createdAt": "2026-04-11T00:00:00.000Z" }GET/v1/manifestBearer tokenDiscovery
/v1/manifestReturns available tables and last snapshot timestamp for your plan tier.
{ "snapshot": "snapshot_20260411", "last_updated": "2026-04-11T00:00:00Z", "tables": [...] }GET/v1/sample/manifestPublicSample (No Auth)
/v1/sample/manifestPublic sample tier manifest — no token required. Includes the table list, the column schema and an upgrade message.
{ "snapshot": "snapshot_20260411", "tables": [...], "upgrade": "…" }GET/v1/sample/{table}PublicSample (No Auth)
/v1/sample/{table}Parquet stream from the public sample bucket (5-year S&P 500 slice). No token required. Use the bare table name with no .parquet suffix (a suffix on a monolithic table returns 400). The tables served are listed in /v1/sample/manifest.
application/octet-stream — raw Parquet bytesGET/v1/sp500/{table}Bearer tokenData
/v1/sp500/{table}Parquet stream for the Benchmark universe (current and former S&P 500 constituents). Requires a Benchmark (sp500), Pro, or Institutional token. Full history, 500+ tickers. Without an Authorization header the gateway answers 302 and redirects to /v1/sample/{table}.
application/octet-stream — raw Parquet bytesGET/v1/pro/{table}Bearer tokenData
/v1/pro/{table}Parquet stream from the Pro bucket. Requires pro or full plan token. Full active + delisted US universe (28,000+ entities), 15-year rolling history (2011→present). Without an Authorization header the gateway answers 302 and redirects to /v1/sample/{table}.
application/octet-stream — raw Parquet bytesGET/v1/full/{table}Bearer tokenData
/v1/full/{table}Parquet stream from the Institutional bucket. Requires `full` plan token. 28,000+ US-listed entities plus foreign issuers, active + delisted, 1993→present. Without an Authorization header the gateway answers 302 and redirects to /v1/sample/{table}.
application/octet-stream — raw Parquet bytesGET/v1/{plan}/{table}/{cik}.parquetBearer tokenData
/v1/{plan}/{table}/{cik}.parquetOne company's rows of a per-CIK partitioned table (fact, ratio, filing, earnings_signals, stock_price, stock_price_daily; Institutional adds insider_transaction, insider_ownership and institutional_holding). {plan} is sp500, pro or full; {cik} is the 10-digit zero-padded CIK. Much smaller than the whole table. institutional_holding also has /v1/full/institutional_holding/by_issuer/{cik}.parquet and /by_filer/{cik}.parquet.
application/octet-stream — raw Parquet bytesGET/v1/presign/{table}Bearer tokenData
/v1/presign/{table}Returns a short-lived signed URL for a table (or /v1/presign/{table}/{cik}.parquet for one company's partition), so a client such as DuckDB can read it directly. Expiry is capped by rateLimits.presignMaxExpiry.
{ "url": "https://…", "expires_in": 3600 }GET/v1/plansPublicDiscovery
/v1/plansPlans, tiers and rate limits. The canonical source for the numbers on this page.
{ "version": "…", "plans": { "sample": {…}, "sp500": {…}, "pro": {…}, "full": {…} } }GET/v1/usageBearer tokenAnalytics
/v1/usageReturns daily API call counts, error rates, and per-table breakdowns for the last N days (default 7, max 30).
Query Parameters
daysNumber of days to return (1–30). Defaults to 7.{ "period_days": 7, "total_calls": 1420, "error_rate": 0.012, "daily": [...], "table_breakdown": {...} }Python Example
Download a Parquet table and query it locally with DuckDB in under 10 lines.
# pip install duckdb requestsimport os import duckdbimport requests token = os.environ["VALUEIN_TOKEN"] # your Bearer tokenurl = "https://data.valuein.biz/v1/sp500/fact" r = requests.get(url, headers={"Authorization": f"Bearer {token}"}, stream=True)r.raise_for_status() with open("fact.parquet", "wb") as f: for chunk in r.iter_content(chunk_size=8192): f.write(chunk) conn = duckdb.connect()df = conn.execute( "SELECT * FROM read_parquet('fact.parquet') LIMIT 5").df()print(df)Available Tables
13 core tables ship on every tier. Pass any table name as the {table} path segment. Institutional adds 11 more (insider Forms 3/4/5/144, 13F/13D/13G holdings and Form ADV), listed below. See Parquet Schema Reference for full field definitions.
| Table | Description |
|---|---|
references | Derived flat join of entity + security. One row per security. Start here for cross-company queries; JOIN index_membership on cik = cik for membership filters. |
entity | Company profiles: name, sector, SIC code, location, CEO, founding year, description. One row per CIK. |
security | Exchange listings: ticker, exchange, FIGI, valid date range (SCD Type 2). Multiple rows per company. |
filing | SEC EDGAR filing index: accession ID, form type, filing date, acceptance timestamp. Links entity to facts. |
fact | 120.5M+ financial data points: XBRL concept values with accepted_at timestamps for PIT accuracy. |
ratio | Pipeline-computed financial ratios per entity per fiscal period. Carries an append-only accepted_at vintage, so ratios are point-in-time reconstructable like fundamentals. Filter by category for grouped screens. |
factor_scores | Cross-sectional factor scores (value, quality, momentum families) per entity — powers universe screens. |
earnings_signals | Per-entity earnings signals: surprises, revision momentum, and report-timing features. |
stock_price | Coarse price series per entity, available on every tier: one close at each fiscal period end plus month-end closes, each carrying the priced listing's total_return_index — so the ratio of two month-end rows is the total return between them, dividends included. |
stock_price_daily | Pro and Institutional only — the free tiers do not carry the daily series (licensed market data). Full daily OHLCV bar series with total_return_index and the raw corporate-action factors — join to fundamentals for PIT valuation multiples. Use total_return_index for total return; adjusted_close is a sparse vendor passthrough. Prices are licensed market data, not EDGAR: no bars in 1993, earliest bar is 1994 or later and differs per security. |
index_membership | Historical index constituents (SP500, RUSSELL1000, RUSSELL2000, RUSSELL3000) with effective_date / removal_date for PIT universe construction. Keys on cik. |
taxonomy_guide | Mapping of 350+ standard_concept labels to raw XBRL tags and human-readable descriptions. |
standard_concept | The standardization layer itself: every canonical concept with definition, statement type, and review status (rule-based mappings checked by accounting identities). |
Paid-tier and Institutional tables
Not in the free sample. Institutional tables answer on /v1/full/ only; tables marked monolithic have no per-CIK partitions.
| Table | Tier | Description |
|---|---|---|
filing_registrant | Paid tiers | Every registrant of an SEC accession. filing has one row per accession, owned by the first registrant, so a combined (co-registrant) filing appears under one CIK there. Join to filing on accession_id to list a company's filings including the combined ones. Not in the free sample manifest. |
insider_party | Institutional | Directory of every insider or reporting person on Forms 3/4/5/144. Joins to insider_transaction via insider_party_id. |
insider_filing | Institutional | Form 3/4/5, SC 13D/13G and 144 disclosure filings (and amendments), rooted on the issuer in subject_cik. |
insider_transaction | Institutional | Form 3/4/5/144 line items: one transaction, initial holding or proposed sale per row. Also served per CIK. |
insider_ownership | Institutional | SC 13D / 13G beneficial-ownership disclosures (5%+ stakes), one row per filing and reporting person. Also served per CIK. |
insider_footnote | Institutional | Form 3/4/5 footnotes, verbatim: the receipt behind a 10b5-1 plan or a weighted-average price. Joins to insider_filing on accession_id. Monolithic file. |
insider_ownership_purpose | Institutional | Schedule 13D Item 4 Purpose of Transaction, verbatim raw text. 13D and 13D/A only. Joins to insider_filing on accession_id. Monolithic file. |
institutional_filing | Institutional | Form 13F-HR / 13F-NT filings (and amendments), rooted on the manager. |
institutional_holding | Institutional | Form 13F-HR holdings: one row per filed information-table line. Also served per CIK, by issuer or by filer. |
institutional_other_manager | Institutional | 13F manager lists: other included managers and managers reporting for the filer. Joins to institutional_filing on accession_id. Monolithic file. |
investment_adviser | Institutional | Form ADV Part 1A advisers, keyed on crd_number (cik is a nullable soft link). Exempt Reporting Advisers omit Items 5.A to 5.F by obligation. |
investment_adviser_private_fund | Institutional | Private funds advised by a registered adviser (Form ADV Schedule D 7.B.1). Joins to investment_adviser on crd_number. |
Manifest Response
Call GET /v1/manifest to discover available tables and the current snapshot timestamp for your plan. Check this before downloading tables to detect updates.
{ "snapshot": "snapshot_YYYYMMDD", "last_updated": "YYYY-MM-DDTHH:MM:SSZ", "tables": ["earnings_signals", "entity", "fact", "factor_scores", "filing", "filing_registrant", "index_membership", "industry_classification", "ratio", "references", "restatement_events", "security", "standard_concept", "stock_price", "taxonomy_guide"]}Rate Limits & Retries
Limits are per Bearer token and enforced at the Cloudflare edge. Every response includes the standard rate-limit headers — read them before retrying so your client never busy-waits.
Response headers on every request
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 117
X-RateLimit-Reset: 1735680000
Retry-After: 42Retry-After appears only on 429 responses (seconds). X-RateLimit-Reset is the Unix epoch when the window rolls over.
Recommended retry policy
429— sleep forRetry-Afterseconds, then retry. Never retry sooner.503— exponential backoff: 1s, 2s, 4s. Cap at 3 retries.5xxother — retry once after 1s, then surface the error.4xxother — never retry. Fix the request.
curl with retry-aware streaming
# --retry honors Retry-After on 429 and exponentially backs off on 5xx.# --retry-max-time bounds total elapsed time; --retry-connrefused covers# the cold-start case after a cache eviction.curl https://data.valuein.biz/v1/sp500/fact \ -H "Authorization: Bearer $VALUEIN_TOKEN" \ --retry 5 --retry-max-time 120 \ --retry-connrefused \ --output fact.parquetParquet responses are a single binary stream — no pagination, no cursor tokens. The gateway sets Content-Length on every data response so clients can show progress and pre-allocate buffers. For incremental reads, query the latest snapshot from /v1/manifest and download only when the timestamp changes.
Bulk Data API vs MCP Server vs Python SDK
Three channels, one Bearer token, same warehouse. Pick by access pattern.
| Channel | Best for | Returns | Latency |
|---|---|---|---|
| Bulk Data API | Loading full tables into DuckDB, Spark, or a warehouse. Periodic syncs. | ZSTD Parquet (full table) | Edge stream · MB-scale |
| MCP Server | Single-fact lookups from AI agents. Conversational queries. | JSON tool responses | Sub-100ms typical |
| Python SDK | DataFrame-shaped queries from notebooks and scripts. | DuckDB over R2 Parquet · as-filed vs latest restatement columns | Local DuckDB · ms |
Use the SDK first if you're writing Python — it wraps this API and the MCP server with sensible defaults. Use the raw Bulk Data API when you're in a non-Python stack or building a partner integration.
Error Codes
| Status | Meaning | Common Cause |
|---|---|---|
| 200 OK | Success | Request succeeded. Parquet bytes or JSON body in the response. |
| 400 Bad Request | Invalid table | The table name in the path is not in the valid tables list. Check spelling and trailing slashes. |
| 401 Unauthorized | Missing or invalid token | No Authorization header, malformed Bearer token, or token not found in KV store. |
| 403 Forbidden | Plan too low | Your token exists but its plan does not grant access to this bucket (e.g. sample token accessing /v1/sp500/). |
| 429 Too Many Requests | Rate limit exceeded | You have exceeded your per-minute or per-hour rate limit, or your plan's daily bulk-download limit (resets at UTC midnight). Slow down and retry after the Retry-After header; a higher plan has higher limits. |
| 503 Service Unavailable | Snapshot loading | The R2 snapshot is being refreshed. Retry after 30–60 seconds. This is rare and brief. |
Get your API token
Subscribe to the free Benchmark plan, Pro, or Institutional to receive a Bearer token instantly. The sample tier is always free — no credit card required.
Index names are trademarks of their owners; Valuein is not affiliated with or endorsed by any index provider.