Daily Racing Data Scraper

STARTING VOLUME AND URGENCY — PLEASE READ BEFORE QUOTING
Latest saved queue audit (17 October): 94,359 race-level collection jobs covering 825,298 runner rows. This comprises 90,207 result-link discovery/matching jobs (790,848 runner rows) and 4,152 direct-result jobs (34,450 runner rows). These are queue counts, not a newly verified distinct-page total; reconcile overlaps, existing captures and unpublished fields before fixing the final manifest. The work includes finding/matching correct Racing Post result links, not only downloading supplied URLs.

We need completion in days, not months. State measured sustained completed race pages/hour, separate discovery overhead, realistic operating hours and source-imposed pauses, expected elapsed days for this starting backlog, first useful batch date, full completion date, and any volume-related price changes. Three sample races prove initial quality only; do not extrapolate an unsupported speed promise. Provide a staged plan and disclose if this scale/timeframe cannot be met.

I have Racing Post Ultimate. The existing Windows/Python collector uses an attached authenticated Chrome session and keeps getting locked out; a saved status records BROWSER_SOURCE_REFUSAL_406. Diagnose session expiry, rate limiting and source rejection separately. No credential sharing is required. A saved-HTML parser test alone does not prove live retrieval reliability.

UPDATED SCOPE: RACING POST HISTORICAL RESULTS / MISSING-DATA BACKFILL

I already have a horse-racing database and an existing process for collecting daily Racing Post data. The main requirement is a Python collector to retrieve missing historical race results and horse performances from Racing Post and deliver structured JSON for import into my existing system. Please quote against this updated scope, rather than a new general daily feed.

1. INPUT AND TARGETED COLLECTION
Accept a CSV/JSON list of missing race IDs/URLs, horse IDs/URLs or dated performance records. Support British and Irish Flat, all-weather, hurdle and chase racing, with configurable dates. Retrieve only missing records and supporting pages needed to complete them. When a race result is requested, account for every runner. When horse history is requested, retrieve the agreed missing performances. Distinguish missing races, missing runner performances and missing fields. The historical range and volume will be agreed after examining a sample gap list; do not assume an unlimited archive download.

2. RACE DATA
Collect where published: Racing Post race ID and URL; course ID/name; race date and scheduled time; actual off time if available; race title, code/type, class and grade/group/listed status; handicap and age restrictions; distance; turf/all-weather surface; going; number of starters; prize money and currency; winning time and comparison with standard; final race status. Collect sectionals/split times only where the source actually publishes them, and identify unavailable coverage.

3. RUNNER / HISTORICAL PERFORMANCE DATA
Collect where published: horse ID/name/profile URL; race ID; finishing position and dead heats; non-completion codes (PU/F/UR etc.); non-runners and disqualifications; beaten distance and cumulative beaten distance where available; starting price (original and decimal), favourite indicator; draw; age; weight carried; jockey ID/name and claim; trainer ID/name; headgear and first-time indicators; published performance/in-running comments.

CRITICAL: Include the actual Racing Post Rating (RPR), Topspeed (TS) and Official Rating (OR) for each historical performance. These must be separate fields. Do not substitute in-house ratings or confuse OR with RPR/TS. Use race-specific historical trainer, jockey, weight and ratings, not current profile values. State explicitly which fields are accessible, which require a subscription and which you cannot provide. Never invent inaccessible data or bypass subscription access.

4. OUTPUT AND VALIDATION
Provide JSON plus CSV exports for inspection, with separate race and runner-performance records linked by stable IDs. Include source URLs, collection timestamps, status and field-level missing-data explanations. Supply a field dictionary covering types, units and null conventions. Preserve original text alongside useful conversions for distance, weight, odds and beaten distances. Missing ratings must not become zero. Preserve non-finisher/non-runner status. Identify the source of any supplementary data; do not silently mix sources.

5. OPERATION AND DELIVERABLES
Run locally on Windows using Python, via command line or a simple launcher. Include full source code, requirements/dependencies, README and setup/run instructions. Provide configurable output paths and request rates, bounded retries/backoff, logging, checkpoint/resume, duplicate prevention, cached completed records, a retryable failed-record list and a collection summary. Re-running the same input must not duplicate performances or unnecessarily fetch completed records. Write to staging files initially; do not overwrite my production database. Support manually triggered backfill, with optional scheduled runs if needed.

Disclose all ongoing costs: APIs, subscriptions, proxies, hosting or other services. Explain the access method and what permissions/licences it requires. State source-code ownership, delivery time and maintenance/defect-fix terms. No new dashboard or prediction model is required.

6. WORKING TEST BEFORE THE FULL PROJECT
Propose a separately priced small demonstration before the full build is awarded. I will select approximately 10-15 historical races covering at least 100 runner performances: recent and older races, UK and Ireland, Flat turf, all-weather, hurdles and chases, with non-runners/non-finishers and unusual results where available.

Supply actual JSON/CSV files and source links, not just screenshots/video. Demonstrate populated RPR and Topspeed wherever accessible on the selected source records. Show extracted values against Racing Post for selected runners: IDs, position, OR/RPR/TS, SP, weights, jockey/trainer and comments. Account for every starter and explain every missing required field.

Also demonstrate: (a) a mixed input of existing and missing records, skipping completed records; (b) interruption and successful resume; (c) an invalid/unavailable record logged without stopping the remaining job; (d) two runs of the same input with no duplicates. Report elapsed time, request/page count, paid-service usage and estimated running cost per 1,000 runner performances. Define whether any quoted 'result' charge means a race, runner performance or page.

Acceptance requires correct source-matched values in all manually checked records, complete accounting of sample runners, explicit missing-data reporting, working resume/selective collection, no fabricated values or duplicates, and usable delivered files/code.

PLEASE ANSWER IN YOUR BID
Have you collected Racing Post historical data before? Can you demonstrate actual RPR and Topspeed? What access/subscription is required? Which fields cannot you provide? Will I own and receive the full working source code? What are the demonstration price, full-project price, ongoing costs and support period? Existing bidders: please confirm or revise your quote for this historical backfill scope.