Build MQTT + SQLite + Hadoop Sales Data Pipeline (Proof of Concept, 5 Days)

PROJECT OVERVIEW
We need a small, working proof of concept that collects retail sales
transactions over MQTT, stores them locally in SQLite, loads them into
Hadoop (HDFS + Hive) and produces basic sales reports. Everything must
run on one machine with Docker so we can demo it and build on it later.

DATA FLOW
1. A sales simulator (Python) publishes transactions to an MQTT broker (Mosquitto)
2. A collector subscribes and saves every message in SQLite (it keeps working if Hadoop is down)
3. A loader moves new SQLite rows to HDFS on a schedule and on demand
4. Hive tables on HDFS and SQL queries produce the sales reports

DELIVERABLES
1. docker-compose.yml: Mosquitto, single-node Hadoop 3.x (HDFS + YARN), Hive. One command starts it all.
2. Sales simulator
- 5 stores, 200 products, configurable rate (default 20 messages/sec)
- Topic: sales/{store_id}/transactions
- JSON fields: transaction_id, store_id, product_id, category, quantity,
unit_price, payment_method, timestamp (UTC, ISO 8601)
3. MQTT to SQLite collector
- QoS 1, reconnects automatically
- Validates each message. Bad messages go to a "rejected" table instead of being dropped silently.
- A repeated transaction_id is ignored (primary key)
4. SQLite to HDFS loader
- Uploads only rows not yet uploaded, and marks them as uploaded only after HDFS confirms the write
- Partitioned by date, e.g. /data/sales/dt=YYYY-MM-DD/
- Safe to re-run: never creates duplicates in HDFS
5. Hive tables and five reports, each exported to CSV:
- Daily revenue per store
- Top 10 products by revenue and by quantity
- Hourly sales trend
- Revenue by category
- Revenue by payment method
6. README covering setup, running each component, configuration and how to extend it
7. A 5-10 minute screen recording of the full pipeline working

ACCEPTANCE CRITERIA
- On a fresh machine, "docker compose up" plus the README steps runs the full pipeline
- The simulator sends 50,000 transactions and the Hive row count matches exactly
- Hadoop is stopped for 10 minutes while the simulator runs. After restart, no data is lost and nothing is duplicated.
- Running the loader twice in a row creates no duplicates
- All five reports run, and their totals match a spot-check against SQLite
- Clean code, delivered in a Git repository

TECH STACK
Python 3.10+ (paho-mqtt, sqlite3), Eclipse Mosquitto, Apache Hadoop 3.x,
Apache Hive, Docker Compose. Spark SQL instead of Hive is acceptable if
you explain why.

TIMELINE (5 DAYS)
Day 1: Docker environment running, simulator publishing to MQTT
Day 2: Collector writing to SQLite with validation and duplicate handling
Day 3: Loader to HDFS with date partitions and safe re-runs
Day 4: Hive tables and all five reports
Day 5: Failure testing, README, recording, handover

MILESTONE PAYMENTS
30% - MQTT and SQLite working, shown in a live demo (end of Day 2)
40% - Data in HDFS, reports running (end of Day 4)
30% - Final delivery, after all acceptance criteria pass

OUT OF SCOPE
Production or multi-node cluster, cloud deployment, web dashboard.
MQTT TLS and authentication are welcome but not required.

HOW TO BID
- Start your bid with the word "PIPELINE" so we know you read this
- Describe one Hadoop/Hive or MQTT project you have done (a link or a few lines)
- Confirm your fixed price and that you can deliver in 5 days
- Suggest any change you would make to this design

Bids missing these items will not be considered.