Right now you're either paying an intern to copy/paste prices into a spreadsheet, or you're not tracking at all and finding out from customer service tickets on Mondays. Here's the 6-step framework we install to track competitor prices daily — automatically, reliably, and at a cost that makes sense for an SMB.
Get a free competitive pricing auditMost founders won't say this out loud, but they don't actually know what their top three competitors charged yesterday. They have a guess. They have a number from "the last time someone checked." They might have a screenshot an intern took for a board deck three weeks ago. What they don't have is a chart that updates every morning, with red flags whenever a competitor's price moves more than a few percent on a SKU that matters. So they live with a permanent, quiet anxiety — are competitors running a flash sale right now, did they drop prices on the SKUs we share, are we leaving margin on the table on the things they don't carry — and the answer is always "we'll check on Monday." Before you scrape anything, you should check the target's robots.txt under RFC 9309 — respecting it keeps the scraper polite and your business out of trouble.
The fix is a small scraper plus a database designed for this specific job: raw scrape data lands in a schema-flexible store (so a layout change doesn't break a year of history), clean prices get derived and written to a typed price-history table, and only meaningful changes — surfaced in near real-time using MongoDB change streams — trigger an alert. Below is the exact 6-step framework, with the actual MongoDB and Postgres snippets we use. The on-call rotation that picks up those alerts at 11pm on a Friday is the same one we describe in how we handle alerts at 3AM so you don't have to.
Define the SKUs, URLs, variants, and regions worth tracking.
Not everything is worth scraping daily.
Scrape resilient to layout changes — extract from JSON-LD when available, fall back to selectors.
A flaky scraper is worse than no scraper.
Currency, bundles, discounts, shipping, and taxes all roll into one effective price.
Otherwise the alerts are noise.
Avoid false alarms from wrong-variant comparisons.
Match on UPC first, then fuzzy title + brand + size.
Raw scrape goes to MongoDB (flexible). Clean price events go to Postgres (typed, queryable).
Best of both worlds.
Action-only notifications: real drops, real stock-outs, real shipping changes.
No "they raised price 1¢" noise.
Raw scrape data lands in MongoDB (schema-flexible). The extractor derives clean prices and writes a typed history row to Postgres only when something changed.
MongoDB · raw scrape capture (NoSQL — schema flexible)// Scraper writes the entire response, schemaless,
// so we can re-extract later if the page layout changes
db.scrapes.insertOne({
competitor: "competitor_x",
sku: "ACME-1234",
variant: { size: "M", color: "navy" },
url: "https://...",
html: rawHtml, // full page, gzip-compressed
json_ld: extractedJsonLd, // structured data if present
scraped_at: new Date(),
http_status: 200,
user_agent: "DevOpsAm/1.0"
});
Postgres · only insert on real change (SQL — typed history)-- Slowly-changing dimension Type-2: only write a new row when
-- the price, stock, or shipping actually changed.
INSERT INTO competitor_price_snapshots
(competitor, sku, variant_key, price, currency, in_stock,
shipping_cost, observed_at)
SELECT $1, $2, $3, $4, $5, $6, $7, NOW()
WHERE NOT EXISTS (
SELECT 1
FROM competitor_price_snapshots
WHERE competitor = $1
AND sku = $2
AND variant_key = $3
AND (price, in_stock, shipping_cost) = ($4, $6, $7)
AND observed_at = (
SELECT MAX(observed_at)
FROM competitor_price_snapshots
WHERE competitor = $1 AND sku = $2 AND variant_key = $3
)
);
The trigger points we look for — if any of these have happened, the cost of doing nothing is real:
The framework above isn't theoretical — it's a checklist. Each gate takes a day or two to install for the SKUs and competitors that matter to your business, and once it's running it runs without you. We typically run the scrapers on a schedule with AWS Lambda's pay-per-invocation pricing, which keeps the monthly bill in dollars rather than hundreds of dollars — the same dollar-tightening discipline we describe in how we cut a client's AWS bill by 38%. The point is not to obsess over every penny competitors charge; the point is to stop being surprised — to know within hours, not days, when something has actually moved.
Done right, this is the cheapest competitive intelligence your business will ever buy. Done wrong (or not at all), it's the silent reason your Friday revenue keeps mysteriously underperforming. The intern with a spreadsheet is not a substitute — she's a symptom. (For the legal question that always comes up: the Ninth Circuit's hiQ Labs v. LinkedIn ruling established that scraping publicly accessible web data is generally not a Computer Fraud and Abuse Act violation, but you should still respect robots.txt and rate-limit politely.)
Send us your top 10 SKUs and the 1-3 competitors who matter most. Within 48 hours we'll send back a clear report: where you sit relative to them today, how much their prices have moved this quarter, and what a daily tracker for your full catalog would cost to run.
Show me where I sit (free)