What the system does, in one paragraph and one transformation.
Five public platforms surface what Vietnam is paying attention to; Shopee's own search trends say what people are ready to buy. Most of the rest is news noise with zero shopping intent. This system filters that out automatically and converts what survives into fields a creator can act on the same day — user need, Shopee niche + products, a seed keyword, and finally the specific SKUs to feature, joined from our own product big data. Everything ranked, nothing hand-sorted.
Why a manual, single-platform approach leaves money on the table.
An automated pipeline that ingests all five platforms nightly, discards the noise, and hands Ops a ranked shortlist of scored product opportunities — each already mapped to a Shopee niche and a ready-to-paste search keyword — before the market has moved.
Each source occupies a distinct stage of a demand funnel, from first whisper to money at the till. We don't treat their numbers as equal — we treat them as different grades of evidence, and weight them accordingly in the score.
Shopee rising search + any one upstream platform fast-tracks straight to a brief — demand is already at the till and you only need the story. Google intent + one of {YouTube research, TikTok commercial} clears the bar without it. A single-source spike with no commercial echo stays noise until a second source confirms it.
Shopee search does double duty: paired with catalogue depth it exposes the gap between demand and supply. Rising Shopee searches against thin, poorly-listed or out-of-stock results is the highest-margin signal in the whole system — real buyers, no competition. Flag these separately; they outrank a bigger trend in a saturated category.
A nightly batch job: ingest everything, normalise to one shape, filter the noise, map survivors to product fields, score, and route. Ops wakes up to a ranked feed.
Stages 3 and 4 are the product; everything else is plumbing you can assemble from off-the-shelf tools. Stage 3 decides what is worth money, Stage 4 decides which money — the exact niche, products and keyword. Build and tune these two before widening the source list.
Shopee enters the pipeline twice, deliberately. At Stage 1 it's an input — its own top and rising searches are trend signal in their own right, and can surface an opportunity no public platform shows. At Stage 5 it's the validator — every generated keyword is checked against live results for volume, supply depth, price spread and commission before anything reaches a creator. Nothing ships on inference alone.
What we capture, how we pull it, and how often. Phase 1 favours official and free sources; paid analytics come online in Phase 2.
| Platform | Signal captured | How we pull it | Refresh |
|---|---|---|---|
| Google Search | Trending Now feed; rising & related queries; interest over time | Trends CSV export · pytrends · SerpApi / DataForSEO Trends API for reliability | Daily |
| TikTok | Trending hashtags, sounds & creators; TikTok Shop best-sellers (VN) | Creative Center (free) · Kalodata / FastMoss / EchoTik for Shop analytics | Daily |
| YouTube | Most-popular chart (regionCode=VN); review & haul velocity; autocomplete | YouTube Data API v3 (official, free quota) · VidIQ / TubeBuddy | Daily |
| Products under active ad spend; buy/sell & deal-group demand | Meta Ad Library API (free) · Content Library API · social listening for groups | Daily | |
| Threads | Trending topics; emerging Gen-Z conversation & sentiment | Threads API (limited) · Talkwalker / Meltwater · manual scan in Phase 1 | Daily |
| Shopee anchorsource and validator | Top / rising search keywords; search autocomplete; category best-sellers & Flash Sale; trending affiliate products | Shopee Ads keyword planner · search autocomplete endpoint · Affiliate & Open Platform API · seller-centre category reports | Daily |
| Shopeeenrichment pass | Supply depth, listing quality, stock, price bands, commission % per candidate keyword — the supply-gap calculation | Shopee Affiliate / Open Platform API · category catalogue | On demand |
| Shopee product feed internalour own big data | SKU-level sales, 6-month sales trends, commission, price, ratings, shop tier, category path — the deal layer | Internal big-data platform (already in place) — see §07 | Daily |
Google and Facebook have no official "trending topics" API, and Threads' trending data is thin in Vietnam. Shopee's richest keyword data sits behind the Ads keyword planner and Seller Centre rather than a clean public API — so plan for an authenticated seller/affiliate account and a scraping-tolerant fallback for autocomplete. Budget for third-party providers (SerpApi, DataForSEO, Kalodata, a social-listening seat) and respect each platform's terms. These are line items, not free.
The four moves that turn a raw trend string into the three output fields — Stages 3 and 4 in detail.
Tag every trend and drop the dead ones. A lightweight LLM classifier plus keyword rules assign one label: news/event (drop), entertainment (weak — keep only if merchandisable), seasonal (strong), product-native (strongest, the query is nearly a product), evergreen-need (steady demand).
For survivors, answer the job-to-be-done question: who is searching this, and what are they trying to accomplish? A football match becomes "fans want to watch and support at home." Admission results become "new students must equip a dorm and study setup." This is the empathy step that makes the product mapping non-obvious and defensible.
Match the need against the Shopee category taxonomy to pick one niche and three to six concrete product types. A rules-plus-retrieval layer drives it: a curated need→category map for known patterns, semantic matching against the live catalogue for new ones. Output favours the 100k–500k₫ band and in-stock, high-commission items.
Produce the exact Vietnamese query a shopper would type into Shopee — not the news phrase. "việt nam vs campuchia" becomes áo đấu việt nam. Each candidate is then run against live Shopee search: does the keyword appear in Shopee's own rising searches, how many results come back, what's the price spread, the commission, and the supply gap? Kept, swapped or dropped on that evidence — never on the model's guess alone.
| Bucket | Test | Action | Example from this week's VN feed |
|---|---|---|---|
| News / event | Reports something; nothing to buy | Drop & archive | xổ số miền nam (2M+), arrests, obituaries |
| Entertainment | Show or celebrity; merch only | Keep if merchandisable | K-dramas, badly in love s2 |
| Seasonal | Time-boxed demand wave | Prioritise — runway matters | tra cứu xét tuyển (500K+), nghỉ lễ 2/9 |
| Product-native | Query is nearly a product | Fast-track | iphone 15 pro, xe chạy điện, vua quạt |
| Evergreen-need | Steady, recurring demand | Always-on inventory | nước dừa, phone cases, fans |
Every survivor gets one number, 0–100, so Ops can rank a hundred candidates in seconds. Five weighted components; bar length shows each component's share.
Momentum sums each source's volume against its weight from §02, so the same raw number counts for more when it comes from Shopee or TikTok than from Threads. The Shopee-confirmed multiplier applies whenever the seed keyword is independently rising in Shopee's own search — the single strongest predictor in the model.
The trend layer stops at a keyword. Our internal Shopee product feed carries it the last mile — from "search this" to "push these exact SKUs." This section covers what the feed gives us, what it can't be trusted on, and how a deal gets scored.
historicalSoldunitsSoldmonthSalesTrends[]revenue
commissionRatepriceminPricemaxPricesamePrice
ratingratingCountreviewRateshopTypeisOfficialShopisShopeeVerifiedfavorite
cateIdPathcateIdcateNames[]productName
genTimeapprovedDatecbOptionshopLocationbrandName
pidproductUrlimageUrlshopIdshopName
The feed has no stock level, no voucher or campaign data, and no shipping terms. That means "deal" in the discount sense can't be computed from this record alone — we can rank opportunity but not discount depth. Either add a live price-history snapshot to detect genuine markdowns, or restrict the wording in briefs to "trending product" rather than "deal." Recommend the former.
Product feeds lie in predictable ways, and the single record supplied demonstrates three of them. Every gate below exists because this record failed it.
₫488.18m ÷ 488,180 = ₫1,000.00 exactly. Revenue is derived as price × units, not observed. It carries zero independent information and inherits every price error.
Never rank on it
₫1.00k for an iPhone case, with minPrice = maxPrice and samePrice: true. A bait or decoy listing — real cases run ₫20k–80k. Ranking by revenue here understates true GMV by 20–80×.
Quarantine
528,515 — 40,335 more than the all-time historicalSold of 488,180. April alone (490,740) exceeds the all-time total. Internally impossible.
Quarantine
0, interleaved with 31,985. That reads as a scrape gap or temporary delisting, not genuine zero demand. Treat zeros as missing, never as data.
Impute, flag
commissionRate: 0.0. However well it sells, the MCN earns nothing. This is a hard reject, not a score penalty.
Reject
genTime 2018-09-10 — nearly eight years old. Cumulative lifetime sales are not a demand signal for this month; only the cleaned recent window is.
Downweight
Preferred + isShopeeVerified: 1, rating 4.96 across 97.64k ratings. The one dimension this record passes cleanly.
Pass
| Derived field | Computation | Why it exists |
|---|---|---|
| units_int | Parse "488.18k" → 488180; handle k / m / b suffixes and the ₫ prefix | Every numeric arrives as a display string; nothing is sortable until parsed |
| recent_velocity | Mean of last 3 months, zeros treated as missing, anomalies winsorized | The real demand number. Replaces historicalSold for ranking |
| momentum | recent 3 months ÷ prior 3 months, on the cleaned series | Rising beats big-but-flat — same principle as the trend layer |
| commission_value | commissionRate × effective_price = ₫ earned per conversion | The number the MCN is actually optimising. A 3% cut of ₫400k beats 15% of ₫30k |
| price_percentile | Rank price within its cateId distribution across the full corpus | Detects bait prices and outliers — you hold all Shopee data, so compute this per category |
| listing_age_days | today − genTime | Separates a fresh riser from an eight-year-old evergreen |
| quality_flags[] | Set by the gates above | Drives the quality multiplier; also the QA backlog for the data team |
| Gate | Rule | Sample record |
|---|---|---|
| Affiliate viable | commissionRate > 0 | Fail — 0.0 |
| Creator-safe shop | isOfficialShop = 1 OR (shopType ∈ {Mall, Preferred} AND isShopeeVerified = 1) | Pass |
| Review credibility | rating ≥ 4.5 AND ratingCount ≥ 50 AND reviewRate within 3–35% | Pass |
| Price plausible | price within P10–P90 of its category | Quarantine |
| Series integrity | Σ months ≤ historicalSold AND max(month) ≤ historicalSold | Quarantine |
| Live listing | Non-zero sales in at least 2 of the last 4 months | Marginal |
Survivors of the gates are scored 0–100. This is a separate, product-level score — the Opportunity Score in §06 ranks trends, this ranks SKUs within a trend.
Quality is a multiplier, not a component, and that is deliberate: a bad-data record with strong sales would otherwise average its way into a brief. Quarantine must be able to zero a deal outright.
Not every deal is suggested for the same reason, and different creators need different kinds. Each surfaced deal carries one archetype so the creator manager can match it to the right talent.
The trend pipeline and the product feed run as parallel lanes and meet at a single join. That join is the integration.
Before the join, a brief said "make a video about back-to-school, search balo laptop sinh viên." After it, the brief says "feature these four products, here are the affiliate links, this one is a momentum riser at ₫18k commission per sale." That difference is the whole reason to integrate the feed — it removes the creator's sourcing step entirely.
One record per opportunity — the three headline fields the system exists to produce, plus the metadata Ops needs to act.
| Field | Meaning | Example value |
|---|---|---|
| trend_id | Merged trend across platforms | vn-2026-0812-backtoschool |
| sources[] | Where it was seen | google, tiktok, youtube, facebook, shopee |
| momentum | Volume and growth, weighted by source | 500K+ · rising |
| shopee_confirmed | Keyword independently rising in Shopee search | true |
| supply_gap | Demand vs. quality listings available | Moderate — 1.15× uplift |
| intent_bucket | Stage-A label | Seasonal |
| user_need | Output 1 — the job to be done | New university students equipping a dorm and study setup |
| shopee_niche | Output 2 — niche + products | Dorm & study: laptop backpack, desk lamp, mini fan, storage |
| seed_keyword | Output 3 — paste into Shopee | |
| deals[] | Output 4 — ranked SKUs attached from the product feed | 4 products · top Deal Score 91 · ₫24.6k comm/sale |
| score | 0–100 plus band | 88 Go now |
| window | Seasonal runway | now → mid-Sep |
Any Go now record spawns a one-page creator brief automatically: the hook (why it's trending), the user need, the seed keyword, and the ranked SKUs from §07 with live affiliate links, prices and commission per sale — plus a publish-by date. The creator never sees the pipeline, and never has to source a product themselves; they just film.
The engine run end to end on four signals from this week's Vietnam feed — including one only Shopee's own search data would have caught — plus one the filter correctly kills. Click any keyword to copy it.
Ship a working loop in two weeks on two platforms, then widen. Don't wait for all five to prove the model.
The feedback loop in Phase 3. Once you track which scored picks actually converted to GMV, the score stops being a guess and becomes a model trained on your own network's results — something no off-the-shelf trend tool can replicate.