From Raw SIP Trades to Clean, Chart-Ready OHLCV Candles

Core idea. A raw SIP trade feed is not a ready-made chart. Building defensible OHLCV requires interpreting trade semantics, timestamps, reporting context and price/volume eligibility before aggregation.

A candlestick looks simple.

For a one-minute interval, take every trade in that minute and calculate:

Open = first trade | High = highest trade | Low = lowest trade | Close = last trade | Volume = sum of shares

That sounds reasonable.

For U.S. equities, it is also wrong.

Raw consolidated SIP trade data is not a ready-made chart. It is a stream of reported market events carrying information about price, size, venue, timestamps, reporting facility, corrections and - critically - sale conditions.

Some trades are valid executions but should not establish a new high or low. Some should contribute to volume but not price. Some represent average-price transactions or other special executions. Odd lots can appear in the consolidated tape without being appropriate inputs for the same chart logic as ordinary trades. And during extended hours, an execution can be reported to the SIP minutes - or historically even hours - after it actually occurred.

The result is important:

Raw SIP completeness and chart accuracy are not the same objective.

When we started reconstructing intraday OHLCV from raw SIP trades for Microcap Engine, this distinction became one of the most important parts of the entire market-data architecture.

How Do Raw SIP Trades Become Chart-Ready OHLCV?

The naive way to construct one-minute bars is approximately:

group trades by minute → first / max / min / last / sum(size)

That works only if every trade belongs to the same analytical universe.

They do not.

A raw U.S. equity trade can contain fields such as:

Consolidated trade feeds therefore contain more information than a chart should display.

Massive documents the participant timestamp as the time a trade occurred at the exchange or TRF and the SIP timestamp as the time the SIP received the event. It also exposes sale conditions, TRF identifiers and TRF timestamps. Massive trade schema

That distinction becomes critical once the market gets sparse, fragmented or heavily dependent on off-exchange reporting - exactly the environment frequently encountered in small- and micro-cap stocks.

The First Mistake: Treating Every Reported Trade as Price-Forming

Consider an unusual transaction printed far away from the surrounding market.

A simplistic system sees:

price = $1.20 → Low = $1.20

But that trade may carry a condition indicating that it is an average-price transaction, an odd lot, a contingent transaction or another execution category whose treatment differs from a normal price-forming trade.

The consolidated processing rules explicitly distinguish between components of OHLCV. For example, Massive documents Average Price Trade and Odd Lot Trade condition behavior where consolidated volume may update while open/close or high/low do not. Massive trade conditions

That provider rule is an input to Microcap Engine, not automatically the final Microcap Engine inclusion policy. In the frozen v1.19 methodology, a specific class of provider volume-only sale conditions is intentionally removed from the canonical analytical universe before field-level folding.

So both of these simplistic models are wrong:

trade → always include everything

trade → use one generic valid / invalid flag

The actual flow is closer to:

trade → whole-event safety/exclusion checks → independent field eligibility for surviving events

Price and volume can still have different eligibility for surviving conditions and temporal states, but some event classes are deliberately removed before either reaches the reducer.

This is why two platforms can start with substantially the same consolidated market activity and still publish different candles.

Why This Matters Much More in Microcaps

In a highly liquid stock, one unusual 12-share execution is surrounded by thousands of other trades. Removing it may barely change the candle.

In a thin premarket microcap, that same trade can become the high of the minute, the high of the entire premarket session, the input to ATR or range calculations, the trigger for a breakout condition, or even the apparent price at which a historical strategy thinks it could have traded.

Once that contaminated one-minute candle becomes an input to a five-minute candle, a scanner, an indicator or a backtest, the error propagates.

A single print can therefore change much more than one pixel on a chart. It can change the strategy itself.

SIP Timestamp vs Participant Timestamp: Why It Changes OHLCV

Sale conditions are only part of the problem.

The second major problem is time.

A raw trade may expose at least two conceptually different timestamps:

• Participant time tells us when the underlying execution occurred.
• SIP time tells us when that information entered the consolidated information stream.

Those times are usually extremely close. But not always. And the exceptions are exactly where naive candle reconstruction can break badly.

A Real Example: ANNA, March 20, 2026

During our validation work we found a particularly clear example in ANNA.

If we simply group all trades by their SIP timestamp, both transactions suddenly appear inside the 08:00 minute.

But neither trade occurred at 08:00.

They were executions from around 07:09-07:10 that became publicly reported around 08:00.

That distinction is not academic. Imagine that the contemporaneous market around 08:00 is trading near $4.20. A delayed $5.00 execution can make a naive 08:00 candle show:

High = $5.00

A chart now appears to say: “The stock traded to $5.00 around 08:00.”

What actually happened: “A transaction executed roughly 51 minutes earlier was reported around 08:00.”

Those statements are not equivalent.

Why So Many Historical Artifacts Clustered Around 08:00

The timing was not random.

Before March 30, 2026, FINRA Trade Reporting Facilities did not begin their normal reporting session at 04:00 ET. FINRA amended the TRF schedule so the facilities would begin operating at 04:00 ET starting March 30, 2026, noting that the change would reduce the need to queue trades for later reporting.

That creates an important historical market-structure regime. Trades could legitimately be executed before the reporting facility opened and subsequently appear as a burst of reports when the facility became available.

This behavior was visible across multiple symbols in our corpus.

For DRMA, for example, we observed Form T trades reported just after 08:00 whose participant timestamps reached much further back into the morning.

One 100-share execution around $1.50 occurred at approximately 04:38:53 ET but reached the SIP around 08:00:07 ET. Another observed trade had occurred around 04:15:52 ET and was similarly disseminated shortly after 08:00.

These are not millisecond networking differences. They are completely different temporal phenomena. And that is exactly why a single generic concept of “latency” is insufficient.

Regulatory source: FINRA Regulatory Notice 25-15

Why Not Re-Bucket Delayed Trades by Participant Timestamp?

At first glance, the obvious solution seems to be: if the trade happened at 07:10, move it back into the 07:10 candle.

For historical chart reconstruction, that produces something visually attractive.

For point-in-time backtesting, however, it introduces a serious problem.

At 07:10, the trader did not yet have that report. The market did not receive it through the SIP until approximately 08:00.

Moving the trade backwards in time would insert information into a historical candle before that information became available. For a research platform, that can create a subtle form of look-ahead bias.

Our approach therefore separates two questions:

• When did the information become observable?
• Should this report be allowed to redefine the technical price state of the minute in which it became observable?

Those questions do not need to have the same answer.

A delayed execution can remain attached to its real information-arrival chronology without being allowed to create a misleading new technical high or low. This distinction became one of the core principles of our OHLCV methodology.

Should Form T Trades Be Filtered From Premarket OHLCV?

It would also be wrong to conclude: Form T = bad trade.

Form T identifies an execution outside regular primary-market hours. Most premarket and after-hours trading is supposed to exist.

If every Form T trade were removed, extended-hours charts would largely stop representing extended-hours trading.

The real issue is context.

A contemporaneously reported extended-hours execution is fundamentally different from a transaction that occurred much earlier and was later disseminated into a different market state.

This is why robust filtering cannot be built from one condition code alone. The system needs to understand the combination of sale condition, session, report timing, reporting venue/facility, and the trade’s role in the analytical output.

That is substantially different from a generic “remove bad ticks” filter.

Should Odd-Lot Trades Affect OHLCV Candles?

Odd lots are a good example of the difference between the public tape and the canonical analytical universe.

Historically, odd-lot executions were not included in the consolidated public tapes. The SEC notes that before December 9, 2013, NMS-stock odd-lot transactions were not publicly disseminated; from that date onward they became part of the public consolidated tape.

Massive’s provider condition reference identifies Odd Lot Trade as condition 37. Its consolidated rule can treat that condition as volume-eligible while not updating open/close or high/low.

Microcap Engine v1.19 makes a stricter product decision:

If condition 37 is present, the entire event is excluded from canonical analytical OHLCV.

That means an explicit odd-lot event contributes to none of Microcap Engine canonical O/H/L/C, FullVolume, TradeCount, FullDollarVolume, VWAP, Microcap Engine Current Price, parent/session aggregates, scanner inputs or ordinary backtest inputs.

Just as importantly, Microcap Engine does not infer odd-lot status from trade size.

A 50-share or 10-share execution is not automatically classified as an odd lot by Microcap Engine merely because it is small. The authoritative marker is the disseminated condition 37 / SIP I. If that marker is absent, the canonical aggregation layer does not synthesize it from size < 100, a historical round-lot lookup, price, or TRF status.

That matters because NMS round-lot assignments changed over time, including the 100/40/10/1-share regime effective in 2025. Reconstructing odd-lot status from a fixed quantity heuristic would therefore create historical inconsistencies.

Before December 9, 2013, Microcap Engine accepts the historical source limitation: it does not try to reconstruct odd-lot executions that were never disseminated on the public tape.

Other charting systems also apply technical-chart rules that differ from a raw consolidated-volume interpretation. TradingView, for example, publicly documents odd-lot filtering in its North American intraday history.

The broader lesson is the same: being visible in a raw feed is not the same thing as belonging in one canonical technical candle.

Background sources: SEC odd-lot transparency note | TradingView intraday methodology note

The Dangerous Alternative: Percentage-Based Outlier Removal

Once bad-looking wicks are visible, the tempting solution is to create a price filter: ignore trades more than 10% away from the previous price, or remove isolated prints outside a moving range.

We strongly rejected that approach.

Microcaps routinely make legitimate moves that would look impossible in more liquid securities.

A large wick is not evidence that the trade is bad. A large percentage move is not evidence that the trade is bad. An isolated-looking print is not evidence that the trade is bad.

Filtering based on the appearance of the chart creates an enormous risk of deleting the exact extreme moves that a microcap research system is supposed to study.

Instead, rejection should be based on market-data semantics and temporal evidence.

We built an explicit negative-control case to test this.

Negative Control: RDGT, March 31, 2026

RDGT produced exactly the kind of premarket chart that an aggressive price-outlier filter might incorrectly “clean.”

Between approximately 08:51 and 09:05 ET, the stock showed substantial price movement and large wicks.

So we inspected the underlying trade stream rather than deciding from the visual appearance of the chart.

Those numbers tell a completely different story from the earlier delayed-report examples.

The executions were contemporaneous. They were not hours-old transactions suddenly appearing in the premarket.

The extreme prices were also supported by broader execution activity.

Around 08:53, a low near $4.50 included a contemporaneous TRF execution with only about 2.57 ms between participant and SIP timestamps, while surrounding executions supported the same move.

Around 08:56, the move toward approximately $5.40 was supported by non-TRF executions.

Around 09:04, a high near $5.85 was not dependent on one tiny anomalous print: multiple subsequent 100-share, non-odd-lot, non-TRF executions confirmed trading at the level.

The correct conclusion was therefore: keep the wick.

This negative control is at least as important as the fake-spike examples. A cleaning system is not successful simply because it removes suspicious candles. It also has to prove that it preserves legitimate extreme volatility.

What Our Validation Corpus Showed

We expanded the research beyond individual screenshots.

One validation corpus covered approximately 350,000 raw trades across a mixture of known artifact cases and genuine-volatility controls.

The extended-hours portion contained 162,046 PM/AH trade events.

Across those extended-hours events, we observed zero missing-condition states, zero empty condition arrays and zero lone “regular/no-special-condition” placeholders in the tested sample.

Regular trading hours behaved very differently: the same corpus contained 101,770 RTH trades with missing condition fields, where the absence of a special condition was normal rather than suspicious.

That result matters because it demonstrates another important principle: the same raw representation can have different meaning depending on session context.

A production-grade candle builder cannot necessarily apply one universal interpretation to every session.

This is exactly the kind of edge case that disappears when development is based only on highly liquid regular-hours data.

Can Delayed Prints Distort Premarket Volume?

Yes, but not every delayed report should be treated the same way.

Suppose 60,000 shares were executed gradually during a period when a reporting facility could not disseminate them normally, and the reports appear together after the facility reopens. A naive SIP-time aggregation may make the reopening minute look like a sudden 60,000-share burst of current market participation.

That can contaminate RVOL, dollar-volume filters, volume-rate calculations, float rotation, scanner ranking, and any feature intended to measure activity happening now.

The frozen Microcap Engine methodology therefore distinguishes ordinary stale TRF price reports from structurally deferred TRF backlog.

For an ordinary PM/AH TRF report executed while the relevant TRF was open, the initial production stale-price threshold is 500 ms. If the SIP report age exceeds that threshold, the event is prevented from updating technical price, VWAP and Microcap Engine Current Price, while technical Volume remains governed by the surviving condition rules.

A structurally deferred TRF event is treated more aggressively. If the participant execution occurred while the effective reporting facility was closed or paused, the report is disseminated after reopening, and the report age is at least 60 seconds, Microcap Engine excludes the event from technical OHLC, FullVolume, TradeCount, FullDollarVolume, VWAP and Microcap Engine Current Price.

The raw event is still valuable evidence. It is simply not allowed to masquerade as contemporaneous technical activity in the SIP minute where it finally appeared.

This distinction is historically versioned. The relevant TRF operating window changed from an 08:00 ET opening to a 04:00 ET opening on March 30, 2026, so the structural classifier is calendar/profile-driven rather than one permanently hardcoded clock rule.

And Microcap Engine still does not move the event backward into the participant-time candle. The canonical information-arrival chronology remains SIP-time based.

TradingView has publicly documented historical premarket volume distortions caused by late OTC prints and the use of additional filtering for technical-analysis data. The important point is not that every platform should copy one vendor. It is that reported volume and contemporaneous technical activity can answer different questions.

Sale Conditions Are Not a Simple Blacklist

A production rule cannot be built by collecting a few suspicious condition codes and deleting whichever one appears first.

Microcap Engine v1.19 uses two stages.

First comes a whole-trade exclusion gate. If any of these promoted conditions is present, the event is removed from all canonical analytical products before field folding:

• 2 — Average Price Trade
• 7 — Cash Sale
• 13 — Extended Hours, Sold Out of Sequence
• 20 — Next Day
• 21 — Price Variation Trade
• 29 — Seller
• 37 — Odd Lot Trade
• 52 — Contingent Trade
• 53 — Qualified Contingent Trade

Condition 12, Form T / Extended Hours, is the deliberate exception to that provider volume-only class rule. A contemporaneous Form T execution is needed for legitimate PM/AH price formation, so condition 12 by itself is not dropped. But the exception is not dominant: [12,37], [12,2] or [12,13] is still a whole-trade drop because another excluded condition is present.

If the event survives the whole-trade gate, Microcap Engine evaluates independent:

• Open/Close eligibility (OC)
• High/Low eligibility (HL)
• technical Volume eligibility (V)

Known surviving rule-bearing conditions are folded field-by-field with restrictive NO winning. Known metadata-only conditions are neutral. Microcap Engine-specific transforms, such as the PM/AH Form-T rule, are applied before that restrictive fold.

Unknown numeric condition IDs, explicit NULL or malformed condition representations fail closed for the entire event contribution rather than being ignored.

Session context matters too. A missing/empty/no-special-condition representation is normal Regular Sale behavior in RTH. In PM/AH, the same semantic state is treated as a condition anomaly: technical price, VWAP and Microcap Engine Current Price fail closed, while technical activity remains allowed unless another rule vetoes it. Explicit known PM/AH conditions without Form T are still evaluated normally; “no Form T” is not itself a rejection rule.

A trade should be filtered because of what the event represents under the promoted methodology, not because its price looks inconvenient.

Reference: Massive aggregate construction note

Why Can the Raw Last Trade Differ From the OHLCV Close?

This leads to another subtle but important consequence.

Imagine the last event received during a minute is an execution that is not eligible to form technical price.

The raw feed may correctly say: latest reported trade = $2.01, while the candle correctly says: Close = $2.04.

Those values are answering different questions.

The latest raw trade asks: what was the price attached to the latest trade event we received?

The chart close asks: what was the latest execution eligible to form the technical close of this candle?

Conflating those concepts is dangerous in scanners.

A scanner using raw-last price while a backtest uses filtered candle close can silently create live-vs-backtest divergence even if both systems claim to use the same market-data provider.

For that reason, the price state used by the scanner, chart and backtest must come from explicitly defined canonical semantics.

What Happens When No Trade Is Eligible to Form Price?

Another edge case occurs when raw trades exist during a minute, but none of them should create a technical price candle.

The tempting solution is to carry the previous close forward and draw O = H = L = C = previous close.

That makes the chart visually continuous. It also creates a trade that never happened.

For analytical data, we prefer an explicit absence of price over a synthetic candle.

This distinction becomes particularly important for ATR, high/low calculations, moving averages, entry models, time-in-trade logic, and higher-timeframe aggregation.

A visual layer may decide how to display a missing price interval, but a backtest should not silently convert “no eligible price-forming execution” into “the stock traded at the previous close.”

How Should Higher Timeframes Inherit Filtered OHLCV?

Cleaning the one-minute candles is not sufficient if five-minute candles are then reconstructed independently from the raw tape.

That can create a bizarre situation: the one-minute chart excludes a problematic trade, but the same trade reappears inside the five-minute high or volume.

A consistent architecture instead creates a canonical lower-timeframe representation first and derives higher timeframes from that same state.

raw trades → normalize → classify → canonical 1m → 5m / 15m / hourly / daily analytical state

The exact implementation can vary. The invariant matters more: the chart should not tell one version of history at 1 minute and another version at 5 minutes. The same is true for the backtest and live scanner.

The Most Important Rule: One Canonical Market History

A surprisingly common source of research error is not a bad formula. It is having multiple slightly different definitions of the market inside the same product.

For example: historical backtest uses vendor aggregates, live scanner consumes raw trades, chart uses a third data source, volume parameters use raw consolidated volume, and price indicators use filtered candles.

Each component can look individually reasonable. Together, they are no longer testing the same market.

One canonical technical market history shared by charting, scanning, derived parameters and backtesting.

Only then can a historical signal and a live signal be meaningfully compared.

What We Explicitly Do Not Do

Our filtering is not designed to make charts look prettier.

• We do not remove a trade simply because the candle has a huge wick.
• We do not apply a generic percentage-distance rule.
• We do not smooth extreme microcap moves.
• We do not assume every off-exchange/TRF trade is bad.
• We do not assume every Form T trade is bad.
• We do not simply move delayed trades backwards into an earlier candle if doing so would change what information was historically knowable at that time.

These constraints are important because aggressive cleaning can be just as damaging as no cleaning at all. A research dataset that removes genuine volatility is worse than a noisy dataset if the purpose is to understand extreme market behavior.

Our Practical Pipeline

The final v1.19 architecture is deliberately ordered. In simplified form:

1. Ingest and strictly parse the raw trade event, preserving exact identifiers, timestamps, reporting-facility state, raw conditions, correction state and exact quantity when available.
2. For historical/recovery REST data, apply the promoted active-row lifecycle selector before aggregation.
3. Normalize condition representation without erasing distinctions such as missing, empty, null, malformed or unknown.
4. Apply the v1.19 whole-trade condition exclusion gate.
5. For surviving events, fold OC / HL / V independently, apply the Form-T/session transform, then apply session-specific condition-less behavior.
6. Apply temporal eligibility: ordinary PM/AH stale-TRF price veto and the separate structurally-deferred-TRF activity veto.
7. Keep candle ownership on SIP/report time. Participant time is evidence for temporal eligibility, never a reason to re-bucket the event backward.
8. Reduce the surviving ordered events into one canonical 1-minute state using exact quantity and nullable price fields.
9. Compute technical activity, VWAP and Microcap Engine Current Price from their own explicit contracts rather than treating them as aliases of candle Close or raw Last.
10. Derive 3m / 5m / 10m / 15m / 30m / 1h / 4h states from the canonical 1m history.
11. Feed charts, scanners, indicators and historical research from that same canonical lineage.
12. In realtime, separate immediate INTRABAR evaluation from operationally finalized CANDLE_CLOSE evaluation without creating a second OHLCV truth.

For this article we are publishing the material aggregation rules rather than hiding the production policy behind a generic “cleaning” label.

What the Final Microcap Engine v1.19 Contract Does

The source of truth for this article is the v1.19 final implementation-freeze methodology. The important point is not that one heuristic cleans a chart. It is that each stage has an explicit contract and the same semantic package is intended to govern historical, realtime and recovery paths.

The condition map is versioned, but Microcap Engine owns the final inclusion policy

Microcap Engine uses the authenticated Massive stock-trade condition reference as the provider mapping, including consolidated updates_open_close, updates_high_low and updates_volume semantics.

The exact provider snapshot is versioned and monitored for semantic changes. A provider-map change is reviewed and promoted rather than silently changing the analytical universe mid-session.

But the provider map is not the final product policy. The v1.19 whole-trade gate deliberately removes conditions 2, 7, 13, 20, 21, 29, 37, 52 and 53 from all canonical analytics before field folding.

Form T condition 12 remains session-specific rather than being blanket-blacklisted.

Odd Lot means explicit condition 37, not “small trade”

Odd-lot classification is based on the disseminated condition marker only.

Microcap Engine does not say:

size < 100 → odd lot

and it does not reconstruct a historical round-lot table merely to decide whether a trade belongs in the candle.

Condition 37 present means the event is a canonical whole-trade drop. Condition 37 absent means Microcap Engine does not invent it.

This is an important correction to simplistic implementations because round-lot rules changed historically and TRF status is independent of odd-lot status.

Canonical time is SIP time; temporal eligibility is a separate question

Candle ownership follows the SIP/report timestamp.

A trade with participant time 09:10:59.800 and SIP time 09:11:00.100 belongs to the 09:11 candle. It never gets moved into 09:10 merely because the execution economically happened earlier.

For cross-source parity, the canonical ordering domain uses millisecond SIP time plus sequence and stable provider identity. Native finer precision is retained for audit, but REST is not allowed to create a different historical ordering rule from the live WS path.

Participant time remains crucial for PM/AH TRF temporal tests.

Ordinary stale TRF and structural backlog are different states

The initial production ordinary PM/AH stale-price rule is:

report age > 500 ms

for a TRF execution reported while that facility was otherwise open.

That veto removes technical price, VWAP and Microcap Engine Current Price contribution, while Volume remains condition-driven.

Structural backlog is a different mechanism. Microcap Engine checks the effective reporting-facility operating calendar. If the execution occurred during a closed/paused facility interval, was disseminated after reopening, and is at least 60 seconds old, the event contributes to none of technical OHLC, FullVolume, TradeCount, FullDollarVolume, VWAP or Microcap Engine Current Price.

The difference matters: a late price report is not automatically fake volume, and an hours-old reopening backlog should not become “activity now.”

Technical FullVolume is intentionally not raw SIP-reported volume

Microcap Engine defines FullVolume as post-canonical technical volume.

For every event that survives the whole-trade gate and finishes with V=YES:

• FullVolume adds exact quantity
• TradeCount adds one disseminated canonical print
• FullDollarVolume adds price × exact quantity

AverageTradeSize is then derived as:

FullVolume / TradeCount

The same additive components are summed into parents and session windows. Microcap Engine does not compute dollar volume as Close × Volume, and it does not average child AverageTradeSize values.

Raw/source evidence can be retained for audit, but it is not a second competing scanner volume truth.

Quantity is exact and presence-sensitive

Where the feed supplies exact fractional quantity, Microcap Engine uses the exact decimal quantity for arithmetic.

The methodology treats missing, null, malformed and out-of-range exact-quantity states differently rather than silently coercing them into a valid integer amount. The February 23, 2026 fractional-quantity source regime is therefore part of the data contract rather than an implementation footnote.

A valid fractional trade with integer size=0 is not zero activity.

VWAP is built from raw eligible executions, not child-bar averages

Microcap Engine keeps exact:

Σ(price × quantity)

and:

Σ(quantity)

for eligible executions.

The product uses a 04:00-anchored VWAP, a 09:30-anchored VWAP and the completed DailyVWAP. Eligibility is anchor-specific, so one global vwapEligible flag is insufficient.

Valid contemporaneous TRF executions are not excluded merely for being TRF. The v1.19 whole-trade drop happens before VWAP, and surviving restrictive VWAP conditions can still veto an anchor.

Higher-timeframe/session VWAP is reconstructed by summing exact numerator and denominator components, never by averaging child VWAP values.

Microcap Engine Current Price is not Raw Last and not candle Close

RawLastTrade, Microcap Engine Current Price, BarClose and PreviousRTHClose are separate concepts.

Microcap Engine Current Price is the latest canonical event whose final eligibility has:

OC=YES AND V=YES

after the condition, session and temporal rules.

It is ordered by the canonical event key rather than whichever worker happened to process a message last.

The value may carry across minute boundaries until a newer eligible execution arrives, but that carry-forward state is never injected back into analytical OHLC, parent highs/lows or historical candles.

Historical corrections and cancels are reconciled before canonical aggregation

The final methodology does not treat every REST lifecycle row as a new trade.

For the REST active-row selector:

• correction state missing, explicit 0 and explicit 1 are kept
• 7, 8, 10, 11 and 12 are dropped
• null, unknown or malformed lifecycle states fail closed for the affected history/recovery scope

This same active-row concept is used for certified historical builds and REST recovery before the canonical reducer.

Microcap Engine intentionally does not maintain a second full LIVE_AS_SEEN historical OHLCV database or a permanent intraday correction ledger solely to reproduce every later cancellation in point-in-time historical research. The point-in-time simulation keeps only information that was actually available at the decision time; later correction/cancel knowledge is not allowed to rewrite that earlier simulated decision. Later certified REST history can therefore legitimately differ.

Ordering, deduplication and recovery are deterministic

Sequence number alone is never a dedupe key.

The live path uses an exact event identity containing session date, event type, symbol, SIP millisecond, sequence, exchange, typed TRF identity and provider trade ID. Duplicate exact identity with equivalent payload is a no-op; a conflicting payload is an integrity problem.

Reconnect/overflow recovery is rebuild-and-REPLACE, not additive replay. A recovery epoch can span multiple reconnects and is not declared healthy merely because the socket reconnected.

Exact REST+WS overlap merging is only allowed after cross-transport identity has been validated. Otherwise the safe fallback gives affected completed minutes exclusively to REST and hands control back to WS at a clean later minute. This avoids fuzzy dedupe.

Realtime close finality is separate from candle ownership

INTRABAR scanner logic can react immediately to canonical state updates.

For CANDLE_CLOSE, the beta contract uses a 100 ms delivery grace plus an ordered finalization barrier. The grace covers provider/network delivery tail; local worker backlog is handled by the barrier rather than by stretching grace.

A late event that belongs to an already evaluated SIP-time candle can repair canonical/chart/dependent state, but it never creates a retroactive trading signal for that old close.

Parent timeframes add no second grace after their final required canonical 1m child is finalized.

Historical coverage is PIT-universe driven and certification-aware

Historical raw ticks come from Massive stock Trades REST, not provider aggregate bars or Massive Flat Files.

The expected historical set is built from the point-in-time product universe, warmup dependencies and the sealed trading calendar. Stable instrument identity is separate from date-effective query symbol, so rename, ticker reuse and delisted cases are not resolved from today's ticker list.

A successful zero-trade day is different from a failed or unfetched day. Pagination must be exhausted, expected symbol-day coverage is exact-set based, and selected historical materializations carry semantic/provenance fingerprints.

Corporate-action adjustment sits above the raw eligibility layer: canonical storage remains raw/unadjusted where specified, while analytical feature views use split/corporate-action state knowable as of the relevant decision time.

Higher timeframes inherit field presence, not display fiction

Canonical 1m Open is the first OC=YES event; High/Low come only from HL=YES events; Close is the last OC=YES event.

If there is no eligible technical price, Microcap Engine does not manufacture O=H=L=C=previous close inside analytical data.

Parents fold the actual nullable canonical child fields:

• first eligible child Open
• maximum eligible child High
• minimum eligible child Low
• last eligible child Close
• sum of exact technical activity components

A UI may choose visual continuity, but display carry-forward never becomes analytical history.

The core invariant is not “make the chart look clean.” It is: the same normalized event sequence, under the same promoted profiles and ordering, must produce the same canonical technical state everywhere that state is consumed.

How Does Microcap Engine Treat Corrections for Real-Time Simulation?

For point-in-time real-time simulation, Microcap Engine does not let later correction/cancel knowledge rewrite an earlier trading decision. Lifecycle rows that were not part of the information set available at that moment are excluded from that simulation path; later REST reconciliation/certified history is kept as a separate post-day truth.

How Are Historical Halts and LULD States Handled?

Microcap Engine also maintains historical trading-halt and LULD state data. That market-state history is kept separate from trade-condition filtering so a backtest can distinguish a genuine trading pause/reopening from an ordinary period with no prints.

The halt/LULD layer therefore provides point-in-time market-state context around the canonical OHLCV series rather than being used as a price-cleaning heuristic.

There Is More Than One Legitimate OHLCV Methodology

The Microcap Engine methodology is not the only defensible way to build candles from U.S. trade data.

A system trying to reproduce a provider's official aggregates, a broker chart, a market-microstructure research feed, or a point-in-time backtest may make different legitimate choices about sale conditions, volume, delayed prints and session treatment. More than one methodology can be “correct” if its objective is explicit, its rules are internally consistent, and the same definitions are applied across live and historical data.

Microcap Engine optimizes for one specific objective: a point-in-time technical history that is consistent across charts, scanners and backtests while preserving legitimate microcap volatility.

Why Can OHLCV Candles Differ Between Data Vendors?

When two data platforms disagree on a high, low or volume value, the immediate assumption is often: one provider must have missing data.

Sometimes that is true. But often the difference is interpretation.

Massive explains that two reconstructions can differ because sale-condition eligibility determines which events update price or volume.

TradingView publicly documents two additional examples in its own North American intraday history: odd-lot filtering and filtering of historical late-print behavior that distorted premarket volume.

So there is no universal law stating that every platform must produce an identical candle from every raw report.

Before comparing OHLCV datasets, the better question is: What market events does each candle actually represent?

Related vendor methodology: Massive - why market data may differ | TradingView intraday data note

Why This Matters for Backtesting

Imagine a premarket strategy that buys when price breaks the premarket high.

A single stale report produces an artificial high at $5.00. The contemporaneous market never trades there during the observed period.

The backtest may now conclude that the breakout never occurred, the entry occurred much later, the stop distance was larger, the stock had a larger premarket range, ATR was higher, or the strategy should not have traded the stock at all.

Now consider the opposite mistake.

A real RDGT-style $5.85 spike is deleted because it looks extreme.

The backtest now studies a market that never existed.

Both errors originate before the strategy code runs.

Data methodology is not just infrastructure. It is part of the trading model.

The Broader Lesson

The biggest surprise from this research was not that raw data contains unusual prints. That is well known.

The bigger lesson was how many decisions are hidden inside something as simple as a candle.

A one-minute OHLCV bar is not a raw market fact. It is the result of a methodology.

Someone has already decided which trades count, which timestamp defines chronology, which conditions affect price, which conditions affect activity, how late reports are handled, what happens when no eligible price exists, and how larger intervals inherit those decisions.

Most traders never see those choices. They see five numbers: O H L C V.

But for serious backtesting - particularly in thin, volatile premarket microcaps - those hidden choices can materially change the research result.

That is why our objective is not to create the cleanest-looking chart. It is to build a candle history that is deterministic, point-in-time defensible, market-structure-aware, and identical everywhere the strategy uses it.

Because before you can test whether a trading strategy has an edge, you first need to know that the market history you are testing actually makes sense.

This research was conducted as part of Microcap Engine’s work on building point-in-time, chart-ready market data for microcap backtesting and scanning.

FAQ

Methodology and Scope Note

This article summarizes the Microcap Engine Canonical OHLCV Aggregation Blueprint v1.19-final-implementation-freeze (2026-09-17) and the validation evidence that led to it.

The ticker examples, trade-level timestamps and validation-corpus statistics in this note come from Microcap Engine internal validation datasets. Public sources below are used for market-data semantics, vendor methodology and regulatory context.

The ANNA and RDGT chart screenshots are five-minute visualizations of the relevant event windows. The underlying validation described in the text was performed against raw trade-level records and timestamps, not inferred from the screenshots alone.

Where this article describes Microcap Engine rules, it describes the frozen canonical methodology. Operational rollout still has explicit validation/certification gates, including broad-corpus regression, WS↔REST identity/visibility testing, production finalization-grace calibration, end-to-end parity/chaos/load validation and the clean historical rebuild.

Selected Public References

• Massive - Stocks Trades WebSocket schema
• Massive - Trade Conditions glossary
• Massive - Why is Massive's market data different from other providers?
• Massive / Polygon - How does Massive create OHLCV aggregate bars?
• FINRA - Regulatory Notice 25-15: TRF operating hours extended to 4:00 a.m. ET effective March 30, 2026
• U.S. SEC - Odd Lot Rates in a Post-Transparency World
• TradingView - Why do the values on the TradingView intraday charts differ from other sources?