A Practical Guide to Fixing Duplicate Tick Data in HK Real-Time API

Introduction

When building quantitative trading systems, market data collectors, and running strategy backtests for Hong Kong stocks, most traders encounter a tricky yet overlooked problem: their calculated volume metrics, tick-based indicators, and candlestick charts never fully match official exchange data.

Most people blame flawed strategy logic at first. In practice, the inconsistency usually stems from duplicated tick records being repeatedly processed in real-time WebSocket data streams.

Redundant trade data distorts volume, turnover ratio and order flow statistics. It creates over-optimistic backtest results, interferes with parameter tuning, and causes inconsistent performance between backtest simulation and live trading. This article breaks down why duplicate HK stock ticks occur, delivers practical deduplication logic, and explains how to stabilize your quantitative research and live data ingestion.

Core Causes of Duplicate Trade Records

Hong Kong equity market data is widely distributed via persistent WebSocket connections. Data travels through network layers, message relays and local consumer services. Duplicate ticks are mainly caused by three common scenarios:

• Network reconnection & message retransmission Brief network drops will trigger server-side data replenishment. Trades executed during the offline window are resent, creating identical duplicate records.
• Process restart state loss If your system does not persist processed trade markers, every program restart will re-consume historical tick data and recalculate finished trades.
• Multi-layer data forwarding without unique identification Market data often passes through multiple gateway modules. Without unique trade identifiers, the same tick may be forwarded and consumed multiple times.

Critical Research Pitfall: Never Deduplicate Only by Timestamp

HK stocks feature ultra-high intra-day matching frequency. Multiple independent trades can share the exact same timestamp. Timestamp-only filtering mistakenly removes valid transactions, creates data gaps, and ruins backtest integrity.

Build Reliable Deduplication Keys

The most accurate solution is using the native unique trade ID provided by real-time APIs.

If native trade IDs are unavailable, generate a unique data fingerprint by combining core trading fields: stock symbol, exact timestamp, trade price and trade volume. Combined MD5 hashing ensures unique identification for every single tick.

```python
import hashlib

def generate_key(trade):
    text = (
        trade["symbol"]
        + str(trade["timestamp"])
        + str(trade["price"])
        + str(trade["volume"])
    )
    return hashlib.md5(text.encode()).hexdigest()

data = {
    "symbol": "00700",
    "timestamp": "2026-08-17 10:30:20",
    "price": "380.50",
    "volume": "300"
}

print(generate_key(data))
```

Note: Insufficient combination fields increase hash collision risks, which misclassify different trades as duplicates and introduce backtest deviations.

WebSocket Stream Deduplication Implementation

For stable quantitative data pipelines, decouple data reception and data calculation. Isolate deduplication as an independent preprocessing module. Only verified, non-repetitive tick data flows to downstream candlestick generation, factor calculation and trading signal logic.

The following practical example uses AllTick API WebSocket subscription for real-time HK market data verification, ideal for strategy prototyping and backtest validation:

```python
import websocket
import json

cache_ids = set()

def on_message(ws, message):
    data = json.loads(message)
    trade_id = data.get("id")
    if trade_id in cache_ids:
        return
    cache_ids.add(trade_id)
    print(
        data.get("symbol"),
        data.get("price"),
        data.get("volume")
    )

ws = websocket.WebSocketApp(
    "wss://apis.alltick.co/ws/stock",
    on_message=on_message
)
ws.run_forever()
```

Important Reminder for Live & Backtest Usage

In-memory caching is only suitable for local testing. High-frequency tick streams cause unbounded memory accumulation. For production-level data collection and large-scale backtesting, use TTL-enabled distributed cache to auto-expire outdated fingerprints and maintain stable system performance.

Two Key Production & Research Pitfalls

Many strategies work perfectly in local tests but produce biased live data due to two neglected details:

Optimize Cache TTL Based on Market Trading Windows

A too-short TTL cannot cover message replenishment periods after network reconnection, leaving residual duplicate data. An over-long TTL accumulates massive invalid fingerprints, raising query and storage costs. Configure TTL according to HK stock trading hours and maximum retransmission windows.

Avoid State Loss From Service Restarts

In-memory cache resets after every restart, causing repeated recalculation of historical ticks. For long-running data collection systems, persist deduplication fingerprints in external cache middleware to decouple trading logic and data verification status.

Conclusion

Data quality fundamentally determines the credibility of quantitative backtesting and live trading performance. Although tick deduplication is a basic preprocessing step, it directly affects volume-based strategies, candlestick accuracy and factor reliability.

Given the high-frequency matching characteristics of the Hong Kong stock market, traders must integrate idempotent data verification into daily data collection and strategy research workflow.

For HK quantitative strategy development, traders can leverage Tick-level streaming data from AllTick API. With standardized data cleaning and deduplication mechanisms, researchers can eliminate hard-to-reproduce data anomalies and significantly improve the stability and authenticity of backtest results and live strategy performance.