Twitter’s Trending Topics in .NET: Real-Time Stream Processing, Locality-Sensitive Hashing, and Geospatial Clustering

1 The 500-Million-Tweet Challenge: Architecting for Velocity and Volume

Every second, thousands of tweets flood the internet — news updates, memes, breaking events, and bots fighting for visibility. To extract meaningful trending topics from this chaos, we need to analyze velocity (how fast something is growing), novelty (whether it’s new or resurfacing), and locality (where it’s happening). This is not a simple COUNT(*) GROUP BY hashtag. At Twitter scale — roughly 500 million tweets per day — traditional analytics systems collapse under the data’s velocity and cardinality.

This section builds the foundation for an end-to-end real-time trend detection system built on Azure and .NET, showing how to handle unbounded streams, probabilistic frequency estimation, and event-time aggregation in a production-grade, cloud-native way.

When developers first attempt to identify trending topics, the instinctive approach is to aggregate hashtags over time:

SELECT hashtag, COUNT(*) AS Count
FROM Tweets
GROUP BY hashtag
ORDER BY Count DESC

This naive approach breaks immediately at scale for several reasons:

Velocity Matters: A hashtag that consistently appears (like #MondayMotivation) may always top the count, but trends are about momentum — how quickly mentions rise over short windows.
Novelty Counts: A new hashtag suddenly gaining traction is more meaningful than one that’s always popular. We need to detect the rate of change, not just raw totals.
Locality Is Key: A hashtag might trend in Tokyo but not in Toronto. Without location-aware aggregation, we lose the regional nuance that defines “trending.”

Trending is a derivative problem — it’s about acceleration. That means we need temporal context (what changed in the last few minutes) and spatial context (where it changed).

At small scales, you could store everything in a relational database and run hourly aggregations. But for hundreds of millions of tweets, those approaches can’t keep up. To handle the data firehose, we need streaming analytics — systems that process data as it arrives, continuously updating trend scores in real time.

1.2 The Scale Challenge: Ingesting 500M Tweets/Day

Let’s break down the math before diving into architecture.

500 million tweets per day = ~5,800 tweets per second on average.
Peak events (sports finals, elections, celebrity news) can spike beyond 100,000 tweets per second.
Average payload size: 1–2 KB per tweet including metadata. → At peak load, the system ingests over 200 MB per second.

If we tried to store every tweet and compute exact counts per hashtag, we’d quickly hit impossible memory and compute limits:

10 million unique hashtags/day would require millions of counters.
Each counter (64-bit integer) consumes 8 bytes → 80 MB per minute in just counters.
Keeping even 24 hours of data means terabytes of hot storage.

Running exact-count queries on this firehose is infeasible. Instead, we need approximate, streaming-friendly data structures — like Count-Min Sketch — to estimate frequencies in constant time and space. Combined with Azure’s event ingestion and PaaS compute services, we can scale to billions of events with minimal infrastructure management.

1.3 The Solution Blueprint: End-to-End Serverless and PaaS Architecture

The goal is a fully managed, event-driven architecture that can handle tweet ingestion, near-real-time analytics, and visual reporting — without relying on custom servers or manual scaling. The Azure ecosystem provides an ideal backbone for this.

1.3.1 Ingest: Twitter API → Azure Event Hubs

Azure Event Hubs serves as the ingestion layer — a distributed event streaming platform capable of millions of events per second.

Why Event Hubs?
- Native partitioning for horizontal scalability.
- Built-in checkpointing for reliable replay.
- Integration with Stream Analytics, Azure Functions, and Databricks.

Each tweet (in JSON form) is pushed to an Event Hub partition based on a key such as tweet.id % partitionCount, ensuring load distribution. Event producers can be a small .NET service pulling from the Twitter Streaming API:

var producer = new EventHubProducerClient("<connection-string>", "tweets");
using EventDataBatch eventBatch = await producer.CreateBatchAsync();

foreach (var tweet in tweets)
{
    eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(tweet))));
}
await producer.SendAsync(eventBatch);

This setup buffers tweets efficiently and can handle transient API interruptions gracefully.

1.3.2 Process: Event Hubs → Azure Stream Analytics / Azure Functions

Azure Stream Analytics (ASA) is a fully managed stream processing engine that lets us express complex logic using SQL-like syntax (SAQL). It’s built for real-time aggregations, windowing, and anomaly detection.

Typical use:

Filter out non-English or spam tweets.
Extract hashtags, user mentions, and coordinates.
Compute per-minute aggregates using tumbling windows.

Example:

SELECT
    hashtag,
    COUNT(*) AS Count,
    System.Timestamp AS WindowEnd
INTO
    TrendingHashtags
FROM
    TweetStream TIMESTAMP BY CreatedAt
GROUP BY
    TumblingWindow(minute, 1), hashtag

For custom logic (like Locality-Sensitive Hashing or advanced ML inference), Azure Functions acts as a complementary processor. It integrates natively with Event Hubs and can scale elastically.

1.3.3 Analyze & Serve: Cosmos DB + Azure Maps + Power BI

Processed results (trending topics, counts, geolocations) are persisted in Azure Cosmos DB, which supports:

Low-latency reads (<10 ms typical).
Global distribution, enabling regional dashboards.
Native geospatial types for trend mapping.

Cosmos DB stores documents like:

{
  "hashtag": "#Elections2025",
  "region": "New York",
  "trendScore": 87.3,
  "coordinates": { "lat": 40.7128, "lon": -74.0060 },
  "timestamp": "2025-11-10T14:45:00Z"
}

Visualization layers:

Azure Maps: Interactive web visualizations for geospatial clustering.
Power BI Streaming Dataset: Dashboard integration for business teams to monitor real-time trends.

1.3.4 Archive: Azure Data Lake Storage Gen2

For historical trend analysis, model training, and offline reprocessing, all tweets and intermediate outputs are archived to Azure Data Lake Storage (ADLS) Gen2. ADLS integrates seamlessly with Azure Synapse and Fabric, allowing batch queries using T-SQL, Spark, or Kusto Query Language (KQL).

Example of batch processing in Synapse:

SELECT hashtag, COUNT(*) AS DailyCount
FROM OPENROWSET(
    BULK 'https://trendslake.blob.core.windows.net/tweets/2025/11/10/*.json',
    FORMAT='csv', FIELDTERMINATOR=',', FIELDQUOTE='"'
) AS Tweets
GROUP BY hashtag;

By separating ingestion, processing, serving, and archiving layers, this architecture achieves elastic scalability, fault isolation, and data lifecycle efficiency — the foundation for the next stages: probabilistic frequency estimation and temporal velocity analysis.

2 Probabilistic Power: Frequency Estimation with Count-Min Sketch in C#

2.1 Why Exact Counts Fail

In a traditional relational system, counting hashtag frequencies requires storing each distinct tag as a key and incrementing a counter. But at Twitter scale:

The number of unique tokens (hashtags, n-grams, URLs) can exceed hundreds of millions daily.
Memory usage grows linearly with the number of unique keys.
Maintaining distributed counters causes locking and network contention.

Even distributed systems like Redis or Cassandra struggle with unbounded keys. You either drop old data (losing accuracy) or shard aggressively (increasing complexity).

Approximation algorithms like Count-Min Sketch (CMS) solve this elegantly: They provide constant time and space frequency estimation with bounded error, using a 2D matrix of counters and multiple hash functions. For stream analytics, this is the difference between “impossible” and “real-time.”

2.2 Concept Deep Dive: The Count-Min Sketch (CMS) Algorithm

CMS maintains a small 2D array of integers with dimensions d × w, where:

d = number of hash functions (rows).
w = width (number of counters per row).

Each hash function maps an item to one column in its row.

2.2.1 Adding an Item

For each item x, we hash it with all d functions and increment the counters:

for i in 1..d:
    count[i, hash_i(x)] += 1

2.2.2 Estimating Frequency

To estimate the frequency of x, we take the minimum count across all hash rows:

estimate(x) = min(count[i, hash_i(x)] for i in 1..d)

This guarantees:

No underestimation (never smaller than the true count).
Small overestimation due to hash collisions, bounded by configuration parameters.

2.2.3 Error and Space Complexity

Given parameters ε (error) and δ (confidence):

Width w = ceil(e / ε)
Depth d = ceil(ln(1 / δ))
Space complexity: O(w * d) (constant, independent of number of unique items)
Time per update/query: O(d) (constant)

For example, with 1% error (ε=0.01) and 99% confidence (δ=0.01), CMS requires only:

w ≈ 272, d ≈ 5 → ~1,360 counters total.

That’s trivial memory for millions of unique items.

2.3 Practical .NET Implementation: Building a `CountMinSketch<T>` Class

We can build a reusable, type-safe C# implementation for general-purpose frequency estimation.

2.3.1 Basic Structure

public class CountMinSketch<T>
{
    private readonly int _width;
    private readonly int _depth;
    private readonly int[][] _table;
    private readonly Func<T, int>[] _hashFunctions;

    public CountMinSketch(int width, int depth)
    {
        _width = width;
        _depth = depth;
        _table = new int[depth][];
        _hashFunctions = new Func<T, int>[depth];

        var rand = new Random();
        for (int i = 0; i < depth; i++)
        {
            int seed = rand.Next();
            _table[i] = new int[width];
            _hashFunctions[i] = item => (item.GetHashCode() ^ seed) & int.MaxValue % width;
        }
    }

    public void Add(T item, int count = 1)
    {
        for (int i = 0; i < _depth; i++)
        {
            int index = _hashFunctions[i](item) % _width;
            _table[i][index] += count;
        }
    }

    public int Estimate(T item)
    {
        int min = int.MaxValue;
        for (int i = 0; i < _depth; i++)
        {
            int index = _hashFunctions[i](item) % _width;
            min = Math.Min(min, _table[i][index]);
        }
        return min;
    }
}

2.3.2 Example Usage

var cms = new CountMinSketch<string>(width: 500, depth: 5);

cms.Add("#Azure");
cms.Add("#Azure");
cms.Add("#DotNet");

Console.WriteLine(cms.Estimate("#Azure"));  // ≈ 2
Console.WriteLine(cms.Estimate("#DotNet")); // ≈ 1

This structure fits naturally into a streaming environment, allowing us to track millions of hashtags in a few kilobytes.

2.4 Integration: Using CMS Inside Azure Stream Analytics with C# UDFs

Azure Stream Analytics supports C# User-Defined Functions (UDFs), allowing you to extend SAQL with custom logic.

2.4.1 Registering the Function

In the ASA job, upload the compiled assembly containing the CountMinSketch class. Then register the function:

CREATE FUNCTION CountEstimate
ASSEMBLY 'Trends.CountMinSketch.dll'
TYPE 'Trends.CountMinSketch.Functions'

2.4.2 Using the Function in a Query

SELECT
    hashtag,
    UDF.CountEstimate(hashtag) AS ApproxCount,
    System.Timestamp AS WindowEnd
INTO
    TrendingHashtags
FROM
    TweetStream TIMESTAMP BY CreatedAt
GROUP BY
    TumblingWindow(minute, 1), hashtag

By combining CMS with windowed aggregation, ASA can maintain approximate counts without materializing huge state tables. The trade-off — small bounded overcount — is acceptable for real-time trend scoring, where ranking relative magnitudes matters more than exact values.

3 Finding the “Now”: Temporal Trend Detection with ASA Tumbling Windows

3.1 The Power of Stream Analytics Query Language (SAQL): Thinking in Time Windows

In stream processing, data is unbounded — it never stops arriving. Instead of static tables, we think in windows: slices of time that represent “current context.” Azure Stream Analytics Query Language (SAQL) provides SQL-like semantics for expressing temporal logic over event streams.

When we ask, “What’s trending now?”, we’re really saying, “What has changed within the last N minutes?” This requires:

Aggregating over fixed time intervals.
Comparing metrics between consecutive windows.
Deriving velocity (rate of change).

3.2 Concept Deep Dive: Tumbling vs. Hopping vs. Sliding Windows

Stream Analytics supports three main window types:

Window Type	Description	Example Use Case
Tumbling	Fixed, non-overlapping intervals (e.g., every minute)	Calculating per-minute counts
Hopping	Overlapping intervals with a hop size smaller than window length	Smoothing rolling averages
Sliding	Updates continuously whenever an event arrives	Precise latency monitoring

For trending topics, tumbling windows are ideal — each minute forms a discrete, complete snapshot of activity. This avoids double-counting tweets and simplifies velocity calculations between adjacent intervals.

3.3 Practical SAQL: Finding Volume and Velocity

We’ll now combine everything into a working example.

3.3.1 Step 1: Get Volume

Compute total mentions per hashtag for each minute:

WITH HashtagVolume AS (
    SELECT
        hashtag,
        COUNT(*) AS Volume,
        System.Timestamp AS WindowEnd
    FROM
        TweetStream TIMESTAMP BY CreatedAt
    GROUP BY
        TumblingWindow(minute, 1), hashtag
)
SELECT * INTO VolumeOutput FROM HashtagVolume;

3.3.2 Step 2: Calculate Velocity

Use LAG() to compare current and previous windows:

WITH TrendVelocity AS (
    SELECT
        hashtag,
        Volume,
        LAG(Volume, 1) OVER (PARTITION BY hashtag LIMIT DURATION(minute, 2)) AS PrevVolume,
        System.Timestamp AS WindowEnd
    FROM
        VolumeOutput
)
SELECT
    hashtag,
    Volume,
    (Volume - ISNULL(PrevVolume, 0)) AS Velocity,
    WindowEnd
INTO
    VelocityOutput;

3.3.3 Step 3: Compute Trend Score

Blend absolute volume with relative growth to define “trending” strength:

SELECT
    hashtag,
    (Volume * 0.2 + Velocity * 0.8) AS TrendScore,
    WindowEnd
INTO
    TrendingHashtags
FROM
    VelocityOutput
WHERE
    Volume > 5 AND Velocity > 0;

This scoring formula weights velocity (change rate) more heavily than absolute count, which helps surface emerging topics early. You can tune the coefficients dynamically based on data volatility.

4 De-duping the Stream: Near-Duplicate Detection with Locality-Sensitive Hashing (LSH)

When analyzing massive, fast-moving streams like Twitter data, duplicates are inevitable. Retweets, quote tweets, and slightly modified reposts all refer to the same content but appear as distinct text. Without handling these near-duplicates, trend detection becomes skewed—one viral post replicated across hundreds of minor variations can overpower truly unique trends. The challenge is recognizing textual similarity at scale without introducing expensive pairwise comparisons or fuzzy matching logic that can’t keep up with the stream velocity.

4.1 The “Retweet” and “Slight Variation” Problem

Imagine three tweets arriving within milliseconds:

“Elon Musk launches new rocket.”
“Elon Musk launches new rocket!”
“Breaking: Elon Musk launches a rocket 🚀.”

Each is technically different in text, but they’re semantically identical. In practice, users retweet and modify phrasing slightly—adding punctuation, emojis, or short prefixes like “RT” or “BREAKING.” To our counting system, these appear as unique tokens. Without de-duplication, frequency metrics inflate artificially, giving an illusion of higher engagement for a topic.

Exact string matching (tweet.Text == previous.Text) fails immediately. Cosine similarity or edit distance (Levenshtein) are better but computationally heavy: O(n²) comparisons for a set of n tweets per window. At hundreds of thousands per second, this is infeasible.

The solution is Locality-Sensitive Hashing (LSH), an approximate technique for quickly identifying similar items by hashing them into the same bucket. Unlike traditional hash functions that maximize entropy (ensuring dissimilar inputs produce distinct outputs), LSH intentionally increases the chance that similar inputs share a hash. This allows us to detect near-duplicates efficiently.

4.2 Concept Deep Dive: Locality-Sensitive Hashing (LSH)

Locality-Sensitive Hashing works by projecting items into a lower-dimensional space using hash functions that preserve similarity. If two items are close in the original feature space, they’re likely to collide in at least one LSH bucket. This property enables scalable approximate nearest-neighbor search or duplicate detection in high-dimensional datasets.

4.2.1 From Text to Shingles

Before applying LSH, we convert tweets into shingles—fixed-length sequences of characters or words. For example, with word-level 3-shingles:

Tweet: "Elon Musk launches new rocket" Shingles: [Elon Musk launches, Musk launches new, launches new rocket]

Each shingle is hashed to an integer. The set of hashed shingles forms the document signature.

4.2.2 MinHash for Text Similarity

The MinHash algorithm approximates the Jaccard similarity between two sets:

J(A, B) = |A ∩ B| / |A ∪ B|

Instead of comparing sets directly, we compute a small signature vector that statistically preserves this ratio. For each of k hash functions, MinHash records the smallest hash value of the document’s shingles. Two documents with similar content will produce similar MinHash signatures.

Here’s a simplified C# implementation for generating MinHash signatures:

public static class MinHashGenerator
{
    public static int[] ComputeSignature(IEnumerable<int> shingles, int numHashFunctions)
    {
        var rand = new Random();
        var signatures = new int[numHashFunctions];
        for (int i = 0; i < numHashFunctions; i++)
        {
            int a = rand.Next(1, int.MaxValue);
            int b = rand.Next(0, int.MaxValue);
            int minHash = int.MaxValue;
            foreach (var shingle in shingles)
            {
                int hash = (a * shingle + b) & int.MaxValue;
                if (hash < minHash)
                    minHash = hash;
            }
            signatures[i] = minHash;
        }
        return signatures;
    }

    public static double JaccardEstimate(int[] sigA, int[] sigB)
    {
        int match = sigA.Zip(sigB, (x, y) => x == y).Count(equal => equal);
        return (double)match / sigA.Length;
    }
}

Example usage:

var textA = "Elon Musk launches new rocket";
var textB = "Elon Musk launches a new rocket!";
var shinglesA = GetWordShingles(textA, 3);
var shinglesB = GetWordShingles(textB, 3);

var sigA = MinHashGenerator.ComputeSignature(shinglesA, 100);
var sigB = MinHashGenerator.ComputeSignature(shinglesB, 100);

Console.WriteLine(MinHashGenerator.JaccardEstimate(sigA, sigB)); // ~0.85

A similarity score above a configurable threshold (say 0.8) can be treated as a duplicate.

4.2.3 Banding and LSH Buckets

To avoid comparing every pair of signatures, we use banding. We divide each signature into b bands of r rows. Each band is hashed into a bucket. Documents sharing at least one band hash are considered potential duplicates.

This drastically reduces comparisons while retaining high recall for similar items. The probability of two documents being grouped depends on their similarity and the chosen b and r values—a tunable trade-off between precision and recall.

For streaming data, each new tweet is hashed into LSH buckets. If a near-identical signature already exists, we treat it as a duplicate and increment the corresponding hashtag’s counter rather than creating a new entry.

4.3 The Integration Pattern: The “ASA → Function → ASA” Decoupling

While Azure Stream Analytics excels at windowing and aggregation, it’s not designed for heavy in-memory operations like MinHashing or shingle comparisons. Implementing LSH directly as a UDF would overload the streaming job, increasing latency and state size. Instead, we decouple the flow into a multi-stage architecture that combines ASA and Azure Functions.

4.3.1 Architecture Overview

ASA Job 1 – Initial filtering and projection Extract hashtags, language, and cleaned text fields; push cleaned tweets to Event Hub 2.
Azure Function (LSH Processor) – Deduplication Implements MinHash and LSH logic in C#, marking tweets as duplicates or originals.
ASA Job 2 – Trend analytics Consumes deduplicated stream from Event Hub 3, running Count-Min Sketch and tumbling-window logic as before.

4.3.2 Azure Function Implementation Example

public static class DeduplicationFunction
{
    private static readonly LshEngine _lsh = new LshEngine(100, 20); // 100 hash funcs, 20 bands

    [FunctionName("DeduplicateTweets")]
    public static async Task Run(
        [EventHubTrigger("tweets-cleaned", Connection = "EventHubConnection")] EventData[] events,
        [EventHub("tweets-deduped", Connection = "EventHubConnection")] IAsyncCollector<string> output)
    {
        foreach (var e in events)
        {
            var tweet = JsonConvert.DeserializeObject<Tweet>(Encoding.UTF8.GetString(e.EventBody));
            bool isDuplicate = _lsh.IsDuplicate(tweet.Text);

            if (!isDuplicate)
            {
                _lsh.Add(tweet.Text);
                await output.AddAsync(JsonConvert.SerializeObject(tweet));
            }
        }
    }
}

The LshEngine maintains a rolling in-memory cache of recent tweet signatures. To prevent unbounded growth, we apply a sliding expiration (e.g., retain 10 minutes of history). This balances deduplication accuracy with memory footprint.

4.3.3 Event Flow

Event Hub 1 → ASA Job 1: Parse raw tweets into structured form.
ASA Job 1 → Event Hub 2: Emit cleaned events.
Event Hub 2 → Azure Function: Apply LSH deduplication.
Function → Event Hub 3: Output unique tweets only.
Event Hub 3 → ASA Job 2: Compute frequency, velocity, and trend scores.

This pattern isolates the computationally heavy text-similarity logic inside Functions, where horizontal scaling is automatic and independent of Stream Analytics job throughput. The result is a robust, serverless pipeline that maintains low latency even under massive bursts of traffic.

5 Fighting the Noise: Real-Time Spam/Bot Detection with ML.NET

Even with deduplication, we’re still vulnerable to coordinated spam or bot activity. A malicious botnet can flood the stream with identical or near-identical hashtags, mimicking organic growth. The challenge is differentiating between genuine, viral interest and synthetic manipulation in real time. That’s where machine learning comes in.

5.1 The Problem

Traditional filters—like IP rate limiting or identical text matching—don’t work well in a distributed network like Twitter’s ecosystem. Bots can distribute across accounts, vary message structures slightly, and schedule posts to appear natural.

From an analytics perspective, the symptom is a sudden, unnatural spike in mentions for a single hashtag or topic, typically without corresponding diversity in text or location. If we don’t detect and penalize such anomalies, our trending algorithm over-represents noise as legitimate popularity.

The solution is to introduce a real-time anomaly detection layer. It evaluates historical mention rates per hashtag and flags statistically abnormal spikes. In practice, we can accomplish this efficiently with ML.NET’s built-in time-series anomaly detection capabilities.

5.2 The Tool: Using ML.NET for Anomaly Detection

ML.NET is Microsoft’s open-source, cross-platform machine learning framework for .NET developers. It includes several ready-to-use algorithms, including DetectIidSpike, which identifies unexpected increases or drops in numerical series.

For streaming data, we can feed ML.NET the rolling counts per hashtag (from our 1-minute tumbling windows). The model continuously evaluates whether the most recent value deviates significantly from historical patterns.

Install the required package:

dotnet add package Microsoft.ML

5.3 Practical C# Implementation: `DetectIidSpike`

Below is a self-contained example that demonstrates how to detect anomalies in real time using a moving window of counts.

using Microsoft.ML;
using Microsoft.ML.Data;

public class TweetCount
{
    public float Count { get; set; }
}

public class SpikePrediction
{
    [VectorType(3)]
    public double[] Prediction { get; set; }
}

public class AnomalyDetector
{
    private readonly MLContext _mlContext;
    private readonly ITransformer _model;
    private readonly int _historySize;

    public AnomalyDetector(int historySize = 30)
    {
        _mlContext = new MLContext();
        _historySize = historySize;
        var dataView = _mlContext.Data.LoadFromEnumerable(new List<TweetCount>());
        var pipeline = _mlContext.Transforms.DetectIidSpike(
            outputColumnName: nameof(SpikePrediction.Prediction),
            inputColumnName: nameof(TweetCount.Count),
            confidence: 95,
            pvalueHistoryLength: _historySize);
        _model = pipeline.Fit(dataView);
    }

    public bool IsSpike(List<float> values)
    {
        var data = values.Select(v => new TweetCount { Count = v });
        var transformed = _model.Transform(_mlContext.Data.LoadFromEnumerable(data));
        var results = _mlContext.Data.CreateEnumerable<SpikePrediction>(transformed, false).ToList();
        return results.Last().Prediction[0] == 1; // 1 indicates spike
    }
}

Example usage:

var detector = new AnomalyDetector();
var counts = new List<float> { 10, 12, 13, 11, 14, 13, 60 }; // last value spikes
bool spikeDetected = detector.IsSpike(counts);
Console.WriteLine(spikeDetected); // true

In production, we maintain a short history of counts for each hashtag, fed from the 1-minute tumbling window output. If a spike is detected, we compute a spam score or anomaly penalty.

5.4 Integration: Calling the ML.NET Model

There are two ways to integrate this anomaly detection into our streaming architecture, depending on latency and complexity requirements.

5.4.1 Option 1 – Embedded in the LSH Azure Function

We can extend the same DeduplicationFunction from section 4 to include anomaly detection. Each time a new window completes, we update the history for that hashtag and check for anomalies:

public static class TrendFunction
{
    private static readonly AnomalyDetector _detector = new AnomalyDetector();
    private static readonly Dictionary<string, List<float>> _history = new();

    [FunctionName("DetectAnomalies")]
    public static async Task Run(
        [EventHubTrigger("trends-input", Connection = "EventHubConnection")] EventData[] events,
        [EventHub("trends-output", Connection = "EventHubConnection")] IAsyncCollector<string> output)
    {
        foreach (var e in events)
        {
            var record = JsonConvert.DeserializeObject<TrendRecord>(Encoding.UTF8.GetString(e.EventBody));

            if (!_history.ContainsKey(record.Hashtag))
                _history[record.Hashtag] = new List<float>();

            var list = _history[record.Hashtag];
            list.Add(record.Volume);

            if (list.Count > 30) list.RemoveAt(0);

            bool isSpike = _detector.IsSpike(list);
            record.SpamScore = isSpike ? 1.0 : 0.0;

            await output.AddAsync(JsonConvert.SerializeObject(record));
        }
    }
}

This pattern keeps spam detection near the data source. Because Azure Functions scale automatically, the cost of running ML.NET in-process is manageable.

5.4.2 Option 2 – Integrated with Azure Stream Analytics

Azure Stream Analytics offers a built-in ANOMALYDETECTION function that detects outliers directly within SAQL queries. It’s simpler, though less flexible than ML.NET. Example:

SELECT
    hashtag,
    ANOMALYDETECTION_SpikeAndDip(Volume, 95, 60) OVER (PARTITION BY hashtag LIMIT DURATION(hour, 1)) AS SpikeResult,
    System.Timestamp AS WindowEnd
INTO
    SpamDetectionOutput
FROM
    TrendingHashtags;

This detects statistically abnormal changes based on historical values over a one-hour sliding window. For smaller pipelines or prototypes, this approach avoids maintaining a separate ML model entirely.

5.4.3 Applying the Spam Penalty

Once we have a spam score (0 for normal, 1 for likely spam), we adjust the final trend score used for ranking:

record.FinalScore = (record.Volume * 0.2) + (record.Velocity * 0.8) - (record.SpamScore * 50);

In Stream Analytics, the same logic can be applied declaratively:

SELECT
    hashtag,
    (Volume * 0.2 + Velocity * 0.8 - SpamScore * 50) AS FinalScore,
    WindowEnd
INTO
    CleanTrendingHashtags
FROM
    SpamDetectionOutput;

This ensures that artificially inflated counts are penalized immediately in the ranking pipeline, maintaining data integrity for dashboards and alerts.

6 “Where is this Happening?”: Geospatial Clustering with Azure Maps

Understanding what’s trending is only half the story. The real question many teams ask next is: Where is it happening? A hashtag might surge globally, but its origins often reveal much more—regional sentiment, event proximity, or even coordinated campaigns. To unlock this insight, we extend our real-time pipeline into geospatial analysis, enriching tweets with coordinates and visualizing them on an interactive map.

This section focuses on how to transform textual or partial geographic data into usable coordinates, how to store and query geospatial data efficiently, and how to visualize live trend clusters using Azure Maps and Cosmos DB.

6.1 From Text to Coordinates: The Data Challenge

Twitter’s geolocation fields are inconsistent by design. Depending on user privacy settings and tweet context, location data can come from several sources:

geo field – Precise latitude and longitude coordinates (rare, often disabled by users).
place field – A bounding box defining a region or city.
user.location field – Free-form text (“NYC”, “Bangalore, IN”, “Somewhere on Earth”).

Each source needs different parsing logic. A robust pipeline needs to handle all three gracefully, fallback when data is missing, and still perform efficiently at streaming scale.

Here’s an example of a tweet payload that includes different location possibilities:

{
  "id": "17502354321001",
  "text": "Massive crowd at Times Square for New Year's Eve! #NYC2025",
  "geo": { "coordinates": [40.758, -73.9855] },
  "place": {
    "full_name": "New York, NY",
    "bounding_box": {
      "coordinates": [[[-74.0267, 40.6839], [-73.9104, 40.6839], [-73.9104, 40.8771], [-74.0267, 40.8771]]]
    }
  },
  "user": { "location": "Manhattan" }
}

We start with the most precise data available. If the geo field exists, we use it directly. If not, we approximate using the centroid of the place.bounding_box. Finally, if only the text location exists, we feed it through the Azure Maps Search API to resolve it into coordinates.

Here’s the pseudocode to illustrate this extraction logic in C#:

public static (double lat, double lon)? ExtractCoordinates(Tweet tweet)
{
    if (tweet.Geo?.Coordinates != null)
        return (tweet.Geo.Coordinates[0], tweet.Geo.Coordinates[1]);

    if (tweet.Place?.BoundingBox?.Coordinates != null)
    {
        var box = tweet.Place.BoundingBox.Coordinates.First();
        double lat = box.Average(c => c[1]);
        double lon = box.Average(c => c[0]);
        return (lat, lon);
    }

    if (!string.IsNullOrWhiteSpace(tweet.User?.Location))
        return GeocodeLocation(tweet.User.Location);

    return null;
}

This hybrid approach keeps the data pipeline resilient and flexible. Each layer of fallback maintains partial precision without breaking the flow.

6.2 The Geocoding Pipeline

The Azure Maps Search API provides the missing link between text and geospatial data. It translates a human-readable location like “SF” or “Tokyo” into latitude and longitude. The endpoint we use is:

GET https://atlas.microsoft.com/search/address/json?api-version=1.0&query={location}

For scalability, we handle geocoding asynchronously inside an Azure Function, backed by the same Event Hub stream used in earlier stages. The function batches incoming location lookups to minimize API calls and respects the Azure Maps rate limits.

public static class GeocodeFunction
{
    private static readonly HttpClient Http = new HttpClient();
    private static readonly string AzureMapsKey = Environment.GetEnvironmentVariable("AZURE_MAPS_KEY");

    [FunctionName("GeocodeLocations")]
    public static async Task Run(
        [EventHubTrigger("tweets-for-geocode", Connection = "EventHubConnection")] EventData[] events,
        [EventHub("tweets-geocoded", Connection = "EventHubConnection")] IAsyncCollector<string> output)
    {
        foreach (var e in events)
        {
            var tweet = JsonConvert.DeserializeObject<Tweet>(Encoding.UTF8.GetString(e.EventBody));
            if (tweet.Geo == null && tweet.Place == null && tweet.User?.Location != null)
            {
                var (lat, lon) = await GeocodeAsync(tweet.User.Location);
                tweet.Latitude = lat;
                tweet.Longitude = lon;
            }
            await output.AddAsync(JsonConvert.SerializeObject(tweet));
        }
    }

    private static async Task<(double, double)> GeocodeAsync(string location)
    {
        string url = $"https://atlas.microsoft.com/search/address/json?api-version=1.0&subscription-key={AzureMapsKey}&query={Uri.EscapeDataString(location)}";
        var json = await Http.GetStringAsync(url);
        dynamic result = JsonConvert.DeserializeObject(json);
        var pos = result.results?[0]?.position;
        return (pos.lat ?? 0.0, pos.lon ?? 0.0);
    }
}

Each geocoded tweet is enriched with its coordinates and forwarded to the next Event Hub. Downstream analytics jobs can now perform spatial aggregations and clustering using precise geometry, instead of brittle string matching.

Batching and caching are essential here. Geocoding the same city thousands of times per minute wastes API calls. You can store recent resolutions in Azure Cache for Redis with a 24-hour TTL to reduce lookup overhead.

6.3 The Clustering & Visualization Pipeline

Once each tweet carries a latitude and longitude, the next step is to store and visualize them. Azure Cosmos DB supports native geospatial indexing through the Point type, making it ideal for storing location-based data at scale.

We define a document schema optimized for querying by location and trend score:

{
  "id": "hashtag_#NYE2025_2025-12-31T23:59:00Z",
  "hashtag": "#NYE2025",
  "trendScore": 87.5,
  "location": { "type": "Point", "coordinates": [-73.9855, 40.758] },
  "region": "New York",
  "timestamp": "2025-12-31T23:59:00Z"
}

To support visualization, we expose Cosmos DB data through the Azure Maps Web SDK. This SDK supports multiple layers, including bubble maps and heatmaps, ideal for showing intensity and clustering.

Here’s an example of a Blazor WebAssembly component that renders the heatmap layer:

@page "/trends-map"
@inject IJSRuntime JS

<h3>Trending Topics Map</h3>
<div id="map" style="height:600px;width:100%;"></div>

@code {
    protected override async Task OnAfterRenderAsync(bool firstRender)
    {
        if (firstRender)
        {
            await JS.InvokeVoidAsync("initAzureMap");
        }
    }
}

The associated JavaScript (referenced in _Host.cshtml) initializes the map:

async function initAzureMap() {
    const map = new atlas.Map('map', {
        center: [-95, 40],
        zoom: 3,
        authOptions: {
            authType: 'subscriptionKey',
            subscriptionKey: '<AZURE_MAPS_KEY>'
        }
    });

    const datasource = new atlas.source.DataSource();
    map.sources.add(datasource);

    const heatmap = new atlas.layer.HeatMapLayer(datasource, null, {
        radius: 10,
        color: ['#00f', '#0f0', '#ff0', '#f00'],
        intensity: 0.8
    });
    map.layers.add(heatmap);

    // Load initial data from Cosmos DB REST API
    const response = await fetch('/api/trends');
    const trends = await response.json();
    trends.forEach(t => datasource.add(new atlas.data.Feature(
        new atlas.data.Point([t.location.coordinates[0], t.location.coordinates[1]]),
        { hashtag: t.hashtag, score: t.trendScore }
    )));
}

The resulting dashboard visualizes the density of trending hashtags in near real time. As Azure Functions push new trend records into Cosmos DB, the frontend refreshes through WebSocket or SignalR updates.

In production, clustering can also be handled server-side. Cosmos DB supports spatial queries like “find all trends within 50 km of a point,” enabling geo-filtered views directly in the query layer.

7 Advanced Geospatial Analysis with Kusto Query Language (KQL)

Once you’ve mapped live activity, the next challenge is exploring the spatial and temporal relationships within that data. Architects and analysts often need to ask higher-order questions—queries that combine geography, time, and metadata in one operation. Azure Data Explorer (ADX), also known as Kusto, brings this capability to the pipeline. It’s optimized for high-ingest, low-latency analytical queries on structured and geospatial data.

7.1 Beyond Simple Maps: Answering Deeper Questions

With Kusto, we can move beyond visualization and into true spatial reasoning. Example scenarios include:

Event proximity: “Show me all trends within 50 km of a specific coordinate where a major concert occurred.”
Regional overlap: “Which hashtags were trending within my defined sales region polygon?”
Comparative intensity: “How do trends in Tokyo differ from Osaka over the last 10 minutes?”

These questions require fast joins between dynamic spatial datasets and historical trends, something KQL excels at. ADX maintains compressed columnar storage with built-in support for geometry functions.

A simple example query looks like this:

Trends
| where Timestamp > ago(10m)
| where geo_distance_2points(Latitude, Longitude, 35.6895, 139.6917) < 50000
| summarize count() by Hashtag
| top 10 by count_

This query finds the top 10 trending hashtags within 50 km of Tokyo in the last 10 minutes, executing in milliseconds even over millions of rows.

7.2 The Modern Data Stack: ASA → Azure Data Explorer (ADX/Kusto)

Azure Stream Analytics can output directly into ADX, creating a near-real-time analytical pipeline without additional ETL. The connection is configured as an output sink in ASA:

OUTPUT TO "adx_trends"
WITH (Datasource = "ADX",
      Database = "TrendAnalytics",
      Table = "Trends",
      Cluster = "adxcluster.eastus.kusto.windows.net")

This integration allows the same tumbling window outputs we built earlier to stream continuously into ADX, where analysts can run advanced queries without impacting real-time ingestion.

ADX’s native geospatial support includes:

geo_point_to_s2cell() for spatial clustering.
geo_point_in_polygon() for region containment.
geo_distance_2points() for distance filtering.

Each function operates directly on coordinate fields, leveraging spatial indexes for sub-second response times.

7.3 Practical KQL: Geospatial Clustering on the Fly

To cluster geospatial data dynamically, we use the S2 geometry system, which divides Earth’s surface into hierarchical cells. KQL exposes this via geo_point_to_s2cell(latitude, longitude, level).

Here’s a practical example:

Trends
| where Timestamp > ago(30m)
| extend s2cell = geo_point_to_s2cell(Longitude, Latitude, 8)
| summarize Count = count(), AvgScore = avg(TrendScore) by s2cell
| extend center = geo_s2cell_to_central_point(s2cell)
| project Latitude = todouble(center.lat), Longitude = todouble(center.lon), Count, AvgScore
| render scatterchart with(kind=map)

Each S2 cell represents a region of roughly 3–4 km² at level 8. The query groups nearby tweets into spatial buckets, calculates aggregate scores, and visualizes the results directly on the map interface inside ADX or Power BI.

We can also apply polygonal filtering for region-specific analytics:

let region = dynamic({"type":"Polygon","coordinates":[[[139.6,35.5],[139.9,35.5],[139.9,35.8],[139.6,35.8],[139.6,35.5]]]});
Trends
| where geo_point_in_polygon(Longitude, Latitude, region)
| summarize TotalVolume = sum(Volume), AvgVelocity = avg(Velocity)

This example returns all trends within a polygon (roughly covering central Tokyo), summarizing their volume and velocity. These queries can power BI dashboards, alerting systems, or geospatial machine learning models.

8 Conclusion: The Assembled System and Final Scoring Algorithm

8.1 The Final Architecture Diagram

By this point, our architecture integrates every stage of a high-performance, cloud-native real-time analytics system:

Ingest – Twitter API → Azure Event Hubs
Stream Process – Azure Stream Analytics (windowing, counting)
Probabilistic Estimation – Count-Min Sketch UDF for approximate frequency
Duplicate Filtering – LSH Azure Function
Anomaly Detection – ML.NET for spam/bot detection
Geocoding – Azure Maps API via Function
Geospatial Storage – Cosmos DB (for live dashboards)
Advanced Analytics – Azure Data Explorer (for ad-hoc spatial and time-series queries)
Visualization – Blazor or Power BI with Azure Maps SDK

This decoupled, PaaS-first model scales elastically from thousands to millions of tweets per second without manual tuning. Each stage performs a distinct role and communicates via Event Hubs, ensuring independent scalability and fault isolation.

8.2 The Final Scoring Algorithm (Putting It All Together)

To deliver a unified, meaningful “trend score,” we combine all major metrics into one formula. This can run as an Azure Function or within ASA using custom expressions:

public double ComputeFinalScore(double volume, double velocity, double spamScore)
{
    const double w1 = 0.2;  // volume weight
    const double w2 = 0.8;  // velocity weight
    const double w3 = 0.5;  // spam penalty weight
    return (volume * w1) + (velocity * w2) - (spamScore * w3);
}

This logic can also be expressed directly in SAQL:

SELECT
    hashtag,
    (CMS_Volume * 0.2 + ASA_Velocity * 0.8 - MLNET_SpamScore * 0.5) AS FinalScore,
    Latitude, Longitude, System.Timestamp AS WindowEnd
INTO
    CosmosTrending
FROM
    CleanTrendingHashtags;

The resulting scores are written to Cosmos DB, where they serve both as the source for real-time dashboards and as a feed for further offline analysis or machine learning.

8.3 The Dashboard

The final user experience surfaces this complex pipeline as a simple, intuitive interface. In Power BI or a Blazor web app, analysts can:

View live-updating maps of trending topics.
Filter by region, hashtag, or sentiment.
Zoom into a city to observe emerging micro-trends.
Hover over heatmap clusters to inspect the underlying data.

The dashboard queries Cosmos DB and ADX simultaneously. Cosmos serves the low-latency real-time view, while ADX powers time-based aggregations and regional comparisons.

8.4 Key Takeaways for Architects

Designing real-time systems at this scale requires balancing precision, cost, and performance. The architecture here demonstrates several enduring principles:

Probabilistic data structures like Count-Min Sketch enable accurate-enough analytics at massive scale.
Stream-first thinking—processing data in motion rather than at rest—reduces latency and infrastructure cost.
Serverless decoupling between ASA and Functions allows independent scaling and clean fault boundaries.
Geospatial enrichment adds critical context, turning raw trends into actionable insights.
PaaS-managed services minimize operational overhead, letting teams focus on algorithmic innovation rather than infrastructure.

By combining .NET’s expressiveness, Azure’s managed data stack, and modern probabilistic and ML techniques, we’ve built a complete blueprint for detecting, scoring, and visualizing trending topics in real time across the globe. It’s fast, scalable, and most importantly, architecturally elegant—a model pattern for any developer building real-time insight systems in the cloud.

Twitter's Trending Topics in .NET: Real-Time Stream Processing, Locality-Sensitive Hashing, and Geospatial Clustering