PayPal/Data Engineer/Data Structures & Algorithms

A PayPal data engineer is building a real-time analytics dashboard to show trending search queries and their frequencies. How would you design a data structure to efficiently track and query the top-K most frequent queries from a continuous, high-volume stream?

PayPalData Engineer3–5 YearsData Structures & Algorithms

For efficiently tracking top-K frequent items in a high-volume, continuous stream, a combination of a Count-Min Sketch and a Min-Heap is ideal. The Count-Min Sketch provides an approximate frequency count for each item with bounded error and memory footprint, crucial for high-throughput streams where exact counts for all items are impractical. The Min-Heap, constrained to size K, maintains the current top-K frequent items, allowing for quick updates and retrieval. When an item arrives, its frequency is estimated using the Count-Min Sketch. This estimate is then used to decide if the item should be added to or update its position within the Min-Heap.

Leveraging Count-Min Sketch and Min-Heap

The Count-Min Sketch is a probabilistic data structure consisting of a 2D array (table) and multiple hash functions. Each hash function maps an item to a specific column in its respective row. To update an item’s count, we hash it with each function and increment the corresponding cells. To query an item’s count, we hash it with each function and take the minimum value across the corresponding cells, which provides a high-probability upper bound on its true frequency. The Min-Heap stores (frequency, item) pairs, where the root holds the Kth most frequent item. If an incoming item’s estimated frequency is greater than the heap’s minimum, it is inserted, potentially evicting the current minimum to maintain the top-K.

Best practice

For production systems, tune the Count-Min Sketch parameters (width and depth) based on desired error bounds and probability of accuracy. Wider sketches reduce collision probability and thus error, while deeper sketches with more hash functions improve accuracy. Also, consider decaying counts for items in the Count-Min Sketch if trends are time-sensitive, allowing older counts to diminish and newer trends to emerge. Ensure hash functions are independent and well-distributed to minimize collisions.

Edge case interviewers probe for

Interviewers often ask about handling items that “fall out” of the top-K but later become frequent again, or how to manage sudden spikes in less frequent items. Since the Count-Min Sketch maintains a global approximate count, these items will still have an updated estimate when they re-emerge. If their frequency surpasses the current Min-Heap’s minimum, they will naturally be re-added. The Min-Heap dynamically adjusts to reflect current trends.

Common mistake

A common mistake is attempting to store exact counts for all distinct items in memory, which quickly becomes infeasible for high-volume streams with a large cardinality. Another error is not understanding the probabilistic nature of the Count-Min Sketch, leading to overreliance on its exactness or misinterpreting its error bounds. Simply using a hash map for counts and sorting, or a fixed-size reservoir sampling, would not scale or provide continuous top-K tracking efficiently.

What the interviewer is checking

The interviewer is assessing your understanding of streaming algorithms, approximate data structures, and how to combine them for practical problems. They are looking for your ability to handle massive datasets under memory and processing constraints, explain trade-offs between accuracy and resource usage, and design a robust solution for real-time analytics. Your reasoning for choosing each component and parameter tuning is key.

Imagine a giant lost and found office at a huge airport. Thousands of items are dropped off every minute, and people constantly ask, “What are the 10 most common things people lose?” It would be impossible to perfectly count every single unique item and then sort them all the time. Instead, we have a special system. When an item comes in, we quickly add it to a grid that keeps a ‘hazy’ count of how many times things like it have appeared, not perfectly accurate, but good enough.

Simultaneously, we have a small display board that only shows the 10 currently most frequently lost items. If a new item comes in and its ‘hazy’ count from our grid suggests it is more common than the least common item on our display board, we update the board, replacing the least common item with this new, more common one. This way, we always have a good approximation of the top 10 lost items, without needing to know the exact count of everything ever lost.

Why interviewers ask this

Interviewers ask this to gauge your ability to design efficient, scalable solutions for real-time data processing, a common challenge at companies like PayPal. It tests your fundamental knowledge of data structures, algorithms, and how to apply probabilistic approaches when exact solutions are not feasible due to resource constraints.

What a strong answer signals

A strong answer demonstrates a deep understanding of streaming algorithms, the trade-offs between accuracy and resource usage, and the ability to combine different data structures (like Count-Min Sketch and Min-Heap) to solve complex problems. It signals practical engineering judgment and an awareness of system limitations.

Common follow-ups

  • How would you handle a scenario where you need exact top-K counts?
  • What are the memory implications of your chosen solution for an extremely high-cardinality stream?
  • How would you adapt this approach to find the top-K items within a sliding time window (e.g., last hour)?

Advanced variation

A more advanced variant might involve designing this system for a distributed environment. How would you handle stream partitioning, merge Count-Min Sketches from different nodes, and aggregate top-K lists from multiple sources while maintaining accuracy and efficiency?

Consider an e-commerce platform like PayPal building a feature to show “trending products” to users in real-time. Instead of processing massive logs for exact counts, a Count-Min Sketch can efficiently track approximate purchase frequencies for millions of products from a continuous stream of transaction events. A Min-Heap then maintains the top-K products, allowing the dashboard to update dynamically as popular items emerge and recede, providing a responsive and relevant user experience without overwhelming backend systems.

frequency_tracker.py
import hashlib
import sys
import heapq

class CountMinSketch:
    def __init__(self, width, depth):
        """
        Initializes the Count-Min Sketch with specified width and depth.
        width (m): number of columns in the sketch (controls error rate).
        depth (k): number of hash functions (controls probability of error).
        """
        self.width = width
        self.depth = depth
        self.table = [[0] * width for _ in range(depth)]
        self.hash_seeds = [i for i in range(depth)] # Simple seeds for hash functions

    def _hash(self, item, seed_index):
        """
        Generates a hash for the item using a specific seed.
        In production, use a more robust hash function.
        """
        return (hash(str(item) + str(self.hash_seeds[seed_index])) % self.width)

    def add(self, item):
        """
        Adds an item to the sketch, incrementing its approximate count.
        """
        for i in range(self.depth):
            col = self._hash(item, i)
            self.table[i][col] += 1

    def estimate(self, item):
        """
        Estimates the frequency of an item.
        """
        min_count = sys.maxsize
        for i in range(self.depth):
            col = self._hash(item, i)
            min_count = min(min_count, self.table[i][col])
        return min_count

class TopKFrequentTracker:
    def __init__(self, k, cms_width=1000, cms_depth=5):
        self.k = k
        self.cms = CountMinSketch(cms_width, cms_depth)
        self.min_heap = [] # Stores (frequency, item) tuples
        self.item_frequencies = {} # To quickly check if item is in heap

    def add_item(self, item):
        self.cms.add(item)
        estimated_freq = self.cms.estimate(item)

        if item in self.item_frequencies:
            # If item is already in our heap, update its frequency
            # This is a simplification; a true heap update would involve removing and re-adding,
            # or using a more complex data structure. For interview, this approximation is often accepted.
            for i, (freq, current_item) in enumerate(self.min_heap):
                if current_item == item:
                    self.min_heap[i] = (estimated_freq, item)
                    heapq.heapify(self.min_heap) # Re-heapify after update
                    self.item_frequencies[item] = estimated_freq
                    return

        if len(self.min_heap) < self.k:
            heapq.heappush(self.min_heap, (estimated_freq, item))
            self.item_frequencies[item] = estimated_freq
        elif estimated_freq > self.min_heap[0][0]:
            # If current item is more frequent than the smallest in heap
            smallest_freq, smallest_item = heapq.heappop(self.min_heap)
            del self.item_frequencies[smallest_item]
            
            heapq.heappush(self.min_heap, (estimated_freq, item))
            self.item_frequencies[item] = estimated_freq

    def get_top_k(self):
        # Returns top K items sorted by frequency (highest first)
        return sorted(self.min_heap, key=lambda x: x[0], reverse=True)

# Example usage:
tracker = TopKFrequentTracker(k=3)
search_queries = ["laptop", "mouse", "keyboard", "laptop", "monitor", "mouse", "laptop", "keyboard", "laptop", "headphones", "mouse", "laptop"]

print("--- Processing queries ---")
for query in search_queries:
    tracker.add_item(query)
    print(f"Added: '{query}'. Current Top {tracker.k}: {tracker.get_top_k()}")

print("n--- Final Top K queries ---")
print(tracker.get_top_k())

# Expected actual frequencies: laptop: 5, mouse: 3, keyboard: 2, monitor: 1, headphones: 1
Input Stream Count-Min Sketch Hash Functions (h1, h2, h3…) 2D Array (Table) Min-Heap (Top K Items)
  1. 1Real-time top-K frequency tracking in high-volume streams requires approximate data structures due to memory constraints.
  2. 2A Count-Min Sketch provides efficient, probabilistic frequency estimates with bounded error for all stream items.
  3. 3A Min-Heap of size K effectively maintains the top-K most frequent items based on estimates from the Count-Min Sketch.
  4. 4Tuning Count-Min Sketch parameters (width, depth) balances accuracy, probability of error, and memory footprint.
  5. 5This combined approach demonstrates mastery of streaming algorithms and practical trade-offs in data engineering.