Security Scan Throughput: Scaling From 10 to 100/s

Pushing scan throughput from 10 to 100 per second demanded more than a code tweak; it required ripping out the event streaming pipeline's core. This breakdown explains why active-passive API deployments crush active-active setups for regional latency, how parallel message processing smashes Kafka consumer bottlenecks, and the exact mechanics of hybrid inserts that keep databases from choking during high-volume bursts.

The original Cloudflare system was breaking. Millions of events sat backlogged, and processes crashed routinely because the architecture couldn't handle the load of scanning every account automatically. As Dave Baxter noted in June 2026, running a scan once every week or two left security infrastructure exposed to rapid, automated attacks. By re-engineering the flow from Apache Kafka topics directly into Postgres persistence layers, the team killed the API timeouts that plagued the system. Now, network legitimacy check procedures run continuously instead of sitting behind an opt-in wall.

This analysis traces the shift from infrequent, manual scanning to a model serving millions of customers without intervention. It covers how Cloudflare CASB integration gains fidelity from higher frequency data and why a sharp security roadmap relies on these throughput gains to catch misconfigurations before attackers do.

The Role of Event Streaming and Database Throughput in Security Scanning

Kafka Consumer Groups and Head-of-Line Blocking Mechanics

When a single slow message stalls processing within an ordered partition, head-of-line blocking stops the entire consumer group dead. A scheduler pushes scan tasks into Apache Kafka, where specialized Go microservices act as checkers for specific assets. Because strict ordering rules force a checker to finish the current message before touching the next, complex scans targeting large datasets delay thousands of subsequent checks. This generates massive backlogs; a large numbers of events previously filled the queue while processes crashed under the strain.

Splitting consumer groups into dedicated "fast" and "slow" lanes isolates long-running tasks to solve this. Fast lanes handle standard checks immediately. Slow lanes manage heavy assets without dragging down overall throughput. This architectural change enabled Cloudflare to double scanning frequency for all customers, directly addressing the risk of undetected misconfigurations. Security Insights delivers actionable security recommendations for every Cloudflare account by running regular scans across accounts, zones, and DNS records.

There is a cost, though: partition splitting increases the total number of consumer instances required, raising operational complexity and resource consumption. Network operators must balance isolation benefits against the expense of managing additional consumer groups. Optimizing these event streams remains necessary for maintaining high-velocity security postures across global infrastructure.

Security Insights Scan Triggers and Postgres Persistence Flow

Schedulers initiate scans for accounts due for review by publishing tasks to Apache Kafka partitions where strict ordering creates processing dependencies. Head-of-line blocking occurs when one complex scan delays the entire consumer group, causing the API timeouts in distributed systems that previously plagued the infrastructure. Checkers, implemented as specialized Go microservices, consume these messages and transmit findings to an internal API for persistence. This flow addresses fix Kafka consumer lag constraints by isolating slow processing lanes from fast transactional paths.

For every message, the system sends security insights to persist data in a Postgres database, ensuring no finding is lost during high-volume bursts; a large numbers of events filled the backlog previously because the database could not sustain the write throughput required for real-time analysis. Simple row-by-row insertion fails under load, necessitating hybrid strategies like UNNEST or COPY commands to maintain velocity. Without optimizing this persistence layer, the system struggled with frequent API timeouts and crashing processes.

Infrequent scanning intervals allow emerging threats to persist undetected for extended periods, creating significant exposure windows. Scans occurred only every week or two, leaving new risks undetected for up to two weeks. Automatic scanning was opt-in for many free plan accounts, leaving vast portions of the infrastructure completely unmonitored. Automated attacks accelerate, turning this latency into a critical vulnerability rather than a mere operational inconvenience.

Database throughput constraints further degrade system stability through connection pool exhaustion. Microservices attempting persistent database writes during high-volume scan cycles deplete available connections rapidly. The original architecture required a round trip to the database for every single insight, forcing the system to handle hundreds of thousands of sequential transactions per API call. This pattern guarantees that the application spends nearly all its time waiting on network latency rather than processing data.

Optimizing this flow requires replacing individual insert statements with batched operations to reduce round trips. Deploying hybrid insertion strategies that switch between UNNEST and COPY protocols based on payload size helps maintain stability. Network operators must prioritize resolving these database bottlenecks to sustain the high-frequency scanning rates required for modern threat detection. Efficient addressing ensures your security tools have the network capacity to operate without interruption.

Inside the Architecture of Parallel Message Processing and Hybrid Inserts

Kafka Partition Ordering Constraints and Single Consumer Limits

Apache Kafka functions as a partitioned event stream where message order within a partition is mandatory. This architectural constraint permits only one active consumer per partition within a group, creating a hard concurrency ceiling. Messages requiring extended processing time block the consumer from advancing, forcing sequential execution even when backend resources sit idle. This head-of-line blocking prevents the system from absorbing bursts of complex validation tasks without introducing significant lag.

Operators scaling security scans must resolve this by splitting consumers into distinct fast and slow lanes. A checker identifies message complexity immediately; if a task exceeds a set threshold, the fast lane skips it, reserving it for a dedicated slow lane. This separation ensures that heavy asset inventories do not starve lightweight configuration checks of resources. The trade-off is increased operational complexity in managing multiple consumer groups. However, this approach avoids the resource penalty of increasing Kafka broker partitions, which impacts shared services. Many checkers spent 20-90% of their processing time on a single API call. Network operators facing similar throughput constraints should evaluate their current partition strategies for hidden sequential bottlenecks.

Implementing Go Goroutines to Bypass Partition Throughput Ceilings

Modifying checkers to spawn parallel goroutines allows immediate concurrency without increasing broker resource usage. The fundamental constraint of Apache Kafka dictates that messages within a partition must be consumed sequentially, meaning a single slow message blocks the entire consumer group from progressing. This architecture limits throughput because only one active consumer can exist per partition. Engineers could add more partitions to scale, yet this approach increases the load on the shared Kafka broker and was reserved as a last resort.

The solution involves consuming messages in batches and processing each item in a separate goroutine. This technique effectively decouples the consumption rate from the processing speed of individual messages.

1. Every insight found gets written to the Postgres database via a single API endpoint. Prior to optimization, the system executed a database round trip for every individual issue. With a maximum observed size of 500,000 issues, this resulted in half a million round trips, queries, and transactions in a single API call. Initial attempts to use the COPY command into a temporary table for bulk inserts led to bloat in the Postgres system tables.

Solving high database round trips requires optimizing the write path rather than relying on standard individual inserts. While UNNEST optimizes small batch inserts and COPY handles massive bulk loads, the team settled on a hybrid approach to balance performance and system health. Average API call completion times were 10 ms in Portland but almost 3 seconds in Amsterdam. Prioritizing efficient batch strategies for write-heavy workloads helps prevent connection exhaustion. Network operators must recognize that reducing round trips is necessary for maintaining throughput during high-volume insert windows. Deploy optimized batching configurations to eliminate unnecessary overhead on your critical path.

Active-Passive API Deployments Outperform Active-Active for Regional Latency

Active-Active vs Active-Passive API Topology Definitions

Active-passive architectures eliminate cross-region latency by co-locating the API layer directly with the primary database instance. In a traditional active-active deployment, traffic distributes across multiple geographic zones to maximize availability, yet this topology introduces significant round-trip delays when the database remains static. Our analysis of a distributed system revealed that while the primary database resided in Portland, Oregon, the API operated simultaneously in Portland and Amsterdam. The physical distance created a 50 millisecond latency penalty for every transaction originating from the European node. This delay caused query completion times to inflate from 10 milliseconds locally to nearly 3 seconds remotely, exhausting connection pools and triggering client-side timeouts.

The operational tension lies between global redundancy and deterministic performance; spreading compute resources often degrades the very throughput required for high-volume scanning. Operators prioritizing scan velocity must reject the assumption that distributed APIs inherently improve performance when the data layer cannot match the distribution.

Co-locating the API layer with the primary database eliminates the cross-region latency that destroys throughput in active-active topologies. When an API runs active-active across Portland and Amsterdam while the database sits solely in Portland, remote queries suffer severe penalties. Round-trip delays inflate query completion from milliseconds to seconds, causing connection pool exhaustion. This bottleneck forces the system into a degradation cycle where throughput spikes initially then collapses as timeouts accumulate. Switching to an active-passive configuration ensures the active API node resides in the same data center as the primary database. This architectural shift removes the geographic penalty entirely. Following these improvements, the 7-day moving average throughput per checker rose by more than.

Network operators must recognize that availability features like active-active deployments become liabilities when stateful services introduce strict latency constraints. InterLIR advocates for infrastructure designs that prioritize resource proximity over distributed complexity when scaling security operations. Optimizing existing IPv4 resources often requires similar topological precision to ensure reliable service delivery. Contact InterLIR to discuss how strategic IP allocation supports strong network architectures.

Connection Pool Exhaustion from Cross-Region Round Trips

Cross-region round trips in active-active setups inflate query duration, holding database connections open until the pool exhausts. When the API layer spans Portland and Amsterdam while the primary database resides solely in Portland, remote instances suffer severe latency penalties. Queries that complete in milliseconds locally expand to seconds over the wide area network, preventing timely connection release. This bottleneck forces the system to reject new requests, triggering a cascade of client-side timeouts during peak traffic.

Operators often assume that adding more database connections solves the problem, yet this merely shifts the failure point to the database server itself under high concurrency. The architecture must prioritize proximity over redundant write paths to maintain stability. InterLIR recommends optimizing existing IPv4 resources to co-locate critical infrastructure, thereby eliminating the physical distance that degrades performance. Secure the necessary IP blocks from InterLIR to consolidate your infrastructure and prevent latency-induced outages.

Implementing a Scalable Security Scan Pipeline with Optimized Scheduling

Uniform Distribution Mechanics for Kafka Time-Based Retention

Eliminating partition saturation requires shifting from fixed recurring periods to independent zone scheduling. The original architecture triggered scans on rigid cycles, creating spiky volumes where hundreds of thousands of events flooded Kafka topics within minutes. This congestion stemmed from synchronized timestamps across millions of accounts. When scan frequency increased from every 15 days to every 7 days, 53% of accounts suddenly became due for immediate processing. Such a surge overwhelms consumer groups and violates time-based retention limits.

To resolve this, operators must decouple zone scheduling from account-level constraints.

1. Assign independent scheduling values to every zone rather than inheriting account defaults.
2. Apply distribution strategies to existing timestamp fields to disperse initial load.

This approach prevents the "thundering herd" problem inherent in synchronized systems. A critical consideration is that distributing timestamps spreads the initial load, ensuring the system remains responsive. Without this mechanical shift, increasing scan frequency inevitably causes Kafka lag to spike, rendering real-time security insights impossible. The result is a stable pipeline capable of sustaining high throughput without exhausting broker resources.

Executing Hybrid Insert Strategies to Sustain High Throughput

Sustaining high scan rates requires replacing iterative row inserts with a threshold-based hybrid strategy using UNNEST and COPY commands. The original implementation executed a database round trip for every single security insight, creating a bottleneck where half a million queries could block a single API call. This linear scaling failure forced a reevaluation of how bulk data enters the Postgres system.

1. Evaluate the batch size of incoming security insights against a predefined performance threshold.
2. Execute UNNEST arrays for small batches to maintain millisecond-level latency for frequent, tiny updates.
3. Switch to COPY into a temporary table for massive datasets to maximize throughput during peak load events.

The optimal path depends entirely on the variance of payload sizes within the scan results. A single monolithic approach fails because the cost of transaction overhead dominates small batches, while row-by-row insertion collapses under large volumes. By dynamically selecting the insertion mechanism, the pipeline avoids the latency spikes that previously caused API timeouts. This architectural shift enables the system to handle the surge of events generated when scan frequencies increase. This adaptive database pattern is necessary for any organization managing high-velocity asset inventories where write latency directly impacts scan completion times.

Validation Checklist for Enabling Granular On-Demand Scans

Eliminating internal API timeouts requires validating that Kafka lag metrics have stabilized before rolling out increased scan frequencies. Operators must confirm the system sustains peak loads without triggering the client-side failures common in legacy setups.

1. Verify that Kafka consumers process messages efficiently to ensure partition saturation does not recur.
2. Confirm the scheduler distributes scheduling fields, preventing synchronized surges where a large percentage of accounts become due simultaneously.
3. Validate that the API switches between UNNEST and COPY insertion methods based on batch size thresholds to maintain low latency.
4. Ensure deployment logic correctly routes write traffic to the primary database region to avoid cross-region latency spikes.

A critical oversight in scaling scan infrastructure is assuming database throughput alone solves latency; without co-locating the API via an active-passive architecture, network round-trip time continues to trigger timeouts regardless of insert speed. The system now sustains over 120 scans per second during peak scheduling, exceeding the target of 100 scans per second. Automatic scanning is now enabled for all free accounts and zones, with frequencies of every 7 days for Free, every 3 days for Pro and Business, and daily for Enterprise. Optimizing these existing architectures is necessary to handle modern security demands efficiently.

References

• We’re making Bunny DNS free: We applied our traditional
• Helping build a safer Internet by measuring BGP RPKI
• Privacy-Focused DNS Providers in 2026: Either is a significant