IP Functions are Generally Available, bringing high-performance network analytics to the Lakehouse

Every firewall, load balancer, VPN, CDN edge, DNS resolver, Kubernetes cluster, and application server all emit a stream of records keyed on one thing: an IP address. For a large enterprise, these streams collectively generate tens of billions of events a day and are the foundation of some of the most valuable analytics an organization runs, including threat detection, fraud investigation, and network observability. Historically, the industry treated these network observability use cases as specialized use cases that required a specialized stack, leading to silos, fragmented governance, and lock-in.

That changes now. Today, IP Functions are available Generally Available. With this launch, IP address analytics becomes a first-class, high-performance SQL workload on the lakehouse, letting security teams parse, enrich, and analyze their highest-volume IP data alongside the rest of their analytics, under one governance model.

In the past, IP addresses were deceptively hard to handle in SQL. An IPv4 address looks like a string but behaves like a 32-bit integer; IPv6 is 128 bits. A CIDR block like 10.0.0.0/8 isn't a value at all - it's a range of 16 million addresses. Asking "is this IP inside that subnet?" is a range containment problem hiding behind a piece of text.

The result was a lose-lose. SQL-only analysts were locked out of basic IP filtering because it required procedural code. Data engineers burned cycles maintaining UDF libraries and CIDR-expansion jobs. Above all, the critical workloads, like enriching billions of events against threat-intelligence and geo-IP tables, would run for over an hour when the business needed answers in minutes. At a petabyte scale, that gap is the difference between catching an intrusion in progress and reading about it in the post-mortem.

Databricks introduces a complete set of built-in IP functions that make network data a first-class citizen of the lakehouse. They handle IPv4 and IPv6 uniformly, accept both human-readable STRING and compact BINARY representations, understand CIDR notation natively, and are implemented in the engine itself so the optimizer and Photon can accelerate them.

The example below enriches raw network flow logs with threat intelligence and then identifies suspicious source networks that are scanning large numbers of destinations and ports. What previously required custom parsing logic and bespoke IP libraries can now be expressed directly in SQL using native IP and CIDR operations.

No UDF. No regex. No integer gymnastics. It reads like the question the analyst is actually asking.

ip_cidr_contains(cidr, needle) - tests whether an IP address or another CIDR block falls within a CIDR block. This is the single predicate behind CIDR block joins and high-volume filtering, and the function the entire optimization effort centers on.

ip_host(ip) - normalizes an IPv4 or IPv6 address to its standard form (e.g., collapses 2001:0db8:0000::1 to 2001:db8::1).

ip_cidr(cidr) - produces the canonical representation of a CIDR block.

ip_network(cidr) / ip_network_first(cidr) - returns the first (network) address of a CIDR block.

ip_network_last(cidr) - returns the last address of a CIDR block.

ip_prefix_length(cidr) - returns the prefix length (the number after the /).

ip_as_binary(ip_or_cidr) - converts an address or CIDR to its canonical, compact binary form (4 bytes for IPv4, 16 for IPv6). Storing and joining on BINARY skips repeated parsing and shrinks storage.

ip_as_string(ip_or_cidr) - converts a binary representation back to human-readable text for reporting.

try_ip_host(ip), try_ip_cidr(cidr), try_ip_as_binary(ip_or_cidr), try_ip_as_string(ip_or_cidr) - identical to their counterparts, but return NULL instead of erroring on invalid input. Essential when ingesting raw logs where a fraction of records are always malformed, so one bad row never fails a billion-row job.

These native functions compose naturally with the rest of SQL, they're available to every SQL user, with no setup, and the optimizer understands them, which is what makes the performance story possible.

Rearc, which helps enterprises develop GenAI, Data, and Cloud platforms, is leveraging the IP Functions to build network observability use cases for large scale customers.

We built a product for a major financial firm that regularly processes more than 30TB of data per day, where performance and cost efficiency are critical. With IP functions native to the Databricks engine, we were able to parse, validate, and join IP data directly in SQL, replacing a previous ad-hoc implementation with something far more elegant and maintainable. Because the functions are built into the engine, we got this without sacrificing the performance our workloads demand at this scale. They've made network analytics on the lakehouse simpler and faster for us to deliver." —Dara Kharabi, Practice Lead, AI & Data at Rearc

A common IP analytics challenge is finding a single address or sub-CIDR within a much larger range, which is critical for quickly detecting threats, investigating fraud, and monitoring network activity at scale. This use case represents a range join, which engines have historically struggled with because standard join algorithms rely on equality. Databricks’ engine supports an optimized range join for IP addresses via ip_cidr_contains.

Databricks’s ip_cidr_contains outperforms traditional warehouses on price and speed across all scales of probe (i.e. “needle”) and block (i.e. “haystack”) tables. We benchmarked ip_cidr_contains across five representative scenarios:

Size of probe table (i.e. number of needles)

Size of probe table (i.e. number of needles)

Size of CIDR block table (i.e. number of haystacks)

Size of CIDR block table (i.e. number of haystacks)

The results show that Databricks’ IP Functions’ performance is much stronger than competitors, even at increasing scales. Once the number of probes exceeds 10B IP addresses and the number of blocks goes beyond 1M CIDRs, query speed begins to flatline.

The cost gap is just as stark. Even as workloads scale, Databricks remains 2x to as much as 6.4x cheaper.

"Databricks' native IP functions let us work with IP and CIDR data directly in SQL. We no longer rely on brittle string parsing or manual bitwise logic. Being able to parse, validate, and join IP data as first-class SQL operations has made this work simpler and more maintainable for our team. It's a natural fit for running network and traffic analytics alongside the rest of our data on the lakehouse."—Stanby, Data Engineering Leader

Ultimately, these highly performant IP Functions enable an entire class of network workloads to be built directly on the lakehouse.

CIDR enrichment joins at scale - tag every event with GeoIP, ASN, threat-intelligence, or ownership metadata in seconds, so downstream detection and investigation queries run against enriched data.

High-volume, real-time filtering - "show me every connection from this suspicious /16 in the last 24 hours" becomes an interactive query instead of a batch job.

Unified IPv4 and IPv6 analytics - mixed-protocol tables work out of the box, so modern traffic is analyzed, not dropped.

CIDR-in-CIDR matching - check whether an entire subnet falls within another, at the same performance as IP-in-CIDR, for network-topology and policy analysis.

First class SQL accessibility - analysts get IP filtering and joins with plain SQL, no procedural code or UDF libraries required.

With these functions natively supported on the lakehouse, these network workloads share one governed copy of the data with the rest of the enterprise - no separate specialized system to license, secure, and keep in sync.

Native IP functions are available now Generally Available on Databricks Runtime 18.3 and above.

See the IP functions reference documentation for the full function list and signatures. The network firehose has always been one of your biggest datasets - now it can finally live where the rest of your analytics do.