Businesses share data assets across organizations efficiently and securely with BigQuery Analytics Hub. However, multiple industries need to share real-time streaming data with their partners and customers. These businesses want the ability to deliver real-time access to their streaming data to internal teams or external customers. For example, a financial institution wants to share and monetize its proprietary real-time market data with its enterprise customers that would be making real-time financial decisions based on the streaming market data.
To help, today we’re introducing the public preview of Pub/Sub topics sharing in Analytics Hub, to help organizations curate, share, and monetize their streaming data assets through Analytics Hub. Pub/Sub is Google Cloud’s simple, highly scalable and reliable global messaging service and is leveraged by customers across the globe to reliably publish and subscribe to streaming data at scale. Analytics Hub, the data exchange platform built on BigQuery, is used by thousands of companies to securely share hundreds of petabytes across organizational boundaries in a given week with zero-copying at scale.
Sharing Pub/Sub topics through Analytics Hub provides a number of benefits:
-
The ability to curate and share valuable streaming data externally with your customers for data monetization or internally with other intra-org teams
-
An integrated experience for centrally managing accessibility to your organization’s streaming data
-
The ability to search and subscribe to valuable Pub/Sub topics shared by other organizations
Streaming data sharing is a well-established pattern with multiple use cases across industries:
-
Retailers can share inventory levels in real-time with Consumer Packaged Goods (CPG) enterprises to provide real-time fulfillment visibility.
-
Financial Services enterprises can share and monetize financial data (e.g., instruments pricing) with customers, offering a flexible pay-as-you-go model for market data distribution and consumption.
-
Advertising enterprises can share advertising data insights in real-time with advertisers to help them understand the campaign effectiveness.
-
Healthcare professionals can power predictive algorithms to monitor patients and analyze risk in real time.
How does it work?
Sharing Pub/Sub topics through Analytics Hub lets you curate and distribute a library of streaming data across multiple internal and external organizational boundaries. Real-time streaming data is shared by leveraging Analytics Hub Exchanges and Listings, allowing you to logically categorize and group sets of Pub/Sub topics and provision access at scale.
To start sharing Pub/Sub topics in Analytics Hub, first create or use an existing exchange. This exchange allows you to group listings that reference shared topics. A shared topic is the unit of sharing a Pub/Sub topic in Analytics Hub, and you can create or use an existing Pub/Sub topic to distribute message data to your Analytics Hub subscribers. Analytics Hub does not replicate the source Pub/Sub topic.
When subscribing to a listing with a shared topic, a linked Pub/Sub subscription is created in the subscriber project. Once message data is delivered through your mechanism of choice, you can begin consuming the data directly in your own managed project. You can manage Pub/Sub subscriptions through Analytics Hub, or directly in Pub/Sub.
Getting started with sharing Pub/Sub topics
You can start sharing Pub/Sub topics by following a few simple steps.
Create a listing
Creating a listing with a shared Pub/Sub topic looks very similar to creating a listing for a BigQuery dataset.
First, choose Pub/Sub Topic as the resource type.
Then, either select an existing Pub/Sub topic or create a new one to create the shared topic.
Finish populating the required fields to publish the listing, including adding appropriate roles for viewing and subscribing. Once published, your shared topic referenced in the listing becomes available for your subscribers to create linked Pub/Sub subscriptions.
Search and subscribe
You can search for listings of shared topics through the Analytics Hub catalog. Depending on how the listing was tagged when it was created, you can also use the filter criteria to further limit the scope of your searches.
Once you’ve identified the shared topic, simply select Subscribe to create a linked Pub/Sub subscription (assuming you have permissions to do so). You can then configure your subscription properties, including delivery type.
Take the next step
The ability to easily share Pub/Sub topics with Analytics Hub helps support a wide variety of streaming data sharing and monetization use cases. You can now simplify your streaming data sharing experience with Pub/Sub topics and Analytics Hub. As we incorporate feedback from preview and move to general availability, we will be adding additional capabilities. In the meantime, explore sharing topics in your own projects, or if you’re new to BigQuery, spin up a project using the BigQuery sandbox and Pub Sub’s free tier. This is just the beginning of Analytics Hub’s multi-asset data sharing journey. Learn more about Analytics Hub’s built-in data sharing capabilities.