Collect the Data You Need, Right Where You Need It

In times of endless data, it is crucial to collect smarter.

As an analytics expert and practitioner, I know first-hand that collecting data across multiple digital assets and channels can be daunting. This is especially the case when the number of devices connected to the global internet exceeds 21 billion in 2023. Thankfully, our current Iinternet addressing system can handle a lot of these devices, namely up to 3.4×10E38 (that’s 34 followed by 37 zeroes). 

Out of these 21 billion devices, about 66% is made up of Internet of Things (IoT) devices, all of which generate data about their operation, features and settings. Call it connected black boxes or telemetry on steroids, but these devices are sending data home to service providers who use that data for product enhancement.

Such a scale of data collection provides not only the ideal fuel for AI and machine learning, but also the means to establish performance baselines and outliers. Feature usage models, insights and action plans can all be derived from such an unfathomable well of information.

(Re)introducing the Measurement Protocol.

How do these devices measure activity, you ask? This post is a perfect excuse to look at Google Analytics 4's Measurement Protocol as an alternative data collection method that can help you measure all the IoT data you need—and make it compatible with the flat data model you have come to adopt and love. The Measurement Protocol was introduced in the early 2010s with the former version of Google Analytics, the now sunsetted Universal Analytics. Back in the day, the Measurement Protocol was used in very creative ways, so seeing it reborn for GA4 is a great opportunity to (re)discover this lesser-known yet powerful feature in Google Analytics.

In essence, the Measurement Protocol is an API that allows you to send events directly to Google Analytics servers, bypassing the need for bulky software development kits and complex integrations. The minimal software footprint of the Measurement Protocol means it is easily embeddable in every system that can call a URL. As you can imagine, this can be used for all IoT—everything from kiosks to points of sales to IoT devices. Some clear advantages include:

• Standard protocol, so it is compatible with a wide range of devices and platforms
• Easy to use, even for developers with limited experience
• Scalable, so it can be used to collect data from large numbers of users
• Security, through the use of data collection secret keys

Because of its lightweight approach, using the Measurement Protocol means you can collect just the data you need. The lack of an explicit user consent mechanism on most IoT devices will encourage you to adopt a privacy-first approach, so focus on telemetry and not on personal data. 

Uncovering the Measurement Protocol’s inner workings.

How does it all work? Well, when creating a Google Analytics 4 (GA4) property for your IoT project, you first need to create a new web property and then simply click on this newly created data stream to access the Measurement Protocol API secrets panel.

The next step is to create a key, which you will reference in your Measurement Protocol API calls. All you need to do is provide a nickname for your key and you can use the provided ID in your API calls. As you can see from the list below, our Data.Monks use it quite a lot!

Once your keys are set up, make note of your GA4 Measurement ID for your IoT stream and use code to create a URL to the Measurement Protocol service that combines everything we need, including event parameters. In the example below, our connected fridge will send an event when the fridge door is open.

The desired URL should look like this:

https://www.google-analytics.com/mp/collect?measurement_id={your ID}&api_secret={your key}

Now we need to send the above URL as a POST request, with a JSON payload containing the event parameters we want to send along. Keep in mind that, because this is not like a GA4 event sent from a browser or a mobile app, there is no automatic detection and collection of extra elements, as with GA4’s enhanced measurement. In fact, the Measurement Protocol only measures what you send it. From there, post the request in your favorite programming language—Python, in my case.

Sure enough, the event registers in the GA4 real-time interface and subsequent hits will become part of your GA4 reports—and live on to BigQuery if you’ve linked your property to Google Cloud Platform.

And of course, as I’m sure you can already guess, creating dashboards on your devices’ activity is a breeze in Google Looker Studio. That’s all there is to it!

Time to try out the Measurement Protocol yourself.

We have seen that the Measurement Protocol, like other event-level data collection platforms, uses an API-friendly format to send data out to Google Analytics. From a technical standpoint, this is a very easy and efficient implementation, so feel free to get creative for all your IoT projects.

We’ve discussed using the Measurement Protocol primarily for IoT devices (or any device that isn’t a computer, mobile phone or gaming console). Bearing that in mind, you can also use it as a data exchange method in a cloud environment as an API callback after a process completes. This means the Measurement Protocol works great with Cloud Functions or messaging queues like Google Pub/Sub or Kafka.

Finally, circling back to the remark I made about AI, this kind of measurement is indeed an ideal way to collect fuel for an AI/ML model, but AI can also be used to trigger the right event at the right time, and with the right data payload. At this point, AI can improvise and improve on your intended data collection plan, start sending events outside of the scope of its original program, and unlock even more insights. Coupled with Google Cloud Platform’s Cloud ML, the results may surprise you! 

In short, here are your key takeaways about the Measurement Protocol:

• Simple mechanism: any system that can generate a URL can use it
• Encourages concise, compact, privacy-friendly data collection
• Can be used on anything, about anything
• Leverages the power of the Google Analytics 4 flat data model
• Small software footprint: very limited resource consumption
• Complements an AI strategy and unlocks new opportunities