Is real-time processing a core requirement for Customer Data Platforms? Some customers say yes and some say no; not surprisingly, the answers correlate closely to their own use cases and needs. The real answer depends on the use cases your company expects a CDP to support. But no one likes “it depends” as an answer.
A general answer might turn on what fraction of buyers have use cases that require real-time. CDP Institute research finds that real-time profile delivery ranks just beneath requirements we’ve included in the RealCDPTM checklist. This puts it right on the border.
Yet the question is more complicated than that. There’s more than one type of real time processing associated with a CDP. Our survey asked two questions: one about real time profile delivery and another about real-time recommendations. A more comprehensive view would include at least two more topics: real-time updates and real-time identity resolution. Let’s look at all four of those in turn.
- Real-time updates. This refers to ingesting data from source systems at speed and scale, loading it into the CDP, using it to update customer profiles, and making those profiles available. A typical use case is making events in one system visible to another system, such as letting a call center agent see an online buyer’s shopping cart. This usually needs new information to be posted in one or two seconds, although some people might allow longer. Completing the ingest/load/update/share process that quickly requires engineering optimized for individual transactions. It also requires complementary capabilities outside the CDP: source systems must be able to send real-time updates as events occur and target systems must either ingest real-time updates from the CDP or read the live real time CDP data directly. The good news here is that’s often a middle group between true real-time updates and a periodic batch update. For example, a lag of several minutes is acceptable in many re-targeting scenarios, where the task is to send a follow-up message after an event has occurred or to remove someone from a follow-up list.
- Real-time lookup. This is letting an external system read the customer profile as it currently exists in the CDP. The access usually involves API call that includes a unique identifier such as a customer ID and specifies which data elements to return. In cases like showing customer history to a call center agent, the CDP may simply display its profile on a user screen. In other cases, the profile data may be extracted so the calling system can use it in a process such as Web site personalization. A CDP may support real-time lookup even if it only updates the underlying profiles at intervals such as nightly. The data available for real-time lookup is often copied from the main CDP data store into a special format optimized for real-time interaction, such as an indexed file or in-memory database. This means users must specify in advance which elements are made available. The acceptable response time for real-time lookup depends on the use case: there’s little problem in making a call center screen or Web site wait one full second but programmatic ad bidding might require response in less than 50 milliseconds.
- Real-time identity resolution. Real-time updates and profile requests usually include a unique identifier such as a customer ID, device ID, or email address. This lets the CDP quickly find the related customer profile by searching for an exact match. But when the input lacks a clear ID, the CDP may need to run an identity resolution process that looks for matches based on a combination of data elements (name, address, etc.) and/or behavioral information (location, time of day, device type, etc.) This can require considerable processing to test different matches, apply different match methods, and to look up external reference data or identity graphs. A comprehensive identity process would also reassess existing match sets (groups of IDs assigned to the same person) after each update to see if the new data implied a split or merger. This sort of identity resolution can be difficult even without time constraints. So it’s important to understand what kind of matching is needed when judging whether a CDP can support a particular use case.
- Real-time interactions. This refers to processes where the CDP is updated with new data during the course of an interaction. It combines real-time lookup with real-time updates. A typical use case is website personalization, where the CDP receives an initial message with customer information, finds the customer profile, and returns an offer recommendation. The customer then takes an action which is reported back to the CDP, which updates its profile and returns a new offer recommendation. This cycle continues until the customer leaves the Web site. Depending on the details, the CDP may only update its profile with the new information or it may do more work such as rescoring a predictive model. These interactions are usually executed by copying the current profile into memory at the start of the interaction, updating the in-memory profile as the interaction proceeds, and then copying the final profile back into the main CDP data store when the interaction is complete. One drawback to this approach is that customer actions taken through a different channel may not make their way into the in-memory copy of the profile.
Which of these real-time capabilities do you need? Again, the answer depends on your situation. Whatever you decide, ensure the system you buy can do what you expect. Remember that CDPs vary widely and it’s not safe to assume that any particular system has any particular capability. Take your time to do thorough research: not only will you avoid mistakes, but you’ll be better prepared when you acquisition is complete and it’s time to start deployment.
RELATED ARTICLES
What’s Needed to be a RealCDP?
Building vs. Buying a CDP? Why That’s Not the Only Question You Should be Asking
Be in-the-know with all the latest customer engagement, data management, and Redpoint Global news by following us on LinkedIn, Twitter, and Facebook.
