The Fields In the Event Tracking Dictionary

The baseline fields in the dictionary are:

• Event Name – What the action is called. Best named using specific phrases that a sophisticated user might use to describe their actions

• Triggers when… – A specific API response, user action, or event that acts as the snapshot for this event and its properties being sent to our logs.

• Screen – A screenshot or image that illustrates where the user is when the action is triggered

• Properties – A list of property names that will be tracked along with this event (e.g. source, isLoggedIn)

• Example Property Values – Best when done exhaustively, the list of potential values that fall under each property above. In cases where there is a limited set of potential values (e.g. potential Sign Up sources like Facebook, Email, Google), it’s best to list them here.

• Data / Property Type – Each property should be constrained to one data type such as a boolean, string, numerical, latitude & longitude, or float.

• Description – How would you describe this event being captured to someone who has never used the product before? Use this field to remove any possibilities of misalignment between business teams using this in the future and engineering teams implementing these specifications.

• Technical Comments – OAuth, APIs, and internal services can have their own quirks that you want to detail out here. Specifications like aggregating 2XX responses into a single “Success” value can go here.

• Testing Comments – This is a living, breathing document. When new features are released, it’s good to pass through QA and ensure that events are firing as necessary. Communicating changes and issues here makes it quick to resolve issues.

What Makes A Good Event Tracking Dictionary

You might have a different set of fields vs the template. But there are a few keys I look for that make it a good asset to align the team.

1. Simple

   At a minimum, the dictionary needs to be simple and easy to understand. Things like clear naming conventions are the basics. But the true test is if someone entering the company for the first time is able to look at the Event Tracker and quickly map the events to their actions in the product with out needing to read every event’s definition.

2. Actionable

   A business user of the event tracker need to be able to move from the sheet to the data to decisions without tons of involvement from a data analyst. A lot of times this comes down to the right level of comprehensiveness. Too few events tracked, and you will have incomplete information to make the decision. Too many events tracked, and it will be overwhelming.

3. Visual

   Every event tracked should have screenshots. While people can interpret the names and descriptions of events in a lot of different ways, the visual representaiton of the screen ends up being much more specific and clear.

1 – Mindset of The Business User

As I mentioned earlier, the end user of most analytics system are business users. We need to build something that empathizes with that end user. This means humanizing data and analytical processes. This influences how we choose the tools to use, the events to track, how to name the events, and what what properties are needed. It is worth spending a meaningful amount of time here, just as we would in customer research for a new product.

To get into the mindset of the business user, there are four levels of questions I go through. For each question I have provided some examples from a product I recently worked with named Honeydu which is a way for companies to send and receive invoices online for free.

1. What are the business goals and objectives?

   What are the key results and metrics that the business and executive team is optimizing for? Sources of this information would be current and historical OKR’s, quarterly and yearly planning documents, and board decks.

   Example One: X new users receiving/sending an invoice by end of Q4 2020

   Example Two: X% of invoices sent to new users lead to a new user sign up

   Example Three: X recurring invoices active by end of Q4 2020

2. What are the objectives and goals of each team?

   The high level goals of the company isn’t enough. Each team will typically have a set of objectives and goals that ladder up to the company level metrics. To understand these objectives and goals you can seek out each team’s OKR’s, talk to the team leaders, etc. Here you want to both understand what they were for a few time periods historically and what the team leaders are thinking they might be for a few time periods in the future.

   Example One: (New User Growth) 2 invoices sent within the first 7 days

   Example Two: (Payment Integrations) 85% of new payment methods added are successfully verified

   Example Three: (NUX) % of New Users starting with an invoice template

3. What are the product experiences around these goals and objectives?

   Next, for each of the goals/objectives I identify the product experiences that correspond with driving those goals/objectives. It is important to not just identify what the specific screen of the product or the funnel, but the context of the experience that might influence the goal or action. For example, in a ride sharing product like Uber, if the product experience was booking a ride, in addition to the funnel of booking a ride I might want to understand how many drivers were on the map? Or, what the estimated time was?

   Step 1: In Honeydu, lots of different factors could lead to a user sending their first invoice, our core action. We’d ask ourselves:

   • When a user is selecting a contact to send an invoice to, are they more likely to be successful when a contact is available on the user’s list of historical Businesses or when they need to search?

   • What are the supporting actions that help users create and send their first invoice? Are invoice templates a good way of accelerating the time-to-send? Or is it more important that their contacts are imported first?

   Step 2: The next step is to think through the experiences that could prevent users from reaching our goals and objectives. In Honeydu’s case, I’d ask: Why didn’t a new user successfully create their first invoice? Did they look at different templates and not find one that was relevant to them? Did they try creating an invoice from scratch and find it too difficult to go back to our template directory? What actions did our activated users perform that non-activated users did NOT perform?

   Step 3: Finally, imagine that any event could be the last event that we’d ever track from the user in our product. What would we want to know about this experience? We would need to know if they were given a “No Results Found” page after a contact search, or an error while adding a new payment method, and use the popularity of these events to begin triaging issues in our user experience.

4. What are the questions/hypotheses I/they might want to answer around these goals and product experiences?

   Next, I think about what questions or hypotheses they (or I) might have around those goals or objectives. Similarly here, talk to team leads or individual contributors on the team about what questions they faced. But also think through yourself, come up with hypotheses of questions, and validate the importance level of those questions with that team.

   Example One: One of the key goals and product experiences on Honeydu is when someone sends their first invoice. I’d ask the question, what experiences do I think need to happen in order for someone to feel confident about sending their invoice to a business? Hypotheses like “they need out with one of our industry-approved templates” or “they see the business already listed in the Honeydu network” indicate experiences that we need to be able to track in order to quantify and move on from hypotheses to confidence in correlation/causation.

   Example Two: The more users that pay an invoice through Honeydu, the more revenue we generate. We should track at what point users are most likely to pay an invoice, asking ourselves “When are users most likely to pay an invoice through Honeydu? When it’s due today? Tomorrow?” By tracking the daysTillDueDate property on the Pay Invoice Successful event, we can inform and build our push notification and email strategy for users who don’t organically reach the experience of seeing an invoice’s due date without spamming them outside of their natural inclinations.

2 – Journey’s Instead of Metrics

One of the keys I discussed earlier was getting to the right level of abstraction in events. The foundation of this is tracking journey’s, not metrics.

While we use the goals and objectives from step one as an input into what we track, stopping there ends up in bad event tracking. Bad event tracking often looks like the individual asked themselves “can I calculate all my OKRs with these metrics” e.g. # users clicking sign up, # completing order, conversion between Sign Up and Completed. The problem with this is that it’s going to lead to dead ends. You will be able to know “50% of my users signed up” but won’t be able to ask “why?”

To answer the question of “why?” you first need to understand the journey of intent, success, and failure and then the context for each of those events in the journey (which we’ll track with properties in step 3).

A couple of quick examples showing the event journey of Intent → Success → Failure:

• Example One

  • Intent: Add New Payment Method Selected & Add New Payment Details Submitted

  • Success: Add New Payment Method Successful

  • Failure: Add New Payment Method Failed

• Example Two

  • Intent: Create Invoice Selected → New Invoice Started → Contact Search

  • Success: Recipient Added to Invoice → Invoice Sent

  • Failure: Invoice Draft Saved (a default action)

2a – Success Events

I first think through the success events. A success event is when an action in the product has been completed successfully. These events stem from the business goals that I collect in step one. Examples of success events might include:

• Payment Successful

• Signup Successful

• Invoice Sent

• Booking Completed

In order to not go overboard and track everything, I pressure test each event with a question. “Imagine I did track this, and 99% of users did it, what would I do about it? What does it tell me?” If I’m unable to to find something actionable on the extreme, then the event is likely unhelpful.

2b – Intent Events

For each success event, I then think through the intent events. An intent event is often a step required as the precursor to any success event. Tracking intent events are critical to understanding the “why” around the success events.

Intent events inform me of how “educated” and “motivated” the user is to complete some step for the action I want them to complete. Everything is actually an intent event of some next event — but we often only think of them as “goals” which prevents us from tracking them accurately. For example, in a ride sharing app Selecting a Destination is a goal, but requires an intent/setup event of selecting a ride type (in the old Lyft/Uber flow). Then, booking the actual ride becomes the goal, but requires the intent/setup event of finding a destination from search/history… and so on

In order to come up with the intent events I ask – what are the steps I have to complete in order to complete the success event? Common examples include:

• In our first Journey example, we noted the intent events of “Add New Payment Method Selected” and “Add New Payment Details Submitted”

  • Note that we have two levels of intent here — high intent, where a user is actively submitting their payment details, and low, but indicative intent, where a user is selecting whether to add their payment details by Bank or Credit card. Drop-offs in between these events lead to actionable next steps for a team: either the user found the input fields daunting, or didn’t have this information available at the time. We now know if they had selected a bank or credit card payment method, and can follow up with more information and personalized content that helps the user get through this step.

I also use Intent Events to identify the paths that a user naturally takes while completing an action. For example, with our invoicing and bill payment app, do users start sending invoices by importing the contact first, or creating the invoice first? How does this behavior change as our bill payment network grows?

Similarly, in Gojek’s food delivery product, in the early days, we noticed our most successful users were those who already knew what they wanted to eat and were coming to Gojek simply to fulfill the delivery service. The intent event of these users would be searching for a specific restaurant, finding the menu items they wanted, and finally, setting their delivery details. However, as these users matured, we noticed the most prevalent user intent journeys shift as users began using Gojek more as a means of new restaurant discovery rather than fulfillment of restaurants they already knew. Now, intent events would be things like scrolling through their friends’ food feeds, browsing discounted deals, or using the Nearby feature.

These intent events are crucial for understanding what Bangaly Kaba (Reforge EIR, Former Head of Growth at Instagram) calls our adjacent users. As users mature and markets expand, these journeys evolve over time and our product should, too, by matching the intent of both new and mature users.

2c – Failure Events

A failure event is something that happens between an intent event and success event that prevents the user from achieving success. In-between intent events and success events live a number of failure paths that users could encounter. Understanding the failure paths is just as critical to understanding the success paths as they give us actionable information on how to improve the success event. To come up with failure events I ask – What are the possible things that prevent the user from completing the goal? There are two types of failure events:

• Implicit Failures are the drop-offs that occur before the successful completion of the goal. The user just “disappears” from our journey. In example two of our journeys above, events are tracked in a way that provides two implicit failure indicators:

  • Users that perform Create Invoice Selected but do not perform New Invoice Started within 5 minutes indicate a failure in our activation journey.

  • Users that perform Contact Search but do not perform Recipient Added to Invoice within 5 minutes indicate a failure in our search results or search history. Users may not feel motivated enough to create a contact from scratch or found confusing results.

• Explicit Failures are events where the intended experience goes wrong.

  • Events like “Ride Cancelled” on Lyft or “Order Cancelled — Restaurant Closed” when ordering food delivery are examples of Explicit Failures

  • In Honeydu, Add New Payment Method Failed and Pay Invoice Failed are two examples of events that often get forgotten in the event tracking exercise because they are responses to a user’s behavior, rather than actual actions taken within the product. However, if your web/mobile app receives errors and displays them to your user, these should be easy to track and log for monitoring. Storing these error response messages as an event property is an easy way to quickly diagnose why a common user journey may suddenly be failing.

3 – Properties

Once we have our success, intent, and failure events the next step is to figure out the properties that we want to associate with the event. Properties are once again a key to achieving two of our main goals, providing the right level of abstraction and making the data actionable.

Properties are essentially things ways that I might want to segment the events. A key mistake is tracking a segmentation as an event itself. For example:

• Good: Signup Up Selected (event), Source (property), Facebook (property value)

• Bad: Facebook Signup Selected

A key source to understand which properties you might need to track are the questions and hypotheses you uncovered in step one.

• Question Example: How do users prefer to add their contacts?

  • Property Example: source → History / Import / Manual Entry

• Hypothesis Example: Users that are new to freelancing are more likely to use templates to get started, and will require more onboarding to get to core value than users that choose to Build Their Own invoice.

  • Property Example: templateName → (null)/Basic Invoice/etc.

Some high level questions I like to ask to determine which properties are important:

• How will I be able to segment users who became frustrated versus disinterested?

• How will I be able to identify the different paths that mature versus casual users use?

• Does this give me enough nuanced data to compare and contrast successful users from dropped off users?

• If this were the last event that I ever tracked from a user, what would I want to know about the user’s experience on this screen?

Properties tend to fall into one of a few common buckets. To make sure I’m being thorough, I use these buckets to see if I’m missing anything:

User Profile Properties

The most common set of properties are those related to the profile of the user. This could demographic or firmographic information. Some examples:

• City

• Age

• Company size

• Role

• Product Tier

Often times these are things that you want to be able to permanently segment both users and events onwards until the property is changed. Some platforms like Mixpanel contain a feature like super-properties that allows you to do this easily. Questions to ask to figure out which of these properties to track:

• If I were to be a personal assistant for this user, what are the preferences I’d need to know about them in order to be helpful?

• What demographic information might influence the behavior of a user?

Marketing Properties

The second most common set of properties are those related to marketing that might impact or influence the user’s behavior. This could include things like:

• Source

• Campaign

• Entry Page

User Action Properties

Another set of properties are those related to actions of the user. For example:

• First Purchase Date

• First Service Type

• Total Orders

Here it is important to distinguish between two types of properties:

• Set and Forget – These are properties you set once, but don’t change after. Examples would be things like First Purchase Date, First Sign Up Attribution, and Birthdate.

• Append/Increment – These are properties you use to segment and create relevant, personalized user experiences. Incremented attributes could be things like Total Purchases, Total Revenue, and so on. Appending (and removing) user attributes allows teams to quickly identify relevant users for promotions, new updates, and experiences they’ve already indicated interest for. Examples could be things like a list of restaurants you’ve already purchased from (food delivery), a list of stores you’ve favorited (hyperlocal POIs), or features the user has used.

Type Of Action Properties

Most events have a type associate with them. Distinguishing between the type is important to getting actionable data. Some examples:

• Ride Cancelled: User Initiated vs System Initiated

• Payment Selected: Credit Card vs Wire Transfer

• Photo Uploaded: Camera vs Gallery

• Login Successful: Google vs Facebook vs Email vs Phone

To figure out the types I ask questions like:

• Who is responsible for this conversion (or failure)?

• What is responsible for this conversion (or failure)?

• What are the preferences this user has in completing this action?

• How would I describe the most important user journey path for this action?

• What additional information could I use to predict future actions of this user based on this action?

Contextual Properties

Contextual properties are those that help me understand what might influencing the motivation of the user to complete or not complete the goal. Some examples:

• The number of drivers on the screen

• The types of merchants displayed

• The # of search results

Questions that I find helpful to uncover contextual properties might include:

• What could influence motivation of the user to complete the goal?

• How could I differentiate an increase or decrease in motivation?

• Imagine this was the last event that we ever track from the user. What would we want to know about the experience?

4- Pressure Test

Once you have your set of events and properties defined, you should pressure test understanding and actionability.

Testing For Understanding

An event or property might make sense to us, but the question is does it make sense to your end audience? If people don’t understand the basics like what an event is, then doing actual analysis isn’t going to happen. The key here is that you should get business users, not developers, to sign off.

At Gojek, the first time things were instrumented by our mobile developers. They created event names for our mobile application like “Signup Handler” or “Cached Result Feed.” Those events made sense to them, but were a foreign language to the business users. Because of the obscurity, no one used any of the analytics tools we had implemented.

There are a few audiences I recommend testing for understanding:

1. Business Users Close To Product Development

   The first group you should test with are business users close to product development. This is typically product managers. Those that are closer to product development tend to have more knowledge of how the product works. So if things are confusing to them, it will be confusing to those with less knowledge.

2. Business Users Farther Away From Product Development

   The next group are business users that are farther away from product development. This is typically marketers or those in customer support. They typically have less detailed knowledge of how the product works, but should be able understand all of the events and properties.

3. New Employees

   The next group are new employees. New employees haven’t gotten accustom to the companies lingo or acronyms yet. Ultimately you are looking for these new employees to be able to understand the events and data with out tons of hands on training.

   At Gojek I implemented a “quick quiz” for users to get access to our analytics tools. I knew our event tracking was starting to become a mess when users wouldn’t be able to “pass” the quiz. So I recommend people to flip “onboarding and testing” of new employees on its head and use it to test how good your “internal event instrumentation product” is instead!

In any of these cases, if you find something that is confusing to the end user, a simple question to ask to uncover new naming would be “What would you call when a customer [does action] on this [part of the product]?”

Testing For Actionability

Understanding the data is the first step, but we need to pressure test whether or not the data is actionable. Here, we should go back to list of questions and hypotheses that we generated in step one, choose a sample of them, and pressure test how we might answer those questions and hypotheses.

At Gojek a common question that came up across the product, operations, finance, and research teams was “What makes the [best/happiest/most productive] driver?” The key here is that what “happy” is for finance might be different from what “happy” is for the Driver Benefits team. Or what “most productive” is different from Transport Team than it is from the Food Delivery team.

So we needed to test for actionability across these teams and use cases by coming up with hypotheses of how we might analyze the question from the various perpsectives. Could we segment drivers into human-understood groups like 5 Star Drivers, Long Distance Drivers, Service Type, and more.

5 – Track “Decisions Made Without Data”

No matter how thorough you are with the above process, there will always be changes you need to make. The business, goals, and product are constantly changing creating new needs. You can never anticipate 100% of the questions and hypotheses that need to be answered.

One exercise that I recommend the data team run regularly is what I call “Decisions Made Without Data.” It’s pretty simple. Every quarter we kept a list of decisions that the broader team made without data.

For example, while at Gojek a question emerged whether or not paying drivers more quickly drove driver retention. At the time we weren’t tracking the proper data to inform this question. Someone had come up with a threshold based on their own intuition and rationale. Uncovering that decision made without data led us to start tracking some new things in the data so that we could confirm speed of payment did drive retention and develop specific product and marketing initiatives around it.

The exercise helps you uncover things that you either overlooked, didn’t anticipate, or changed in the business. To be clear, I’m not saying that everything must be decided only when data is available. The reality of any fast growing company is that some decisions will need to be made with out data. But the list helps uncover situations where we lacked data that could have been helpful in a key decision.

Ongoing Signals of Success

Creating an excellent system of data in an organization is always an ongoing effort of iteration. When you are working on internal tooling and systems where your customer is another set of employees vs the end customer for the company, you don’t have the visceral feedback of revenue or other metrics to tell you whether you are doing a bad, good, or great job. Here are some signals you can use to understand how things are going:

Bad signals:

• There is a single person who knows how to do tracking — no one else knows how to write the event specs themselves

• Data analysts are needed to do basic analysis around key goals and objectives Event names and property names are duplicated, in multiple casings (e.g. Sign up and Sign Up)?

• The number of decisions made with out data each quarter is growing

• Usage of the analytics tools is low

• When new features and products get added, event tracking isn’t updated to reflect the new paths

• Funnel data is logically impossible (e.g. more users doing Step 2 than Step 1)

Good signals:

• Lots of teams are using the event tracking sheet and analytics tool (track usage of your tools like you track usage of your product)

• Other teams are contributing and trying to track new parts of the product development

• The event tracker gets updated as new features/ways of doing things in the app are implemented

• It’s easier for a business team to pull the data from the analytics platform than to write transactional queries to find the answers to their questions

Great signals:

• Event tracking is embedded within your regular goal and objective setting to make sure the team is armed with the proper information.

• Teams further and further away from the development process (i.e. customer support) are using the tooling on a regular basis with out tons of detailed training.

• Even when the product has a major redesign, the event tracking is able to leverage the original event names and properties logic.

• Teams can put their money behind it — the event tracking can be trusted to segment users and allocate user incentives (e.g. referrals, discounts, promotions).