Getting the most from our research data

In this series we will go through the steps that we took and we'll share some of the actions that we are taking based on the insights gathered. We have followed the Spotify DIBBs (Data -> Insights -> Beliefs -> Bets) model so each post in this series will go into one of each of these steps.

1. Selecting our Systems
2. Analysing the Data
3. Gathering Insights
4. Documenting our Beliefs
5. Placing our Bets

Like many people we have struggled with getting the balance right between investing in the right level of research upfront and delivering "value" now. Our company objective for Q4 2019 was to redress this imbalance and set the company up with the right data and systems to enable data-informed decision making which will enable us to deliver the highest quality events for our customers.

Setting up for Success

For the past three months I have been experimenting with different tools and structures that would enable the collation and interrogation of the data from all of the different sources. We took a lean approach to this, trying many different tools and structures, iterating on the ones that worked, before settling on a set up that works for us. In this first post I'm going to go through how we experimented with our systems in order to be able to make the most of our data and has been the foundation which has enabled us to make the most of our data. Hopefully you can get some value from our experience.  

Collating data across sources

We have our ticketing software, spreadsheets that track feedback surveys, sli.do surveys, a database for talk feedback, individual user feedback, our chat tool, a physical question wall at UXDX 2019 and more. Each source holds bits of data that combine to tell the bigger picture about the problems our customers face and how well we are delivering on their needs. But we struggled with the best way to consolidate these into a single view that we can use to derive valuable insights.

Spreadsheets

Like most companies, we started with Excel / Google Sheets. The flexibility of spreadsheets makes them a great option but it didn't give us the level of data interrogation that we wanted. For example, we ask people what their goals for attending the conference are. We intentionally leave this as an open-ended question as we have been surprised by some of the responses that come up - for example we didn't expect "looking for a job" to be as high as it is but looking to meet new perspective employers comes up a bit. I tried multiple different approaches for creating tags to classify these goals but it was always messy and didn't really enable the reporting that we wanted.

Database

Linking different records together was the main challenge - goals and goal classifications being one example.  Databases are built around joining records so we tried this next. We started creating a few tables, loading in our data and we started teaching the team some basic SQL queries to be able to interrogate the data. The effort involved meant that people weren't using it as much as we should. The fact that you couldn't really browse the data easily meant that it was hard to generate the insights that we wanted. And it was slower than we wanted because everything from table design and creation to interrogation took a lot of thought and effort. We needed something more visual.

SQL Reporting Tools

I've previously worked with some data warehouse tools like Oracle BI, QlikView, Tableau and SQL Server Reporting Services (SSRS). We have a lot of data, but not crazy amounts, so we needed to ensure there was a cost effective solution. I looked into options like Google Data Studio and Amazon Redshift but the cost and effort / reward balance didn't align for a company of our size.

New SaaS Spreadsheet / DB hybrids

A colleague in Aer Lingus introduced SmartSheet to be able to track the product backlog for the Aer Lingus website (thanks Sophie). This ticked a lot of the boxes we were looking for. I investigated a few different options in this space including Coda, Smartsheet and Airtable. All of them are great but they each target slightly different use cases. We decided that Airtable met our needs the best.

Airtable gives us the speed and ease of use of a spreadsheet with the joins and linking of records that a database provides. What I really like is the ability to view the data in many different ways - from spreadsheets, to kanbans, to calendars etc. It took a bit of time to collate, cleanse and enter the data, which I will go into in more detail in another article, but the ease of creating and moving columns made this easier than it was using the databases approach. Trial and error, mixed with patience, was the key here.

Airtable isn't perfect and there are a few things we'd love. The way it works is you create bases (think of them like separate spreadsheets) - we have separate ones for sales, marketing, production, research, the UXDX Framework etc. We could have created one base for all of these as there is some information that is shared across all of them but we have chosen to separate them for better usability (to keep the number of tabs under control) as well as to keep under the 50k record limitation. This creates a bit of overhead for us to keep things in sync which is annoying - and we're not alone - the No. 1 feature request for Airtable is to be able to sync data across bases. Some other competitors offer this but I think the challenge for Airtable is that enabling this would mess with their pricing model. Hopefully they can find a solution as it is very frustrating at the moment. Apart from that, after spending the time to figure out how it works the only things that cause us some issues are the lack of offline editing and their forms offering is a little limited (but its great to have forms so can't complain too much).

Automating Integration

It took a long time to collate all of the data and the last thing we wanted is for it to go stale. This meant that we needed to integrate all of the different sources into Airtable so that the data stays up to date and we don't have to keep investing so much time in maintaining the data. Again we investigated a few options before landing on the best approach for us.

Manual Integrations

Most of the systems we work with have APIs and webhooks (Airtable actually doesn't have webhooks, but they integrate with Slack and Slack has webhooks so we use that workaround). We started by writing a few lambda functions to keep systems in sync. These worked and were free to run which is a great bonus (the number of invocations is well below the free thresholds offered by the cloud vendors).  But each integration was very time consuming to write and test and, since I was the one writing them, they were limited in their quality - they worked but I didn't implement a good logging and error reporting solution which meant that sometimes the data would get out of sync.

Integration Services

Nearly every service we use promotes Zapier as a way of integrating between systems so we started by looking into this. This looked really promising but when we started calculating the number of integration "tasks" that we would need, the number started to rise quite a bit and with them the monthly price. We really liked the idea of making the integrations simple though so we looked for competitors and that's when we found Integromat. It is similar to Zapier but with better pricing. After playing around with both we decided to go with Integromat.

Again, no tool is perfect. Integromat doesn't have as many options for connecting systems as Zapier but they do have most of the ones we care about and they are adding new ones at a great pace. Also, you can create custom webhook / http integrations for the systems that they don't have. The other challenge is around Upserts for Airtable - these are quite complicated to build in Integromat because you have to search for records first and then branch for the Create / Update actions. Integromat doesn't have an easy way to merge these branches back again yet. This makes your integrations "messier" and more difficult to decipher but this is something Integromat are working on fixing.

Lessons Learned

The right tools let you multiply the value of your data

We had always collated and reviewed the data that we captured but we lacked a way to really get the insights that we needed to make the best data-informed decisions on where to take our products. The ability to scan through the data in Airtable as well as the reporting capabilities makes it really easy to get the insights that we need.

Data Consistency

We experiment with the questions that we ask at each ticket purchase / post-event feedback survey which has been a great way to uncover some of the psychology of how people answer questions - but it made it a nightmare to try to consolidate all of the data. We wanted to make sure that we didn't lose this capability to experiment but also enabled some consistency for reporting. Airtable lets us do this by creating multiple different forms with different labels but they all map to the same columns.

There is still going to be a manual step in classifying goals, value received, challenges remaining etc. but by having a consistent structure the consolidation of items will be simpler.

The cost of manual / bespoke work

One of the key trends that we track and highlight at UXDX is the move to Serverless or "Service-full" systems (where you piece together multiple SaaS products). We have seen first hand how much time and money this really saves. With Integromat we were able to integrate a lot more systems in much less time than it would have taken writing each integration manually, and we have good error reporting built-in. We have also taken this approach with our re-design of our website which we will share later in the year when the re-design launches.

New use cases

Having gone through the process of consolidating our data it sparked new ideas on how we can link data. We have moved a lot of our systems from spreadsheets and other SaaS products into Airtable and it is really helping us to move faster. I always think of the push back on automated testing because companies used to only test 3 or 4 times a year. Why should you automate that? You would never get an ROI? But now we know that automated testing opens up so many more opportunities for companies that they previously didn't have - such as letting companies improve cash flow by releasing developed products earlier. By investing the time in getting our data right we were able to uncover many more uses for the systems and tools we discovered.  

Next Steps

With the structure agreed we need to move on to actually working with our data which you can read in our next post in this series.

Let me know if you have any comments or feedback on our approach as we are always learning and would love to hear more.