Dan discusses what led him to create a simple recommendation engine. 

The goal was to learn and try out concepts quickly.

Dan discusses what happens when someone takes an action on an artifact. 

Dan discusses using customer.io for a reverse ETL job.

Dan discusses showing examples and proof concept before creating a more reliable solution. 

Some people will say “You’re building tons of stuff on a shaky foundation and going to hurt yourself in the long term”. I agree it could hurt in the long term, but that I’d rather experiment with a lot of various things before figuring out what truly matters and then coming back to harden those features. The hard part is to know when to come back to update key parts. 

You can’t operate in the “move fast” move forever, I’ve been burned in the past by not paying down my tech debt. But I’d rather build something people want that I need to fix / update than spend time building things that don’t move the needle.

Dan Wolchonok

I created a simple recommendation system as a part of the early alpha for Reforge (it operated under workartifacts.com when it was live). I built all of this using separate tools from our standard stack. 

as a part of the early alpha for Reforge (it operated under 

 when it was live). We built all of this using separate tools from our standard stack. The benefits of using a different stack were that we didn’t have to coordinate with other teams about stepping on others’ toes, and we were able to move as fast as possible. 

simple service that would email someone when they took action on an artifact

 to see if we could bring them back days later. The concept was “Since you saved X, you might like Y” one day later. If someone saves 10 artifacts in one session, it shouldn’t send them 10 emails, it should only send one recommendation email.

We send events from our app (server side and client side) into Segment. The engineering team implements these when building and deploying features.

Segment forwards those events to other services (Amplitude, Customer.io) and populates our snowflake data warehouse with rows for each event.

I built a segment in customer.io that looks for any activity (upvote, artifact reaction, note reaction, save) in the past 3 days.

When people enter the segment, it triggers a workflow

I populate the recommendation data via a reverse ETL job

 configured in customer.io. I was impressed that the reverse ETL functionality was included in their lowest pricing tier. It syncs data from our data warehouse hourly to contacts.

 (work_artifacts_app.artifact_action_recs) that looks for any artifacts you've interacted with (across all of those actions) and then finds artifacts where you haven't interacted to make recommendation

I use the action to customize the email (upvote, save, etc)

We were pretty happy with the results of the campaign. There’s obviously room to improve this, but we were happy with a 20% CTR.

This is very much an alpha version of recommendations. Future improvements:

The data warehouse isn’t real time. As we move to bring this back to our production infrastructure, the data will only be stale by as much as an hour. This isn’t really an issue today because we delay the recommendation email by 1 day.

We need to move the snowflake view logic into a DBT model. Right now it’s a manually edited view in snowflake that isn’t in source control.

We rely on the manually curated related artifact data. In the future we could make this much more sophisticated, but it was more than enough for the initial launch.

We should setup monitoring of the data in these systems via Metaplane so that we are alerted when there are data issues. Examples might be: freshness of data is stale, events stop firing, etc.

It isn’t using subscriptions to help people manage which types of emails they’d like to receive. It isn’t clear whether the 3 day threshold is the right period of time.

We should look at taking engaged users behavior and using it to bring back inactive users.

Simple Recommendation Engine for Reforge

Simple Recommendation Engine for Reforge

Overview

Goal

Documentation

Sign up to see more