• Problem Definition
• Optimization Objective
• Optimization Strategy
• Key Metrics
• Structure of Solutions
• Problems, Solutions & Impacts
• ML Solutions in Details
• Overview - How do they iterate?
• From Manual Scores to GBDT Model (2015 - 2017)
• GBDT Regression Model
• GBDT Ranker
• From GBDT to Deep Learning Model (2018)
• Failed Models
• Hotel Embedding & Query Embedding
• Simple NN
• Lambdarank NN
• NN model with features from GBDT and FM
• Deep NN
• Improve Deep Learning Model (2020)
• Two Tower Architecture
• Cold Start for new Listings
• Position Bias
• Optimize on All Listings, Improve Diversity (2023)
• Tricks and Lessons Learned from Airbnb
• Next Step
• References

Problem Definition

Optimization Objective

Optimization Objective is set to focus on bookings(or conversion).

1. prioritizing listings that are appealing to the guest but at the same time demoting listings that would likely reject the guest

Optimization Strategy

To achieve the above objective, there are 2 stages for the optimization strategy:

1. First stage (before 2020): treat every listing independently and assume that the booking probability of a listing could be determined independently of other listings in search results.
2. Second stage (after 2020): Optimize on all listings, treat the all listings as one; try to diverse the result

Key Metrics

1. Offline evaluation
1. NDCG (Normalized Discount Cumulative Gain)
2. Online test
1. Main metrics:
1. booking
2. revenue
2. Other metrics such as engagement metrics: listing viewed; listing saved

Structure of Solutions

Classic two steps recommendation system:

• Step 1: Retrieve (or candidate generation):
1. location based queries (e.g. New York): Rule based: fetch everything near by
2. broad queries (e.g. Skiing in France): Model based
• Step 2: Ranking of listings:
1. Model based: query, user and listing features
2. Trained using user actions: clicks, wishlists, inquires, bookings and rejections
3. Tradeoffs between Guest and Host preferences

two steps recommendation system

Problems, Solutions & Impacts

The table below shows the detailed evolution process. The following steps were mentions to have brought significant booking gains:

1. Replacing the manual scoring function with a gradient boosted decision tree (GBDT) model in 2015
2. Replacing GBDT regression model with GBDT ranker model in 2017
3. In-session recommendation with hotel embedding in 2018
4. Replacing GBDT ranker with deep learning in 2020

ML Solutions in Details

Overview - How do they iterate?

From Manual Scores to GBDT Model (2015 - 2017)

1. Assign labels to different feedbacks from search session
1. booking: 1
2. click: 0.01
3. impression: 0
4. reject: -0.4
2. Two ways of training: point-wise and pair-wise training

GBDT Regression Model

1. Point-wise training
2. Loss: RMSE

GBDT Ranker

1. Pair-wise training
2. Within the same search session, generate listing pairs
3. Loss: cross entropy
4. In order to pushing good outcomes to the top, and at the same time, push bad

outcomes to the bottom.

From GBDT to Deep Learning Model (2018)

Failed Models

1. Listing ID as sparse feature of the deep learning model
2. Multi-task learning model to predict both booking and views

Hotel Embedding & Query Embedding

In Session Recommendation with Hotel Embedding

1. Steps
1. Offline training: Based on users search session and skip-gram(word2vector), get listing embedding
2. Online in-session recommendation: memorize listings that users liked or clicked, then recommend similar items based these info
2. Special modifications
1. Better negative-sampling: instead of sampling globally, sample negative listings from the same city
2. More weight to booked listings: in the booking sessions, the booked listing will always be the context listing
3. Cold start of new listings:
1. We use the provided meta-data about the listing to find 3 geographically closest listings (within a 10 miles radius) that have embeddings, are of same listing type as the new listing (e.g. Private Room) and belong to the same price bucket as the new listing (e.g. $20 − $25 per night).
2. Next, we calculate the mean vector using 3 embeddings of identified listings to form the new listing embedding.

Query Embedding

1. Steps
1. Offline training: Based on users' search queries sequence and skip-gram(word2vector), get query embedding
2. Online free search: autocomplete users' free search terms
2. Example:
‣
Users’ search query sequence
‣
Example of free search

Simple NN

1. Simple single hidden layer NN with 32 fully connected ReLU

activations

2. Features: Same features as GBDT
3. Loss: L2 regresson loss
4. Target:
1. 1 -> booking
2. 0 -> not booking
5. Performance: -> neutral bookings as GBDT

Lambdarank NN

1. Pair-wise NN
2. Loss: cross entropy
3. Target: same as previous model

NN model with features from GBDT and FM

1. The production model is NN.
1. From the GBDT model, we took the index of the leaf node activated per tree as a categorical feature.
2. A factorization machine (FM) model that predicted the booking probability of a listing given a query, by mapping listings and queries to a 32-dimensional space.

Deep NN

1. What did they do?
1. scaling the training data 10x
2. moving to a DNN with 2 hidden layers.
2. Some special features: Features output from another model:
1. Price of listings that have the Smart Pricing feature enabled,

supplied by a specialized model

2. Similarity of the listing to the past views of the user, computed based on co-view embeddings (hotel embdding)

Improve Deep Learning Model (2020)

Two Tower Architecture

Cold Start for new Listings

1. Main problem:
1. the absence of user generated engagement features like number of bookings, clicks, reviews etc.
2. Solution
1. Predict engagement features
2. Feed these features to NN model

Position Bias

1. Solution
1. Introduce position as a feature, and this feature will be set to 0 while doing inference
2. To reduce model’s dependence to this feature, this feature is regularized by dropout while training. Dropout rate is set to be 0.15 based on offline evaluation

Optimize on All Listings, Improve Diversity (2023)

1. Model is still two-towel structure DNN
2. Instead of building a single model, build models for each position
1. Position 0: reuse the regular pairwise model
2. From position 1 to N: model has addition input which is the listing from position 0 to position N-1

Tricks and Lessons Learned from Airbnb

While modelling, consider the specific property of their platform

1. two-sided marketplace, they need to meet the needs of guests and hosts
1. Data label: -0.4 for rejected listing
2. a listing can only be booked by one guest at the same time
1. If including listing id as feature, it might cause overfit
3. user never book the same listing for twice
1. in-session recommendation

Training Data

How to iterate

To improve our chances of success, we abandoned the {download paper 7→ implement 7→ A/B test} loop. Instead we decided to drive the process based on a very simple principle: users lead, model follows.

Next Step

1. Not all of the topics are covered in this summary. For example
1. AirBnb category
1. Building Airbnb Categories with ML and Human-in-the-Loop - Part 1
2. Building Airbnb Categories with ML & Human in the Loop - Part 2
2. More work related to diversity
1. Managing Diversity in Airbnb Search (2020)
2. Search & recommendation from Booking.com, Agoda ….