Overall

The overall shape of the distribution is about what I expected. I imagine that most restaurants that hover around the 1-2 star range don’t stay in business very long, so they aren’t very well represented here. And even the best restaurants have off nights and off customers, so it’s not surprising that there aren’t a ton of 5 star restaurants.

By State

Does this distribution look different state to state? If I filter out the states that only have a small number of restaurants, the distributions look pretty similar state to state. All state distributions are negatively skewed with very similar averages. Just from a first look, it doesn’t seem like you see a lot of variation in how state’s restaurants are rated.

# From initial plots, I know many states are included that have only
# a handful of reviews. Eliminate these from plotting since they don't
# contain enough data to plot meaningful distributions
df <- data.frame(state = c("AZ", "NC", "NV", "OH", "PA", "WI"), 
                 full = c("Arizona", "North Carolina", "Nevada", "Ohio", "Pennsylvania", "Wisconsin"))
avgs <- restaurants%>%distinct(business_id, .keep_all = TRUE)%>%group_by(state)%>%select(stars)%>%summarise(avg =round(mean(stars), 2))
df <- inner_join(df, avgs, by =  'state')
# Hackish way to put avg rating on plot facets
df <- bind_cols(df, 
                pos = data.frame(y = c(2000, 750, 1500, 1000, 750, 325)))


restaurants%>%
  # remove duplicate businesses (from the categories ennumeration)
  distinct(business_id, .keep_all = TRUE)%>%
  filter(!(state %in% c('AK', 'CA', 'CO', 'IN', 'NY', 'VA', 'IL', 'SC')))%>%
  # join my avg's df to be used for facet labeling
  inner_join(df, by = 'state')%>%
  group_by(full)%>%
  ggplot(aes(stars))+
  geom_histogram(fill = 'midnightblue', color = 'black', alpha = .7, bins = 9)+
  facet_wrap(~full, scales = 'free')+
  my_theme()+
  labs(x = 'Ratings',
       y = '# of Restaurants',
       title = 'Distribution of Yelp Ratings by State')+
  geom_text(data = df,
            aes(x = -Inf, y = y, label = paste0("Avg Rating: ", avg)),
            hjust = -0.1,
            vjust = -1,
            size = 2.5,
            family = 'AvanteGarde')

By Category

When I look at how ratings are distributed by category, I start to see some variation. Below are the 6 most common restaurant categories and their ratings distributions. While ‘Food’ and ‘Sandwiches’ look similar to the distributions I have seen so far, ‘Fast Food’ is quite different. The skew is reversed–a majority of restaurants have 2-3 star ratings instead of 3-4– and the distribution is much wider. ‘Bars’ and ‘Nightlife’ are narrower distributions with more pronounced peaks and ‘American’ is much closer to normal than the others.

The data in the table is exactly what’s visible in the plot. ‘Fast food’ has a lower average rating with the highest variance and ‘Bars’ and ‘Nightlife’ have the lowest variance of all the categories included. Variance in data is a good thing here as it indicates restaurant categories are a useful predictor of restaurant ratings. I can explore this idea a bit more by taking a look at the highest variance restaurant categories and comparing to the lowest variance ones.

Notice that automotive is high variance, but the number of restaurants with the automotive tag is very low, so it’s very possible that this distribution doesn’t have enough data to be trustworthy. The others make sense though– at least in my opinion. Aside from automotive, these are all fast, casual food types where you can see a wide variety of customers and a wide variety of quality. A good burger to one person is awful to another just because of the peculiarities of taste.

Conversely, if I look at low variance categories I see they tend to be more niche. My guess is that the clientele at these sort of restaurants knows, generally, what to expect, and so they are less often ‘surprised’ (in both good and bad ways) by the food. That’s just speculation though, so who knows. Note: the low variance plot is actually a sample of the low variance categories because there are ~30 categories with a similar variance at the low end. Make sure you use the same seed I did if you want to replicate the plot.

This variation by category makes some intuitive sense and is something that could be included in a standard regression sort of ratings prediction algorithm. I probably won’t extend the NLP based predictions to include outside variables such as categories, but it could be done and is something that should certainly be looked at if you were looking to really fine tune your model. For now, it’s enough to say that there is a relationship between restaurant category and restaurant rating.

Alternative plot for Categories

While viewing the distribution this way is informative, ultimately I’m using the overall rating for all restaurants of category X to build the plots. I actually have individual review level data though, so I should take advantage of it. I can gather up the many thousands of reviews about, for instance, Chinese restaurants and use those to display the breakdown of each categories reviews. This will give a nice tidy view of how reviews of each restaurant category vary.

There are over 600 unique categories, so first I have to find the most common categories and filter down the dataset. Here I choose categories in the top 72%(because that made the prettiest plot!), but you could play around with any number to make a plot to your liking. From there, it’s just a matter of grouping the restaurants by category and rating to get review totals and then using that information to calculate averages for each category. Simple, right?

The final plot below is ordered by average category rating–the top category has the highest average rating and the bottom the lowest. You can see from this the higher the average rating, the higher the proportion of 4 and 5 star reviews. This is something that intuitively we know, but the plot hammers home the point.

# Select a color palette for the plot
colors = c(brewer.pal(9, 'PuBu')[c(5, 6, 7, 8, 9)])

# I need information from both df's so I join them on business_id
joined_df <- inner_join(restaurants, sample_reviews, by = 'business_id')


t <- joined_df %>%
  # Group by category and sum the # of unique restaurants of each category
  group_by(categories)%>%
  mutate(t = n_distinct(business_id))%>%
  # Need to ungroup to filter down to the top 3/4's of the restaurants
  ungroup()%>%
  filter(t >= quantile(t, .28))%>%
  # Now we regroup by category and star so we can calculate the total number
  # of reviews category X had for 1, 2, 3, etc stars
  # Looks as follows eg:
    # Chinese: 1 star, 1234 reviews
    # Chinese: 2 star, 1494 reviews
    # Chinese: 3 star, 2003 reviews
    # Chinese: 4 star, 2500 reviews
    # Chinese: 5 star, 750 reviews
  group_by(categories, stars.y)%>%
  summarise(count = n())%>%
  # redo grouping by just category
  ungroup()%>%
  group_by(categories)%>%
  # grab the avg rating for each category NOTE this way sucks, I need a better way to do it
  mutate(working = stars.y*count, avg = sum(working)/sum(count))%>%
  # clean up df
  select(-working)%>%
  # sort by categorey and rating
  arrange(categories, stars.y)

# Use the df created above to get a ranking of the categories by avg rating
# This is used to order the categories in the final plot
rankings <- t %>%
  distinct(categories, .keep_all = TRUE)%>%
  select(categories, avg)%>%
  arrange(desc(avg))

# Reorder categories by avg so that you get the cascade effect seen in the final plot
t$categories <- factor(t$categories, levels = rev(rankings$categories))

# actual plotting
ggplot(t, aes(x = categories, y = count, fill = as.factor(stars.y)))+
  geom_bar(stat = 'identity', position = 'fill', alpha = .8)+
  coord_flip()+
  # this percent labeling requires scales package
  scale_y_continuous(labels = percent)+
  my_theme()+
  theme(legend.title = element_blank(),
        legend.position="top",
        legend.direction="horizontal",
        legend.key.width=unit(0.75, "cm"),
        legend.key.height=unit(0.1, "cm"),
        legend.margin=margin(0, 0, -0.1, -2, "cm"))+
  scale_fill_manual(values=colors, labels = c("1 Star", "2 Star", "3 Star", "4 Star", "5 Star"))+
  labs(y = 'Proportion of Reviews',
       x = '', 
       title = 'Yelp Reviews by Restaurant Category')

All reviews(from the sample data)

So what sort of reviews are people leaving on Yelp?

Interesting! People are actually more likely to leave good reviews, especially 5 star ones. If I think back to the overall distribution in 3.1.1 I can imagine how this glut of 4 and 5 stars reviews is moderated by the 1 star reviews to bring the average restaurant level review down to about 3.5. Just mulling over how I imagine people use Yelp, I’d imagine you see so many good reviews because a good restaurant experience is something people are excited about and probably motivated to share. A mediocre experience isn’t something–at least personally–that people are as compelled to post like a really good or bad review.

Revisiting variance, but on the individual review level

Before, we saw that certain types of restaurants had a lot more variation in their overall ratings than others. Does this hold true when we start to dig down into the individual review level? If I extract the highest variance categories on the individual review level and plot them I see some old friends from our distribution plots in 3.1.

joined_df%>%
  # Could use this filtering to get the exact columns as before, but they aren't strictly the highest variance at this level.
  # filter(categories %in% c("Burgers", "Chicken Shop", "Chicken Wings", "Fast Food", "Hot Dogs", "Donuts"))%>%
  group_by(categories)%>%
  mutate(total = n_distinct(review_id), var = var(stars.y))%>%
  ungroup()%>%
  # Had to play around with the # of reviews to filter on here. Could be changed
  filter(total >= 250)%>%
  filter(var >= quantile(var, .97))%>%
  group_by(categories, stars.y)%>%
  summarise(total = n())%>%
  ggplot(aes(stars.y, total))+
  geom_col(fill = 'firebrick4', color = 'black', alpha = .7)+
  facet_wrap(~categories, scales = 'free')+
  my_theme()+
  labs(x = 'Ratings',
       y = '# of Reviews',
       title = 'Yelp Ratings for Restaurant Types with \n High Variance')

These restaurants all have significantly more 1 star reviews than a typical Yelp restaurant. It’s a much flatter plot than, for example, this group of low variance categories shown below.

This reinforces the relationship discovered above between restaurant category and rating. We see it on both the restaurant and individual review level. This is definitely a feature I’d want to include in a more standard, tabular data sort of supervised model. I’m curious whether the text in the reviews of these various categories differs to a degree that we can observe. Further still, are these categories so different that a text based model will perform at different levels on different categories. Or to clarify by example–it’s commonly known that many facial recognition neural networks are significantly less accurate when trying to identify members of ethnic minority classes. Am I going to see similar issues when dealing with minority restaurant categories since they appear to behave differently in the way they are reviewed? Let’s look into the review text itself to see if we can gather some more insight.

Review Length By Rating

If I zoom in a bit and look at the past two years faceted by ratings, differences start to become apparent.

avgs <- total_words_by_review%>%
  group_by(stars)%>%
  summarise(`Average word count` = round(mean(total),2))%>%
  rename(Stars = stars)

total_words_by_review%>%
  filter(date >='2015-12-11')%>%
  group_by(weekly = floor_date(date, 'weekly'), stars)%>%
  summarise(avg_len = mean(total))%>%
  ggplot(aes(weekly, avg_len))+
  geom_line()+
  stat_smooth(size = .5, se = FALSE, color = 'red')+
  my_theme()+
  facet_wrap(~stars, scales = 'free')+
  labs(x = '',
       y = '# of Words in Average Review',
       title = 'Average Yelp Review Length by Rating, by Week')

Good reviews tend to be significantly shorter and 5 star reviews are the shortest of all by a good amount. Perhaps people like to vent when they have a bad restaurant experience, but when they have a good one they are more succinct – ie “Great Food!” could be a full review for a 5 star restaurant.

Review Length By Category

I want to briefly examine the issue discussed at the end of section 3.2.2. Am I able to discern differences in the text of reviews between categories. Using information I gathered above, I’ll filter down to reviews for a couple of the highest variance (Fast Food and Chicken Wings) and a couple of the lowest variance(Vegan and Vegetarian) categories.

It appears that the higher variance categories (Chicken Wings and Fast Food) tend to have shorter poor reviews than our sample low variance categories. At the high end of the ratings there isn’t much difference though. This is a good start to determining if you can tell the difference between categories by review text alone. It seems like review length could be one helpful indicator.

Has the Average Sentiment of a Review Changed Over Time?

One fascinating area to explore is whether reviews have become more positive or negative throughout the years. If I group the reviews by month and rating, I can examine whether, perhaps, bad reviews have gotten more extravagantly negative throughout the years.

word_per_review%>%
  inner_join(get_sentiments('afinn'), by = 'word')%>%
  inner_join(sample_reviews, by = 'review_id')%>%
  filter(date >='2009-01-01')%>%
  group_by(monthly = floor_date(date, 'monthly'), stars)%>%
  summarise(sentiment = mean(value), words = n())%>%
  ggplot(aes(monthly, sentiment))+
  geom_line()+
  geom_smooth(color = 'red', size = .5)+
  facet_wrap(~stars, scales = 'free')+
  my_theme()+
  labs(x = '',
       y = 'Average Monthly Sentiment',
       title = 'Average Sentiment of Yelp Reviews 2009-2017, by Rating',
       subtitle = '(Data Grouped by Month)')

(Note: I decided to filter out reviews before 2009.) This is a pretty amazing insight. It seems that negative reviews have gotten more negative and positive reviews have gotten more positive. Interestingly, if you don’t facet this plot, the average sentiment has stayed pretty much constant since 2009. I’m a little bit suspicious of this drastic of a change in sentiment in the reviews. My gut is that the shortening length of 5 star reviews over time is giving greater emphasis to the words you’d expect in a 5 star review –amazing, fantastic, etc. So the presence of these words in shorter and shorter reviews is pushing up–or down in the case of negative words–the averages over time.

Review Length versus Review Sentiment

I can explore this a tiny bit by doing a scatter plot of individual reviews with review length on the X axis and sentiment for that review on the Y axis.

length_sentiment <- word_per_review%>%
  inner_join(get_sentiments('afinn'), by = 'word')%>%
  group_by(review_id)%>%
  summarise(sentiment = mean(value), words = n())%>%
  inner_join(total_words_by_review, by = 'review_id')

length_sentiment%>%
  ggplot(aes(total, sentiment))+
  geom_point(alpha = .1)+
  geom_smooth(color = 'red', se = FALSE, size = .5)+
  geom_hline(yintercept = 0, color = 'steelblue', linetype = 2)+
  facet_wrap(~stars)+
  my_theme()+
  labs(x = '# of Words in Review',
       y = 'Review Sentiment',
       title = 'Yelp Review Length vs. Review Sentiment Score',
       subtitle = 'Grouped by Rating')

So short reviews seem to be more hyperbolic. The shorter the review the more likely it will be scored with a very high or very low sentiment rating. That seems to fall in line with what I mentioned above about a handful of very positive or negative words carrying a ton of weight in shorter reviews.

But one question you should have when looking at this plot is ‘How come 1 star and 5 star reviews both have so many high and low sentiment reviews?’ And that’s a good point. Why are we seeing low sentiment 5 star reviews at all? This is an issue with using basic sentiment analysis. ‘Feta-freaking-licious. Holy shit. That’s all I can say.’ is a 5 star review, but has a -4 sentiment score. There are several reasons for this. The ‘afinn’ sentiment dataset is relatively small and many words aren’t captured by it. This reduces the number of words that are scored, again giving more weight to those words that are scored. One of the words that is included is ‘shit’, which happens to receive a -4 score. Any human that reads this review will recognize that ‘Holy shit’ is a good exclamation in this context, but our naive sentiment scoring can’t pickup on that. This sort of sentiment analysis will also mishandle negation of words – ‘not like’ is obviously negative to us, but will be see as ‘like’ to the sentiment analysis(at least in it’s present form).

And similar things happen in the negative reviews that receive high sentiment scores. ‘Would make for an outstanding buffets for dogs, raccoons or other animals.’ is obviously a negative review to humans, but all our sentiment analysis sees is ‘outstanding’. There are a number of ways you could go about strengthening this sentiment analysis–for instance, by using bi and trigrams to gather more context from each review–but I won’t deal with that here. Since my model is a completely different NLP approach I’ll just point out the limitations and gather what information I can from the sentiment analysis.

Even with all that bad news about sentiment analysis, I can still use it to gain some valuable, subtle insights. I can see that shorter 4 and 5 star reviews tend to be very positive. Looking into those reviews, you see a lot of ‘Outstanding food, and some random sentence of praise here’ reviews. Short, sweet, and very positive. As we get longer reviews, it’s not that these ‘outstandings’ disappear, it’s that they are moderated by a plethora of other words in the longer review. It’s essentially a regression to the mean effect. You see that in the trend line drifting down to a slightly lower sentiment. Similar effects are seen in the 1 star reviews, but in the opposite direction. Reviews gradually trend more positive as the length of a bad review grows–all the starkly negative words are averaged out with more moderate sentiment words.

Average Sentiment by Category

Another step towards understanding the difference seen between categories is looking at the average sentiment of their reviews. Here, again, it’s obvious that sentiment of these high variance review restaurants is generally lower. Now it’s important to note that Fast Food and Chicken Wing restaurants on average receive lower ratings than Vegan and Vegetarian restaurants, so we’d have to control for this rating difference to see if this sentiment difference is real or is an artifact of the rating difference. I think this could be an instance where propensity score matching would help balance the dataset so we could more accurately examine the difference in sentiment rating between restaurant types, but I’ll leave that as an exercise for the reader.

For now, I can say that sentiment for Chicken Wings and Fast Food has trended down over the past several years while Vegan and Vegetarian may have seen a mild increase in average sentiment.

What are the most influential words in good vs. bad reviews

One last way I will examine review sentiment is to see which words have the greatest influence on the sentiment of reviews. I’ll do this by looking at the number of times that a word occurs in a review and multiplying that by that words sentiment score.

As you can see below, the contributions look quite different for each separate rating. 1 star reviews are overwhelmed by words such as bad, worst and horrible, while 5 star reviews are dominated by positive words such as amazing and love. It’s important to, again, point out that sentiment analysis by word won’t catch certain intricacies of the English language. I have a feeling if I were to look at bigrams of review text, I’d see that what actually occurs most often in 1 star reviews is ‘didn’t love’ or ‘not recommend’ rather than ‘love’ and ‘recommend’. This sort of negation is missed by word level tokenization, but could be easily captured by bigrams if you were so inclined.

Still, the visualization is quite informative. You see more and more positive influence as you go from 1 to 5 star reviews, which tells me I am capturing the general idea of a review even if I am missing some details. In general, this gives me confidence that there are actual word usage differences between ratings that my model will be able to identify and use for prediction.

t <- word_per_review%>%
  anti_join(stop_words)%>%
  inner_join(sample_reviews, by = 'review_id')%>%
  select(-text)%>%
  inner_join(get_sentiments('afinn'), by = 'word')%>%
  group_by(stars, word)%>%
  summarise(occurences = n(), contribution = sum(value))%>%
  top_n(15, abs(contribution))%>%
  ungroup()%>%
  arrange(stars, contribution)%>%
  mutate(order = row_number())


ggplot(t, aes(order, contribution, fill = contribution > 0))+
  geom_col(show.legend = FALSE)+
  facet_wrap(~stars, scales = 'free')+
  scale_x_continuous(
    breaks = t$order,
    labels = t$word,
    expand = c(0,0)
  )+
  coord_flip()+
  my_theme()+
  labs(x = '',
       y = 'Contribution toward Sentiment',
       title = 'Words with the Highest Contribution Towards Sentiment')

What words are influencing our selected restaurant categories?

And finally I decided to do the same sort of sentiment contribution analysis, but focus on our selected restaurant categories. The differences aren’t as noticeable here, but some word contributions still stand out. Fast food has a lot of contribution from ‘clean’ and ‘fresh’ and ‘wrong’ while those words don’t show up in any of the other 3 categories. I imagine Fast Food reviews focus on slightly different signals than other restaurants. Since fast food is standardized across restaurants, the differentiating factors are things like the restaurant being clean, or the fries being fresh, or your order being wrong. These are all things that are taken for granted–more or less–at non-fast food restaurants and so maybe these factors rise to a higher level of importance for the fast food category, resulting in their appearance in more fast food reviews.

If there are similar identifiers in other categories then it bodes well for there being textual differences amongst the different restaurant categories. They may be subtle, but hopefully our neural network will pickup on them.