When buyers search on Etsy, they need to quickly and easily find the perfect item. At the same time, sellers need to be confident their unique products are being seen by the right customers. Our Ads Search ranking model, which is built on a multitask learning foundation, is the critical link in this connection. Recently, we identified an opportunity to drive more meaningful buyer engagement by enhancing our model’s ability to predict purchase intent. We achieved this via a dual-pronged improvement: introducing Multigate Mixture of Experts (MMoE) to our model architecture and leveraging add-to-cart as an auxiliary signal. By providing our downstream systems with more accurate predictions, we improved matching in our marketplace, surfacing more relevant listings for buyers while helping sellers reach customers who are genuinely interested in their products.BackgroundWhen a buyer searches for an item on Etsy, we want them to find exactly what they’re looking for from our inventory contain

Etsy has maintained a sharded MySQL architecture since around 2010. This database cluster contains most of Etsy’s online data and is made up of ~1,000 tables distributed across ~1,000 shards. Over the last 16 years, it has grown significantly: combined, these tables have over 425 TB of data and receive roughly 1.7 million requests per second. Etsy engineers access our MySQL data through a proprietary object-relational mapping (ORM). The ORM has a corresponding model for each MySQL table. When a table is sharded, its rows are partitioned among different databases known as shards. Each shard has identical table schemas and contains a distinct subset of all rows. Sharded tables require a unique id field to be designated on the model as the “shardifier id” which determines the shard for each record. Shardifier ids are designed to co-locate related data in order to minimize the number of databases we need to rely on for a given query. For example, all records related to a single shop or use

Ever searched for something specific, only to be met with results that are close, but not quite? On Etsy’s Search Relevance team, that frustration is exactly what we are tackling. Our goal is simple yet ambitious: to help buyers find exactly what they’re looking for, and to help sellers reach the people seeking their special products.Search plays a central role in that mission. Historically, Etsy’s search models have relied heavily on engagement signals – such as clicks, add-to-carts, and purchases – as proxies for relevance. These signals are objective, but they can also be biased: popular listings get more clicks, even when they’re not the best match for a specific query.To address this, we introduce semantic relevance as a complementary perspective to engagement, capturing how well a listing aligns with a buyer’s intent as expressed in their query. We developed a Semantic Relevance Evaluation and Enhancement Framework, powered by large language models (LLMs). It provides a comprehen

In 2021, we published a blog post titled “Increasing experimentation accuracy and speed by using control variates,” describing how we reduce the variance of metrics using CUPED in our experimentation platform. This is a follow-up on how CUPED has evolved at Etsy since then. Spoiler – It’s changed a lot, decreasing our average experiment duration by 3 days!Etsy’s mission is to Keep Commerce Human. To achieve this, we need to understand the impact each change to our platform has on our buyers' and sellers' experience. Whether that involves changing the color of the “Buy Now” button on the Etsy app or updating elements of how our algorithms rank search results, we leverage large-scale online experimentation to iterate on and improve the things we build.However, running an experiment can be a long process. From design and setup to running the experiment and analyzing results, the entire experimentation process can take weeks to months. Experiments must run long enough to collect sufficient

Rarely are there opportunities for big, bold, game-changing improvements in web performance. The Speculation Rules API (SRA) is a recent browser development that offers just such an opportunity. This post details a joint effort between the search and the web performance teams at Etsy to implement SRA on Etsy search pages and drastically improve the performance of product listing pages with some metrics seeing 20-24% improvements and some dropping all the way to 0ms values.Prefetching OptionsThere are two main ways to predictively fetch resources for the next page:<link rel="prefetch"> allows us to instruct the browser to download a resource we believe we'll need soon. The resource can be anything, like a static asset or an HTML page. Speculation Rules API (SRA) is a recently updated browser API which allows for a JSON definition to dictate what page A wants to do with page B. The "do"-ing can be either prefetching (just download the HTML) or prerendering (load the page, including its s

For more than 20 years, Etsy has been the destination for human creativity online. Our marketplace is home to more than 100 million special items made, handpicked and designed by more than 5 million sellers. These items and the real people behind them are what set us apart. But while the huge variety of Etsy’s inventory is one of our greatest strengths, it also creates fundamental engineering challenges specific to our marketplace. The challenge: Etsy’s unique inventoryWith millions of creative items across thousands of categories – many of which are unique – it’s difficult to accurately capture all possible product attributes, which range from standard attributes like “color” and “material”, to niche attributes like “bead hole size” and “slime additives.” The range of possible attributes and their values is so broad that it’s a challenge even to enumerate them, let alone label listings with specific attribute data. Unlike other online retailers (that may also have enormous inventories

Imagine you have a fabulous idea to drive more sales on Etsy by giving out free ice cream with every purchase. How would you know if it will actually work?One way to test this out is to run an experiment! An experiment is a learning tool that can help us see if a new change really affects user behavior and key metrics we care about. At Etsy, we run nearly 2000 experiments a year for testing new ideas to improve the user experience for our buyers and sellers.For our ice cream idea, we could set up our experiment as an A/B test on the listing page, where one group of users (the “control” group) sees the usual “Buy it now” button, while the other (the "treatment" group) sees a shiny new version that says, “Buy it now and get a FREE ICE CREAM!”.Comparison between the “Buy it now” button in the control and treatment groups in our example experiment. Disclaimer: Example for illustrative purposes only.We would then track how many people click the “Buy” button in each group to see if the new b

Every day, shoppers from Etsy's community of nearly 90M buyers visit our marketplace to search for unique, handmade, and vintage items. But with over 100 million listings, how do we help each buyer find exactly what they're looking for? Traditional search and recommendation systems often fall short of capturing the nuanced interests that make each Etsy buyer unique, from specific styles to aesthetic preferences. At Etsy, understanding our buyers' interests is central to delivering engaging, personalized experiences. Recently, we explored enhancing our personalization by leveraging large language models (LLMs) to create detailed buyer profiles based on buyers’ browsing and purchasing behaviors. We strive towards privacy by design and build these exploratory models with buyer privacy in mind. While this work is still in experimentation, we’re excited to share our early efforts and what we’ve learned.Understanding Buyer ProfilesBuyer profiles anonymously capture nuanced interests for each

This post investigates the benefits and limitations of prompt engineering in two instances of AI-assisted onboarding relying on large language model (LLM) technology. Of particular interest is how truthful (and therefore reliable) LLM-generated answers turn out to be in the context of Etsy-specific question answering. Among other insights, we find that asking the LLM to identify specific source snippets is a good way to flag potential hallucinations.Over the past few years, pre-trained large-scale/foundation language models such as OpenAI’s o-series [1] and Google’s Gemini family [2] have revolutionized the field of natural language processing (NLP). Trained on vast amounts of text, images, code, audio, and videos, such models encapsulate a great deal of world knowledge, which can be called upon to perform a wide range of downstream tasks, such as sentiment analysis, language translation, and natural language inference, among many others.The canonical way to improve the task performanc

In the past, sellers were responsible for managing and fulfilling their own tax obligations. However, more and more jurisdictions are now requiring marketplaces such as Etsy to collect the tax from buyers and remit the tax to the relevant authorities. Etsy now plays an active role in collecting tax from buyers and remitting it all over the world. In this post, I will walk you through our tax calculation infrastructure and how we adapted to the ongoing increase in traffic and business needs over the years.The tax calculation workflowWe determine tax whenever a buyer adds an item to their Etsy shopping cart. The tax determination is based on buyer and seller location and product category, and a set of tax rules and mappings. To handle the details of these calculations we partner with Vertex, and issue a call to their tax engine via the Quotation Request API to get the right amount to show in our buyer's cart. Vertex ensures accurate and efficient tax management and continuously updates t

One of our Guiding Principles at Etsy is that we “commit to our craft.” This means that we have a culture of learning, in which we’re constantly looking for opportunities to improve and learn, adopt industry best practices, and share our findings with our colleagues and our community. As part of that process, Etsy recently adopted Jetpack Compose – Android's modern toolkit for defining native UIs – as our preferred means of building our Android app. The process of adoption consisted of a gradual expansion in the size and complexity of features built using Compose, eventually culminating in a full rewrite of one of the primary screens in the app. The results of that rewrite gave us the confidence to recommend Compose as the primary tool for our Android engineers to build UIs going forward.AdoptionOur engineers are always investigating the latest industry trends and technologies, but in this case a more structured approach was warranted due to the central nature of UI toolkits in the dev

At Etsy, we’re focused on elevating the best of our marketplace to help creative entrepreneurs grow their businesses. We continue to invest in making Etsy a safe and trusted place to shop, so sellers’ extraordinary items can shine.Today, there are more than 100 million unique items available for sale on our marketplace, and our vibrant global community is made up of over 90 million active buyers and 7 million active sellers, the majority of whom are women and sole owners of their creative businesses. To support this growing community, our Trust & Safety team of Product, Engineering, Data, and Operations experts are dedicated to keeping Etsy's marketplace safe by enforcing our policies and removing potentially violating or infringing items at scaleFor that, we make use of community reporting and automated controls for removing this potentially violating content. In order to continue to scale and enhance our detections through innovative products and technologies, we also leverage state-

In 2020, Etsy concluded its migration from an on-premise data center to the Google Cloud Platform (GCP). During this transition, a dedicated team of program managers ensured the migration's success. Post-migration, this team evolved into the Etsy FinOps team, dedicated to maximizing the organization's cloud value by fostering collaborations within and outside the organization, particularly with our Cloud Providers.Positioned within the Engineering organization under the Chief Architect, the FinOps team operates independently of any one Engineering org or function and optimizes globally rather than locally. This positioning, combined with Etsy's robust engineering culture focused on efficiency and craftsmanship, has fostered what we believe is a mature and successful FinOps practice at Etsy.Forecast MethodologyA critical aspect of our FinOps approach is a strong forecasting methodology. A reliable forecast establishes an expected spending baseline against which we track actual spending,

Etsy features a diverse marketplace of unique handmade and vintage items. It’s a visually diverse marketplace as well, and computer vision has become increasingly important to Etsy as a way of enhancing our users’ shopping experience. We’ve developed applications like visual search and visually similar recommendations that can offer buyers an additional path to find what they’re looking for, powered by machine learning models that encode images as vector representations.Figure 1. Visual representations power applications such as visual search and visually similar recommendationsLearning expressive representations through deep neural networks, and being able to leverage them in downstream tasks at scale, is a costly technical challenge. The infrastructure required to train and serve large models is expensive, as is the iterative process that refines them and optimizes their performance. The solution is often to train deep learning architectures offline and use the pre-computed pretraine

Easily the most important and complex screen in the Buy on Etsy Android app is the listing screen, where all key information about an item for sale in the Etsy marketplace is displayed to buyers. Far from just a title and description, a price and a few images, over the years the listing screen has come to aggregate ratings and reviews, seller and shipping and stock information, and gained a variety of personalization and recommendation features. As information-rich as it is, as central as it is to the buying experience, for product teams the listing screen is an irresistible place to test out new methods and approaches. In just the last three years, apps teams have run nearly 200 experiments on it, often with multiple teams building and running experiments in parallel.Eventually, with such a high velocity of experiment and code change, the listing screen started showing signs of stress. Its architecture was inconsistent and not meant to support a codebase expanding so much and so rapid

Balancing Engineering Ambition with Product RealismIntroductionIn July of 2023, Etsy’s App Updates team, responsible for the Updates feed in Etsy’s mobile apps, set off with an ambitious goal: to revamp the Updates tab to become Deals, a home for a shopper’s coupons and sales, in time for Cyber Week 2023.The Updates tab had been around for years, and in the course of its evolution ended up serving multiple purposes. It was a hub for updates about a user’s favorite shops and listings, but it was also increasingly a place to start new shopping journeys. Not all updates were created equal. The most acted-upon updates in the tab were coupons offered for abandoned cart items, which shoppers loved.We spotted an opportunity to clarify intentions for our users: by refactoring favorite-based updates into the Favorites tab, and (more boldly), by recentering Updates and transforming it into a hub for a buyer’s deals.Technical OpportunityWhile investigating the best way to move forward with the De

A little while ago, Etsy introduced a new feature in its iOS app that could place Etsy sellers' artwork on a user's wall using Apple's Augmented Reality (AR) tools. It let them visualize how a piece would look in their space, and even gave them an idea of its size options. When we launched the feature as a beta, it was only available in "wall art"-related categories, and after the initial rollout we were eager to expand it to work with more categories. What differentiates Etsy is the nature of our sellers’ unique items. Our sellers create offerings that can be personalized in numbers of ways, and they often hand-make orders based on demand. Taking the same approach we did with wall art and attempting to show 3D models of millions of Etsy items – many of which could be further customized – would be a huge undertaking. Nevertheless, with inspiration from Etsy's Guiding Principles, we decided to dig deeper into the feature. What could we improve in the way it worked behind the scenes? Wha

IntroductionEach year, Etsy hosts an event known as “CodeMosaic” - an internal hackathon in which Etsy admin propose and build bold advances quickly in our technology across a number of different themes. People across Etsy source ideas, organize into teams, and then have 2-3 days to build innovative proofs-of-concept that might deliver big wins for Etsy’s buyers and sellers, or improve internal engineering systems and workflows.Besides being a ton of fun, CodeMosaic is a time for engineers to pilot novel ideas. Our team’s project this year was extremely ambitious - we wanted to build a system for stateful machine learning (ML) model training and online machine learning. While our ML pipelines are no stranger to streaming data, we currently don’t have any models that learn in an online context - that is, that can have their weights updated in near-real time.Stateful training updates an already-trained ML model artifact incrementally, sparing the cost of retraining models from scratch. O

IntroductionPersonalization is vital to connect our unique marketplace to the right buyer at the right time. Etsy has recently introduced a novel, general approach to personalizing ML models based on encoding and learning from short-term (one-hour) sequences of user actions through a reusable three-component deep learning module, the adSformer Diversifiable Personalization Module (ADPM). We describe in detail our method in our recent paper, with an emphasis on personalizing the CTR (clickthrough rate) and PCCVR (post-click conversion rate) ranking models we use in Etsy Ads. Here, we'd like to present a brief overview.Etsy offers its sellers the opportunity to place sponsored listings as a supplement to the organic results returned by Etsy search. For sellers and buyers alike, it’s important that those sponsored listings be as relevant to the user’s intent as possible. As Figure 1 suggests, when it comes to search, a “jacket” isn't always just any jacket:Figure 1: Ad results for the que

IntroductionThe Feature Systems team at Etsy is responsible for the platform and services through which machine learning (ML) practitioners create, manage and consume feature data for their machine learning models. We recently made new real-time features available through our streaming feature platform, Rivulet, where we return things like “most recent add-to-carts.” While timeseries data itself wasn’t new to our system, these newer features from our streaming feature service would be the first timeseries inputs to be supplied to our ML models themselves to inform search, ads, and recommendations use cases.Not too long after we made these features available to users for ML model training, we received a message from Harshal, an ML practitioner on Recommendations, warning us of "major problems" lying in wait.Figure 1. A user message alerting us to the possibility of "major problems for downstream ML models" in our use of the timestamp datatype.Harshal told us our choice to export real-ti