A/B Testing

    A/B Testing for Bidding

    To measure the impact of bidding in relation to waterfall, run an A/B test to split your audience into two segments. This ensures that changes you make are measured effectively, rather than relying on pre-post tests that might have fluctuating CPMs caused by differences in the number of users, updates to the app, and other factors.

    It's difficult to measure success by comparing performance before and after implementing bidding due to the difficulty of isolating other factors; however, A/B testing shows much clearer performance gains from bidding. The value of bidding shouldn't be measured on performance alone, as there are also operational efficiencies when moving to bidding.

    Test on Mediation Platforms

    If possible, run A/B tests through your mediation or analytics platform, as these platforms are set up to control factors effectively and give accurate results. Refer to your platform's documentation to set up any tests.

    If your mediation or analytics platform doesn't support A/B tests, follow the steps and best practices below to help measure the difference between bidding and waterfall.

    Create an A/B Test

    Best Practices

    • Test all traffic on one app at the same time, including all ad formats. If you don't want to test a whole app, you can test either iOS or Android, as long as there is sufficient traffic.
    • Test with at least 30,000 daily impressions in each group (60,000 daily impressions overall). Less than this can give inaccurate results.
    • Test a period of 14 days or longer to adequately compare performance, as metrics can fluctuate during the first 14 days.
    • Move as many of your existing waterfall ad networks to bidding at the same time. However, don’t introduce new ad networks at this stage, as any increase in revenue may be due to the new demand. You can run an additional test afterwards to see the impact provided by new networks.
    • Include several ad network demand sources to increase competition.
    • Ensure that your integration is free from errors before you start to measure performance.

    A/B Test Setup

    To attribute revenue uplift to bidding, everything in the experiment needs to be equal except for bidding vs. waterfall.

    An A/B test is set up with two groups:

    • Control group: A traditional waterfall
    • Test group: A validated mix of waterfall and bidding

    Mediation Platform Setup

    Control Group

    Set up your waterfall for your placements as you would normally would.

    Test Group

    For mediation that supports bidding

    • For Audience Network, reuse the existing waterfall placement with the highest revenue (per OS and ad format) if the mediation platform allows. If the mediation platform doesn’t support the same placement ID in both the test and control group, you will need to create a new Audience Network placement ID.
    • The only placement setting that's relevant to bidding is the ad format. For example, any waterfall price targets at placement level won't be applied to bidding.
    • For other ad networks, refer to their documentation to determine whether you should reuse an existing placement or create a new one. In most instances you should keep the existing placement as it will be calibrated against historical performance.

    For ad networks that don't support bidding

    • Duplicate your waterfall placements, so that different placements IDs are used in both the test and the control groups. Ensure that you keep the same configuration, such as price floors. This will allow you to measure revenue separately for both the test and the control groups.
    • Some mediation platforms will allow you to automatically duplicate your current waterfall so you won't have to manually set up the existing ad networks.

    Measure A/B Test Results

    • The key metric to measure performance in bidding is Average Revenue Per Daily Active User (ARPDAU), which measures overall revenue, not performance by network. This is different from waterfall where the primary success metric is network CPM, with fill rate and overall payout ratio being used as secondary measures.
    • It's expected to see network CPMs changing when you start bidding. This is because ad networks will get access to larger opportunities and display more impressions. While on waterfall, they were contained in their waterfall position to a certain outcome; in a bidding environment, ad networks might deliver additional impressions at a lower or higher CPM, which will impact their overall CPM. Rather than focus on CPM fluctuations, it's important to look at ARPDAU.
    • You might also have a slight fluctuation in Impressions Per Daily Active User (IMP/DAU). This might be explained by two major reasons:
      • A drop in latency when moving to bidding, especially when you have multiple Facebook Audience Network price floors.
      • IMP/DAU can increase if there is no backfill option in the waterfall. If impressions were previously not being served in the waterfall because they were below the lowest price floor, those impressions will now be filled by the bidders.
      • If you are concerned by an increase in IMP/DAU, set up an auction-wide floor at the same level as the lowest price floor in your waterfall. Please note that this will not improve performance because the Audience Network bidder will ignore floors in its auction.