Creative Sets: Testing within Apple Search Ads and it’s Limitations

The introduction of Creative Sets gives Apple Search Ads users a great opportunity to align various creatives to different keywords. On top of that, now app publishers can test Apple Search Ads assets. However, it doesn’t mean that Apple’s update made full-scale A/B tests possible within the App Store itself.

In this post, we will figure out how Creative Sets can be used for Apple Search Ads assets testing and what are the limitations of this experimental method.

How to Test Apple Search Ads Assets with Creative Sets

Basically, a Creative Set is an assets pack for Apple Search Ads which consists of screenshots and app previews chosen from your app’s product page in the Store. The primary goal of these Creative Sets is to add more variation to Apple Search Ads aligning various assets to ad group keywords and audience segments.

The number of Creative Sets within Apple Search Ads is limited to 10. Each set has an image number limit as well:

• iPhone: 4 or more screenshots/app previews in the portrait mode and 2 or more assets in the landscape one;
• iPad: 3 or more screenshots/app previews in the portrait mode and 2 or more assets in the landscape one.

The process of testing Apple Search Ads Assets with Creative Sets is pretty straightforward:

at creating different Creative Sets and aligning them with various Ad Groups or Campaigns, you are to track their performance (based on conversion and tap-through rate).

When creating Creative Sets try to avoid using screenshots/app previews denoted as “default”. The thing is the default Apple Search ads with the first creatives from your store product page is already there. It makes more sense to compare the performance of your default ads with the one of your Creative Sets.

Apple also recommends using the same number of assets for each Creative Set under testing. Thus, the same scheme of your Creative Sets will make the results of comparison clearer. Try to use a maximum of:

• 3 portrait or 1 landscape screenshot/app preview on iPhone;
• 2 portraits or 1 landscape asset on iPad.

Limitations of Testing with Creative Sets within Apple Search Ads

The latest Apple Search Ads update and the introduction of Creative Sets induced the speculation about ‘A/B testing within the App Store itself’. However tempting, it’s not true.

Here is the list of limitations which preclude calling the new Creative Sets feature a tool for proper A/B testing:

1. Creative Sets allow tests only in the context of Apple Search Ads

It’s a good method of optimizing Apple Search Ads banners but the results can’t be treated as the universal truth of user interaction with your app. In fact, the difference in user behavior is quite substantial.

For instance, according to our latest study on TTR benchmarks, App Store visitors now detect  Apple Search Ads way better and it changes their behavior. For instance, the average tap-through rate of Apple Search Ads banner is about two times lower compared to the results of genuine top position in the App Store Search (7.11% against 19.29%).

2. It’s impossible to test app’s product page within Apple Search Ads

Experimenting with Creative Sets, one can draw conclusions regarding App Store search at most. The results of this testing can’t be applied to product pages of apps. Furthermore, such Apple Search Ads ‘tests’ don’t allow you to experiment with such extremely important product page elements as an app icon, name, promo text, etc.

Apple Search Ads is based on keywords and match types so its targeting method is dramatically different from Facebook, Google AdWords and in-app ads. Creative Sets don’t allow mobile marketers to use external traffic sources.

That’s why one should be extra careful with the takeaways from experimenting with Creative Sets if they rely on paid traffic. In such cases, fully-fledged optimization of App Store product page with help of specialized mobile A/B testing platform is a must.

3. The restrictions of screenshots and app previews within Creative Sets

As it was mentioned above, one can use only screenshots and app previews from their product page preparing Creative Sets. It imposes serious restrictions on the testing ability of experimentation with Apple Search Ads Assets.

Thanks to the recent update of screenshots limit, an app product page in the App Store can contain up to 10 images. This limit seems more than sufficient unless you plan to test screenshots with help of Creative Sets within Apple Search Ads.

We shouldn’t forget that screenshots and app previews within Creative Sets have the same order as on your product page. It means you can’t play around their order to its full extent. For example, such fixed order won’t allow you to put the second screenshot before the first within one of your Creative Sets.

This Creative Sets limitation also doesn’t allow you to test polar opposite hypotheses and creative concepts. Indeed, there is barely a publisher that’ll allow their product page to be packed with differently designed screenshots with various color schemes.

4. Creative Sets are aligned to keywords

Creative Sets are bound to different keywords which makes results interpretation tricky. It’s impossible to tell which screenshot/app preview combination outperforms others in all respects and should take first positions on your product page.

Let’s assume that Creative Sets testing determined different winners for two different keywords, this result is quite ambiguous. However, in such cases, you can check Creative Sets performance on the ad group level to grasp average results and identify the winner.

5. The limited number of storefronts

For now, Apple Search Ads storefronts are limited to only 7 countries:

• Australia;
• Canada;
• Mexico;
• New Zealand;
• Switzerland;
• the United Kingdom;
• the United States.

Presumably, this list will keep growing but for now the storefront limitation is a serious testing obstruction. It’s also impossible to localize your app’s assets via Creative Sets testing.

6. It’s hard to measure the statistical significance of Creative Sets testing

Apple Search Ads doesn’t have a built-in calculator of results statistical significance as testing is not the primary goal of Creative Sets launch. So you are responsible for all these calculations yourself. In case of specialized A/B testing platforms, all work is done for you ensuring quality results.

7. The sample size for testing within Apple Search Ads should be calculated manually

Sample size is another important ingredient of a successful experiment which Apple Search Ads testing lacks. You should calculate sample size manually as well, otherwise your tests won’t be valid and trustworthy conclusions will be out of your reach.

8. No insights into user behavior

Conversion and tap-through rate are the only metrics that Creative Sets testing can provide. These metrics can barely tell why App Store visitors like or dislike your app and its assets. To enhance your marketing strategy, you need way more advanced and exquisite analytics.

For instance, when it comes to app preview analytics, Apple Search Ads testing won’t tell you the average view duration, or how many users installed your app after watching the app preview, or what episode of the video triggers the majority of installs. In its turn, SplitMetrics A/B testing platform can provide all these data and more.

That’s why it’s better to opt for specialized A/B testing solutions that provide in-depth analytics for all product page and search page elements along with the expertise on running smart experiments.

10. Creative Sets testing low capacity

Experimenting with Creative Sets within Apple Search Ads doesn’t give that much room for manoeuvre. The number of tests you can run simultaneously is limited by the number of existing  Creative Sets. Moreover, it’s quite time-consuming as the change of concept under the test calls for product page update and Apple review.

11. It’s impossible to run pre-launch tests

As you can see, the word about free App Store A/B testing doesn’t have much ground. Moreover, Creative Sets testing can’t be qualified as a free method as you still have to pay for Apple Search Ads.

Nevertheless, Creative Sets is a great way of making your ads more relevant, for instance, if you have a dating app, it makes sense to run ads with screenshots depicting male and female characters for corresponding gender groups. What’s for experimenting with Creative Sets, SearchAdsHQ is committed to create the tool for convenient assets testing within Apple Search Ads.

On top of that our team prepared Apple Search Ads Benchmarks report which provides key App Store ads performance metrics (TTR, CVR, CPT, CPA, etc.). Thus, you can compare your own marketing stats with the industry benchmarks to find growth opportunities and better manage your ads on the App Store.

When it comes to comprehensive ASO strategy, Creative Sets can’t be the only method of testing as you’ll need more scale, flexibility, and analytics. SplitMetrics A/B testing platform provides all that along with the expertise on best ASO practices.