RSS

Automated Testing Using the LINE Ad SDK

by LINE Engineer on 2016.11.24


This post introduces how to run tests with the ad client module provided for the LINE Platform. The LINE ad client module runs on both mobile and the web. This post will focus on testing with a mobile client.

LINE Ads Platform overview

The structure of the LINE Platform is quite simple as shown below. Various types of protocols can be used for server-client communications. This post will focus on testing with the HTTP protocol.

Test environment overview

The LINE Ads Platform is structured on a server-client model and we need to test the different behaviors of different clients. But we may face some obstacles on the way when running tests in such environment. Whether tests are done by a person or by an automated system, the target servers are expected to do certain behaviors in each test.

However, adjusting the servers each time to produce the expected behaviors is not efficient. Moreover, configuring a test environment that can trigger server failures is not easy.

To solve these problems and run tests more easily, we use a tool to mock server behaviors. Among many HTTP-based mock server tools, we chose WireMock for the testing of the LINE ad client module. The following diagram shows how the test environment using WireMock has been set up.

The role of each element is as follows.

  • Mobile Device: A device where a test target application is installed and test code is run. The device can be either an actual smartphone or an emulator/simulator.
  • Client App: An application that uses the LINE ad client module. Test code, by manipulating the UI of this application, prompts the LINE ad client module SDK to carry out certain actions to test various scenarios.
  • SDK: The LINE ad client module.
  • Testing App: Binaries containing test code.
  • Testing Helper: A module used by test code. This module specifies expected behaviors on the servers and asks the servers to provide information necessary for checking the validity of each request.
  • Server PC: A PC on which the mock server is running.
  • Server App: A server application that implements server behaviors required for client testing. It is written in Java. It mainly uses WireMock’s HTTP Mock feature and has some additional capabilities for automated testing.
  • WireMock: An open-source tool that helps testers easily define and use HTTP-based server APIs.
  • Mappings: Files used to define responses for certain requests. They are used by WireMock and written in JSON format.
  • Response files: Files used to provide response definitions contained in Mappings in the form of external files. They are used by WireMock and created as text files. The LINE ad client module only uses JSON format files.

About WireMock

The official WireMock website introduces the tool as a simulator for HTTP-based APIs. It also says that users may consider it a service virtualization tool or a mock server.

WireMock provides the following features.

  • Stubbing: Defines HTTP responses for requests and returns appropriate responses based on the pre-defined matching information.
  • Verifying: WireMock remembers every request information it receives. This makes it possible to verify receiving of a specific request and check detailed information of the received request.
  • Request Matching: Identifies a request using attributes such as URL, HTTP methods, and headers.
  • Proxying: Provides proxying to forward a certain request to another host.
  • Record and Playback: Records requests and responses being sent out and saves them as a file.
  • Simulating Faults: Simulates behaviors such as sending an HTTP response with an error code, sending an HTTP response in a wrong format, or delaying a response.
  • Stateful Behaviour: Defines states and, based on the states, defines responses differently.
  • HTTPS: WireMock optionally accepts requests over HTTPS.

How to use WireMock

To create an application for use with WireMock, you must set a dependency to WireMock.

In a project for a Maven build, add a dependency as follows.

Maven

<dependency>
    <groupId>com.github.tomakehurst</groupId>
    <artifactId>wiremock</artifactId>
    <version>x.x.x</version>
</dependency>

In a project for a Gradle build, add a dependency as follows.

Gradle

testCompile "com.github.tomakehurst:wiremock:x.x.x"

To use WireMock APIs during code writing, import the following statement.

import static com.github.tomakehurst.wiremock.client.WireMock.*;

You can run WireMock using Java commands. You don’t need to create a separate application for that. Download the appropriate version of the JAR file from the repository and run the following command. Go here to see other command line options.

$ java -jar wiremock-standalone-x.x.x.jar

User interface testing

UI testing lets you find and interact with UI elements of an application and validate the state and properties of each UI. You may use different UI testing tools for different mobile platforms. The most common UI testing tools for each platform are as follows.

  • Android: Espresso, UI Automator, Robolectric
  • iOS: XCTest, KIF

The LINE ad client module uses Espresso and XCTest.

About Espresso (Android UI testing framework)

Espresso is a UI testing framework developed by Google used for testing a single application.

How to use Espresso

Google recommends using Android Studio for writing Android application tests. When you create a project using Android Studio, it creates source sets and sample code for testing as well so you can get started with your test creation quickly. The following build.gradle file and sample test code are created upon project creation. These files help you find out what is required to use Espresso.

build.gradle

apply plugin: 'com.android.application'

android {
    compileSdkVersion 24
    buildToolsVersion "23.0.2"
    defaultConfig {
        applicationId "com.minhwang.myapplication"
        minSdkVersion 14
        targetSdkVersion 24
        versionCode 1
        versionName "1.0"
        // Below line should be added to use Espresso.
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 
            'proguard-rules.pro'
        }
    }
}

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    // Below line should be added to use Espresso.
    androidTestCompile('com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    compile 'com.android.support:appcompat-v7:24.2.1'
    compile 'com.android.support:design:24.2.1'
    testCompile 'junit:junit:4.12'
}

ExampleInstrumentedTest.java

package com.minhwang.myapplication;

import android.content.Context;
import android.support.test.InstrumentationRegistry;
import android.support.test.runner.AndroidJUnit4;

import org.junit.Test;
import org.junit.runner.RunWith;

import static org.junit.Assert.*;

/**
 * Instrumentation test, which will execute on an Android device.
 *
 * @see Testing documentation
 */
@RunWith(AndroidJUnit4.class)
public class ExampleInstrumentedTest {
    @Test
    public void useAppContext() throws Exception {
        // Context of the app under test.
        Context appContext = InstrumentationRegistry.getTargetContext();

        assertEquals("com.minhwang.myapplication", appContext.getPackageName());
    }
}

To prevent unexpected errors during test runs, you need to turn off animation settings on the emulator or device you are running tests.

About XCTest (iOS UI testing framework)

Introduced at the WWDC 2015 – UI Testing in Xcode session, iOS UI testing is based on the two core technologies: XCTest Framework and Accessibility.

  • XCTest: XCTest provides a framework for UI testing and is integrated with Xcode. Similar to implementing unit tests using XCTest, you can implement UI tests by creating a test target and test classes and functions. You use assertions to validate the outcomes. XCTest supports both Objective-C and Swift.
  • Accessibility: Accessibility is the core technology that gives disabled people the same rich experience and provides important data about the UI. UI testing uses that information to do its work.

There are many testing frameworks available for the iOS platform. At first, we planned to use the KIF framework. However, the ad client module failed to function normally on this framework for unknown reasons so we decided to use an alternative iOS UI testing framework.

How to use XCTest

You need Xcode 7 and iOS 9 or higher.

Creating a project by choosing options

The easiest way to create a project is by adding UI tests. As you can see in the screenshot below, select the Include UI Tests checkbox when you choose options for your new project. This will create a UI testing target and an implementation file containing sample tests.

Adding a test by creating a test target

Select File > New > Target… and choose the iOS UI Testing Bundle template. Enter the name of the target and select the application you want to test from the Target to be Tested drop-down list. Same as above, Xcode will create an implementation file containing sample tests.

Example of writing ad client module tests

The following is one of the scenarios used for ad client module testing.

Purpose Check if the client behaves correctly when DISPLAY POLICY changes from WEIGHTED_RANDOM to SORTED.
Preconditions
  1. Ad list cache is empty and a new request is needed.
  2. When the client requests for the first SHOWCASE, a response is received containing ordering = WEIGHTED_RANDOM and the smallest “expire” value so that the ad can expire during testing.
  3. When the client requests for the second SHOWCASE, a response is received containing ordering = SORTED.
  4. Two ads are listed in the ad list received from the server. The weight value of the first ad is 0 and the weight value of the second ad is 100000000.
  5. The client screen displays only one ad view.
Steps
  1. Display the screen showing only one ad view.
  2. Refresh the screen showing the ad view.
Expected results
  1. WEIGHTED_RANDOM mode should work and the content of the second ad is displayed since the ad in the second response has a larger weight value.
  2. SORTED mode should work and the content of the first ad is displayed on the screen.
  3. The SHOWCASE request count should be 2.

Example of writing a mapping file for ad server mocking

The LINE ad client module sends and receives requests and responses mainly through three types of URLs.

The LINE ad client module requests a “configuration” URL to receive configuration details and requests a “banner” URL to receive information of the ad to be displayed on a screen. User actions and ad view events are collected and sent to an “events” URL.

The following code is an example of responding to a request that matches the specified criteria. The response has a 200 status code and contains the specified file content.

ad_configuration.json

{
    "request": {
        "urlPattern": "/line/advertise/v[0-9]+/config",
        "method": "POST"
    },
    "response": {
        "status": 200,
        "bodyFileName": "display_policy/4_1/ad_configuration.json"
    }
}

In this test scenario, the server provides different information in each of the first and second SHOWCASE requests. The following is an example of a response received after the first request that matched the specified criteria. The response responds with the specified file content and changes the status. WireMock’s scenario feature is used.

ad_banner_first.json

{
    "scenarioName": "SHOWCASE",
    "requiredScenarioState": "Started",
    "newScenarioState": "First SHOWCASE",
    "request": {
        "urlPattern": "/line/advertise/v[0-9]+/showcase",
        "method": "POST"
    },
    "response": {
        "status": 200,
        "bodyFileName": "display_policy/4_1/ad_banner_weighted.json"
    }
}

Once the status is changed, the server sends another response with different content.

ad_banner_second.json

{
    "scenarioName": "SHOWCASE",
    "requiredScenarioState": "First SHOWCASE",
    "request": {
        "urlPattern": "/line/advertise/v[0-9]+/showcase",
        "method": "POST"
    },
    "response": {
        "status": 200,
        "bodyFileName": "display_policy/4_1/ad_banner_sorted.json"
    }
}

Every event being sent out is responded with a 200 status code only.

ad_event.json

{
    "request": {
        "urlPattern": "/line/advertise/v[0-9]+/stat",
        "method": "POST"
    },
    "response": {
        "status": 200
    }
}

Example of writing test code for an iOS client

Below is the app screen that shows how the ad client module testing is configured on iOS.

On iOS, the basic testing steps are as follows:

  1. Configure internal cache required for each test
  2. Select a screen to be tested
  3. Execute an action required for testing
  4. Verify whether the action produces an expected result through Accessibility of the UI

The JSON files listed on the Cache configuration screen are included in a test project as a bundle. You can put file content into the cache by tapping the file name.

Below are the functions written for the above test scenario. Launch the testing app using a setUp() function before executing any test functions.

func testGivenDisplayPolicyChangedToSorted_WhenViewReloaded_ThenShowTheFirstAd() {

        // Specify testStub to indicate the path of the mapping files 
        required for each test.        
        let testStub = AdDisplayPolicyTestStub(adStubID: 
        .testGivenDisplayPolicyChangedToSorted_WhenViewReloaded_ThenShowTheFirstAd)

        self.adViewTestingHelper!.prepareTest(testStub)
        
        let app = XCUIApplication()
        let tablesQuery = app.tables
        
        // Tap the table row which includes Ad list on the screen.
        tablesQuery.staticTexts["Ad List"].tap()
 
        // Tap the table row which includes Clear AD list on the screen 
        to delete the cache content.
        tablesQuery.staticTexts["Clear AD List"].tap()
 
        // Tap Testing Options to go back to the previous screen.
        app.navigationBars["line_ios_advertise_test_force.AdListCacheView"].
        buttons["Testing Options"].tap()

        // Tap the table row which includes the screen name 
        to enter the testing screen. The screen showing the first ad view is displayed 
        and the ad title is specified as the Accessibility label for the ad view.
        tablesQuery.staticTexts["Ad Display Policy Testing View"].tap()
        sleep(2)

        let firstShowingAd = XCUIApplication().otherElements["lineAdView_1"].label

        // Move to another screen and then 
        go back to the original screen to refresh the ad
        app.navigationBars["line_ios_advertise_test_force.AdDisplayPolicyView"].
        buttons["Next"].tap()
        app.navigationBars["line_ios_advertise_test_force.EmptyView"].
        children(matching: .button).matching(identifier: "Back").
        element(boundBy: 0).tap()

        // Check the SHOWCASE request count and the ad title is displayed 
        to verify whether the action has produced an expected result.
        let result = self.adViewTestingHelper?.
        getRequestCountFromServerFor(AdRequestURLs.AdBannerUrl.rawValue)
        XCTAssertNil(result?.error)
        XCTAssertEqual(2, result?.count)
        XCTAssertEqual("2_in_first_response", firstShowingAd)
        XCTAssertEqual("1_in_second_response", XCUIApplication().
        otherElements["lineAdView_1"].label)
}

Example of writing test code for an Android client

The test code flow of Android is similar to that of iOS but they are slightly different as Android and iOS use different APIs. Perform cache initialization on setUp().

@Test
public void testGivenDisplayPolicyChangedToSorted_WhenViewReloaded_ThenShowTheFirstAd() 
throws IOException, JSONException {
    String firstShowingAd;

    Given: {
        AdDisplayPolicyTestStub.AdStubIDs stubID = AdDisplayPolicyTestStub.AdStubIDs.
        testGivenDisplayPolicyChangedToSorted_WhenViewReloaded_ThenShowTheFirstAd;
        AdDisplayPolicyTestStub testStub = new AdDisplayPolicyTestStub(stubID);
        this.testingHelper.requestServerToPrepareStub(testStub);
    }

    When: {
        this.activity = this.testingHelper.launchActivity();
        SystemClock.sleep(2000);

        firstShowingAd = this.activity.getAdTitle();

        // Refresh the contents of the advertisement
        InstrumentationRegistry.getInstrumentation().runOnMainSync(new Runnable() {
        @Override
         public void run() {
          InstrumentationRegistry.getInstrumentation().callActivityOnPause(activity);
          InstrumentationRegistry.getInstrumentation().callActivityOnStop(activity);
          InstrumentationRegistry.getInstrumentation().callActivityOnRestart(activity);
          InstrumentationRegistry.getInstrumentation().callActivityOnResume(activity);
         }
        });

        SystemClock.sleep(2000);
    }

    Then: {
        Assert.assertEquals("2_in_first_response", firstShowingAd);
        Assert.assertEquals("1_in_second_response", this.activity.getAdTitle());
        Assert.assertEquals(2, this.testingHelper.
        getRequestCountFromServerFor(AdRequestURLs.AD_BANNER_URL.getURL()));
    }
}

Continuous Integration

Every test is run on Jenkins according to a schedule. Reports are generated for test results and coverage analysis.

Closing words

In this post, we have gone over how to test a mobile client on the LINE Ads Platform. I also introduced a mobile client test tool – WireMock, and explained how to use the following testing frameworks – Espresso for Android and XCTest for iOS. Lastly, I briefly talked about how test results and coverage analysis are automatically reported back to CI servers.

About the authors

Hwang Min: He is a test engineer responsible for writing test code to verify the LINE SDK.