Introduction to CTS Sep 2, 2010 Jingtao. agenda back ground CTS history CTS work flow download, install and run CTS check CTS report typical fail.

introduction to CTS

Sep 2, 2010Jingtao

agenda

back ground

CTS history

CTS work flow

download, install and run CTS

check CTS report

typical fail cases

5

background

Google define an “Android compatibility” device as one that can run any application written by

third-party developers using the Android SDK and NDK.

Device that are properly compatible can seek approval to use Android trademark and Android

Market client software.

6

how to verify compatibility

the Android Compatibility Definition Document(CDD) spells out the specific device

configuration that will be considered compatible.

CTS(an automatically compatibility test tool) test

Third party application test

7

the process to get certification

Google release CTS according to each Android platform publicly.

manufacturer run CTS on their device.

manufacture fix any bug found by CTS.

if manufacture think any CTS case is wrong, he should upload the CTS report and detail description to Google's database.

if Google accept manufacture's description, he will grant wavier to that device.

Google will think the device is Android compatible if it pass all the CTS case or all the fail cases have waiver.

requirement in CDD

• software API/UI compatibility

• reference app test

• application packaging compatibility

• multimedia compatibility

• developer tool compatibility

• hardware compatibility

• performance compatibility

• security model compatibility

• system update

hardware compatibility• display

• keyboard

• Non-touch navigation

• screen orientation

• touch-screen input: must have a touch screen

• USB: must support adb and mass storage mode

• navigation keys: function of home, menu and back must be available to user at any time.

• wireless data networking: must support for at least one wireless data standard capable of 200kbit/sec or greater.

• camera: must include a rear-facing camera, at least 2 megapixels

• accelerometer: must include a 3-axis accelerometer and must be able to deliver events at 50 Hz or greater.

• compass: must include a 3-axis compass and must be able to deliver events 10 Hz or greater.

• GPS: must include a GPS receiver

• telephony

• memory and storage: /data partition must be at least 150MB and must have at least 92MB memory.

• application shared storage: must be at least 2GB in size.

• Bluetooth: must include a Bluetooth transceiver and enable the RFCOMM-based Bluetooth API.

What is CTS

Compatibility Test System

open source

commercial-grade(20000+ cases)

Automation compatibility test tool

what does it test

• Signature

• Platform API

• Dalvik VM

• Platform Data Model

• Intents

• Permissions

• Resources

CTS History

• Android1.0 - 16784 cases

• Cupcake - 21383

• Donut - 22756

• Eclair r3 - 23138

• Froyo r2 - 23830

CTS work flow

download CTS http://source.android.com/compatibility/downloads.html

1, git ls-remote --heads git://192.168.1.250/android-mirror/platform/manifest.git

...

refs/heads/android-cts-2.1_r3

refs/heads/android-cts-2.1_r4

refs/heads/android-cts-2.2_r1

refs/heads/android-cts-2.2_r2

...

2, repo init -u git://192.168.1.250/android-mirror/platform/manifest.git -b android-cts-2.2_r2

3, repo sync cts

build CTS

• svn://192.168.1.250/oms/branches/horse

make BUILD_CTS=true cts

install CTS

install Android SDK

unzip android-cts-2.1_r3-x86.zipandroid-cts

|-- docs

|-- repository

`-- tools

|-- cts.jar

`-- startcts

edit android-cts/tools/startcts, set

SDK_ROOT=<absolute fold of Android SDK>

run CTSconnect phone to PC

android-cts/tools/startcts to enter CTS shell

3 mode to start test:

plan mode

package mode

case mode

Sample of CTS commands

• start --plan CTS

• start --plan Android -p android.app.cts.ActivityManagerTest

• start --plan Android -t android.app.cts.ActivityManagerTest#testGetRunningServices

what does CTS do

install test case apk

(adb -s deviceID install -r ...apk)

uninstall test case apk

(adb -s deviceID uninstall ...)

execute test

(am instrument -w -e [para] <component>...)

run CTS under Windows1. unzip CTS package and android SDK package to your windows PC.

2. Modified the SDK_ROOT in cts\android-cts\tools\startcts :

SDK_ROOT=D:/CTS/sdk/android-sdk_r05-windows/android-sdk-windows/tools

3. use below command to enter CTS shell:

java -Xmx512M -cp D:\CTS\cts\android-cts\tools\cts.jar; D:\CTS\cts\android-cts\tools\

hosttestlib.jar;D:\\lib\ddCTS\cts\android-cts\tools\junit.jar;D:\CTS\sdk\android-sdk_r05-windows\

android-sdk-windows\tools\lib\ddmlib.jar com.android.cts.TestHost D:\CTS\cts\android-cts\

repository\host_config.xml

Please change red marked to you current file path.

known issue:

The file (cts_result.css, cts_result.xsl, logo.gif, newrule-green.png ) under CTS result can not

automatically copy in new session folder.

Attention:

If you need to clear the data in addon emulator, Please use: emulator @addon -wipe-data

config phone before run CTS empty SD card

never Screen Timeout

English locale

install and set input method to LatinIME

open WIFI and GPRS

correct date/time

/opl/etc/mountd.conf, system.prop

CtsDelegatingAccessibilityService.apk

test report firefox repository/results/20xx.../testResult.xml

Fail Cases

Case Study: CTS on FBW1.4 Waivered

Internal resource ID: 38 cases

Locale related: 11 cases

Others: 4 cases

Fixed certification time: 3 cases

Fixed camera parameter: 1 case

UI related( For example, check box layout): 6 cases

Mock Locations: 22 cases (just test in eng build)

Waiting for waiver

Google Search dependant: 2 cases

no popup window when application force close

Public Resource ID

• http://developer.android.com/reference/android/R.html

• Frameworks/base/core/res/res/values/public.xml

public static final int accelerate_decelerate_interpolator

Constant Value: 17432580 (0x010a0004)

Internal Resource ID

out/target/common/R/com/android/internal/R.java

com.android.internal.R.id.alertTitle

cupcake: 0x0102014e

borqs: 0x01020167

168         String expected = dateFormat.format(mCalendar.getTime());

169         TextView tv = (TextView) d.getWindow().findViewById(com.android.internal.R.id.alertTitle);

170         assertEquals(expected, tv.getText());

resource ID

• R.id.contentPanel

• com.android.internal.R.id.topPanel

• R.id.alertTitle

• R.id.custom

Mock Location• 84 // test that mock locations are allowed so a more descriptive

error message can be logged

• 85 if (Settings.Secure.getInt(mContext.getContentResolver(),

• 86 Settings.Secure.ALLOW_MOCK_LOCATION, 0) == 0) {

• 87 fail("Mock locations are currently disabled in Settings - this

test requires "

• 88 + "mock locations");

• http://androidappdocs.appspot.com/guide/topics/location/index.html

• during initial development in the emulator, you may not have access to

real data from a real location provider (Network or GPS). In that case,

it may be necessary to spoof some data for your application using a

mock location provider.

Settings-> Applications-> Development

check box

camera• android.hardware.cts.CameraTest#testAccessParamete

rsjunit.framework.AssertionFailedError: expected:<2048> but was:<2560> at

android.hardware.cts.CameraTest.assertParameters(CameraTest.java:392)

381 private void assertParameters(Parameters parameters) {

...

387 final int ORIGINALPICWIDTH = 2048;

388 final int ORIGINALPICHEIGHT = 1536;

389

390 // Before Set Parameters

391 assertEquals(PixelFormat.JPEG, parameters.getPictureFormat());

392 assertEquals(ORIGINALPICWIDTH, parameters.getPictureSize().width);

Case Study: CTS on OPhone 2.0

Case number: 23107 cases Nexus One Oscar-r42369 S1-r49137

Fail cases: 3 34 5

Execut time: 6 h 6 h 4 h

fail case summary on Oscar

• internal resource ID: 17

• UI: 8. (check box, grid layout, list padding, pixel

buffer)

• phone number utils: 4

• network type: 1

• SD card write permission: 1

• CTS case issue: 3

phone number util cases(92585)

• android phone do not have sys.min.match.digits but can pass

• set sys.min.match.digits to 7 can pass the case but phone will not

match incoming call correctly.

• e.g. create user A(tel: 01150539) and user B(mobile:13701150539) in

sequence in Contacts, when mobile 13701150539 incoming call,

phone will always show user A incorrectly on screen.

Test Package Test Cases comments

android.telephony.cts.PhoneNumberUtilsTest testCallMethods

android.telephony.cts.PhoneNumberUtilsTest testCompareLoosely

android.telephony.cts.PhoneNumberUtilsTest testCompareStrictly

android.telephony.cts.PhoneNumberUtilsTest testJudgeMethods

According to CTA, we need

to set sys.min.match.digits

to 11. These 4 cases will

pass if set the value to 7.

Sdcard write permission case

• android.permission.cts.NoSdCardWritePermissionTest

• android.permission.WRITE_EXTERNAL_STORAGE

uid=1000: sysetm user

gid=1015: Sdcard write group

SDcard: android oms(before) oms(fixed)

075 777 775

Ipv6 related fail

• tests.api.java.net.SocketTest#test_getLocalAddre

ss

junit.framework.AssertionFailedError:

ANY address not returned correctly (getLocalAddress) with

preferIPv6Addresses=true,

preferIPv4Stack=false

0.0.0.0/0.0.0.0:49380 at

tests.api.java.net.SocketTest.test_getLocalAddress(Socke

tTest.java:645)

known failure actually passed

• android.provider.cts.ContactsTest#testGroupMembershipTable

bug 2258907, needs investigation

will pass after install gms. the gms package link:

smb://192.168.0.146/share/gms

user: b184

pass: zhanghui

rule to pass CTS

Do not modify framework.

review with SDK team before modify

run related CTS cases after modify

summary

• CTS can block SA

• manufacture will submit ticket relate to CTS

• we can help you setup CTS env

41