Android in the Cloud

Estimated read time: 15 min
What you’ll learn: How exense automated and scaled complex Android scenarios for a large organization in Switzerland.
Ideal profile(s): Application owner, Security Specialist, Load tester

Introduction

With the current trend of moving toward newer more robust security standards and complex login scenarios involving multiple types of devices, a new set of problems are arising and challenging IT teams worldwide. By using a case study, we can illustrate how exense has overcome these problems for a large corporation in Switzerland and enabled the delivery of a series of mission-critical automation projects.

Challenges

In this first section, we’ll look at the context in which the need for automating Android devices in the cloud occurred and the various reasons behind the introduction of a hybrid test setup.

Context

In our use case, the customer’s primary goal was to roll out a set of services designed to support the FIDO2 protocol and a variety of new authentication scenarios. These services were critical to the impact on the customer’s front-facing applications, and they could not be released without completing a variety of functional and non-functional tests at scale (i.e., on many devices).

Security is a critical aspect of a large private corporation’s daily business operations, in addition to the criticality of the project, aggressive deadlines had been determined prior to exense’s engagement which allowed exense to display our ability to work under tight deadlines and make the appropriate technical decisions to move forward.

The FIDO2 Protocol

The FIDO2 protocol significantly improves security by electing to always keep sensitive authentication data on the client, while enhancing convenience through the support of unified authenticators.

The array of hardware and software stacks (browser, OS, device type) as well as authenticators (physical peripheral with pin code, fingerprint, or face recognition, etc.) resulted in many test scenarios and automation needs. In many cases, providers and users will opt for passwordless authentication, which makes the user experience much smoother but adds technical complexity.

These improvements come with a high technical cost and impact existing IT systems significantly. In addition to validating many combinations of software and hardware stacks, providers must implement new services to support the new authentication scenarios, which also need to be tested.

Asynchronous authentication

Among the many technical issues that are inherent in this type of project, mobile platform testing was particularly problematic. Currently, Android and iOS account for a large amount if not most of online traffic, including in our customer’s sector. It might seem simpler to focus on geographical biases for specific platforms, for example, a bias towards IOS. But an initial risk-reward analysis suggested that Android-based automation would provide stakeholders with the necessary and sufficient results, while being a more natural platform to start with, given the technical specificities of the client’s applications.

Adding another degree of difficulty, the authenticator selected for the mobile platform relied heavily on asynchronous messaging, especially as part of the user’s device registration workflow. Testing the Android login required the implementation of Google’s Firebase APIs to push notifications from internal client applications onto the Android devices. These notifications would play a major role in carrying information required to authenticate a mobile user and enabling the use of the client-side cryptographic keys. From a performance standpoint, measuring the propagation time of these notifications turned out to be critical.

An example of the test case’s architecture better illustrates the key factors involved in the design of the test setup.

‍Other areas of concern were the option of physical versus emulated devices, and where and how these could be industrially operated. Sensitive data such as proprietary cryptographic code or customer data could not be allowed to travel beyond strict designated parameters and borders.

Solutions

Over the course of a PoC phase, several R&D iterations, and joint workshops, several critical decisions were made, which led to the choice of a hybrid architecture which will be referred to throughout this section. While the core functions of the automation processes will be operated from the customer’s network, the Android farm will be hosted in exense’s own cloud and made remotely available.

Note: The automation (including script development and execution) relied on exense’s automation platform, called Step. Since a large amount of documentation on Step is provided in our knowledge base, we will instead focus on aspects relating more to the management of the Android farm, the app itself, and other technical challenges such as the handling of asynchronous messaging in a hybrid environment.

Technical overview

Here’s a high-level overview of the final setup chosen to run the tests. In this image, the Push Client and Android App were custom-tailored automation artifacts, while the Wait For Notification and Event Broker components were provided out-of-the-box as part of the automation platform (Step). The other entities represented in the diagram were either third-party components or components which were part of the test case’s system.

The rectangles of different colors represent different network zones, highlighting the hybrid nature of the setup. It involves a mixture of cloud-based and on-premise components communicating with one another through (mostly) asynchronous messaging.

This approach allowed us to respond quickly to the budget and time constraints while satisfying other key criteria such as high technical accuracy, meaning, our ability to reveal bugs and extrapolate the system’s behavior under a heavy workload.

Key decisions

Hosting Android in the cloud

Once the decision to place the Android platform at the center of our automation efforts was made, determining how and where this automation would take place was the next important decision.

Creating and managing a farm of physical devices was deemed impractical, requiring the costly expense of purchasing hardware like smartphones. Additionally, it was unnecessary in our context since the generation of valid cryptographic keys without the use of a physical device was already possible. Therefore, an initial attempt was made to emulate Android devices on-premise, but meeting the hardware requirements on commodity client VMs proved to be exceedingly difficult.

We then focused on exense’s private cloud in which the ability to operate Android devices had already been demonstrated. Operating the Android farm outside of the client’s network also made sense from an architectural perspective, since production users would effectively be using their devices from the internet’s public zone.

Eventually, a set of 50 emulated devices were provisioned and made remotely available. Basic calculations derived from an internal load model led to the conclusion that these 50 distinct devices would provide more than enough capacity to simulate and handle an expected total load of 2000 logins per minute.

To mitigate confidentiality concerns, exense was able to provide assurances regarding the physical location of the data center (Switzerland). This aspect can be key for geographically constrained corporations that are subject to stringent local regulations when it comes to data location.

Using a mock application

To reduce the need for sensitive client code or data, a mock Android application was written and deployed on the Android emulators. A minimal amount of functionality was implemented which included the following:

• The ability to register the underlying emulated device against Firebase
• The ability to receive notifications directed at the test user
• The ability to forward events to the client’s private Test Center
• Events forwarded to the Test Center were of two types:
• an event carrying registration information (i.e., the device token assigned by Firebase upon starting the application)
• an event carrying timestamp information marking the reception of the notification (required for End-To-End measurements)

Interconnecting hybrid components

Connecting the customer’s on-premise Test Center with the Android emulators running in the cloud via the Event Broker proved to be a crucial decision. This communication channel enabled real-time monitoring of the notification’s propagation time as well as a breakdown of each individual device’s performance in one consolidated view in Step.

The following figure shows the average end-to-end propagation time of notifications for each individual device (in milliseconds).

The value is produced by measuring the duration between the moment the notification is pushed by the central application, to the moment it is received by the corresponding user’s device (and forwarded back to the Test Center).

The use of Step’s Event Broker as a messaging proxy for external micro-polling allowed us to work around the strict networking policies of the organization. Some minor white-listing withing exense’s internet domains in the http proxy was the only internal change requested in the client’s test environment.

Operating Android at scale

The remote management of Android emulators was done via Keywords in Step.

All devices can be shutdown, started and initialized automatically.

The management of application deployments, reboots and updates is performed using Appium and also exposed as Step Keywords.

The following figure illustrates the software stack installed on hosts contributing to the Android farm.

Conclusions

The release of FIDO2 services was accomplished within the projected deadlines. In the process, exense built and deployed a remotely-managed Android emulation farm for use in a hybrid cloud/on-premise setup. In addition to providing accurate data regarding the overall performance of the system, the tests of authentication scenarios revealed several important bugs such as message duplication in Firebase and stability issues which would have been very difficult to identify otherwise. Subsequently, more comprehensive coverage of FIDO2 such as the registration and authentication with fingerprint’s keys was implemented and tested on top of the existing solution and new issues have been identified and analyzed.

As a software and services provider, our main take away from this project besides the purely technical success in scaling Android emulation and asynchronous testing, is the benefit of using hybrid solutions in corporate environments. Many clients are currently attempting to move much of their operations to the cloud, but the road to cloud-based IT is potentially very problematic. Therefore, we believe that flexibility at each level of the technical stack remains an invaluable asset at this point. By adding cloud components onto an existing on-premise test platform, our client benefited instantaneously from an external technological upgrade that would have been very difficult to achieve purely on-premise.

Summary: This article documents how Exense automated and scaled complex Android scenarios for a large organization in Switzerland.