It’s common to focus on functional requirements and usability when testing an application, areas which are often where most of the effort is spent once an app has moved into the testing, QA, and UAT phases of a project.
Before even thinking about any functional app code, though, it’s important to test all the app’s non-functional elements first. If those aren’t prioritised, not only will it have a knock-on effect on all other testing, it will also impact the app’s ability to function once deployed.
Defining, and clearly understanding, non-functional testing processes is the key to releasing a defect-free mobile app into production. When separated into two groups, these can be referred to as ‘initial’ non-functional tests, and ‘running app’ non-functional tests. But before we go into more detail, it’s necessary to understand why non-functional testing is unique for mobile.
Why is non-functional testing different for mobile?
A mobile app must be treated more like an integration which communicates with the platform – in our case, the Salesforce platform – via an API. This, along with the challenge of all mobile apps running on ever-changing operating systems rather than a browser, means there are many different moving parts involved which are out of the project team’s control.
It causes costly problems when an app is presented to QA or UAT users, or even worse deployed into production, before it’s been confirmed that the app is stable in terms of its foundations. That’s why these initial non-functional tests must be properly executed before anything else can go ahead.
How to approach non-functional testing for mobile apps
As previously touched on, the MobileCaddy methodology separates the full range of non-functional testing processes into two groups, initial non-functional tests and running app non-functional tests, as demonstrated below:
Initial Non-Functional Tests
Running App Non-Functional Tests
The benefits of initial non-functional testing
By separating all non-functional requirements and placing emphasis on those initial testing processes, it surfaces (and removes) common underlying problems which often cause more recognised components of mobile apps to fail.
Passing initial testing gives the project team full confidence that a sufficient foundation is in place. In other words, it confirms that standards for enterprise-grade network resilience, offline-first architecture, data encryption, security, and other key requirements will all be met.
This will speed up QA and UAT down the line, as there will be less need to look for defects and address errors, which reduces the time and effort spent on the full cycle of testing. This allows the QA team to concentrate on the app’s usability and functional requirements.
That extra attention in the areas which allow the app to deliver true value to the business will often result in a far more intuitive, reliable, performance-focused app being deployed out to the end users.
So when and how should this be done?
Every mobile app project should have known points for these tests to be carried out, a prescribed set of tests and processes, and a controlled set of results. By having these things in place, non-technical and non-project team members can be allowed to run such tests. That reduces the cost and the time spent across all test cycles even further, which will prove to be highly beneficial in other areas.
The initial non-functional tests should be carried out at the following trigger points:
- In every migration of an application to a new environment
- At every point of code or config change
- For every new container update or change
Because of the inevitable time pressures , this initial non-functional testing is often undefined and carried out ad-hoc, if at all, with limited or no tools or reporting. This testing must be given structure and definition, though, to avoid unwanted complications in later stages.
MobileCaddy has a proven set of initial non-functional test processes, as detailed below, which are performed hundreds of times per day. Many of these tests are automated via an internal testing framework named TestCaddy, but can each be followed and carried out manually for your own specific mobile projects:
Container App Installation/OS Alignment
For each of the operating systems that will be supported by the application, the container application is installed from the relevant store on a range of OS versions to be supported. Success here will be an installed application ready for the next step, authentication.
Log in & Authentication
The correct Endpoint is then selected and the test user credentials are used to log in (ideally with a profile/permission set that will be used in production). This fleshes out any issues, both with the authentication process and with user permissions. MobileCaddy has a handy permission set creator to make this easy.
Code Version Check
Once the application is running up, it’s important to check the correct version of the app from the platform is going to be tested against. This means each commit must be tagged with a version number that can be easily surfaced and checked within the app.
Initial Data Check
With the majority of applications having an initial synchronisation of data, we can now confirm that we have data locally, and therefore have had success requesting data from the platform. MobileCaddy provides even deeper insight here with our Connection Sessions on the platform side.
Resume & Sync
Because a mobile app will be used periodically throughout the user’s day, it must be able to resume correctly each time it’s called upon. Each operating system has its own policy here, and some can shut down a backgrounded app, in which case the app should restart as the next step.
Restart & Sync (Online/Connected)
This requires fully closing the application, ensuring that it restarts correctly whilst connected, and then confirming that the initial sync takes place successfully.
Restart & Sync (Offline Restart)
This requires fully closing the application, ensuring it restarts correctly whilst offline, and then confirming that the initial sync takes place successfully after switching back to a connected state.
Restart After Interrupted App Load
It’s often the case that an app will be run up only to be stopped or swiped away. This is easy to replicate during testing, followed by an immediate restart to confirm the expected restart behaviour occurs.
Restart After Forced App Close During Sync
At the point an application is synchronising, force close or stop it, then perform an immediate restart to confirm the expected restart behaviour occurs.
Offline Sync Attempt
With the device set to flight mode (or all connections disabled) attempt a data sync, either from a specific place in the app or using the inbuilt test sync. The synchronisation process should fail gracefully without issue for the user. Then switch to a connected status and confirm successful synchronisation.
Salesforce allows the user’s session to be set from anywhere from 15 minutes to 24 hours. To test the refresh flow of the authentication, either set to a low value and wait for the time out, or use the session management on the platform to revoke the test user’s access. Then perform a sync process whilst online and confirm this is successful (which confirms re-authentication).
Over-the-Air Version Upgrades
A growing number of frameworks allow some form of over-the-air upgrade. For MobileCaddy, this has been baked in since the product was incepted in 2013. It’s crucial to ensure this upgrade process has been configured correctly. By cloning a version, toggling between them, and allowing the device to fully upgrade, it confirms this process will be in good order when you release the next version of the app.
The process you test for, but aim to never need. A fairly new addition to the MobileCaddy Application Delivery Framework, but something you should test for however you build your application. If you have a faulty device or some unrecoverable error, you must confirm how you’ll recover with a process which is well-tested.
Seeing the bigger picture
By appreciating the role of this initial non-functional testing, you create a baseline of acceptance and performance, meaning all other testing will be quicker and of higher value. This, in-turn, will both accelerate and de-risk the journey to deploying a defect-free app into production.
Technology such as the MobileCaddy Application Delivery Framework (ADF) exists to bring all this together, providing a methodology which tells you when and what to test, supported by the tooling to do so sufficiently. As a result, the ADF automates and reduces the time spent on these processes, saving significant time and money for the business.
In a forthcoming article to be released over the coming weeks, we’ll be exploring the full testing processes carried out by our MobileCaddy systems, so keep an eye out for a more detailed guide to full non-functional and functional mobile app testing. In the mean time, you can read here to see examples of how specific tools within the MobileCaddy ADF accelerate the overall mobile app testing process.