Erste Hilfe für Integrations- und Akzeptanztestsuiten

Stefan Hildebrandt / @hildebrandttk

me

theory

automated integration tests

mindset

You must do the things you think you cannot do.

Eleanor Roosevelt

strategy

if it hurts, do it more often

DevOps mantra

selection of problems

  • unknown state
  • degradation
  • slow
  • late feedback
  • less feedback
  • way to many
  • maintenance costs
problem

unknown state

problem

failed tests as "valid result"

  • suite contains failed tests for a long time
    • week result
    • interpretation required
  • only costs
    • runtime
    • multiple times to analyse
  • possibility to miss new bugs
solution

disable all failed and flaky tests

  • analyse
  • plan next steps
    • create tasks to fix application
    • create tasks to fix or delete test
    • plan manual tests until test is fixed
    • more transparency
    • less costs
    • simple result
      • more value
problem

missing link between

disabled test and ticket

solution

Create a plugin for checkstyle or jQAssistant

  • add ticket id to @Ignore-Annotation
  • automated check between ticket and test
    • fail on missing ticket
    • fail or reopen on closed ticket
  • ticket tracker workflow hooks with jQAssistant
  • adjust for feature branches
problem

what break the test

and who has to fix it?

and who has to pay for?

  • nightly/weekly builds
    • many changes in one test run
solution

run tests often

... very often

problem

where to run such a mess of tests?

problem

pre prod or development stage

  • complex setup of test data
  • interferences by other users or automated tests
  • only ONE installed version
solution

on a lot of computers

  • in-house data center
    • $(1-2 developer day(s)) ~ $(buy 1 server)
  • external data center
    • $(1 developer hour) ~ $(rent 1 server/month)
  • cloud services
    • pay per use ~ nearly same as above
  • your workstations
    • ci slave as background service
problem

setup and running such a mass of installations

solution

Data Center and DevOps Toolchain

bare metal provisioning

  • wake on lan, dhcp, pxe boot, ..., running bs
  • creates basis for application, vm or container provisioning
  • ironic (openstack)
    • complex, useful for greater installations
  • existing toolchains
    • eg. ESX: expensive, mostly hardware with many cpus and cores
  • linux auto install mode
  • operating system images

application provisioning

ci installation

  • is only an application
solution 2

the docker way

application provisioning

  • install docker
  • build images with dockerfile, maven or gradle

application distribution

  • docker images are a stack of layers
    • reuse of layers
  • registry with compressed layers and meta data of images
  • https transport
  • local storage of images and layers

ci installation

  • creating slave instances for building applications
    • use images with required tools
  • creating/using docker images -> docker in docker installation
    • not easy and special
    • better:
      • install ci and docker with ansible/salt/puppet
      • use tools within docker container
      • everything should work on developer workstation the same way

configuration

  • old-school applications
    • provide configuration files with virtual host names
    • let docker handle system dependencies by aliases
    • auto configuration
      • build isolated network topology with docker
      • use your existing auto configuration setup

isolation

  • use docker-compose project names
    • separates more installations on one host
      • network
      • system aliases
    • support for different image versions
problem

browser installation

solution

docker selenium grid

  • selenium hub
  • firefox/chrome/phantomjs node(s)
  • including virtual X-server w/wo vnc support
problem

test data

  • how to get data
  • transport data to test installation
  • evolving schema
solutions for

get data

  • generate data (RedG, release in a few weeks)
  • copy and obfuscate data from production
    • jailer
    • select over database links
    • backup-restore tools
solutions for

transport data

solution for

evolving schema

  • hold data within databases
  • run schema migration scripts on it
    • implicit test of data migration
  • use ci-jobs for ready to run database containers
  • xslt transformations on dbunit xml
problem

long runtime

solution

no fixed waiting

solution

prepared test data

  • no setup within tests
  • test on existing data
    • contract creation (with existing customer)
    • contract modification (with existing contract)
    • contract termination (with existing contract)
solution

prepared containers

  • on container build
    • database provisioning
    • test data creation
    • execute one time init jobs

example

solution

smart caching of read only containers

  • don't destroy readonly containers
solution

divide into different suites

  • use of consumer driven contract tests
    • no technical limitation to start on producer side
    • pact generated client and server part
solution

faster test interface

service layer for single page apps

  • faster
  • consumer driven tests

internal service layer

  • use of internal api for testing
  • junit
  • spring (boot) test
  • arquillian
solution

UI only test

  • test ui layer with mocked/simulated backend
    • supported by ever js framework
    • Java: modified deployments mit spring/arquillian
solution

acceptance tests

  • not only on ui level
  • combine metrics
  • divide requirements
    • calculations on service/unit level
    • different visualisations on ui-only
    • process integration on system level
solution

mocking/simulating of slow systems

  • use mock services within docker containers and aliases
  • alternative: modify deployment with arquillian

wiremock

  • powerful mocking and asserting of REST requests
  • basic soap support
  • integrated as jUnit rule or standalone

citrus

  • client and server
  • protocols: HTTP, JMS, TCP/IP, REST, SOAP, FTP, SMTP, SSH and many more
  • data types: XML, JSON, and more
problem

Licences for depended systems

  • First Aid: mocking/simulating
  • strategic: place as architectural target
problem

late feedback

solution

parallel runs

problem

stability

already discussed solutions

  • smaller tests
  • defined data and system setup
  • adaptive timeouts
  • mocking
    • "UI only"-Tests
  • test on a lower level
solution

change application

  • crate anchors on html-pages
  • create better interfaces for testing
problem

maintain structural changes on UI's

solution

page object pattern

problem

to many automated integration tests

reasons

easy to create

  • good tooling for creating (not maintain) integration tests
  • less support for mocking and different deployments

duplications

  • developer driven tests on code base
    • unit tests
    • integration tests
    • ui tests
  • QA division driven automated tests
    • interface tests
    • ui tests
    • acceptance tests
    • performance tests
    • penetration tests
  • QA division driven manuel tests
  • -> software testing cupcake anti pattern by Fabio Pereira

solutions

  • co-working / pairing
  • combine measurements
  • agree on metrics
  • align targets
  • build a room of trust
  • trust!!!

Slides

h9t.eu/s/ehia

Links

Stefan Hildebrandt - consulting.hildebrandt.tk

  • Beratung, Coaching und Projektunterstützung
  • Java EE
  • Buildsysteme gradle und maven/ant-Migration
  • Testautomatisierung
  • Coach in agilen Projekten
  • DevOps
Datenschutz Impressum