Thursday, October 27, 2005
Acceptance tests and speed
As noted in the previous post, our test group isn't really used to writing fully automated acceptance tests, as far as I know. Our system is made up of many processes, where the data comes into one process, is transformed and moves through several other processes via IPC. To get data from the start to the end can take some amount of time. Not that it takes a *long* time for data to get processed, but to bring up the system, run a small data set through, and then bring the system down and check the results, well it takes a while.
Today I'm going to talk to the team about two things with respect to acceptance testing.
These two things go hand in hand. We have a manager process that starts the other processes and manages them. It makes sure they come up in the right order, watches them and reports if they are having problems, restarts them, etc. This is good - but it takes time. I wrote an acceptance test that had to run parts of the system 4 times (because I'm testing four different test cases in the one test suite, which requires different configuration of the processes). It takes about 60 seconds to run. From my reading on the XP mailing list - that is "a long time".
Now given that you've never written (or run) and automated test. And given you usually just manually change your configuration, get everything set, run the system, and then check all the results by hand - this is a *vast* improvement. But given you are running in an XP environment, where all acceptance tests should be run multiple times a day, and you have 20+ tests, and they all take a minute - you are looking at 20 minutes before you get any feedback on how the latest change effected the system. Too long. And if the test are going to grow over time, it is getting even worse.
So - I have to see if I can reduce the time of my test. However, the reason I need to talk to the team is because the acceptance test that the tester wrote is taking 60 seconds - and it just runs one set of processes, processing a small set of data. So what my test does in about 15 seconds, this test is doing in 60. The reason is, that the testers test is using the manager to bring everything up and bring everything down. While this is the environment in which the system will run, is it really needed for every story, or can this just be run in a test for the story that the system should be automatically managed (or something like that).
Again - when you are running one test and it takes a minute and spits out automated results, that's a great improvement over manual tests. But if we don't pay attention to speed now, it will bite us later. If we have 100 tests and it takes 2 hours to run them - well that just isn't going to be good.
We'll see how it goes...
Today I'm going to talk to the team about two things with respect to acceptance testing.
- How much of the system do we need to run in order to acceptance test a particular story, and
- The speed of the tests
These two things go hand in hand. We have a manager process that starts the other processes and manages them. It makes sure they come up in the right order, watches them and reports if they are having problems, restarts them, etc. This is good - but it takes time. I wrote an acceptance test that had to run parts of the system 4 times (because I'm testing four different test cases in the one test suite, which requires different configuration of the processes). It takes about 60 seconds to run. From my reading on the XP mailing list - that is "a long time".
Now given that you've never written (or run) and automated test. And given you usually just manually change your configuration, get everything set, run the system, and then check all the results by hand - this is a *vast* improvement. But given you are running in an XP environment, where all acceptance tests should be run multiple times a day, and you have 20+ tests, and they all take a minute - you are looking at 20 minutes before you get any feedback on how the latest change effected the system. Too long. And if the test are going to grow over time, it is getting even worse.
So - I have to see if I can reduce the time of my test. However, the reason I need to talk to the team is because the acceptance test that the tester wrote is taking 60 seconds - and it just runs one set of processes, processing a small set of data. So what my test does in about 15 seconds, this test is doing in 60. The reason is, that the testers test is using the manager to bring everything up and bring everything down. While this is the environment in which the system will run, is it really needed for every story, or can this just be run in a test for the story that the system should be automatically managed (or something like that).
Again - when you are running one test and it takes a minute and spits out automated results, that's a great improvement over manual tests. But if we don't pay attention to speed now, it will bite us later. If we have 100 tests and it takes 2 hours to run them - well that just isn't going to be good.
We'll see how it goes...
