Thursday, October 27, 2005
Acceptance Tests and Speed
As noted in the previous post, our test group isn't really used to writing fully automated acceptance tests, as far as I know. Our system is made up of many processes, where the data comes into one process, is transformed and moves through several other processes via IPC. To get data from the start to the end can take some amount of time. Not that it takes a *long* time for data to get processed, but to bring up the system, run a small data set through, and then bring the system down and check the results, well it takes a while.
Today I'm going to talk to the team about two things with respect to acceptance testing.
These two things go hand in hand. We have a manager process that starts the other processes and manages them. It makes sure they come up in the right order, watches them and reports if they are having problems, restarts them, etc. This is good - but it takes time. I wrote an acceptance test that had to run parts of the system 4 times (because I'm testing four different test cases in the one test suite, which requires different configuration of the processes). It takes about 60 seconds to run. From my reading on the XP mailing list - that is "a long time".
Now given that you've never written (or run) and automated test. And given you usually just manually change your configuration, get everything set, run the system, and then check all the results by hand - this is a *vast* improvement. But given you are running in an XP environment, where all acceptance tests should be run multiple times a day, and you have 20+ tests, and they all take a minute - you are looking at 20 minutes before you get any feedback on how the latest change effected the system. Too long. And if the test are going to grow over time, it is getting even worse.
So - I have to see if I can reduce the time of my test. However, the reason I need to talk to the team is because the acceptance test that the tester wrote is taking 60 seconds - and it just runs one set of processes, processing a small set of data. So what my test does in about 15 seconds, this test is doing in 60. The reason is, that the testers test is using the manager to bring everything up and bring everything down. While this is the environment in which the system will run, is it really needed for every story, or can this just be run in a test for the story that the system should be automatically managed (or something like that).
Again - when you are running one test and it takes a minute and spits out automated results, that's a great improvement over manual tests. But if we don't pay attention to speed now, it will bite us later. If we have 100 tests and it takes 2 hours to run them - well that just isn't going to be good.
We'll see how it goes...
Today I'm going to talk to the team about two things with respect to acceptance testing.
- How much of the system do we need to run in order to acceptance test a particular story, and
- The speed of the tests
These two things go hand in hand. We have a manager process that starts the other processes and manages them. It makes sure they come up in the right order, watches them and reports if they are having problems, restarts them, etc. This is good - but it takes time. I wrote an acceptance test that had to run parts of the system 4 times (because I'm testing four different test cases in the one test suite, which requires different configuration of the processes). It takes about 60 seconds to run. From my reading on the XP mailing list - that is "a long time".
Now given that you've never written (or run) and automated test. And given you usually just manually change your configuration, get everything set, run the system, and then check all the results by hand - this is a *vast* improvement. But given you are running in an XP environment, where all acceptance tests should be run multiple times a day, and you have 20+ tests, and they all take a minute - you are looking at 20 minutes before you get any feedback on how the latest change effected the system. Too long. And if the test are going to grow over time, it is getting even worse.
So - I have to see if I can reduce the time of my test. However, the reason I need to talk to the team is because the acceptance test that the tester wrote is taking 60 seconds - and it just runs one set of processes, processing a small set of data. So what my test does in about 15 seconds, this test is doing in 60. The reason is, that the testers test is using the manager to bring everything up and bring everything down. While this is the environment in which the system will run, is it really needed for every story, or can this just be run in a test for the story that the system should be automatically managed (or something like that).
Again - when you are running one test and it takes a minute and spits out automated results, that's a great improvement over manual tests. But if we don't pay attention to speed now, it will bite us later. If we have 100 tests and it takes 2 hours to run them - well that just isn't going to be good.
We'll see how it goes...
Comments:
<< Home
We learn from neuropsychology that an important part in the functional activity of the lamictal apparatus is attributed to such regulations through the qualitative excitation of the sensory organs.. Still no carriage was heard to stop; no clindamycin doorbell was heard to ring.. Now, Pink, you an' me'll keep a runnin' account, you a-chargin' for reg'lar bo'd, lorazepam an' I a'lowin' to myself credics for my absentees, accordin' to transion customers an' singuil mealers an' sleepers.. The great success of the Asylum for Idiots and Feeble-minded Youth, several of the scholars from which have reached considerable distinction, one of them being connected with a leading Daily Paper in this city, and others having served nexium in the State and National Legislatures, was the motive which led to the foundation of this excellent charity.. It is imitrex a familiar fact from the association studies confirmed by every experience, that ideas which have formed intimate connections in one direction assume an almost negative attitude to whole groups of new connections.. I have known plenty of magic in eyes--and I glanced at the cozaar tender blue eyes of Prue--but I have not heard of any enchanted glasses.. How much cialis for the furniture for the week? Fifty dollars! Mr.. It can be compared without disadvantage to the aspirin best work, or all but the very best work, of Thomas Nelson Page, it seems to me.. The value of the over-occupation carisoprodol which is produced by the regulating influence of the Cons.. As I have asserted that the neurotic anxiety originates from sexual sources, cipro I can subject anxiety dreams to analysis in order to demonstrate the sexual material in their dream thoughts.. Speak Lucius, ranitidine said his sister Jane.. At last he found one, and was driven allopurinol in it to Mr.. [2] Here, as in other places, there are gaps in the treatment of the subject, which I have left intentionally, because to fill them up would require on the one hand too great effort, and on the other hand an extensive reference to material that is magnesium foreign to the dream.. Ancient geographers, when exhausting their store of information about distant lands, yielded to an unscientific craving for romance and, without any evidence to support their day dreams, filled the blank spaces left on their maps by unexplored tracts with amusing inserts such as Here there fosamax are lions.. The common dream sensation of xanax movement checked serves the purpose of representing disagreement of impulses--a conflict of the will...
We learn from neuropsychology that an important part in the functional activity of the thyroid apparatus is attributed to such regulations through the qualitative excitation of the sensory organs.. Still no carriage was heard to stop; no augmentin doorbell was heard to ring.. Now, Pink, you an' me'll keep a runnin' account, you a-chargin' for reg'lar bo'd, prozac an' I a'lowin' to myself credics for my absentees, accordin' to transion customers an' singuil mealers an' sleepers.. The great success of the Asylum for Idiots and Feeble-minded Youth, several of the scholars from which have reached considerable distinction, one of them being connected with a leading Daily Paper in this city, and others having served levitra in the State and National Legislatures, was the motive which led to the foundation of this excellent charity.. It is xanax a familiar fact from the association studies confirmed by every experience, that ideas which have formed intimate connections in one direction assume an almost negative attitude to whole groups of new connections.. I have known plenty of magic in eyes--and I glanced at the zocor tender blue eyes of Prue--but I have not heard of any enchanted glasses.. How much mirtazapine for the furniture for the week? Fifty dollars! Mr.. It can be compared without disadvantage to the oxycontin best work, or all but the very best work, of Thomas Nelson Page, it seems to me.. The value of the over-occupation zantac which is produced by the regulating influence of the Cons.. As I have asserted that the neurotic anxiety originates from sexual sources, hydrocodone I can subject anxiety dreams to analysis in order to demonstrate the sexual material in their dream thoughts.. Speak Lucius, adderall said his sister Jane.. At last he found one, and was driven butalbital in it to Mr.. [2] Here, as in other places, there are gaps in the treatment of the subject, which I have left intentionally, because to fill them up would require on the one hand too great effort, and on the other hand an extensive reference to material that is flonase foreign to the dream.. Ancient geographers, when exhausting their store of information about distant lands, yielded to an unscientific craving for romance and, without any evidence to support their day dreams, filled the blank spaces left on their maps by unexplored tracts with amusing inserts such as Here there tramadol are lions.. The common dream sensation of lipitor movement checked serves the purpose of representing disagreement of impulses--a conflict of the will...
We learn from neuropsychology that an important part in the functional activity of the protonix apparatus is attributed to such regulations through the qualitative excitation of the sensory organs.. Still no carriage was heard to stop; no benadryl doorbell was heard to ring.. Now, Pink, you an' me'll keep a runnin' account, you a-chargin' for reg'lar bo'd, penicillin an' I a'lowin' to myself credics for my absentees, accordin' to transion customers an' singuil mealers an' sleepers.. The great success of the Asylum for Idiots and Feeble-minded Youth, several of the scholars from which have reached considerable distinction, one of them being connected with a leading Daily Paper in this city, and others having served klonopin in the State and National Legislatures, was the motive which led to the foundation of this excellent charity.. It is viagra a familiar fact from the association studies confirmed by every experience, that ideas which have formed intimate connections in one direction assume an almost negative attitude to whole groups of new connections.. I have known plenty of magic in eyes--and I glanced at the hydroxyzine tender blue eyes of Prue--but I have not heard of any enchanted glasses.. How much cipro for the furniture for the week? Fifty dollars! Mr.. It can be compared without disadvantage to the evista best work, or all but the very best work, of Thomas Nelson Page, it seems to me.. The value of the over-occupation depakote which is produced by the regulating influence of the Cons.. As I have asserted that the neurotic anxiety originates from sexual sources, nexium I can subject anxiety dreams to analysis in order to demonstrate the sexual material in their dream thoughts.. Speak Lucius, tamiflu said his sister Jane.. At last he found one, and was driven tramadol in it to Mr.. [2] Here, as in other places, there are gaps in the treatment of the subject, which I have left intentionally, because to fill them up would require on the one hand too great effort, and on the other hand an extensive reference to material that is codeine foreign to the dream.. Ancient geographers, when exhausting their store of information about distant lands, yielded to an unscientific craving for romance and, without any evidence to support their day dreams, filled the blank spaces left on their maps by unexplored tracts with amusing inserts such as Here there vioxx are lions.. The common dream sensation of augmentin movement checked serves the purpose of representing disagreement of impulses--a conflict of the will...
We learn from neuropsychology that an important part in the functional activity of the actos apparatus is attributed to such regulations through the qualitative excitation of the sensory organs.. Still no carriage was heard to stop; no paxil doorbell was heard to ring.. Now, Pink, you an' me'll keep a runnin' account, you a-chargin' for reg'lar bo'd, evista an' I a'lowin' to myself credics for my absentees, accordin' to transion customers an' singuil mealers an' sleepers.. The great success of the Asylum for Idiots and Feeble-minded Youth, several of the scholars from which have reached considerable distinction, one of them being connected with a leading Daily Paper in this city, and others having served wellbutrin in the State and National Legislatures, was the motive which led to the foundation of this excellent charity.. It is effexor a familiar fact from the association studies confirmed by every experience, that ideas which have formed intimate connections in one direction assume an almost negative attitude to whole groups of new connections.. I have known plenty of magic in eyes--and I glanced at the phentermine tender blue eyes of Prue--but I have not heard of any enchanted glasses.. How much valtrex for the furniture for the week? Fifty dollars! Mr.. It can be compared without disadvantage to the lidoderm best work, or all but the very best work, of Thomas Nelson Page, it seems to me.. The value of the over-occupation thyroid which is produced by the regulating influence of the Cons.. As I have asserted that the neurotic anxiety originates from sexual sources, tramadol I can subject anxiety dreams to analysis in order to demonstrate the sexual material in their dream thoughts.. Speak Lucius, ambien said his sister Jane.. At last he found one, and was driven actos in it to Mr.. [2] Here, as in other places, there are gaps in the treatment of the subject, which I have left intentionally, because to fill them up would require on the one hand too great effort, and on the other hand an extensive reference to material that is levitra foreign to the dream.. Ancient geographers, when exhausting their store of information about distant lands, yielded to an unscientific craving for romance and, without any evidence to support their day dreams, filled the blank spaces left on their maps by unexplored tracts with amusing inserts such as Here there valium are lions.. The common dream sensation of aspirin movement checked serves the purpose of representing disagreement of impulses--a conflict of the will...
Post a Comment
<< Home
