I've been somewhat alarmed by the enthusiasm with which the agile community has embraced Lean manufacturing principles, especially kanban. While I'm sure that kanban is effective in some circumstances, I am also convinced that it exposes software teams to real risk of impaired function.
Like many of my qa/tester colleagues who were working in the field in the 90s, I used to be a real process-head. I was part of projects that enthusiastically implemented Six Sigma, CMM, ISO9000, as well as other approaches like RUP, spiral development, etc. etc. We thought then that if only we could get the process right, we could crank out great software just like factories crank out cars and engineers crank out bridges.
Six Sigma and ISO9000 were lifted straight from manufacturing. CMM was the product of a lot of analysts who looked at some properties of a selected set of software projects.
Then the Agile movement came along, and with it new processes, the most significant of which turned out to be XP and Scrum together. These were revolutionary, because they were software development processes actually created from scratch by people who were very very good at creating software. Obviously, because these processes were created by smart people creating software, these processes map very well to how we do excellent software development.
We can argue about the details: is it better to have stable velocity or increasing velocity? should we estimate using points or using ideal hours? Regardless of these details, if your team measures velocity and uses velocity to plan iterations, and if your team always does pair programming, test-driven development, and continuous integration, then you will either release excellent software every iteration; or you will quickly expose and face the reasons why not.
Kanban doesn't make it necessary to face your inadequacies. I have two examples:
At one point my dev team had a chance to interview a self-appointed "kanban expert" on the subject of kanban and Lean. At this point the team had successfully shipped something like 25 of the last 27 iterations, so releasing iterations to production was routine, we had the whole process pretty well dialed in. And we had some killer questions for the kanban expert.
Under this questioning, it turned out that the kanban expert's team had been using a Scrum/XP process, but they blew iteration after iteration after iteration. From his answers to our questions, it seemed that his team lacked the skill and discipline to be consistently successful iteration after iteration. So one iteration, when they knew they'd blown it one more time, they simply stopped doing iterations or planning to release features. This is apparently when this person attached "kanban expert" to his resume.
Before we started releasing every iteration, my team had been using a kanban-like process. We would identify a set of features, then branch our trunk code for each feature under development. When a feature was finished, we would merge the feature branch back into the trunk code. This way we always had features being developed, and we could release trunk safely at any time.
We had a lot of very bright developers on the team. Some of them were maybe a little too bright. With no timebox pressure, we had people doing a lot of gold-plating, making architecture changes that had no relation to the feature being developed, making (cool!) experiments in the underlying code, and generally making merging not very fun.
Furthermore, it was an enormous headache for the marketing people. Without timeboxes or estimates, it was difficult to know when a particular feature would be available, which made it almost impossible to do effective marketing for the product. When we switched to planning iterations based on our velocity and our backlog, we made the marketing people very, very happy.
The fundamental difference between software development and manufacturing or engineering is that in software development we do not manipulate physical objects. We manipulate concepts, and algorithms, and process. Processes designed to optimize the handling of physical objects simply do not map to the work we do when we create software. I'm sympathetic to those who see estimation and iteration planning as waste, but I think the risk inherent in not having a mechanism to expose our inadequacies is too much risk to take, even for the best of teams. We have no warehouses to fill up or assembly lines to halt, we have only the contents of our heads and of our code repository. For software development, the only truly useful metric is: can you deliver running tested features on a consistent schedule? Iteration timeboxing exposes that metric in the most obvious way possible.
What we find in both literature and anecdote about the application of processes taken from manufacturing and engineering, puzzlingly, is that for every team for which the process is successful and helpful, there are other teams who implement the same process only to meet with failure.
What I suspect is that ANY process or methodology (that is not actively destructive), applied to a skilled, disciplined, high-functioning, motivated team, will succeed, regardless of the process. Likewise, any process applied to a low-functioning team will likely fail.
With an XP/Scrum sort of iterative process, we will know very soon exactly how and why we are failing. This seems to be not true of kanban or of any other software development process taken from the manufacturing arena.
Saturday, July 11, 2009
Tuesday, July 07, 2009
validate order with regular expressions
Today I wrote some UI tests to validate that certain text appears on pages in a particular order. Since I've done this now in two different jobs using two different tools, I thought it might be of interest to others.
There are a couple of reasons you might want to do this. For example, a page might have a list of records sorted by certain aspects of the record. Here's a crude example:
One set of critical tests validates that name/date/description appear in the right order; that chris/july/tester appears in the right order; and that tracy/june/gardener appears in the right order.
Another critical test is that chris appears above tracy in the display.
The first challenge for the tester is to identify and be able to address the smallest part of the page that contains the text under consideration. Every tool does this in a slightly different way, but Firebug is your friend here.
The next challenge is to make sure that your particular framework knows how to interpret regular expressions. I have seen this in place in Selenium-RC and SWAT, and I have rolled my own in Watir. (But for all I know Watir might have this kind of thing automatically by now. It's been some time since I've used Watir on a real project.)
Now, to check the horizontal order, you construct a test that looks something like
To test that chris appears before tracy on the page:
Regular expressions are for matching text. The first set of tests is fairly straightforward. The "." operator says "match any character". The "+" operator says "match 1 or more instances of whatever". This means that text like "namedate" would cause the test to fail (which is exactly what we want) but text like "name foo date" would pass (which might not be what we want but it's still a decent test, we could write a fancier regex if that happened to be critical).
The second test is a little trickier. The trouble is that by default "." does not match any newline characters. And since chris has to appear above tracy in the page, there will always be a newline involved. Most if not all languages have a way to tell the "." operator to match a newline, but some of them are really awkward. (I'm looking at you, .NET.)
So instead we make a little workaround: the \W means "match anything that's not a word" and "\w" means "match anything that is a word". "Word" is defined as any letters plus any numbers plus "_", but it doesn't matter. [\d\D] (for digits) would have worked as well, since the point is that one of those expressions will always match anything we encounter between one interesting string ("chris") and the other interesting string ("tracy").
There are a couple of reasons you might want to do this. For example, a page might have a list of records sorted by certain aspects of the record. Here's a crude example:
|name |date|description|
|chris|july|tester |
|tracy|june|gardener |
One set of critical tests validates that name/date/description appear in the right order; that chris/july/tester appears in the right order; and that tracy/june/gardener appears in the right order.
Another critical test is that chris appears above tracy in the display.
The first challenge for the tester is to identify and be able to address the smallest part of the page that contains the text under consideration. Every tool does this in a slightly different way, but Firebug is your friend here.
The next challenge is to make sure that your particular framework knows how to interpret regular expressions. I have seen this in place in Selenium-RC and SWAT, and I have rolled my own in Watir. (But for all I know Watir might have this kind of thing automatically by now. It's been some time since I've used Watir on a real project.)
Now, to check the horizontal order, you construct a test that looks something like
|assert_text|id=text_container|name.+date.+description|
|assert_text|id=text_container|chris.+july.+tester |
|assert_text|id=text_container|tracy.+june.+gardener |
To test that chris appears before tracy on the page:
|assert_text|id=text_container|chris[\W\w]+tracy|
Regular expressions are for matching text. The first set of tests is fairly straightforward. The "." operator says "match any character". The "+" operator says "match 1 or more instances of whatever". This means that text like "namedate" would cause the test to fail (which is exactly what we want) but text like "name foo date" would pass (which might not be what we want but it's still a decent test, we could write a fancier regex if that happened to be critical).
The second test is a little trickier. The trouble is that by default "." does not match any newline characters. And since chris has to appear above tracy in the page, there will always be a newline involved. Most if not all languages have a way to tell the "." operator to match a newline, but some of them are really awkward. (I'm looking at you, .NET.)
So instead we make a little workaround: the \W means "match anything that's not a word" and "\w" means "match anything that is a word". "Word" is defined as any letters plus any numbers plus "_", but it doesn't matter. [\d\D] (for digits) would have worked as well, since the point is that one of those expressions will always match anything we encounter between one interesting string ("chris") and the other interesting string ("tracy").
Friday, April 24, 2009
UI test framework design using Watir
I wrote a little toy test automation framework recently.
It's a keyword-driven framework, and the test data is kept in a CSV file. The test data look like this:
goto,http://www.google.com,,
text_field,name,q,pickaxe
button,name,btnG,click
match,Programming Ruby,text
This is the classic Google Search example from the Watir documentation.
I used CSV because it's the simplest implementation. This is, after all, a toy. In the real world this might be HTML tables or wiki tables or a spreadsheet. What we want is for anyone to be able to write a test quickly and easily after a little bit of training on FireBug or the IE Developer Toolbar. (Test design is a different issue. Let's not talk about that now.)
The framework grabs the test data and interprets each row in order. A simple dispatcher invokes the correct execution method depending on what the first element in the row of test data is. This has two advantages.
For one thing, there are a finite number of types of page elements to deal with, so our methods to manipulate them will be limited in number.
For another thing, we can write custom test fixtures using our own special keywords, and make our own little DSL, like I did with the final row above that starts with "match".
Reporting pass/fail status is left as an exercise for the reader.
This simple framework could easily be the start of a real test automation project. It's robust, it scales well, and it can easily be customized, improved, and refactored over time.
[UPDATE: the comment below about leaking memory is interesting. I didn't say it explicitly in the original post, but I think test data (CSV files) should represent fewer than about 200 individual steps. My vision is that this script is invoked for a series of CSV files all testing different paths through the application, and each CSV file is handled by a new process. Paths through the application with more than about 200 steps make it much harder to analyze failures. The fewer the number of test steps, the better.]
It's a keyword-driven framework, and the test data is kept in a CSV file. The test data look like this:
goto,http://www.google.com,,
text_field,name,q,pickaxe
button,name,btnG,click
match,Programming Ruby,text
This is the classic Google Search example from the Watir documentation.
I used CSV because it's the simplest implementation. This is, after all, a toy. In the real world this might be HTML tables or wiki tables or a spreadsheet. What we want is for anyone to be able to write a test quickly and easily after a little bit of training on FireBug or the IE Developer Toolbar. (Test design is a different issue. Let's not talk about that now.)
The framework grabs the test data and interprets each row in order. A simple dispatcher invokes the correct execution method depending on what the first element in the row of test data is. This has two advantages.
For one thing, there are a finite number of types of page elements to deal with, so our methods to manipulate them will be limited in number.
For another thing, we can write custom test fixtures using our own special keywords, and make our own little DSL, like I did with the final row above that starts with "match".
Reporting pass/fail status is left as an exercise for the reader.
This simple framework could easily be the start of a real test automation project. It's robust, it scales well, and it can easily be customized, improved, and refactored over time.
[UPDATE: the comment below about leaking memory is interesting. I didn't say it explicitly in the original post, but I think test data (CSV files) should represent fewer than about 200 individual steps. My vision is that this script is invoked for a series of CSV files all testing different paths through the application, and each CSV file is handled by a new process. Paths through the application with more than about 200 steps make it much harder to analyze failures. The fewer the number of test steps, the better.]
require 'test/unit'
require 'watir'
class ExampleTest < Test::Unit::TestCase
def goto(args)
@ie.goto("#{args[1]}")
end
def text_field(args)
@ie.text_field(:"#{args[1]}",'q').set("#{args[3]}")
end
def button(args)
@ie.button(:"#{args[1]}","#{args[2]}").click
end
def match(args)
assert_match("#{args[1]}",@ie.text,message="FAIL!")
end
def test_google
@ie = Watir::IE.new
@command_array = []
File.open('watir_keywords.csv') do |file|
while line = file.gets
@command_array << line
end #while
end #do
@command_array.each do |comm|
args = comm.split(',')
#arguably 'case' would be nicer here than 'if', but this gets the job done.
if args[0] == "goto"
goto(args)
elsif args[0] == "text_field"
text_field(args)
elsif args[0] == "button"
button(args)
else args[0] == "match"
match(args)
end #if
end #do
end #def
end #class
Sunday, April 19, 2009
a really good bug report
I had occasion to use FireWatir recently and found a strange bit of behavior trying to manipulate a select_list element. I posted an executable bug report to the Watir-general mail list. (Please read the whole (short) discussion.)
I have not been working closely with Watir for some time, so I missed some obvious diagnostic paths. Even so, though I got a reasonable explanation for the observed behavior, I pushed it farther, wondering why I did not receive a NoMethodError trying to do what I did.
Please note: at no time did I ever editorialize; nor criticize; nor invent solutions. I merely discussed observed behavior with interested parties in disinterested tones. And upon examination, I had turned up not one, but two different bugs.
I am pleased once again to have contributed to the Watir project. Furthermore, I think the discussion linked above is pretty close to a Platonic ideal of bug reports.
I have not been working closely with Watir for some time, so I missed some obvious diagnostic paths. Even so, though I got a reasonable explanation for the observed behavior, I pushed it farther, wondering why I did not receive a NoMethodError trying to do what I did.
Please note: at no time did I ever editorialize; nor criticize; nor invent solutions. I merely discussed observed behavior with interested parties in disinterested tones. And upon examination, I had turned up not one, but two different bugs.
I am pleased once again to have contributed to the Watir project. Furthermore, I think the discussion linked above is pretty close to a Platonic ideal of bug reports.
Sunday, April 05, 2009
at liberty
blogging is so old-school. Thanks again Jeff!
I have been amazed by and touched by and thankful for and proud of all the people who have sent nice messages, said nice things, and sent me leads since I was laid off from the best job I've had in a decade.
I am seeking a telecommuting software testing/QA job or related work. I have expert-level skills and experience, a shockingly good resume, and awesome references from some of the best software people in the world.
I have been amazed by and touched by and thankful for and proud of all the people who have sent nice messages, said nice things, and sent me leads since I was laid off from the best job I've had in a decade.
I am seeking a telecommuting software testing/QA job or related work. I have expert-level skills and experience, a shockingly good resume, and awesome references from some of the best software people in the world.
agile COBOL is no oxymoron
Dr. Brian Hanks of Fort Lewis College said on Twitter just now:
"Agile Cobol - the new oxymoron"
I have to disagree. In the late 90s I worked testing a life-critical 911 data-validation system written in COBOL and C. I was the tester when we migrated from key-sequenced data files (basically VSAM, or very close) to an SQL database (albeit one with no relational integrity-- we had to write that into the code).
When I joined the company, system releases were chaos. By the time I left, system releases were orderly, done on a regular basis, with great confidence. We evolved what was essentially an agile process to regularly ship running, tested features that our customers wanted.
But we broke every rule in the book (of the time) to do it. We had customers talking to developers all the time, we had sysadmins reviewing features before release, we had testers reading code and running debuggers, we (quietly) ignored test plans in 5-inch binders.
The Agile Manifesto validated this way of working. It was the first public acknowledgment that we were not alone in thinking that it took people working together all the time to release good software. Suddenly I had somewhere to point to say "See? Someone else thinks that working this way is a good idea!" As the Agile Manifesto became more widely known, any number of old-school mainframe people came out of the woodwork to talk about how they had been doing similar work for a long time. The Manifesto did not come into being from nowhere.
I want to talk about a few test techniques I used in a couple of COBOL code bases that are creative analogs of modern techniques, but using old tools.
Human Unit Testing
My mainframe career was spent all on Tandem systems. Tandem had a GREAT debugger, and a GREAT proprietary scripting language. As a tester, the first thing I did whenever new code got checked in was to crank up the debugger, set a breakpoint at the new code, step back and forth modifying the values of variables trying to cause a failure. When I did cause a failure, I would step back to a system level and use the UI or the batch processes to attempt to generate the data to cause the failure. Most of the time this was possible. Then I filed a bug report.
It is worth noting, now that the statute of limitations has probably run out: I was completely confident that our system would survive Y2K. Not because of the system test plan in the 5-inch binder (a worthless and stupid exercise required by someone in another part of the building whom we never talked to and that I ignored outright) but because I had personally examined the use of every instance of every date in every place in the entire code base by hand, in the debugger. That took significant time, but it was worth it.
Debugger + macros + scripting = automated unit testing
In hindsight I should have done a lot more of this.
Tandem's scripting language allowed you to start a program, call the debugger, and load something called a 'macro'. The macro would drive the debugger, allowing you to set breakpoints, modify and capture values during the course of the run.
I used to use this technique a lot to accomplish what Jonathan Kohl calls "Computer Assisted Testing". Use automation to accomplish the tedious steps of bringing the system to a state where something interesting is just about to happen, then let human beings take over.
If I knew then what I know now, I could have built some awesome standalone regression test suites using this technique.
Golden Snapshot
I worked in another COBOL code base that was a disaster. Actually, the whole company was a mess. To give a sense of how bad it was: the ball of mud was far too big for the compiler. Whenever the size of the main procedure had to be cut, the devs didn't peel off a module or a functional area; they just cut lines 5000-6000 or whatever and called it a day. Utter chaos.
And improving the code base would have threatened the jobs of the most senior devs. The best developers in the company were given impossible assignments, sabotaged along the way, blamed for the failure, and then demoted. Not nice.
The code was so bad it literally could not be automated at a unit or integration level. So I did two things.
First, we designed a "smoke test" set of manual test cases. We had the best tester in the shop run each test very carefully by hand. At the end of each reference test run we took a snapshot of all the system files and stored them in directories named for each test case.
For subsequent manual runs, the tester would execute the steps, then (figuratively) press a button that caused a diff of the saved files vs. the current system files. Any discrepancies between the state of the files was cause to suspect a bug.
In step 2 of this system, I used a tool called VersaTest from a company called Ascert to begin automating the stuff that the manual testers did. This tool was basically a proxy for the UI and allowed me to drive the system guts by bypassing the UI. To this day, I have never had a better customer support experience than I did with Ascert. I was calling them constantly, and one of their customer support people became a real friend I still speak with now and then.
This is pretty much a last-resort automation option. For one thing, it is extremely expensive up front. For another thing, debugging failures is depressingly hard.
Stubs
I was teaching myself Perl at this time also, which came in really handy on several occasions.
At one point I was assigned to a small side project. One of the better devs had run afoul of his peers again and been sentenced to work on this not-sexy side project. This dev is responsible for one of my favorite quotes of all time. After we'd been working together for a while and the project was starting to flesh out a bit, I noticed that his COBOL was quite a lot better than the code in the main code base and the code I'd seen from other devs. He said "Just because it's COBOL doesn't mean it HAS to be ugly".
This project was a transaction system. Our code would send an transaction, and an outside party would send back one of 4 messages: Success; Failure; Queued for Later Processing; or Later Processing Complete. But their test interface was really flaky and was down most of the time. I wrote a little Perl script that would reply to our system with one of the 4 messages depending on whether the last digit in the transaction number in the record was 1, 2, 3, or 4. It was rather clever and simple, and that little script saw use for the whole life of the project. This little side project was also neat because the team was me, the dev, and a business guy, and we were iterating fast, all together, all using the code, all analyzing the work at the same time.
Meanwhile, in the big sexy part of the company, there was a highly visible project to do transactions with a very important partner, but our dev schedule was ahead of theirs, so interaction testing was some way out. When I was informed that the code was "done", I read the spec and implemented a little Perl network server that could read and write messages over TCP/IP. I pointed our code at it, which immediately fell over dead. I started reporting bugs on the code. The dev actually complained to my boss about this. He could not understand how someone could report bugs when the other side of the transaction was not complete. One day though, he turned all the way around and started being nice to me. I suspect someone had reviewed his "done" code and chewed him out.
No Oxymoron
So it is possible to have automated unit testing, integration testing, and system testing in a COBOL/mainframe environment. Furthermore, it is not only possible to develop COBOL according to the Agile Manifesto; I think the points made by the Manifesto came from the cultures of successful mainframe projects.
Agile software development has been around for a long, long time. It just didn't have a name until those guys got together at Snowbird and cranked up the Agile movement.
"Agile Cobol - the new oxymoron"
I have to disagree. In the late 90s I worked testing a life-critical 911 data-validation system written in COBOL and C. I was the tester when we migrated from key-sequenced data files (basically VSAM, or very close) to an SQL database (albeit one with no relational integrity-- we had to write that into the code).
When I joined the company, system releases were chaos. By the time I left, system releases were orderly, done on a regular basis, with great confidence. We evolved what was essentially an agile process to regularly ship running, tested features that our customers wanted.
But we broke every rule in the book (of the time) to do it. We had customers talking to developers all the time, we had sysadmins reviewing features before release, we had testers reading code and running debuggers, we (quietly) ignored test plans in 5-inch binders.
The Agile Manifesto validated this way of working. It was the first public acknowledgment that we were not alone in thinking that it took people working together all the time to release good software. Suddenly I had somewhere to point to say "See? Someone else thinks that working this way is a good idea!" As the Agile Manifesto became more widely known, any number of old-school mainframe people came out of the woodwork to talk about how they had been doing similar work for a long time. The Manifesto did not come into being from nowhere.
I want to talk about a few test techniques I used in a couple of COBOL code bases that are creative analogs of modern techniques, but using old tools.
Human Unit Testing
My mainframe career was spent all on Tandem systems. Tandem had a GREAT debugger, and a GREAT proprietary scripting language. As a tester, the first thing I did whenever new code got checked in was to crank up the debugger, set a breakpoint at the new code, step back and forth modifying the values of variables trying to cause a failure. When I did cause a failure, I would step back to a system level and use the UI or the batch processes to attempt to generate the data to cause the failure. Most of the time this was possible. Then I filed a bug report.
It is worth noting, now that the statute of limitations has probably run out: I was completely confident that our system would survive Y2K. Not because of the system test plan in the 5-inch binder (a worthless and stupid exercise required by someone in another part of the building whom we never talked to and that I ignored outright) but because I had personally examined the use of every instance of every date in every place in the entire code base by hand, in the debugger. That took significant time, but it was worth it.
Debugger + macros + scripting = automated unit testing
In hindsight I should have done a lot more of this.
Tandem's scripting language allowed you to start a program, call the debugger, and load something called a 'macro'. The macro would drive the debugger, allowing you to set breakpoints, modify and capture values during the course of the run.
I used to use this technique a lot to accomplish what Jonathan Kohl calls "Computer Assisted Testing". Use automation to accomplish the tedious steps of bringing the system to a state where something interesting is just about to happen, then let human beings take over.
If I knew then what I know now, I could have built some awesome standalone regression test suites using this technique.
Golden Snapshot
I worked in another COBOL code base that was a disaster. Actually, the whole company was a mess. To give a sense of how bad it was: the ball of mud was far too big for the compiler. Whenever the size of the main procedure had to be cut, the devs didn't peel off a module or a functional area; they just cut lines 5000-6000 or whatever and called it a day. Utter chaos.
And improving the code base would have threatened the jobs of the most senior devs. The best developers in the company were given impossible assignments, sabotaged along the way, blamed for the failure, and then demoted. Not nice.
The code was so bad it literally could not be automated at a unit or integration level. So I did two things.
First, we designed a "smoke test" set of manual test cases. We had the best tester in the shop run each test very carefully by hand. At the end of each reference test run we took a snapshot of all the system files and stored them in directories named for each test case.
For subsequent manual runs, the tester would execute the steps, then (figuratively) press a button that caused a diff of the saved files vs. the current system files. Any discrepancies between the state of the files was cause to suspect a bug.
In step 2 of this system, I used a tool called VersaTest from a company called Ascert to begin automating the stuff that the manual testers did. This tool was basically a proxy for the UI and allowed me to drive the system guts by bypassing the UI. To this day, I have never had a better customer support experience than I did with Ascert. I was calling them constantly, and one of their customer support people became a real friend I still speak with now and then.
This is pretty much a last-resort automation option. For one thing, it is extremely expensive up front. For another thing, debugging failures is depressingly hard.
Stubs
I was teaching myself Perl at this time also, which came in really handy on several occasions.
At one point I was assigned to a small side project. One of the better devs had run afoul of his peers again and been sentenced to work on this not-sexy side project. This dev is responsible for one of my favorite quotes of all time. After we'd been working together for a while and the project was starting to flesh out a bit, I noticed that his COBOL was quite a lot better than the code in the main code base and the code I'd seen from other devs. He said "Just because it's COBOL doesn't mean it HAS to be ugly".
This project was a transaction system. Our code would send an transaction, and an outside party would send back one of 4 messages: Success; Failure; Queued for Later Processing; or Later Processing Complete. But their test interface was really flaky and was down most of the time. I wrote a little Perl script that would reply to our system with one of the 4 messages depending on whether the last digit in the transaction number in the record was 1, 2, 3, or 4. It was rather clever and simple, and that little script saw use for the whole life of the project. This little side project was also neat because the team was me, the dev, and a business guy, and we were iterating fast, all together, all using the code, all analyzing the work at the same time.
Meanwhile, in the big sexy part of the company, there was a highly visible project to do transactions with a very important partner, but our dev schedule was ahead of theirs, so interaction testing was some way out. When I was informed that the code was "done", I read the spec and implemented a little Perl network server that could read and write messages over TCP/IP. I pointed our code at it, which immediately fell over dead. I started reporting bugs on the code. The dev actually complained to my boss about this. He could not understand how someone could report bugs when the other side of the transaction was not complete. One day though, he turned all the way around and started being nice to me. I suspect someone had reviewed his "done" code and chewed him out.
No Oxymoron
So it is possible to have automated unit testing, integration testing, and system testing in a COBOL/mainframe environment. Furthermore, it is not only possible to develop COBOL according to the Agile Manifesto; I think the points made by the Manifesto came from the cultures of successful mainframe projects.
Agile software development has been around for a long, long time. It just didn't have a name until those guys got together at Snowbird and cranked up the Agile movement.
Saturday, January 03, 2009
Ruby net/http+SSL+Basic Auth
Since one post from this blog is (or at least used to be) a little chunk of official documentation for Basic Auth in Ruby, I decided I should write this down also.
Socialtext is switching some of the SaaS servers to be https all the time. So I had to rejigger my selenium test-runner script, because it reads (test data) from and writes (test results) to the wiki.
I found this very elegant example.
My take on it looks like
What is a little peculiar about this is that Basic Auth is very much optional for SSL connections. Normally the client would do a GET, the server would return a cookie, and the client would use that cookie in subsequent transactions. I found a nice example of that here, but as of today the server seems to be down.
Socialtext is switching some of the SaaS servers to be https all the time. So I had to rejigger my selenium test-runner script, because it reads (test data) from and writes (test results) to the wiki.
I found this very elegant example.
My take on it looks like
def put_test_results_to_wiki
http = Net::HTTP.new(@test_host,443)
req = Net::HTTP::Put.new(@put_loc, initheader = {'Content-Type' =>'text/x.socialtext-wiki'})
http.use_ssl = true
req.basic_auth @test_user, @test_pass
req.body = ".pre\n" + @content + "\n.pre"
response = http.request(req)
puts response
end
What is a little peculiar about this is that Basic Auth is very much optional for SSL connections. Normally the client would do a GET, the server would return a cookie, and the client would use that cookie in subsequent transactions. I found a nice example of that here, but as of today the server seems to be down.
Monday, November 24, 2008
artistic testing
I like to point out that historical, well-vetted aesthetic frameworks are useful for evaluating software.
But what if similar aesthetic frameworks could not only validate software itself but also *predict* software test techniques?
For example, let's examine overarching schools of painting: first came realism, where painters attempted to represent as much as possible what appears to the human eye. We can test realistic scenarios, we do this all the time. I would suggest that most bugs are not discovered under what we understand to be realistic circumstances.
Then came expressionism, where painters attempted to express things not seen. Likewise, test techniques moved on to persona testing (imagining the use of software by peculiar users) and soap-opera testing. Such techniques expose significant bugs.
Then came abstract expressionism. Think of Jackson Pollack. Are there abstract expressionist test techniques? Of course there are.
I would really like to hear of any other explorations along these lines...
But what if similar aesthetic frameworks could not only validate software itself but also *predict* software test techniques?
For example, let's examine overarching schools of painting: first came realism, where painters attempted to represent as much as possible what appears to the human eye. We can test realistic scenarios, we do this all the time. I would suggest that most bugs are not discovered under what we understand to be realistic circumstances.
Then came expressionism, where painters attempted to express things not seen. Likewise, test techniques moved on to persona testing (imagining the use of software by peculiar users) and soap-opera testing. Such techniques expose significant bugs.
Then came abstract expressionism. Think of Jackson Pollack. Are there abstract expressionist test techniques? Of course there are.
I would really like to hear of any other explorations along these lines...
Subscribe to:
Posts (Atom)
