Today I found an article about SOA testing at Wells Fargo.
What a mess.
It's hard to read, but make your way to the end of the article where it describes Wells Fargo's SOA testing tools setup. Here's my un-Enterprisey answer:
Instead of SOAPScope, use soapui. Unlike proprietary SOAPScope, soapui is free and open-source, and as Mike Kelly pointed out, soapui works where SOAPScope fails. soapui is fantastic for exploring WSDLs, but I wouldn't necessarily use it as a test driver.
I use Ruby's soap4r and http-access2 libraries to drive tests. And, instead of Mercury TestDirector ($$$$$$$$$$$), I find that a combination of Ruby's test/unit features and a nice source-code repository give me all of the features that Wells Fargo gets from TestDirector. Super reliable, super simple, super understandable, low overhead and zero cost.
Then they mention "Integra Enterprise from Solstice Software" which seems to parse log files and handle HTTPS (no big deal)-- but it also talks to MQ. (no mention of whether this was IBM MQSeries, Microsoft MSMQ, ActiveMQ, or some other message-queue spec. I guess we can assume MQSeries...)
Now this is worrisome. There is no obvious MQSeries library for Ruby (although Python has one, and the MQSeries interface seems to be pretty well documented), but I've worked with both MQSeries and MSMQ in a number of systems, and the fact that the test environment demands an MQ driver at all sets off all sorts of warning bells in my mind.
If you have a SOAP interface, you should be sending text (XML) over HTTP. If your test tool has to talk MQ, then you are testing at the wrong place. If I had a system with MQ involved, I'd stipulate that the UI/acceptance/system/performance tests are adequate proof that MQ is really working. Having your integration/SOAP tool talk MQ is a massive waste of resources.
Finally, they have "Data Management". They built some sort of db-based repository from which the test driver extracts test data at runtime. The specs are a little odd, but they do mention ODBC. There seems to be some sort of data-writer UI and a data-grabber from the test driver.
My favorite source of test data is to simply generate it within the script. Here' s a made-up brain-dead example to create 1000 users using a SOAP api function "createUser" itself:
1.upto 1000 do |num|
@soap.createUser({"UserNumber" => num},
{"UserName" => "Chris"})
end
(For a much more interesting discussion of generating test data from within the script, see my article "Old School Meets New Wave" in the June issue of Better Software magazine.)
Dragging test data out of a db with ODBC has got to be an expensive operation. If I absolutely positively have to keep test data outside the test script itself, I'd use a delimited file or possibly a spreadsheet. ODBC is overkill.
Here is the conclusion of the paper:
So we all know that tight coupling is bad, right? I'd suggest that serious testers be in a position to alter and augment their test tools at the drop of a hat. That's why I like Ruby for so much of my SOAP testing. Accommodating change with a full-featured scripting language is not too hard.
And if Ruby were to somehow fail, I could (plan A) use my existing scripts to drive soapui without too much difficulty. Or I could (plan B) port them to Java or C# without too much fuss. They would (plan C) certainly work from Perl with some improvements to the SOAP::Lite module.
In short:
soapui for exploring WSDLs;
Ruby for test driving, test management, and test reporting;
sane and rational insertion points for test interfaces on the system;
and delimited files (if you need 'em) for test data--
will take you a long, long, long way down the SOA testing road.
What a mess.
It's hard to read, but make your way to the end of the article where it describes Wells Fargo's SOA testing tools setup. Here's my un-Enterprisey answer:
Instead of SOAPScope, use soapui. Unlike proprietary SOAPScope, soapui is free and open-source, and as Mike Kelly pointed out, soapui works where SOAPScope fails. soapui is fantastic for exploring WSDLs, but I wouldn't necessarily use it as a test driver.
I use Ruby's soap4r and http-access2 libraries to drive tests. And, instead of Mercury TestDirector ($$$$$$$$$$$), I find that a combination of Ruby's test/unit features and a nice source-code repository give me all of the features that Wells Fargo gets from TestDirector. Super reliable, super simple, super understandable, low overhead and zero cost.
Then they mention "Integra Enterprise from Solstice Software" which seems to parse log files and handle HTTPS (no big deal)-- but it also talks to MQ. (no mention of whether this was IBM MQSeries, Microsoft MSMQ, ActiveMQ, or some other message-queue spec. I guess we can assume MQSeries...)
Now this is worrisome. There is no obvious MQSeries library for Ruby (although Python has one, and the MQSeries interface seems to be pretty well documented), but I've worked with both MQSeries and MSMQ in a number of systems, and the fact that the test environment demands an MQ driver at all sets off all sorts of warning bells in my mind.
If you have a SOAP interface, you should be sending text (XML) over HTTP. If your test tool has to talk MQ, then you are testing at the wrong place. If I had a system with MQ involved, I'd stipulate that the UI/acceptance/system/performance tests are adequate proof that MQ is really working. Having your integration/SOAP tool talk MQ is a massive waste of resources.
Finally, they have "Data Management". They built some sort of db-based repository from which the test driver extracts test data at runtime. The specs are a little odd, but they do mention ODBC. There seems to be some sort of data-writer UI and a data-grabber from the test driver.
My favorite source of test data is to simply generate it within the script. Here' s a made-up brain-dead example to create 1000 users using a SOAP api function "createUser" itself:
1.upto 1000 do |num|
@soap.createUser({"UserNumber" => num},
{"UserName" => "Chris"})
end
(For a much more interesting discussion of generating test data from within the script, see my article "Old School Meets New Wave" in the June issue of Better Software magazine.)
Dragging test data out of a db with ODBC has got to be an expensive operation. If I absolutely positively have to keep test data outside the test script itself, I'd use a delimited file or possibly a spreadsheet. ODBC is overkill.
Here is the conclusion of the paper:
No one tool can satisfy all of the application, environment, and testing requirements. The only option is to implement a tightly coupled suite of tools to address the requirements.
So we all know that tight coupling is bad, right? I'd suggest that serious testers be in a position to alter and augment their test tools at the drop of a hat. That's why I like Ruby for so much of my SOAP testing. Accommodating change with a full-featured scripting language is not too hard.
And if Ruby were to somehow fail, I could (plan A) use my existing scripts to drive soapui without too much difficulty. Or I could (plan B) port them to Java or C# without too much fuss. They would (plan C) certainly work from Perl with some improvements to the SOAP::Lite module.
In short:
soapui for exploring WSDLs;
Ruby for test driving, test management, and test reporting;
sane and rational insertion points for test interfaces on the system;
and delimited files (if you need 'em) for test data--
will take you a long, long, long way down the SOA testing road.