Adventures with SOAP using Perl: Part 1 ( SOAP::Lite )

The most prevalent of SOAP libraries for Perl is SOAP::Lite it is the oldest and most documented. Though for all of its documentation it can be quite painful to figure out how to use it.

First make sure you've read Part 0 to set up the server. Once that's done let's look at the most simple way to interface with this server.

In our first example we need to send a request to getCountries, which is a method provided by the API. You can determine this by reading namesservice.wsdl and looking for the operations to see what's available. Essentially this means we need to send SOAP request with a Body of <getCountries />. First we need to import SOAP::Lite.
If you look at our SOAP::Lite import you'll notice that were are passing the arguments +trace => [ 'debug' ]. There are various levels and options for +trace, but this turns on full debug printing which will be sent to stderr. You don't normally want to have debug running in production code, but it will be useful to illustrate our examples and the request they send and receive.

Now let's look at creating an actual SOAP::Lite request object. The first option we pass in is readable => 1, adds whitespace to the request sent so that it's easier to read when you're looking at the debug output, you should not enable this in production, as it makes the request bigger, and I believe it is not considered correct SOAP as I've been told something about extra whitespace in SOAP being considered invalid. The second option is proxy => 'http://localhost:8877' This specifies the hostname and port that the HTTP request is sent to. ns => 'http://namesservice.thomas_bayer.com/' is the namespace, which you can find by looking for namespace in the namesservice.wsdl.

Now we need to actually create and send an actual request. For this trivial request we simply need to call the method that we need on the remote server and then return the object. You can see that SOAP::Lite is generating a namespace for your request to use with the XML <namesp1:getCountries xsi:nil="true">, which is just fine in this case.

Of course we want to do something with our response. Please note that I've modified the code to use 5.10, but if you want to use print instead of say this code will work fine on 5.6 and up. valueof, which is documented in SOAP::SOM, returns the first element in scalar context, and an array in array context. So in my code I've shown both. The syntax used in the parameters to valueof is XPath, so an even simpler way to call it in this case would be $res->valueof(//country); and it would do the same thing with this XML.

Next let's look at the getNameInfo method, it's a bit more complex so let's look at the XML in the XSD. Here's the snippet that is really important. This means that we need to send a request with a body that looks like ( note: you can look at the sample data in MyExampleData.pm for other names. ) Set let's take a stab at writing some Perl. There are some important differences to note from our previous script. You'll notice that I call ->getNameInfo() directly on the request object, instead of passing it as a parameter to ->call. This functions basically the same as call and it will end up making the first tag inside of body. We could have doen this in our first example as $req->getCountries; and that would have been it. Now that we've covered the slight differences in calls, let's go over the completely new things.

SOAP::Data objects are used to create any further data structures. Obviously the hash key of name defines the element name, and value defines what you want to go into it, here I have hardcoded "Mark".


If you run this code you'll notice that it returns a faultstring (among other fault properties) "operation getNameInfo for SOAP11 called with invalid data", and details the error as "element `c-gensym3' not processed at {http://namesservice.thomas_bayer.com/}getNameInfo". Now go back and look at the request, you'll see a c-gensym3 element, where did that come from? Well, SOAP::Lite will generate elements for anonymous elements but we can fix this.

The only difference between this and the previous code is that we aren't putting a \ in front of SOAP::Data. I wrote it the first way because I had seen examples of that all over the place, and could not find a solution to getting rid of the gensyms until I asked this question on stackoverflow.

 Unfortunately this is the most complex example that our server API has implemented. As an exercise to the reader I suggest Implementing a request for the method getNamesInCountry, which is no more complex but available.

Adventures with SOAP using Perl: Part 0 ( prelude )

This is a prelude to a series on working with SOAP Requests using Perl. For the past 3 months I have been working on a Perl API for CyberSource's Simple Order API which uses SOAP (I should note, that although I believe most of the API is now stable some area's still need work, and thus I don't expect it to reach 1.0.0 anytime soon).

First I used SOAP::Lite to do my requests, but I found it confusing to construct the requests that I needed to make. I even discovered a bug that lead to the current ( 0.714 ) release of SOAP::Lite.

Next I started using SOAP::Data::Builder to make it easier to build my SOAP::Lite requests. This was good, but frustrating that I had to add my data in a specific order.

Finally I came upon XML::Compile::SOAP. A glance at it's API which used a hash to build requests seemed much better. however, it took me a few weeks and some help from Mark Overmeer (the author) and an update to XML::Compile::SOAP::WSS to get it to work.

If you're planning on starting a new project that requires SOAP I definitely recommend using XML::Compile::SOAP if you have a .wsdl and a .xsd to work with.

I will be covering all 3 of these methods in Parts 1, 2 and 3 of the series.

To get started we'll need a SOAP Server since I haven't been able to find any reliable public services. To do this you can install XML::Compile::SOAP::Daemon (which probably could use some Plack/PSGI love ). You'll also want to download it's tarball directly, extract it and cd examples/namesservice. Once you've done that you can do perl server.pl . Now you should have a Server running on http://localhost:8877 which we can use for testing our client examples.

Please be advised, these tutorials will not be explain XML, XSD, WSDL, or SOAP, but simply the Perl interfaces.

Experience with having a non Dist::Zilla user contribute to a Dist::Zilla project (it's not hard for them or me)

I've heard many times that Dist::Zilla makes it harder for people to contribute to the project. This is not true, it is either unfortunately either ignorance or FUD (much like saying Linux is harder to use than Windows). Truly, there are things that some dzil users do that can make it harder, but it doesn't have to be that way. Michael Schwern recently contributed to one of my dzil projects without ever realizing I was using dzil, until I told him. He more recently stated on twitter, "While your solution works, it seems like it makes more work for you to shield contributors from dzil". This was true in this case because I wasn't sure how to effectively move a series of multiple patches, I now know it's easy to do with git. Here's how you can allow contributors to contribute to your dzil project without causing you or them undo pain.

1. Don't use anything that changes the line numbers of your source

   Examples: don't use DZP::Prepender. Use modules like OurPkgVersion to insert VERSION, and make sure your # ABSTRACT and any pod is below the 1; at the end of your module. This will keep the line numbers of errors in your code from being different from the final build. It's still ok to use PodWeaver as it will save you way more time than it'll hurt, so long as you follow the rules about pod being at the bottom





2. Use a plugin that commits builds to your source control software

   I personally recommend DZP::Git::CommitBuild









3. Make your build branch your default branch

   I'm not sure how to do this with just git, but on github you can go to the admin section of your repository and change the default branch there. This makes it so that when someone clone's your repository the initial checkout is of your build branch. Your build branch shouldn't require dzil, it's the final build.

Now that you have your final build branch as the default branch anyone who wants to contribute can simply clone your repo and start hacking. Their are a couple of mistakes they could make, They could either change or add files that are maintained, or pruned by dzil. This did happen when Schwern sent me patches, one of them was the addition of a .gitignore, which I already had, but is being pruned out of the build branch. I can also see it happening to meta's and makefiles. These patches can simply be rejected as unnecessary, dzil-ified, or if they are truly a bug, then they can be reported and fixed upstream.

So what if someone sends you a pull request from build/master? well if it's just one or two patches, you add their remote, and do a git remote update and then you can git log remote build/master to find the sha1 of this patch. Now that you have the sha1 all you have to do is git cherry-pick [sha1] and it should apply. If there are any conflicts you may have to resolve them with git mergetool. However, the only conflict with Schwern's patches for me was the .gitignore, all other patches applied without assistance and applied correcly, surprisingly even the pod patch applied without issue.

If you have more patches than is comfortable with git cherry-pick then you need git rebase. The command you want is git rebase -i --onto master [sha1 before first sha1 in series] [tip of remote branch checkout]. So in my case git rebase -i --onto master 3f1e3748 schwern. What this appears to do is ends up rewriting my local schwern/build/master checkout and removes all the build commits, and then applies the patches on top. This means I can now do git merge schwern from the master branch, and all of his patches that I want will be successfully merged. For more on this strategy you may want to read this stackoverflow question.

given this is not quite as easy as a git pull that's a fast forward, but reality is it's not that hard once you know how to do the rebase and how it works. Of course this isn't ideal for constant contributors, those should simply learn to use dzil, but for the random contributor it should be ok.

My solution to not using PluginBundle:AUTHOR for dzil is git

I am now tired of updating my dist.ini's for my Dist::Zilla projects. For many people this is when they start using a PluginBundle with their authorname. I discussed why you shouldn't do this a year ago. Now that I'm tired of managing my dist.ini's on an individual basis I'm going to show you how I'm going to solve the same problem everyone else is, which is getting tired of updating your dzil configuration for all of your projects. I'm using git to do it. You probably haven't considered this, or am thinking I'm wrong because you believe that git can't merge branches without a common history, which is not true. I did it with Regen2, Funtoo, Portage, and Sunrise, which have way more files than any perl repo, including Perl.


First let's talk about the advantages and disadvantages of doing this with git.

I'll start with the disadvantages. You'll have another repository to manage. Git isn't completely automatic, you'll have to remember to merge your changes. You'll also have to add the repository to your existing repositories. You will have to resolve merge conflicts at least once, and probably occasionally more, though most should be fast-forwards.

The advantages are... now the changes to your dist.ini are getting recorded in your history. You can now have a master dist.ini, but remove items without filters in your individual modules. You can share even more configuration as merged differences allow you to maintain differences in downstream commits. It doesn't automatically update all of your modules. Wait didn't I list that last one in disadvantages? Yep, it's an advantage because what if you update your PluginBundle and that update breaks one of your modules, but you don't know it because you haven't worked on that module in a while. You can use this for more than just dist.ini.

Ok so the first thing you you want to do is create your master dzil repo. This is not a git or dzil tutorial so go do that. I did it by creating a dzil new project and removing the files I couldn't use and making a few tweaks and amending all of those changes to the initial commit. Check out my repo for inspiration. Remember the directory structure has to be the same as a dzil repo structure for any common files.

Now add a remote from your new repo to an existing dzil project. Next you need to merge the branch from the remote into your project. The first time you do this you'll have to resolve conflicts. If they're like mine then they'll be easy and obvious. Once that's done future changes will be fast-forwards unless you change something in your perl module repo, then you might end up with a simple merge conflict. One thing to remember NEVER EVER try to merge from your module repo to your dzil repo, it will cause you extreme pain in the future. cherry-picking that direction is possible but not merging.

After that, all that is left to do is run your dzil tests and fix any breakage in your module. Happy merging.

Regen2 accounts transferred

I killed regen2 about 2 years ago. Today I was contacted by Rafal Kupiec asking about the project. Since it's dead he asked if he could take over the project. I have transfered the github account, and freenode channel #regen2 to him. The google groups are remaining archived. At some point in the future there should be a site at asiotec.org I have no idea what he plans on doing with it.

Test::Version under maintenance again

About 4 months ago russoz (Alexei Znamensky) sent me a message about quoting the versions generated by Dist::Zilla::Plugin::OurPkgVersion. I was a bit skeptical at first about needing it, and the suggestion seemed to have more to do with style anyways. But after digging a little I found to my horror that OurPkgVersion was generating completely invalid multipart version strings, e.g:

our VERSION = 0.1.0;

Which doesn't work. I was shocked because I was using Test::HasVersion. Well all Test::HasVersion does it checks to see that you have a VERSION defined. My code above will pass a defined test, but you certainly won't get anything if you try to print it.

So I accepted Russoz's patch and fixed up my documentation. Still, I was disturbed that there weren't any Test modules that would have caught this error for me.

So, eventually, I began working on writing a Test module to do this task for me. First it was mostly research and learning which modules and methods I would need. One of the hardest parts was determining how best to extract the VERSION from the module. PPI seemed like an impossible task, probably most due to the fact that I don't understand it. I thought about require-ing modules, but many told me that could lead to a vulnerability in the test. Ultimately decided to use Module::Extract::VERSION which will work for everything except perl 5.12 style versions. Then I had the problem of learning how to write Tests for Test modules, which ultimately doherty (Mike Doherty) contributed some patches to get my first set of tests working.

Once the module was all but finished functionality wise I had to decide on the name, after discussing it on #distzilla it was decided that it might be a good idea to take over an existing module. Test::Version seemed like the most obvious candidate because it has the most obvious namespace for what is trying to be done. So I spent the next few weeks trying to contact particle (Jerry Gay) to get maintainer-ship. Once I did manage to contact him it wasn't an issue to get control of the Module.

So I released a new version of the old Test::Version 0.03 announcing new maintainership, on the off chance anyone was using it in the darkpan, since I'd already searched the CPAN archives for any usage and no current module was. I then released a TRIAL version for testing. The Trial version ended up having several failing tests by cpan tester reports, I thought it was caused by Test::Builder::Tester whitespace sensitivity (and maybe some of it was). rjbs (Ricardo Signes) suggested Test::Tester and I released yet another version with failing tests. This time I analyzed about 5 of the test reports, and come to find out the module seems to run the tests on the perl modules in no particular order. I assume that somewhere there is some speed optimization which results in this behavior. So I had to find a way to write the tests which could handle the fact that I couldn't depend on the order that files were being tested. Ultimately worked that kink out as well in a way that appears to be satisfactory.

So with version 0.07 we now appear to have a working Test::Version. That checks to see if modules have versions, and that they are valid. It's also capable of checking an entire dist. The docs describe use, and if you use Dist::Zilla you can check out Dist::Zilla::Plugin::Test::Version or use Dist::Zilla::Plugin::TestingMania which will be including it.

Going forward I plan to take version 0.07 if it turns out to have no serious bugs, and release version 1.0.0 as it stands. After that I have a few features releases in mind. Including supporting is_strict if you'd like to make sure that your VERSION passes those tests as well. I'd also like to pull in the functionality of Test::GreaterVersion and Test::ConsistentVersion. I'm open to other suggestions as well.

I hope people can make some good use out of this module to avoid silly mistakes like I made.

Happy Testing!

New beginnings

Since I truly last posted a lot has changed. I moved to Houston, TX, USA (from MI) for a position as a Linux System Administrator. As of this week I've been moved to the programming department in my company, coding Perl. I'm also an elected moderator on StackExchange's Unix & Linux. Of course there's always updates to my CPAN dists.

Adding and Deleting subdomains with Plesk on the Command Line

Plesk has the disadvantage that everything is done through the
database. So we can't just modify Bind's Zone files.

To add an A record of a subdomain you have to do the following:

/usr/local/psa/bin/dns -a domain.tld -a subdomain -ip 127.0.0.1

the first -a is for add to zone which is why it must specify the top
level domen, the second is for A record, and must not contain the full
domain only the subdomain. If you use the full domain
subdomain.domain.tld you will end up with a record like
subdomain.domain.tld.domain.tld because plesk does not allow you to
terminate with a .. Thank you plesk for a shitty interface. The record
this create's looks like

subdomain.domain.tld. IN A 127.0.0.1

To delete this record you would use the following

/usr/local/psa/bin/dns -d domain.tld -a subdomain -ip 127.0.0.1

you'll note that it's the exact same thing except you replace the
first -a which is for add, with -d for delete.

Writing a simple Dist::Zilla::Tester test

Hopefully, someone will use the blog post to write an actual doc patch, seeing as how this is undocumented.

I finally wrote A test for DZP::OurPkgVersion with the help of CJM. So I figure it's best to share the knowledge imparted upon me to all those who are writing plugins without tests.

Before we get started I'm going to advise that this test will only check the output that dzil built, if you need it to test anything more sophisticated, you'll have to learn more.

First you'll want to create a corpus repo like /corpus/MyDZTRepo with a basic minimal repo. This repo is simply a repo that you are using to test your plugin against, to make sure it works right. You put it in corpus so that if you have tests that you have to check in your corpus, those tests themselves aren't run when the test suite is run. The dist.ini doesn't need to contain anymore than the basic stuff needed to build. You .pm files need not have anymore data than what you're going to need to make your dzil plugin do its job. In the case of DZP::OurPkgVersion I only needed to test that the output found the # VERSION string correctly in a couple of scenario's. So that meant having # VERSION in the .pm's and [OurPkgVersion] in the dist.ini.






First DZT (Dist::Zilla::Tester) doesn't provide any tests of its own so you still need to use Test::More or some other testing framework. Next you need to initialize the tester object by telling it where the root of your corpus repo is. After that, unless you need to do other work, you can run $tzil->build so that the build is run.

So now lets slurp a file into memory so we can check to see if it was built right. You'll want to look in 'build/*' as the basic root of the build directory. So 'build/t/test1.t' if you need to slurp a test.



Now that we've pulled our build files into memory lets code up what the result should be. We can just do this with a simple heredoc, obviously you can do it another way.



Now that we've gotten that, all we have to do is compare the file that we expect dzil to output and the file that dzil actually built. This is just standard Test::More



Now let's take a look at it all together.


Pretty simple huh? Hope this means more dzil modules getting tested now. Including more of mine.

Making Secure Recoverable Passwords ( part 3 )

If you haven't read Part 1 please do so.

Although the criticism of Part 2 should be taken with a grain of XKCD Salt, and even the primary plaintiff admitted that it would take him 2 months to crack the final salted sha512. Usually your attacker shouldn't have your shadow file, and having stored them as anything less is just plain not secure. I will acknowledge it has some merit.

Doing a base64 transformation on hex only digits is a bad, idea, and does not have nearly enough possible combinations. Nothing is going to be more secure than random, but random isn't really recoverable, if you lose it. That's why I do some kind of transformation.

I believe that somewhere someone suggested that it would be better to convert from the binary digest into base64 as it would be more random than from hex. I believe this is accurate, but the method suggested was in Perl, which is kind of messy, and more importantly hard to remember. So I asked, on unix.stackexchange.com, how I could do this on the command line. Here's the answer I decided to accept:

echo -n `date` | openssl dgst -binary -sha512 | base64

Remember you should slightly modify the result in a way that you can remember in your head to make it random, and probably use something in place of the "date" command, since it's not reproduce-able.

Again: this is not meant to be as secure as random passwords, just secure enough compared to non random alternatives.