"Elsewheres" are nothing more than links to other pages on the internet that I found interesting and think you might like too. They first appeared on this site in January 2005. From then onwards they appeared in the yellow-looking sidebar on the homepage, in their own view and also on the blog document for the day on which I linked to them. They also had their own feed.

Then, last September, I discovered the use of FeedDemon's News Bins. Since then I've still been bookmarking useful pages, just not actually on codestore, but in this feed, as it's so much quicker and easier to drag to the News Bin than to login in here and create documents in this site. I'm assuming some of you subscribe to that feed? If not it's still useful for me personally as an online bookmarking system.

I'd always meant to remove the elsewhere section from the homepage but never got round to it. This made the homepage look stagnant as the last date showing was September 2007. That's not good in webland, is it now!

So, what I've done is write a little scheduled (hourly) Java agent to download the "news bin" and create "elsewhere" documents on this site. Thus keeping that part of codestore alive and kicking.

If you're interested in how the scheduled agent works here's the code.

If you're interested in what I'm interested in then you might want to see the links I, errr, link to. To do this you have three options:

1. Simplest option is to subscribe to the Newsgator feed.
2. Alternatively you can subscribe to the feed on this site which is an hour out of sync.
3. You can visit this site and see them on the right hand side of the homepage. Note that you can also search them on this site to find links you thought you saw at some point (I know I do that still).

So there you have it. Another little change which will hopefully help revive this site a little and is another step towards Codestore v7 (this is 6.5 you're looking at - what, you mean you didn't know that!?).

Until today on codestore you had to actually visit the site if you wanted to read my posting. If you were lucky I'd have remembered to add a one-liner summary to try and tempt you in, but often all you had to go on was the title itself.

The trouble with using the title alone is that (at least I find) it's almost impossible to come up with one that summarises it all in a catchy way. I spend way to much time obsessing over the title of a post sometimes - changing words over and over.

Well now, due to popular demand (you can probably tell if you've got this far and you're still in Google Reader or the likes) I've supplied the full body of the blog entry via RSS. Unless you want to post a comment (for which I given you a handy "button" to press at the bottom of each item) there's no actual need to visit the site.

So, why has it taken me so long to make this change? The answer's in the last paragraph -- that you don't need to visit the site. My concern with full-content RSS feeds is that my visitors never actually, err, visit. This means they never get to see the site I put so much time in to designing. Purely selfish, I know and no reason to force people to visit.

The obvious side effect to this is that I lose visitors who don't like taking the gamble that something is worth their time to read based on what the title tells them. Hence my about turn. I might as well get the content I write to as many people as possible hadn't I?!

There was another, more practical, reason not to supply full content. All the internal links in my blogs are relative to the BASE href tag in the page's Head. Likewise for inline images. Links start with "http://www.codestore.net/store.nsf/unid/" and images with "http://www.codestore.net/store.nsf/rsrc/" both of which are the names of sorted views.

The trouble is that once downloaded to the RSS feed it doesn't know what the BASE href is and so links and inline resource files become broken. Take the link in the last paragraph for example. If you examine it on the site itself you'll see it points to unid/EPSD-52UQZJ?OpenDocument. If you examine the link in the RSS feed you'll see it points directly to http://www.codestore.net/store.nsf/unid/EPSD-52UQZJ?OpenDocument.

So how did I got about this without using nasty hard-coded absolute links in the actual stored content?

All I did was add an @ReplaceSubstring() to the RSS view. Blog posts on this site are just HTML stored in a plain old text field. All I needed to add to the RSS feed's view was this:

@ReplaceSubstring(
        @Implode(Body; @NewLine);
        "=\"rsrc/":"=\"unid/"; 
        "=\"http://www.codestore.net/store.nsf/rsrc/":"\"http://www.codestore.net/store.nsf/unid/"
);

It seems safe to assume that any occurrence of the strings ="http://www.codestore.net/store.nsf/rsrc/" or ="http://www.codestore.net/store.nsf/unid/" are either a link or an image, respectively. Seems foolproof to me and, more importantly, it works. To prove it here's an image, which just happens to be an image of what the feed now looks like in FeedDemon:

Notice the preview image of the laptop that FeedDemon provides when the item is collapsed. All very cool.

So I hope we're all happy now? If you have any more requests just ask.

I even went as far as setting up a page on JustGiving.com where I'd accept the monies and record the generosity of the donors. The target for the fund raising on that page is a (perhaps) overly-ambitious 10 grand.

Since then I've been doing quite a bit of thinking and have decided to make the charitable donations optional and the download of the application completely free. There are a few reasons for this.

Mainly it's that I don't want to suffer the disappointment of the whole thing falling flat on its face. If this were to happen it would probably make me question my devotion to this site and I don't want that to happen.

What I also want to avoid is starting off with a compulsory donation and then moving to an optional one, thus upsetting those who have already shelled out for it.

Another reason is that I want to make all the code I share available to as many people as possible. Requiring a payment would just mean that not as many people get their hands on the code as possibly could.

Something happened recently which brought home the fact that this site doesn't have the pull it once used to. I probably shouldn't but I use the number of comments posted on blog entries as a (very rough) indicator of the site's popularity. Eighteen months ago when Felix was born the Say Hello to Felix post got 140 odd responses. Last week the Say Hello to Minnie post got a less than half that!

While the Minnie post still got a lot of responses and I'm grateful of them all I can't help but feel the difference between the two number indicates that this site's readership has fallen in the 18 months since Felix arrived. It's inevitable that having children means I don't have the time I used to put aside for running codestore. I've tried my best to keep a certain amount of technical postings and code snippets flowing but there have been the odd week of silence here and there and that's bound to have an affect on the popularity of the site.

My plan now is to try and kick the site back in to gear. Despite the arrival of another child I should now - in theory - have more spare time. Whereas before most of my spare time (including down time with work) was spent working on the house, that's all over now, save for the odd little job here and there. I should then have more time to spend keeping up with emerging technologies and fresh techniques, as before. There should be some time left to put this knowledge on here in the form of demos and "How To"s.

The plan with DEXT is that all new techniques and demos are added to one central database (note that it no longer has anything to do with Ext -- see Ext.nd for that). Each time a new demo is added to DEXT a new release will be made available for download. This will make it easier for you (and me!) to find that demo you knew you saw on here somewhere. I too forget where a demo database is sometimes and the server is becoming a mess of demo databases that can be a nightmare to maintain and follow!

Stick with me on this. I need to do some house-cleaning but should be able to make a copy of the database available within a week or so.

Testament to the build quality of the ThinkPads is the fact it was relatively unharmed other than a piece of the casing missing on the corner. The only problem is with the screen. It's very, very, very dull. You can just make out the login box if you get to within an inch of the screen. Logging in or using it with its own screen is out of the question. Since I've moved to the new office I've had it permanently on the port replicator with an external screen. It's not been used as a portable laptop since and won't be until it's fixed.

Now that a new baby has arrived it would be good if I could work from the house a little more. Hence I really need to get it sorted. I just don't know where to start. Do you?

At the weekend I finally got round to taking it apart, as you can see. Believe it or not it's on and running in this shot. I even managed to completely re-assemble it without powering down.

In taking it apart I was hoping to find a loose connection and fix the problem. Alas I didn't and it looks like a hardware fault.

Having done a little googling I narrowed the problem down to either the LCD inverter or the CCFL backlight. As the CCFL is the cheapest part I'm going to buy one of them and hope that works. If not I'll try the inverter. If neither work I might have to consider a new laptop (it is nearly 3 years old after all). What's the equivalent of the T42 now? Is it the T61?

Here's a re-creation of the photo of her brother from the day he was born. See the similarity!?

Things started in the normal way and it looked like our planned home birth was going to go ahead. We probably left it a little late to call the midwife though and before we knew it (and before the midwife had arrived) Karen was in the bath and wanting to push. At this point it dawned on me I could well be doing the delivery myself. Needless to say I didn't fancy the idea and called 999. Within 10 minutes an ambulance arrived. Within another 10 minutes we were in the maternity ward. It's amazing how fast you can get somewhere with blue lights flashing and the rush hour traffic parting to make way.

Within 10 minutes of being in the hosptital Minnie arrived. Daddy was, yet again, reduced to a babbling fool crying like a little baby.

On Saturday night we were next door and the whole of the street was there. We got a sweepstake going on sex and weight. It turns out I was spot on with the weight but got the sex wrong. I'd been convinced it would be another boy. Howlett men just don't have girls. Apparently. Well, I'm glad I did. She's a little beauty and I'm as proud as punch.

Things might be a little quiet for the rest of the week although I'll try and post more techy stuff if I can. I need a couple of days paternity to help Karen out.

Before I go here's a photo of Felix. I promised 18 months ago not to let this site turn in to a glorifed family album, but have realised I hardly post family info or photos at all now. Well, here he is pretending to be daddy:

Isn't he a big boy! You don't realise how much your children have grown until the next sibling arrives...

Despite the fact nobody knew of a way to stream data directly I did manage to get something out of the whole exercise. What came of it is a way to generate, store and (optionally) email automatically generated files without using the server's disk.

So what? Well, not needing to write to the disk means you don't need to worry about who signed the agent last and/or what access the ID has to the server. Any old developer can now write an agent that works with file generation!

As an example, here's a form that will generate and attach a PDF to itself during the WQS event. It can even email it, as you can see to the right.

You'll just have to trust me for the time being that the agent is run without unrestricted rights.

There's also a form that does the same thing with a text file and one that will create an Excel file

I can't show you a CAPTCHA demo online as this server is Linux and without X11 running it can't use the Java graphics libraries.

Although the approach is the same for them all the difference between the Text/Excel and Image/PDF is that they are based on text and binary data respectively.

Generating the text files, as you'd imagine, is easier and can be achieved with a small amount of LotusScript. Here's the code needed in a WQS agent to attach a text file.

Generating binary files like the PDF requires Java (we need to use a ByteArrayOutputStream as a temporary buffer). Here's some commented code that's an abstract from the WQS of the above demo.

There's probably more detail that's worth discussing further and so I'll probably write a full-length article on the topic.

Next week I'll outline how you go about downloading the DEXT app that the demos live in. In the mean time have a play with the demo. Don't abuse the send-email feature though please! I'll be keeping an eye on its usage and will cut it off if used badly.

As I mentioned yesterday the image created in the Java Agent is written to the disk. What I didn't mention is that the file is then added to a Rich Text field on a Captcha form and saved as a document with the answer to the captcha in there too. The file is then deleted from the disk and the agent redirects the browser to the file within the document that was just created.

To do this I added an image in PassThru HTML to the form that needs a CAPTCHA, like so:

The ID is the hash of the DocumentUniqueID assigned to the document at the point of ?OpenForm. There's probably little point to hashing the document's ID but it seemed geekily cool to, so I did.

This ID is stored in an editable field on the CAPTCHA-enabled Form called CaptchaID. The field is type="hidden" so that the value is passed back to the server and the value of the field at ?OpenForm is the same as the value at ?CreateDocument. Otherwise it would change and we'd lose our link between the "session". It's a handy hack of a session, but is it a flaw in the security of the process?

The "captcha.jpg" Agent creates the image and a document in which to store it. It then redirects the browser's request for the CAPTCHA image to a URL such as:

When the document with the CAPTCHA on it is saved the value in the CaptchaID field is used to look in the "captcha database" for the Captcha document with the same key (the one where the image file is stored along with the answer). This answer is tested and the Captcha document is then deleted. If it doesn't match then the user returns to the form and a new image/document is created.

The obvious flaw here is storing both image and answer in the same document to which the Anonymous user has read access and to which a hacker would know the key (but not the UNID). This might seem like a massive hole in security. But is it?

I feel safe in this approach as I have hidden all but one of the database's design elements from the web. The only thing not hidden is the "captcha.jpg" agent. The Form used to store the document and View used for lookups are both hidden. If you traverse back through the image's URL to the following address:

You will get an error 404 saying:

Go back as far as the root of the database:

And you will see "No views found". Guess the name of the view and you'll get another 404.

As far as I can see there's no way whatsoever that an Anonymous web visitor/hacker/bot can get access to the answer which is stored as plain text in the "hidden" document.

As an extra precaution the hidden Form has no actual fields on it when opened in Domino Designer. If the Form wasn't hidden then all they'd see in the browser is a blank page anyway.

How secure is this approach? There's a simple workaround of keeping the file and answer in separate documents and adding Readers field security to the latter, but I don't see the need to, unless you know of one?

I'm writing a CAPTCHA plugin for a Domino site at the moment. Although I've managed to get a working solution I can't help but feel it could have been a whole lot easier to achieve were there not a couple of things missing from the basic toolkit of a Domino developer.

Creating images on the fly using Java Agents is actually a lot easier than you might think. The hard part -- at least with Domino -- is getting the resulting image to the browser.

There are several Java libraries out there for creating CAPTCHAS but they all seemed huge to me and just a little OTT. Not only that but the two or three I looked at (including the most popular JCaptcha seem to rely on you having such a thing as a user "session" that ties separate page requests together. We don't in Domino and so can't use them unless we're willing to suffer writing and deploying servlets (something I'd leave to the very end until I went down that route).

After further searching I found the source to a JSP page where somebody had written a CAPTCHA generator in less than 100 lines of code. That's what I like to see. Lean and to the point.

Not that is was a simple case of copy/paste of course. The code was written using parts of Java, such as ImageIO, which were only introduced in Java 1.4. My target was a Domino 6 server, which only has Java 1.3 (Domino 7 has Java 1.4).

After more searching I found you could write JPEG files using an undocumented package called com.sun.image.codec.jpeg in Java 1.3. The trouble here though is that there's no way to write the JPG file you've produced directly to the browser.

The main way a Java Agent writes to the browser is using the PrintWriter class, but this won't let you work with files and their inner bytes. Just text! Mikkel Heisterberg has written about this limitation on his well-worth-reading blog. He found that there's an undocumented method in Domino 7's Java called "getAgentOutputStream()" which gives us an OutputStream object. This is exactly what we need. The trouble is it's buggy and unusable. Whether it works in Domino 8 or not I don't know?

The only real way (although I hope somebody is going to correct me on this) to get the image to the browser is to save it to the server's disk first!

Saving to the server's disk is easily done and isn't a big deal, as long as you clean up after yourself, but I'd prefer to avoid that in the first place. Not only because it means you have to make sure code is signed with IDs that are allowed to run unrestriced code and the Agent itself needs to have an increased level of access, but it also means you have to make sure the Agent never gets updated by a copy signed with another ID, which would break the code. It's really quite messy.

Why, why, why can't we create an OutputStream in a Domino Java Agent!?

The trouble is it turned in to a two hour job!

Google Analytics adds itself to a page using this line of code:

This adds a Script tag to the document and loads the required code from Google's site. That's the idea. On the DominoWiki page it didn't add a tag though. It just wrote the escaped string to document itself as visible text.

The first thing I did was add some more JavaScript lines to test the result of escape('<'), which it bizarrely returned as '&lt;'. Since when did escaping do that!?

After about an hour of messing about with charset settings and whatnot I started to reverse engineer the page to the point where I found that removing the reference to the wiki.js file solved the problem. Looking about in that file I found the following function:

/*
        unescape
        Decode an XML string
*/
function unescape(s)
{
        var n = s;
        n = n.replace(/&lt;/g, "<");
        n = n.replace(/&gt;/g, ">");
        n = n.replace(/&quot;/g, "\"");
        n = n.replace(/&amp;/g, "&");
        
        return n;
}

Whoever coded this part of the Wiki had re-written the built-in escape/unescape JavaScript functions. I'm sure I don't need to tell you how naughty that is. Not only bad practice but just plain annoying for the likes of me who come in afterward to do anything and spend half their morning struggling to work it all out.

The fix is simple enough. Just add the following two lines to the unescape() function in wiki.js:

//Following two lines required for Google Analytics to work
n = n.replace(/%3C/g, "<");
n = n.replace(/%3E;/g, ">");

Here's a massive list of "reserved" words you should avoid using in your own code as variables or function names.

var Wiki = {};

Wiki.unescape = function(s)
{
        var n = s;
        n = n.replace(/&lt;/g, "<");
        n = n.replace(/&gt;/g, ">");
        n = n.replace(/&quot;/g, "\"");
        n = n.replace(/&amp;/g, "&");
        
        return n;
}

There's then little to no chance of the code conflicting. Not just with others developers code but also with the inbuilt code of the browser.

Right then, back to what I should have been doing...

Their web presence is all but closed off -- access is only by registration and subsequent authentication. It was the registration process (open to Anonymous users) that showed the obvious signs of attack. At first it looked like spam but on closer inspection was in fact an attempt to hack the server.

The main method of attack appeared to be SQL injection and so there was no real cause for concern as Domino is obviously immune to that method.

What got me worried was when I investigated the ticket that stated all their website's login forms had reverted to the "one with a key on it". On opening domcfg.nsf it was immediately obvious the attack was Domino-aware and security had been compromised. Every document in domcfg.nsf had had its field values over-written with the same patterns used by the attack (Google value777.com and you'll see what I mean). Needless to say I was shocked, got on the phone and had them shut down the http task right away.

The attackers must be Domino-aware as there's no other way they could have found domcfg.nsf.

My next action was Googling "public access to domcfg.nsf" which took me straight to Chirs Linfoot's post on the problem. An interesting reply if from Kendall and points out:

domcfg.ntf's [-Default-] entry doesn't have "Write public docs" checked. When adding Anonymous, did you set it to No Access and then set it to Reader? Notes has this annoying "feature" where upgrading access checks boxes you don't check, and downgrading unchecks boxes. Because of this, I've learned to carefully inspect all checkboxes in the ACL when changing entries.

How it happened in this case I don't know, but Anonymous users had the write access to everything in domcfg.nsf. If you run a web server on Domino check your domcfg.nsf now!

No real harm came of it and we were able to restore the old setting from backup but an interesting question was raised by the customer -- were the attackers trying to harvest username and passwords? It go me thinking. Although my immediate reaction was that this isn't possible I can't say I was completely sure of it at first. What if there was some way of manipulating the sign-in form via domcfg.nsf so that all login requests were sent elsewhere without your knowledge. Scary stuff.