Woofy

Hello,

I'm working on converting all my 0.6 definitions to the new format, and I've encountered a problem with: http://www.goominet.com/unspeakable-vault/

This is the definition I've come up with (sorry if it's hard to read, I'm not very good at this):

comic "Unspeakable Vault"
start_at "http://www.goominet.com/unspeakable-vault/vault/1/"

for page in visit("""<a href="(?<content>unspeakable-vault/vault/[0-9]*/)"\s><img\ssrc="uploads/tx_cenostripviewer/next_en.gif"\sborder="0"\s/></a>"""):
download("""uploads/tx_cenostripviewer/\S+\.(?:jpg|jpeg|png)""")

In any case, the definition seems to get the links just fine, but without the preceding / (which isn't included in the links). Now Woofy appends the links to the existing URL, thus duplicating everything after the server address, for example:

going from the first page:
http://www.goominet.com/unspeakable-vault/vault/1/

to the second page this will happen:
http://www.goominet.com/unspeakable-vault/vault/1/unspeakable-vault/vault/2/

instead of:
http://www.goominet.com/unspeakable-vault/vault/2/

Pretty much the same problem happens with the strip download on this site as well.
Is there any way I can fix this? I've tried inserting a / character, but of course then the regex no longer matches. In 0.6 version I could specify the rootUrl and force the correct end result, is there something similar in version 1.x?

Thank you for any help you can provide.

Sorry - I haven't implemented the rootUrl feature in the engine rewrite. I was rather convoluted, not to mention that it complicated the code quite a bit. It did its job, though :)

Anyways, from what I can see, you cannot have Unspeakable Vault working with the new engine. I'll try to research how I can better parse this kind of urls, so that it works in future versions, but I cannot give you a deadline.

Regards,
Vlad

I see, I thought I was overlooking something, but thank you for your reply, and for the nice program as well.

This appears to be the same as the problem I'm having getting Megatokyo in 1.x…

comic "Megatokyo"
start_at "http://megatokyo.com/strip/1"

for page in visit("""<a href="\.(?<content>[^"]*?)">Next</a>"""):
     download("""<img align="middle" src="(?<content>strips/[^"]*?)" """)

doesn't work, because the new engine assumes all files are stored at the same directory level as the current page; whereas Megatokyo loads pages from "/strip/####" and images from "/strips/####.gif".

Thus the definition as written seems like it ought to work but instead steps through each page trying to download "megatokyo.com/strip/strips/####.gif" which obviously does not exist.

There are 2 possible solutions I see. One would be to somehow change the web engine to account for oddball structures like this (giving us essentially a "CD" command within the web site for pages whose "working directory" is different than the one given in the URL.

The other, potentially easier fix would be to allow the definition writer to construct the ultimate download[s] for each page algorithmically (a function that takes a concatenated string and uses it for the download would be perfect). I actually tried to do this in my copy and was given the error that (string) is not valid to send to download(). In the xkcd example there is a method called match() which appears able to return content from the regex as a string variable. If I could write something like:

    image = match("""<img align="middle" src="(?<content>strips/[^"]*?)" """) as (string)
    download_string("http://megatokyo.com/" + image)

That's a really good idea! And rather easy to implement. It should be included in Woofy in the next couple of weeks :)

I've implemented your idea, and now you can pass the result of match directly to the download expression - it should make things a bit easier for everyone.

However, I've also found out why Woofy was unable to generate links correctly for this comic (it actually had to do with an obscure html element - http://www.w3.org/TR/html5/document-metadata.html#the-base-element ), and fixed that too :). So mikius's definition is now working and there's no need to concatenate strings or stuff like that. Just download the latest version from http://vladiliescu.ro/woofy and you're good to go.