Woofy

OK, here's another interesting problem for you.

The page source for the comic "Power Nap" splits its image URLs with a line break, as in the following:

<img src="http://www.powernapcomic.com/
2015/pnap20150406.jpg" border=0><br> </td></tr></table> <!— del comic—>

Now, if I just try to capture the entire URL, the line break is read in (incorrectly, of course) as part of the URL, and Woofy gets a 404 error instead of an image. But I also can't just capture the portion following the line break as if it were a relative URL, since the page's URL is http://www.powernapcomic.com/d/20150406.html. (Note the extra "d" directory which isn't part of the image's URL.)

So, what I need to do is either (a) capture the entire URL but have Woofy either delete the line break character before attempting to download or simply ignore the line break character completely, or (b) capture the "relative" URL and then construct the download URL by prepending "http://www.powernapcomic.com/" to it.

Are either of those possible in a Woofy definition file?

— Darryl

I _think_ the first one is possible, by doing something like this:

comic "..."
start_at "..."
 
for page in visit("""..."""):
    image_url_raw = match("""whatever regex captures the src, same syntax as for download""") as (string)
    image_url = image_url_raw[0].Replace("\n", "")
    download(image_url)

Haven't tried this approach though, so you might run into a couple of issues.

The good news is that the second approach is certainly possible :)

comic "..."
start_at "..."
 
for page in visit("""..."""):
    relative_image_url = match("""whatever regex captures the second line of the src""") as (string)
    image_url = "http://www.powernapcomic.com/" + relative_image_url[0]
    download(image_url)

Still something not quite right using the format you provided in the second example. Adding a "warn" statement to print out image_url confirms that the URL is being assembled correctly, but the "download(image_url)" statement is generating a "no images found" result rather than actually downloading anything.

Could you post your definition so I can take a look at what's happening?

No problem.

comic "Power Nap"
start_at "http://www.powernapcomic.com/d/20110617.html"

for page in visit("""<a href="(?<content>[^"]*?)"><img src="[^"]*?next.jpg" alt="the next comic" border=0>"""):
    relative_image_url = match("""(?<content>20[^"]*?pnap[^"]*?jpg)" border=0><br>  </td></tr></table>""") as (string)
    image_url = "http://www.powernapcomic.com/" + relative_image_url[0]
    warn("{0} {1}" % (page, image_url))
    download(image_url)

You're right and I'm wrong, sorry. download will try to find the url in the page, and since it can't find it (because of the new line), it skips the image. So neither approach will work.

I thought it worked initially, because my tests used a "regular" comic, so concatenating the image url resulted in a string that was already in the page, and download found it, and all went well. Sorry again for the confusion.

If I'm understanding you correctly, the download() function, even if given a specifically-defined URL, only actually downloads the file if the URL appears somewhere in the page's source code…?

Interesting.

And, I assume, something that's not quickly and easily fixed. ;)

Well, a non-Woofy workaround exists for the one comic I've come across which presents this problem, so it's not terribly critical. (All of the image files for "Power Nap" are stored in a set of directories which have index lists which can be publicly viewed, which means a "download all images from this page" browser plug-in can grab them all.)

— Darryl

Yup, download() will search for the regular expression every time (don't know why I thought it would download the link, guess it was just wishful thinking). It's not a straightforward fix, either :(.

Glad you could at least download the images.