Posted Yesterday, 06:26 PM
LiteraTec:
First of all, huge thanks to Il Cucchiaio for those scripts. So wonderful of you to do that. I'm on a Mac running 10.6.8 and it's been working for me. So far I've been able to download several individual threads and one entire subforum. About a dozen more subforums to go and then I can relax a little :-) So here's exactly what I did:
1) Downloaded and installed Python 2.7
Il Cucchiaio instrxn, see copied prvs (below) dlww post:
The scripts are written for Python 2.7. You can download the Python installer here.
/my qstn asked & answered, re wh to install [python Mac OS X 64-bit] and [ActiveTcl 8.5.15.0],
in short:
I downloaded the first one: Mac OS X 64-bit/32-bit x86-64/i386 Installer (2.7.6) for Mac OS X 10.6 and later [2] (sig).
I had some issues with IDLE freezing up. I'm gonna try installing ActiveTcl 8.5.15.0 and see if it helps.
in full: -- replies - nice!
-Hathaway Green, Posted Yesterday, 09:32 PM :
LiteraTec, thank you for your notes on exactly what you did! I am on a Mac running 10.7.4 and am going to follow your steps. I have a question about the first very first step, though. In which format did you download Python 2.7? I'll be grateful if you or anyone here can advise me about which link to click at www.python.org/download/releases/2.7.6/
I figure I want to click on one of two options below:
#1 " Mac OS X 64-bit/32-bit x86-64/i386 Installer (2.7.6) for Mac OS X 10.6 and later [2] (sig). [You may need an updated Tcl/Tk install to run IDLE or use Tkinter, see note 2 for instructions.] "
#2 " Mac OS X 32-bit i386/PPC Installer (2.7.6) for Mac OS X 10.3 and later [2] (sig). "
Do I need to be concerned by note [2] about a TCL/Tk install? Thanks for any help!
Il Cucchiaio, another thanks to you for doing & sharing this. I'd seen your earlier post a bit up thread that you were going to try writing a script, and I'd thought to PM you but don't think I have PM ability. So I was super glad to see that you posted again after finishing the script.
________________________
-Il Cucchiaio:
hathaway green, the Mac users up-thread who've successfully installed Python can probably advise you better on this than I can, but it looks like you probably want to download the 64-bit version and then download Active Tlc 8.5.15.0 from here. And I'm happy to have shared the scripts. I had actually initially planned to write a script that downloaded the entirety of the TWoP forums, but I quickly realized that it would take up too much time and space to download everything. I finished writing the subforums script a few weeks ago, but I didn't think think there was any interest in it around here, and I kind of forgot about it until I noticed that I had a private message from another poster about it. So anyway, I'm glad that there are a few people who've found it useful.
Edited by Il Cucchiaio, Yesterday, 10:23 PM.
hathaway green, the Mac users up-thread who've successfully installed Python can probably advise you better on this than I can, but it looks like you probably want to download the 64-bit version and then download Active Tlc 8.5.15.0 from here. And I'm happy to have shared the scripts. I had actually initially planned to write a script that downloaded the entirety of the TWoP forums, but I quickly realized that it would take up too much time and space to download everything. I finished writing the subforums script a few weeks ago, but I didn't think think there was any interest in it around here, and I kind of forgot about it until I noticed that I had a private message from another poster about it. So anyway, I'm glad that there are a few people who've found it useful.
Edited by Il Cucchiaio, Yesterday, 10:23 PM.
________________________
-LiteraTec:
hathaway green, I downloaded the first one: Mac OS X 64-bit/32-bit x86-64/i386 Installer (2.7.6) for Mac OS X 10.6 and later [2] (sig). Not sure about Tcl/Tk. Mac OS X 10.7 comes with Apple 8.5.9 and according to this site it may work fine as an alternative. Otherwise you'll need ActiveTcl 8.5.15.0. FYI, for anyone else running 10.6, it seems the older alternative that I have (Apple 8.5.7, which came with OS X 10.6) is notoriously buggy, which may be why I had some issues with IDLE freezing up. I'm gonna try installing ActiveTcl 8.5.15.0 and see if it helps.
Edited by LiteraTec, Today, 12:44 AM.
hathaway green, I downloaded the first one: Mac OS X 64-bit/32-bit x86-64/i386 Installer (2.7.6) for Mac OS X 10.6 and later [2] (sig). Not sure about Tcl/Tk. Mac OS X 10.7 comes with Apple 8.5.9 and according to this site it may work fine as an alternative. Otherwise you'll need ActiveTcl 8.5.15.0. FYI, for anyone else running 10.6, it seems the older alternative that I have (Apple 8.5.7, which came with OS X 10.6) is notoriously buggy, which may be why I had some issues with IDLE freezing up. I'm gonna try installing ActiveTcl 8.5.15.0 and see if it helps.
Edited by LiteraTec, Today, 12:44 AM.
2) Right-clicked the links to the scripts and saved the files (dlThread.py and dlSubForum.py) to my desktop.
3) Double-clicked the subforum script and it automatically opened in IDLE.
4) Changed "address" and "file_name" to the ones I wanted. The only problem I had here was that IDLE kept freezing whenever I tried to do any sort of cut/copy/paste of multiple characters. So I went in and backspaced one letter at a time to erase the original address and file name, and then manually typed out the ones I wanted. Once I did that it worked fine.
-forumfish: As some have posted upthread, I find that it works best (on a Mac, anyway) to manually key in the names, as copy-paste causes crashes.
... I did the same thing with the individual thread script. The only difference is you must change the number of pages in addition to the address and file name - don't forget to do that! I forgot the first time and ended up with over a hundred html files for a six-page thread because the default was something like 150. It just kept downloading the last page over and over, lol.
4) After I changed that stuff, I just went to the top menu bar and clicked Run>Run Module. Got a little dialogue box saying "source must be saved" and clicked OK.
5) Presto! Everything started downloading right to my desktop.
to the desktop? okay, bcs th is wh saved the script files. script will save the dl'd threads to that same location = folder (desktop ~is a folder) = 'directory' (I think)
Il Cucchiaio instrxn, see copied prvs below dlww post:
Save the scripts in the folder where you want to store the content you're downloading. The scripts will download the content to whatever folder they're in. The thread downloading script will create a subfolder for the thread it's downloading, and the subforums script will create a subfolder for each thread in the subforum. So if I were downloading the Game of Thrones General Gabbery subforum, I would create a folder called "Game of Thrones General Gabbery," and save the subforums script in that folder before running it. If you're going to download multiple subforums, make sure to move or copy the script to the right folder before you run it each time.
and note:
-alynch:
Il Cucchiaio, just as an FYI, the subforum script has errored out when I tried it on the Arrested Development and Sopranos forums. They work fine at first, but then stall out on certain thread every time. For Arrested Development it's this thread, and for The Sopranos it's this one. Anecdottaly, I wonder if the script has trouble with long thread titles. It's no big deal if it can't be fixed. I've developed a method for saving them manually*** that isn't especially tortuous.
-Il Cucchiaio:
Hm. I just tried downloading the Arrested Development forum and didn't have any problem with that thread. But you could be right that the problem is related to the long thread titles. I know that in Windows, the path + filename can't exceed 225 characters, so if you have the files deeply embedded in a directory, then that could be a problem.
-alynch:
You're right. I just tried it using my desktop as the directory and it worked fine. Thanks.
***
-hathaway green Posted Today, 04:38 PM
alynch, what is your method for saving subforums or threads manually? I'd be glad to know, if you don't mind sharing.
-alynch Today, 04:50 PM
hathaway green, I've been using Autopager. It's a Firefox add-on, so you need to use that browser. I only use this method when the Python scripts error out. It basically allows you to load all the pages of a thread onto a single page. Then you just save the Web Page to your computer. It can be tedious for subforums since you have to do each thread individually, but it takes pretty much exactly as much time as the scripts if you give it your full attention. Here are the instructions I got from another poster who pointed to this tool:
Once you download it it brings you to it’s instruction page and there will be a pop up at the top, click normal mode and then go to TWOP and it should ask you (in another popup) to load the rules that exist already for TWOP (or auto load all rules that exist for every site), accept that and it should work fine :) . Go to page 1 of whatever thread you want and then click on the icon for the add-on in your toolbar and it gives you the option to immediately load 3, 5, 10, all pages (and pause or clean all). If it’s not in your toolbar then go to View, Toolbars, Customize, find the icon and drag and drop it to your toolbar. Or you can right click, select auto pager, immediately load, all pages. It will flash a green symbol on the page that says loading until it's done.
Edited by alynch, Today, 04:50 PM.
-forumfish Posted Today, 04:54 PM Finally had time to sit down and play with Il Cucchiaio's scripts. As some have posted upthread, I find that it works best (on a Mac, anyway) to manually key in the names, as copy-paste causes crashes. So far, my tests have been successful. Many thanks and virtual cookies (the edible kind) to Il Cucchiaio and everyone else who has shared their tips.
hathaway green, I don't know if this is what alynch did, but I've been using the "Re-Pagination" add-on for Firefox, so that I can see up to 100 pages of a thread on a single page (with lots of scrolling, of course). I then hit "save" and choose "Web page, complete" so I end up with an html file plus a folder of supporting files. This works for a thread, not a whole subforum, and threads longer than 100 pages must be saved in sections. It works great but is very time-consuming, which is why I am so excited to have these new scripts!
Edited because alynch posted just before I did. Edited by forumfish, Today, 04:55 PM.
-
No comments:
Post a Comment