twop dlthread & dlsubforum :)

Questions and Concerns about Site Closing - Page 47 - Tech Issues, Questions and Information - TWoP Forums 
Posted May 25, 2014 @ 8:14 AM

-Il Cucchiaio:

Here is the script for downloading a subforum, and here's the one for downloading a thread.

Just a few caveats:

• I'm by no means an experienced programmer, and I haven't thoroughly tested these scripts, so you may run into some bugs.  But I have successfully used them to download entire threads and subforums to my computer.  
• I've only used them on a PC.  I think they should work if you have a Mac, but I can't guarantee it.
• Downloading particularly large subforums will take a while and will take up a fairly substantial amount of space.  For example, the Game of Thrones General Gabbery subforum has 73 threads and more than 1700 pages.  It took me about 45 minutes to download that entire subforum and it takes up about 400 MB of disk space.
• None of the internal links in the downloaded pages are changed by the scripts, so once the site shuts down, these links won't work. And these scripts don't download the style sheets, images, etc. associated with TWoP, so after the site shuts down, the formatting of the pages you've downloaded will be stripped down to the basic html.  In other words, it won't look as pretty, but all of the posts will still be there for you to read.

Instructions:

1. The scripts are written for Python 2.7.  You can download the Python installer here.
2. Save the scripts in the folder where you want to store the content you're downloading.  The scripts will download the content to whatever folder they're in.  The thread downloading script will create a subfolder for the thread it's downloading, and the subforums script will create a subfolder for each thread in the subforum.  So if I were downloading the Game of Thrones General Gabbery subforum, I would create a folder called "Game of Thrones General Gabbery," and save the subforums script in that folder before running it.  If you're going to download multiple subforums, make sure to move or copy the script to the right folder before you run it each time. 
3. Open IDLE (which came bundled with the Python stuff you installed) and then open the script you want to use in IDLE.   [Note: Double-clicking the script from Windows Explorer will run it, which you won't want to do before you've edited it, so don't try to open it this way]
4. There are only a few lines of code you need to change before you run it.  At the top of the subforum downloading script, where it currently says, 

   address="http://forums.televisionwithoutpity.com/forum/1193-game-of-thrones-general-gabbery/"

   change the stuff in quotes to the address of the subforum you want to download.  And change

   file_name = "game_of_thrones_general_gabbery"

   to the name of the subforum you want to download.  If you're using the thread downloading script, you also need to set "numPages" to the number of pages in the thread.
5. To run the script, go to Run-->Run Module, or simply hit F5.

That's it.  As I said above, I can't guarantee there aren't bugs, but if you're having trouble getting it working at all, I'd be happy to try to help.  Good luck!

Edited by Il Cucchiaio, Yesterday, 08:16 AM.   



-BoldandRestless: Grazie mille, Il Cucchiaio! Is it possible to run the script and download two subforums at the same time?

-Il Cucchiaio: Yep, BoldandRestless, you should be able to download two (or more) subforums at a time if you open the script in separate instances of IDLE.


 -