{"id":2537,"date":"2024-03-02T17:14:51","date_gmt":"2024-03-02T17:14:51","guid":{"rendered":"https:\/\/www.jameshatton.co.uk\/blog\/?page_id=2537"},"modified":"2024-03-02T17:14:51","modified_gmt":"2024-03-02T17:14:51","slug":"open-directories","status":"publish","type":"page","link":"https:\/\/www.jameshatton.co.uk\/blog\/open-directories\/","title":{"rendered":"Open Directories"},"content":{"rendered":"\n
Table of Contents<\/div>
<\/span><\/span><\/span><\/span><\/div><\/div>
<\/div>
    <\/ol><\/div><\/div><\/div>\n\n\n\n

    Introduction<\/h2>\n\n\n\n

    If you ever take the time to look at the address in the URL bar of your web browser, you will often see a string of characters separated by “\/” (the forward slash). In a manner similar to the folders used on your computer, a web server uses folders but we tend to use the term directories because that’s the more unixy term and most web servers run on an Unix-like OS. When your browser goes to the address www.someaddress.com\/dir1\/dir2\/ you are simply having the web server look at what is in the directory called dir2 (which is a subdirectory of dir1) and serve it back to you. A web server will serve back the file index.html in dir2 if it is found and if it is not, then it will show the contents of the directory. The latter is a bit of a security risk so most webser admins (including myself) turn this off.<\/p>\n\n\n\n

    There are however a number of webservers darted across the Internet that leave this function on either deliberately or accidentally. ‘Open directories’ refer to these such directories where the contents are visible and even downloadable. Of course the the contents can vary from random images through to ebooks or music. There are goups about that like to find and catalgoue these open directories and one such example would be the r\/opendirectories<\/a> subreddit. Acually, nowadays, these open directories tend to encompass more than just the driectories not locked down by Apache webservers but include folders shared by cloud storage services e.g. Google Drive or One Drive – anything where files can be viewed\/downloaded without authentication.<\/p>\n\n\n\n

    Concerns<\/h2>\n\n\n\n

    There are of course then implications for copyright material to be distributed illegally (depending on local laws) but legality and\/or morality is beyond the scope of this article, where I will focus on the technical aspects and tools. I will however state, that you have to be extremely careful when viewing\/downloading unknown materials from these directories as the files may not be labelled in a way to actually indicate what the contents are. Hence you run the risk of downloading malware or even worse, illegal\/offensive materials (think extreme NSFW) and fortunately where I’ve not been subjected to this, I read a horror story where somebody came across some distrubing contents that had a lasting effect on them. My advice, if in doubt, move on; and hopefully you can see while I will refrain from linking to any such open directories (whose contents may also change after I have linked them).<\/p>\n\n\n\n

    Finding Files and Open Directories<\/h2>\n\n\n\n

    The previously mentioned r\/opendirectories<\/a> subreddit has user submissions of open directories.<\/p>\n\n\n\n

    Both filepursuit.com<\/a> and odcrawler.xyz<\/a> are indexes that allow you to search for specific files in open directories. Anna’s Archive<\/a> is similar but focussed on eBooks, magazines, comics etc. and even academic papers.<\/p>\n\n\n\n

    To donwload files from an open directory individually, you can often just click on the file in a web browser and download it but to download multiple files efficiently will require tools.<\/p>\n\n\n\n

    Downloading Multiple Files<\/h2>\n\n\n\n

    I’ve yet to find a GUI (graphical user interface) application that will basically take the input of a directory (e.g. a URL) and download all the files within so I use two applications, both of which are open source.<\/p>\n\n\n\n

    Listing the Files<\/h2>\n\n\n\n

    First a list of the URIs of the individual files needs to be obtained so OpenDirectoryDownloader<\/a> (by KoalaBear84) will do this. It’s an open source CLI application and you just copy-paste the URL into it.<\/p>\n\n\n

    \n
    \"\"<\/figure>\n<\/div>\n\n\n

    It then spits out a test file which can be read used by either of the next two applications.<\/p>\n\n\n\n

    wg<\/a>et<\/a> is the first main program that many people recommend – it has a CLI (command line interface) as opposed to a GUI (graphical user interface), which will put off most people. wget is an application included on most linux distributions but Windows Powershell uses wget<\/a> as an alias for its Invoke-WebRequest<\/a><\/p>\n\n\n\n

    However, I use aria2<\/a> for its simplicity<\/a> and support for the HTTP(S), FTP, SFTP, BitTorrent, and Metalink protocols. It’s open-source CLI and available for various OS’s. I just stick a copy of the aria2 executable into the same folder as the text file generated by ODD (above) and execute the command:<\/p>\n\n\n\n

    aria2c -i theurlsfile.txt<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
    Introduction If you ever take the time to look at the address in the URL bar of your web browser, you will often see a string of characters separated by “\/” (the forward slash). In a manner similar to the folders used on your computer, a web server uses folders but we tend to use the term directories because that’s the more unixy term and most web servers run on an Unix-like OS. When your browser goes to the address www.someaddress.com\/dir1\/dir2\/ you are simply having the web server look at what is in the directory called dir2 (which is a subdirectory of dir1) and serve it back to you. A web server will serve back the file index.html in dir2 if it is found and if it is not, then it will show the contents of the directory. The latter is a bit of a security risk so most webser admins (including myself) turn this off. There are however a number of webservers darted across the Internet that leave this function on either deliberately or accidentally. ‘Open directories’ refer to these such directories where the contents are visible and even downloadable. Of course the the contents can vary from random[…]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"ngg_post_thumbnail":0,"jetpack_post_was_ever_published":false,"footnotes":""},"class_list":["post-2537","page","type-page","status-publish","hentry"],"featured_image_src":null,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/P89zH1-EV","jetpack-related-posts":[],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/pages\/2537","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=2537"}],"version-history":[{"count":1,"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/pages\/2537\/revisions"}],"predecessor-version":[{"id":2539,"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/pages\/2537\/revisions\/2539"}],"wp:attachment":[{"href":"https:\/\/www.jameshatton.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=2537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}