All right, there has been tons of talk about the need for a library, I've contributed little, unfortunately -- but there is good reason to that...
 One serious issue that is much bigger than getting it to be an effective, useful, search-able, and dynamic library. This issue was copyright. It can't easily be hosted on this site's server, as copyright infringement will take this site down.
While thinking about what the front page might look like to this library, It hit me like a ton of bricks of one possible way to make it useful to all, but not risk the inconveniences of copyright.

A tor site.
Yes, just like the one talked about and shown in the thread about an anonymous auction site.
Read here: http://127.0.0.1/talk/index.php/topic,332.0.html

Here is how one is setup: http://www.torproject.org/docs/tor-hidden-service.html.en
Edit: This might also be useful: http://en.linuxreviews.org/HOWTO_setup_a_Tor-server -- and any other info on setting up a tor server would be appreciated as well.

Anyways, this would offer the following:
1. Anyone (with tor) can see the content of the site - no need to restrict it to just a few individuals
2. The material will not be found by way of Google or other search engines, so the server load would be only for "our" community -- That is, SM, WD, BL, Here, TZ, and the others..
3. Copyright infringement will not be an issue for our side, as no one will know who is hosting it, where it is being hosted, the IP, and also it will not even be found by the copyright law enforcement agencies to begin with!


What I would like to see, if a proper library ever comes into existence is that it is allowed to be copied, or that very extensive archives are produced -- in a sort of Plug & Play archive. By plug and play I mean the archive would not only contain the articles but also  the MySQL and PHP database as well as all that is required to get the library running on another server - with a list of the needed programs, software, and if needed, what OS the server needs to be in order to get it to run.
Of course, if that were done, it would be important to change the password before and after archiving the site -- than give the default passwords (this means changed passwords to the login, the MySQL databases, and all other things -- no reason to slip up and let someone delete it all )

The goal in doing this is that it will create the ability for anyone to host the site anywhere, even not by way of TOR -- than it is their problem to worry about copyright issues. Either way this will make valuable information archived, readily available, and able to be searched effectively.  The information, if it is made so one can make backups shall never be lost again.

One Problem:

If one is to make it dynamic, the backups, and mirrors of the original site will not be up-to-date, or if the cloned websites are able to also be updated as is the original --- they will have different articles added to them, causing a set of different clones of the site with all slightly different content -- What is the solution to that?

I would like to believe this could be easily remedied by connecting something between ALL the servers to allow for them to be updated each time a new article is added. However, I have no idea how this might be done without running into some very very serious security issues, such as the need to reveal the IP of the servers to connect, or if they are linked together, removing articles from one might lead to loss of data by another -- so clearly permissions would need to be set --- another option is to modify the archives/backups so when new articles are uploaded, they are redirected to the main site, in which all the clones/archives are spawned from....

Either way, I think it is very important to make it so it can be backed up and still just as functional - you don't know what is going to happen to the person hosting it on there home server, if they make backups, etc. It would be awfully stupid to go through all of this trouble, and than have it all be gone one morning due to some unfortunate event.


I hope this will lead to some very interesting discussion, and help facilitate the creation of a library -- which I know is more the goals of others than it is mine, but I would like to help as much as I can (with in reason) and I thought this might be one way to go about it -- Please look into it. I would be happy to setup a server.

Also -- I realize that there are a lot of tor nods that are looked at by the government, or at least so they say.. but I'm pretty sure whoever is listening isn't going to care about a copyright article, or know how to do something about it, at least!

--------------------------------Mostly Irrelevant--------------------------------------------------------------------------------------------

Also, and let me apologize in the begging for the next part of this post might just be insane and stupid, and fairly off-topic...
however, I'd disagree because I'm a big fan of symbolism, so I think this would be good to mention as I would like to carry on the bee, wasp, fish, etc type analogues onwards to the library - So here is my input on that subject:

My idea for the name of this library site is The Aranea

According to my sketchy sources, The Aranea means "The Spider's Web"

Why name it The Spider's Web?
Well, in my little imaginary world, spiders are very organized, and they link things together. This is somewhat symbolic of a library, especially this type, the articles will be search-able, organized, and hopefully linked together so it is never lost. Also -- generally spiders will remake (change) their web each day to prepare for the following day so it is more structurally sound, etc --  this could be symbolic for the dynamic library - since it will be updated, adding more articles, and getting stronger with more content.

.. but it gets much deeper than that -- Alright, so Waxworms, as has been discussed before is a parasitic moth larvae  that lives in hives and wasp nests, eating wax, honey,  bee/wasp larvae, that is the very things the nest is made out of - The people who enforce copyright law, which make us take down copyrighted material (thus, destroying (parts of ) our "nest") are fairly well represented in a metaphorical way as waxworms.

as mentioned above, waxworms are the larvae of a moth, moths are captured by spiderwebs, and are devoured.  The Aranea would get rid of the copyright/waxworm problem.

I would like to hear your input on this, as it would be cool if I could add a book to the painting which is being planned to say something like "The Aranea - An abditorium for hiding or preserving articles of value"
Now -- onto the painting I am considering having be produced by the same artist who painted the frontpage picture.... It *might* look similar to the attached image, would that be a good picture for it? I am totally open to suggestion..
What may not be shown in the picture is that the "don't tread on me flag" would be the don't tread on me flag with the hive bee and its trail replacing the snake-- which I think is appropriate as it shows what the place is about, and the "don't tread on me flag" represents our mentality, in that we just want to be able to do want we want to do, and that we don't want people messing with us..

The book, on an angle.. would have a spiders web with a spider in it - probably a black-widow.
 A wasp nest above in a corner to represent us, which would have wasps on it -- assuming The Vespiary can be of use to the making of the library. Perhaps somehow WD via a fish symbol could be added as well?

There would probably be older glassware, and perhaps maybe not the pig skull which died from a bullet to the head....as I'm not to sure that sends the right message if you look at it symbolically... perhaps a dog skull? either way, I am slightly interested in making a reference to animal farm by Orwell -- the dogs, and the pigs are both bad characters for the most part in the book... -- I have both skulls as well as many other skulls such as  lizard, beaver, mountain lion, cat, and many bird skulls to choose from... scales, microscopes, you name it could also be added to the image, so please speak up... but...  Anyways -- Lets try not to get very far off topic with the painting/picture - this thread is about an Onion/Tor Style library, Any suggestions on ways to improve this idea, or why it will not work, or other comments?

PS. Most of this is not me by any means! I've got most of the ideas by reading and talking to what No1uno, and Naf1, as well as others have said.

my 2 cents would be to have a login even if it is just a generic one to keep it a little more obscure and keep prying eyes out.

That is an option, but it would still need to be on a home server I think - most hosting companies will not backup or allow you to backup the site if it is above 30GB, some only 10GB - If it were on a home server, this would reveal IP address, and perhaps cause other problems.
Also if it were password protected, it may seem like it is being used for file-storage, which is always against the terms of service for hosting companies.

I would have with tor is that it grinds everything down to snail speed which could be a problem for large files and full text searching.

I don't think it would be a problem for searching, only navigating and uploading/downloading other files -- I assume that the searching part would use the RAM or processor speed, and not actually have anything to do with the internet speed. I.e you could type "the" and the message would be sent to the server fairly fast, however it would have to find all the articles with "the" in it, which would take a long time if it were a slow computer.

But the slowness of tor is still  a problem,  probably - especially if the upload speed of the server is slow as well, such as only .7Mbps like my internet. How do you think the slowness of Tor could be resolved?

If the server did have a high upload speed, I wonder how slow it really would be? Is there a way to determine the average upload/download speed of Tor in general?

Also -- Can PHP and MySQL be ran via Tor? I assume it can be as the anonymous auction site, if I recall correctly, was running at least stuff similar to PHP and MySQL, but I'm having doubts..

Someone sent me these links which look very useful when it comes to this so I will post them here also...

http://events.ccc.de/congress/2008/Fahrplan/events/2828.en.html

http://events.ccc.de/congress/2008/Fahrplan/attachments/1206_ocat_25c3_r396.pdf

http://www.cypherpunk.at/onioncat/wiki/OnionCat

OnionCat manages to build a complete IP transparent VPN based on Tor's hidden services, provides a simple well-known interface and has the potential to create an anonymous global network which could evolve to a feature- and information-rich network like we know the plain Internet today.

Anyways, to me it seems like Tor is the best option for hosting copyright files, but I could be wrong.

That is pretty far out of my personal reach - I'm not at all educated in setting up something as complicated as that, though I could try and I'm sure I could get help. It would be easier to get an open source journal software and get that running behind Tor on a home-server.

Do you really think that just running a server only through Tor, while hosting ALL of the files on it, for immediate upload would be ineffective?

It takes the copyright enforcers a pretty long time when it is in plain view and indexed by google to find the infringed documents to begin with, running it through tor, where the webpages are not even visible via normal browsing and it also is not  indexed by any search engines.

Plus, they'd have to look at it through Tor to even see it, otherwise won't it just bring up a "Sorry, the page you were looking for cannot be found" page?

Even if they made it that far, what are they going to do? I bet they would have a hell of a time finding who owned the server...

I don't know, what you are saying really would be great, but it is an awfully big task to start up.  

Also some discussion here: http://127.0.0.1/talk/index.php/topic,558.0.html

the government might watch tor, but I doubt they'd really worry about some copyrights -- Are they going to contact all the journals and publishers and tell them? No.
And a lot of the files would likely be public-domain anyways... I.e old books, etc.

They cannot be searched, but if you are referring too just a place for upload where they can be downloaded, like what nk40ouvm said, that is a great idea...

There would be no need to make a more than one torrent really, assuming the torrent contained a list of files with the proper names -- you'd just have to open up the torrent and see the what the file is, and download that individual file.
Additional torrents could be made as updates.

Do torrents ever go offline? get deleted, etc?
The only issue there is if people would be willing to download and allow for uploading. I think a lot of people would do that without much problem.

Torrents are good for transferring mass amounts of data. However it's too slow to run them through Tor, and without Tor every participant gets to see the real IP address of other participants. If everyone here is comfortable with that, and everyone would prefer to get big complete collections instead of individual books or articles, torrents are good.

The IP thing may be an issue, but why aren't torrents a decent option for small files as well?
If you have say a 150GB torrent file, you can click to download that, and than choose the individual files in which you wish to have without needing to download the entire collection.

As far as I am aware torrents can be password protected which isn't all that much of a security blanket...  There are open source enhancements on  the Torrent protocol such as OneSwarm.  It may bypass any necessity for proxies and the like by offering traffic segregation to only those  allowed.  Ultimately though if a member wants to get to know you better and they're capable there wouldn't be a whole lot one could with certainty do to defend against that sort of thing.

http://oneswarm.cs.washington.edu/index.html

This may be of particular relevance to the topic.

http://wiki.oneswarm.org/index.php/Community_server_setup

Why not do like the illegal porn traders and put the large content all into the freenet network, then have a centralized Tor service that indexes all the keys.  The Tor service could also have full copies of all of the content to use for full text searches, but wouldn't need to actually serve the files to the end user as they would get them through freenet which is much faster.

When someone wants to upload a new file, they just put it into the freenet network and post the key to the server.  The server downloads a copy to use for searches, but doesn't need to act as a host which would save a ton of bandwidth.  Require CAPTCHAs and maximum file sizes to keep someone from flooding the server with keys to download and swamping it.

Some kind of moderator would be needed just because lots of people would post keys to random unrelated things to be assholes.

The freenet idea sounds pretty interesting and looks like it could make it a bit simpler - does anyone else have any input on using freenet?

http://freenetproject.org/

Offshore servers is an option I guess, but they are probably not reliable, expensive, and hard to find. I believe you can get servers in china, but I don't know if that is really a good option or not, but the other options, like freenet, tor, torrents, or file-upload sites, etc seem like a  pretty good and cheap option IMO.

I just want there to be a good way to get lots of the files online and stable so that it allows for a library to be built and index them, etc.

Quote from: Dongle on July 13, 2010, 12:57:25 AM

How did the Hive pull it off? I remember there was Hellman, Lilienthal, Rhodium and some others who had everything offshore? Couldn't that be replicated?

They didn't have the fears we have now days. The internet say, 10-12 years ago was a far different place then it is today in the terms of law enforcement. Im pretty convinced if it wasn't for Rhodiums site the hive would have never really came to light the way it did. Look at the issues we have here just from me placing a site up and having copywrite issue all the time. Same effect with Rhodiums site years ago except the flys it attracted where bigger in size with a harder bite.

I've never really heard a good explanation for why the Hive went down and stayed down. It's surely not copyright infringement fears; I've never seen a publisher complain about single articles and the site didn't host any books that were in print. And even with every copyright-infringing bit removed the site would have remained tremendously useful.

As I recall the story (don't know about its veracity), the Swedish web hosting company they were with came under new management, which didn't want to be associated with such a controversial site. But there are certainly other hosts who will provide a home for controversial material if the bills are paid. Even if no host was to be found, it doesn't explain why no static archive of the board was never created and distributed. All that's left to the public is the half-baked partial archive that first surfaced on Synthetikal, plus a handful of content on Hyperlab that was copied over.

I've bugged lugh about the vanished forum data a number of times... he obviously still has a copy because he can fetch old posts on request. I don't know why neither he nor anyone else will release it, and I don't have any pictures of him with midget hookers for blackmail, so who knows if it will ever again see the light of day 

I've bugged lugh about the vanished forum data a number of times... he obviously still has a copy because he can fetch old posts on request.

Yes! Why hasn't there been a release of the entire place? I"m sure its due to some extremely valuable synths that they want to protect. But I thought the whole point of the Hive was its collaborative nature.

With respect to copyright. Why don't you just encrypt all PDFs stored, use a numbering system for the PDFs, and keep the index separate from the PDF archive. This would provide an extra hoop for anyone who sought to determine copyright infringement.

Oops.

Sorry lugh, from the way you can pull useful old posts out of a hat, and how you've talked of bringing the Hive back before, I believed that you and probably other ex-staff of the Hive had full backups that you were keeping private for unknown reasons. If you just had a partial collection of threads saved on your own initiative, I'm sorry for making unwarranted assumptions.

The death of the Hive was very mysterious though and has never been well-illuminated by those in a place to know. Was the Swedish hosting company change of management a real thing? I forget the name of the host now; I think it was posted on SM at one point but censored at Rhodium's request, back when we thought the outage was temporary. Even if that really explained the initial downtime, why has the Hive remained offline for 6 years? It can't have been that hard to find another web host, even for a site as large and controversial as Hive/Rhodium. By now you're probably as sick of these questions as the old "where can I get red phosphorus?" but unlike the phosphorus question I don't think it's really been answered.

I don't know if you are even capable of answering these questions. For all I know a small subset of Hive staff saved the database, pulled the site, and left you in the dark as the rest of us...

There are "seedbox" hosts that cater specifically to high-volume copyright infringement, primarily via bittorrent (hence the name). They might still respond to takedown requests from copyright holders but it is definitely not in their business interests to proactively monitor for infringement, unlike most web hosts. This host, for example, offers unlimited bandwidth at 100 megabit/s and 155 GB disk space for only 16 pounds (about $24 US) per month. And it's not just a torrent-hosting platform, but a full VPS with root access. That's a great price for the resources offered, though it might still be too much depending on your budget.

I don't think Freenet is a good choice for this application. From the Freenet FAQ:

Why can't Freenet store data permanently?

Because we can't find a way to do this without compromising Freenet's other goals. For example, people often suggest that someone's node could just never drop data they want to cache permanently. This, however, won't work because even if the data is still available on their node, there is no way to ensure that requests for that data will be routed to that node. We have considered many other ways that Freenet could store data permanently, but they either won't work, or compromise Freenet's core goals of anonymity, and scalability. However, if you simply reinsert your data regularly, or if it is regularly downloaded (not by the same person as it would be cached on their node and the path to their node), it should remain available indefinitely.

A large library that is always available, and that has very infrequent downloads for most files, is running against the grain of Freenet's design to prioritize popular content.