<< >>
justin = { main feed , music , code , askjf , pubkey };
[ <<< last (older) article | view in index | next (newer) article >>> ]

January 7, 2006
too much data (and cpu)

Got a bit frustrated this evening with the amount of data that I've made myself deal with. There's just so much of it. The mp3 collection and photos are so small and manageable, and the full recordings of all of our jams and rehearsals are somewhat tolerable, but in keeping all of the NINJAM recordings that we host, too much data gets created. And for some extremely stupid reason, I insist on trying to keep it all. And what's the point of keeping it unless you do it redundantly. and I don't just mean on RAID. I mean on RAID, and on another drive, preferably one that doesnt spend all of its time over 100F, etc. So much data, bleh. Nevermind the data that is less big, but scattered around (backing up web servers like this one, cvs databases, etc). There's a lot of irreplaceable bits out there.

Oh yeah my other complaint was that in processing the NINJAM autosongs it ends up using alittle of CPU on my main workstation (which has plenty of CPU to spare), but uses way too much disk bandwidth. So I'm setting up a box just for that processing.

Back to my first paragraph, perhaps it just makes sense to say fuck it. You can't keep information forever. Or if you can, it's not worth bothering. If only we had our 100gb blu-ray recordable discs. if only I can hang on til then, I'll be able to put everything I will have by then on 20 of them or so. times two. plus the online (RAID) version. sigh.

I guess it's not really that much data.. I mean, everything fits within my 1.2tb array. so maybe I should just get another, and keep them synchronized. and set up rsyncs to backup other stuff to them. or just keep getting more firewire drives for the seocnd copies. rOn got one of those usb->IDE cables, that might be a good way to go, and just buy a few 400gb IDE drives..






4 Comments:
Posted by J** S***** on Sat 07 Jan 2006 at 09:54 from 69.209.65.x
Don't stress. I think you're experiencing Google syndrome.


Posted by Andy on Sat 07 Jan 2006 at 11:00 from 142.165.144.x
whip up a couple of these
mashie.org/casemods/udat1.html


Posted by km on Mon 09 Jan 2006 at 10:28 from 69.7.215.x
Why not a p2p Erasure Code based solution? I'd love to see a Nullsoft implemetation of somthing like that. See DIBS or OceanStore:
csua.berkeley.edu/%7Eemin/source_c...
oceanstore.cs.berkeley.edu/info/ov...


Posted by - on Tue 10 Jan 2006 at 08:13 from 64.147.189.x
Nullsoft is dead


Add comment:
Name:
Human?: (no or yes, patented anti crap stuff here)
Comment:
search : rss : recent comments : Copyright © 2024 Justin Frankel