[Yisroel]

I want to build/buy a new system, mainly for all my storage devices. It will be used a real ton for connecting a whole lot of storage devices (USB flash drives, SD cards, and hard drives), but I mean a lot like 20-30 at a time. And it'll also serve to copy over from one device to another, a lot of copying at a time.

 

So my question here to the experts is: What is the main thing I should focus for such a machine that it should be able to handle such kind of stuff (I won't use it for graphics gaming or any intense software) without a blink?

 

Hope someone will help. If anything is unclear please ask me.

[Advertisements]

Gday Yisroel.

Can you give us an idea of what you want to achieve and the workflow you will use?

 

 

without a blink?

What do you mean? or was it a Typo?

 

What type of files are on the flash drive/SD cards?

Will you copy the files to the computer and manipulate them in some way? or just copy to another source for storing them?

What will you do with the stored files?   Just archiving or will they be accessed by a client?

What is your budget?

in which country will you purchase parts? 

[iammykyl]

Gday Yisroel.

Can you give us an idea of what you want to achieve and the workflow you will use?

 

 

without a blink?

What do you mean? or was it a Typo?

 

What type of files are on the flash drive/SD cards?

Will you copy the files to the computer and manipulate them in some way? or just copy to another source for storing them?

What will you do with the stored files?   Just archiving or will they be accessed by a client?

What is your budget?

in which country will you purchase parts? 

[Yisroel]

Gday Yisroel.

Can you give us an idea of what you want to achieve and the workflow you will use?

without a blink?

What do you mean? or was it a Typo?

 

What type of files are on the flash drive/SD cards?

Will you copy the files to the computer and manipulate them in some way? or just copy to another source for storing them?

What will you do with the stored files?   Just archiving or will they be accessed by a client?

What is your budget?

in which country will you purchase parts? 

Without a blink, I meant to say without hesitation. It shouldn't freeze on me or crash when copying multiple things to multiple devices on a time.

 

All kinds, no specifics, just a lot of all kind of them.

Most of the time just from one to another. But sometimes to the PCs drive too.

I'm copying for use in different clients, not my PC.

I would pay up to $1,200-$1,300 for something really good.

The US.

 

And if possible I want more of background information, I should know 'what' are the specs that will make a difference for my situation.

[iammykyl]

I am still trying to determine what class of CPU to spec, so need to fully understand your workflow.   is this correct.

 

You will copy files, (picture/video) from flash/SD media directly to HDD/SSD storage?

 

Is that one source at a time, to NAS or PC ?

More than one source at a time to NAS or PC?

If sent to the PC, what will you do with the Data?

If sent to the NAS, what exactly will happen to it?  i.e. streaming, P2P?

 

How many drives will you need?

Do they need to be in a RAID array?

 

What software will you be using?

[Yisroel]

I'll copy from, and to, USB SD and HDD. Mainly not to the system hard drive.

 

I want to be able to do multiple copy loads simultaneously. Let's say copying SD card O: to USB T: and USB K: to hard drive V:, and so on many more at a time. Also probably sometimes copy form them to a NAS for backup.

 

The more inputs the better, maybe also some hubs to have more inputs. USBs etc. will come and go. But not more than two three steady drives inside the PC, probably not in a RAID.

 

Just leave them alone on the place where it will be copied to.

 

No major programs but the basics. Except some disk recovery software.

[iammykyl]

Sorry for the delay.

Just posting the parts list for you to look over.   Will give more details tomorrow as have to go out again.

> http://pcpartpicker.com/p/9VmzD3

[terry1966]

be interested to know how you plan on connecting 30 hard drives to your pc, but with that many all transferring data at the same time then i'd be wanting the fastest connection/data transfer speeds possible so might be worth looking into a pc with thunderbolt connections on the motherboard or some other connection type like sas.

 

main concern i'd be looking at is saturation of the data bandwidth/connections used and why i'd be looking at the fastest connection types to alleviate any problems in that area, so a lot would depend on how you connect the drives, for instance if they are in some sort of server then a 10GB/s ethernet connection may be the best connection type to use.

 

sorry can't offer any direct recommendations because i have no experience in what your trying to do but the data transfer speeds is what jumped into my head and would be my main concern so thought i'd mention one or two things to consider.

 

 

this link may offer some ideas for you on how your connecting your 30 hd's, i'm thinking your probably going to want a das type solution.. :- http://serverfault.c...into-rack-mount

Edited by terry1966, 15 October 2015 - 09:06 PM.

[Yisroel]

The truth is that I'm confused myself... I never built a similar machine.

 

I wasn't gonna use 30 hard drives. 5 TB of hard drive space is probably enough. I said that between all connected data devices I will sometimes have near 30 devices. Let's say five hard drives, 10-15 USB flash drives, and some SD cards etc.

 

Where I would plug them in? I would prefer the more direct ports on the PC, but obviously i'll use some hubs in case of need.

[terry1966]

to be honest it doesn't sound like you really need anything that special then, i'd probably build a new skylake i7 6700k pc, with a motherboard that uses latest chipset and usb 3.1 standard etc. ( i'm sure iammykyl won't mind putting a good parts list together for you if required. i'd probably go with an m2 ssd boot drive even though they are still expensive in my opinion. )

 

i'd probably add an internal card reader something like this with some more usb3 ports for you to use too :- http://www.neweggbus...m=9b-17-245-002

 

and if you find you still don't have enough usb connection ports on the case and motherboard and card reader (be very surprised if you needed more.) then you can look into adding a pci-e card with more like this :- http://www.neweggbus...m=9b-15-158-412

 

personally my system has 8TB of storage on 6 internal hard drives and when i need to transfer data from them to any external drives for any reason then i use a hard drive dock which connects via esata for fastest transfer speeds my system is capable of. something similar to this usb 3.1 dock :- http://www.neweggbus...m=9b-17-707-398

 

i also have about a dozen usb sticks but can't say i've ever needed or wanted to connect more than 2 at a time, even though my system does have 12 usb ports to connect at least that many devices if i had to, 4 on the case and 8 on the motherboard.

 

[iammykyl]

Thanks for the input terry1966.

I was configuring a Tower build, run as a server/NAS.

The budget is limited, so SAS drives of any decent capacity are out.

Bandwidth is a problem, so gone for a PCI-e to USB, will now have to look for one with more ports that each have their own controller.

I will look at a Skylake build but the Xeon is the same architecture as the i7, same graphics, just not overclockable.   Cost about $100 less. 

 

Gday Yisroel.

With less HDDs, will look for a different MB.

Please walk us through  exactly what you would do when you have 10 source cards connected.

 

 

10-15 USB flash drives, and some SD cards etc.

 

Where I would plug them in? I

That would stress any system and probable run slow.

The case has one front card reader.

Front USB ports can be used with thistype of card reader, > http://www.amazon.co...C_UL70_SR70,50_

One or more  PCI-e to USB adaptor,  4 port.

 

[Yisroel]

Thanks both for your responses. I don't really know if I need something special. I just know that my current PC (With pretty good specs. And i7 with some 16 gigs of RAM) under that kind of load tends to freeze or take very long.

 

Please walk us through  exactly what you would do when you have 10 source cards connected. 

Of course not every time it'll be so much. But in an extreme situation I would - let's say - have 5 copy paste operations, going from one drive/card to another. And a data recovery software running on one device.

[terry1966]

the thing that makes a big difference in your case is the speeds of the drives and the connection speeds, also the size of the files your copying.

 

so for example if the drive you have the file being copied from is only capably of 100MB/s read speeds, and the 5 drives your pasting the file to are capable of 50MB/s write speeds and the connections used to connect those devices are capable of transferring the data at 50MB/s, 5x50 = 250MB/s so to have the transfer not cause a freeze/slow down the original drive where the copied file is located would really need to have read speeds of over 250MB/s.

 

so that's just for the copying of your file to multi devices, then you also want data recovery software running on a device, which itself will depend on what drive your reading and writing to, if your writing to the drive that has the original file on then this is also going to slow things down massively because besides trying to supply 2 and 1/2 times more data than it is capable of reading, it now also has to find time to write data to that drive.

 

your problems aren't with your current pc/cpu or ram amount in my opinion, your problems and freezes stem from not fully understanding the bandwidths and speeds of your drives and connected devices, and what the pc is actually doing, so for example you may find all your problems are solved just by putting in a very fast ssd drive to hold the original files that need to be copied.

 

i believe the best solution for you is to sit down and try and work out what data bandwidth/speed your likely to need and then try and build a system capable of supply it, usually in any pc the thing that most slows it down is not the cpu or ram but the hard drives.

 

the fastest drives today are m2 or pci-e ssd's which i believe can read data around the 1400MB/s mark, so to be safe say you worked out you had 3xusb2 devices and 2xusb3 devices connected for a total of five devices requiring a max of 400MB/s write speeds then you'd find you'd have no slowdowns or freezes using the m2 drive as the file source drive.

 

  

Edited by terry1966, 18 October 2015 - 10:59 AM.

[iammykyl]

Although this is a test on a Mac, transfer from source to external HDD/SSD, it will still hold good for PC.

"While you won’t see a tenfold increase in the transfer speed from USB 2.0 to USB 3.0 in real-world use, USB 3.0 is fast—about three times faster than USB 2.0 with a spinning hard drive, and three to five times faster again with SSD. And using a hub doesn’t impact speeds, even with other (and slower) peripherals attached. USB 3.0" > http://www.macworld....-0-really-.html  So terry sis right that your hard drive is a bottleneck on your existing sytem, but I think there are others and we should try and find them, both hardware and the method of copying.

 

What is the brand/model of your MB and CPU?

Looking at just SD cards, what Class (speed) cards are you using? > http://www.yugatech....their-meanings/

Where are you plugging in the source SD card?   A slot on the case? A card reader? what brand/model?

Where are you plugging in the destination SD card/s?   One at a time or 2//3/4/5? aty a time.

[terry1966]

the easiest way to solve a freeze/slowdown problem when using the pc in your case is probably to use a dedicated drive for the os only where no files are stored that need to be read or written to when your copying, backing up etc.,

 

then the only problem you should encounter is things might take a longer time than necessary if you haven't maximised the potential/ balanced the needs and speeds of the other connected drives and connections for the operations your carrying out.

 

i7 cpu's and ram usually have somewhere between 20 and 30+ GBs data transfer speeds/bandwidth available to them depending on model, so that's a lot of data bandwidth available to be shared around to your devices before your current pc is a problem in my opinion.

 

as iammykwl says though the more info you can supply then the better he or i, can advise you on either best setup of current pc, maybe by just adding an ssd drive or 2 will work out just fine, or if you want a complete new build, maybe a new skylake i7 build and a new motherboard with it's faster connections may well be your best solution, especially with an os drive bandwidth capability of upto 32GBs for an m2 connection for example.

 

really it's just a case of doing a bit of math, knowing a drives abilities and what they are being asked to do, just reading or writing data or doing both at the same time. what connection speeds are being used, usb2, usb3, sata3, sata 6, etc. etc.

 

[Yisroel]

Sorry for my delayed response. Thanks both for your very informative input you helped me alot.
 
In conclusion I see that I was mistaken by thinking that there is something specific that we could pinpoint and say, "this or this is responsible for the operation of reading data from all various devices and copy it." But rather just a combination of everything being good (which is not always in my hands, some devices are from workers or colleagues.)
 
So what I will probably do (I'm not buying the new one just because the old one is slow but because we're moving) is just go with a prebuilt system with top notch specs (i7 skylake, 16 GB of RAM, a SSD for OS and programs, and enough USB 3 ports), and maybe add some expansion i/o for more direct connections without the need for hubs.

Just a few questions:

-I see iammykyl was advising for a Xeon CPU instead of i7, I'm not familiar with the exact differences, but is the Xeon better for me then the i7?

-terry1966, what do you mean by "especially with an os drive bandwidth capability of upto 32GBs for an m2 connection for example"

-Which is a brand for a good card reader (that I actually already saw in the parts list that iammykyl made) and USB hub?