Problems copying to USB disk

In my experience writing to a USB Ext3 is no where near either the 62 GB/hr reported for FAT32 or the 37 GiB/hr I get using rsync/NFS share. When I get to the PC I'll add the actual USB Ext3 speed I get.
 
Black Hole said:
It does look like the custom extension for NTFS write is slugging the transfer rate.
Click to expand...
That would make sense since NTFS write support is implemented using a FUSE (Filesystem in User Space) module. There is additional overhead there.
 
alexp said:
I'm trialling a disk backup in anticipation of hard disk exchange for two aging HDR-Fox T2's.
First, is the fastest transfer time via USB2 rather than LAN, or otherwise? I'm using a Freecom toughdrive.

I installed "auto-unprotect" and "NTFS 3G". Only 4 recordings on my test machine, two are HD, ranging from 1GB to 3GB (30 to 100 minutes). The USB drive was partitioned into a FAT32 and a NTFS. I copied all files to the NTFS partition via the OPT+. The "copying..." message was still there this morning (9 hours later) so I powered it down. Two files copied OK, the third only partly copied, no information on screen. Hmmm? Any ideas?
Click to expand...
I've swapped the internal had drive of an HDR-FOX T2 for different drive. My approach is a bit different to yours. I've just plugged the old drive into one of the HDR-FOX T2's USB slots using a sub £10 USB SATA adapter. The contents are currently copying over at 62GB an hour directly to the target drive, (as auto-unprotect has not been used on the unencrypted HD recordings and everything else is decrypted).

What advantage is it giving you using your approach? I'm puzzled as your first post stated that what you were after is a drive exchange and didn't mention anything else..
 
johnb : I concur that ext3 copies to external USB drive are really slow. I seem to recall about 3MB/s, or ballpark 11GB/hr, having tried ext3 ages ago, and more recently NTFS. Pretty dismal really.

Black Hole : powerline networking has also been poor for me. After investigating, I found that this is not because of bandwidth but because of latency, so the TCP window limits the number of bytes in flight and cripples the bandwidth to maybe 4MB/s for me (well under half what the 100Mb net can do on a wired connection). I am soon going to be trying a "500Mb" powerline pair instead of my current "200Mb" units. (I originally just tweaked the kernel TCP/IP settings to make the TCP window bigger, but IIRC both the samba and FTP connections ignored the change.)

Regarding poor USB transfer speed, a while back I had suspected that the USB interface was just a bottleneck (hardware?). However, alexp's 62GB/hr results with FAT32 are really impressive (and better than Ethernet's best possible performance). Sadly FAT32 is a non-starter for large files, but exFAT would indeed be fun to try (I've never used it myself though).

After installing the CF, I was pleasantly surprised to find that the native internal disk speed is actually really good (in excess of 40MB/s for both read and write for me when I tested again just now - while watching a HD channel).

Also, given alexp's results (which means the USB really is capable of decent speed), I now wonder why USB ext3 is so awful - given that it's a kernel filesystem driver. Perhaps it's the journalling? If so, turning it off might be the best way forward, as I'd expect it to have no more OS overhead than FAT32... I can't easily test this myself right now as I recently reformatted the external ext3 drive I was using to NTFS, after installing the ntfs package on my Humax :)

Luke: fair question. However, I'd have done it the same way as the OP and my reasoning would have been "don't take it to bits until the stuff is backed up"...
 
neilski said:
Luke: fair question. However, I'd have done it the same way as the OP and my reasoning would have been "don't take it to bits until the stuff is backed up"...
Click to expand...
The original HDD will become my backup! The decrypted contents can be read from any HD-FOX/HDR-FOX T2/2000T (or a 1800T if I had one) or a computer I don't see it as risky even if there were recordings I value.
 
exFAT is also likely to be slow, because (like NTFS) the support for it has to be installed via custom firmware. To defeat journalling, try Ext2.
 
Luke said:
I've swapped the internal had drive of an HDR-FOX T2 for different drive. My approach is a bit different to yours. I've just plugged the old drive into one of the HDR-FOX T2's USB slots using a sub £10 USB SATA adapter. The contents are currently copying over at 62GB an hour directly to the target drive, (as auto-unprotect has not been used on the unencrypted HD recordings and everything else is decrypted).

What advantage is it giving you using your approach? I'm puzzled as your first post stated that what you were after is a drive exchange and didn't mention anything else..
Click to expand...

Hi Luke,

I presume you also have to provide power to the drive in addition to the USB-SATA adapter (which I don't have)?
I like the idea of fast direct copy without de-re-encrypting though.

I wanted to avoid getting extra hardware I would probably never use.

I don't need to exchange yet but want to be ready to do so. I've got a 2.5 year old HDR T2 from new, and have just got another two used T2's of unknown age (the disk diagnostics are fine at the moment). These were to replace a fiddly USB disk hanging off the back of a Samsung TV, and an ageing 9200T, which is too slow now even on 4 mux's.

With three T2's, my preferred approach is to use my LAN which extends from PC to TV area hard-wired (properly, with double RJ45 wall sockets). I can take everything off at 37 GB/hr to one of my PC backup desk USB disks (3TB or 1TB). If necessary I can restore to one of the other HDR's temporarily while I fix the broken one - programs will be re-encrypted to the new machine. This way the only thing I have to open is the broken T2 for a new disk, vacuum clean, work the connectors and look for things like bulgy or leaked capacitors...

...so now I just have to wait for dropouts and do a disk diag every few months.
 
Black Hole said:
What does this mean?
Click to expand...

Well, the way I understand it, programs are decrypted on the way off, and when copying back to a different machine they will be encrypted again but according to that machine, so its all portable.

Whereas, if, say, you tried to directly copy from the original disk but to a new disk in a different machine, ain't going to work (not that that was proposed above). A semi-broken disk taken out of the T2 is locked to that machine.
 
alexp said:
Well, the way I understand it, programs are decrypted on the way off, and when copying back to a different machine they will be encrypted again but according to that machine, so its all portable.
Click to expand...
They won't be encrypted again.
 
alexp said:
Well, the way I understand it, programs are decrypted on the way off, and when copying back to a different machine they will be encrypted again but according to that machine, so its all portable.
Click to expand...
You understand wrong then. Nothing is EVER re-encrypted.
 
The OP possibly doesn't realise that both encrypted and unencrypted recordings can co-exist on the internal HDD. There is a flag in the sidecar files which is critical to the process, which marks whether the .ts file needs to be decrypted on playback/streaming/USB-copy. When the file is decrypted in the process of copy-to-USB, the flag is cleared in the external copy of the sidecar files (specifically the .hmt).

The USB copy operation doesn't care whether it is to or from USB; if you (somehow) had an encrypted recording on USB and copied it into the HDR it would be decrypted in the process... but using a different decryption key (ie a different HDR from the one that encrypted the recording) scrambles the recording irreversibly. Of course, by this means it is possible to replace the HDD and mount the old one via USB, and still have access to the recordings on it. Equally, if the flag in the .hmt becomes out of sync with the actual state of the .ts file, chaos ensues.

When the .hmt shows that the recording has already been decrypted (ie it is unencrypted), the playback/streaming/copy operations leave it as it is. It is the original decryption on USB-copy that make the recordings portable.

Meanwhile, the auto-decryption options available using the custom firmware operate to remove the encryption on all recorded content, on the internal HDD.
 
I must say, the logic behind the whole encryption/decryption thing has always had me entirely baffled. Can't see the point in encrypting SD recordings, when the vanilla box just decrypts them if you ask it to copy to an external drive. (HD is another story of course.)

Aaanyway, I was intrigued about the whole transfer speed thing so I did some tests (exFAT, FAT32, ext2, ext3, NTFS). Got a couple of surprises along the way.

Executive summary: when copying TO a USB disk, the fs makes a big difference; when copying FROM it, basically they're all pretty nearly the same. That's when using the Linux command line to do the copying. The biggest surprise was that my memories of ext3+USB sucking really badly were accurate, but only when I used the Humax UI to make the copy. Basically, THAT sucks more badly than even NTFS from the command line. (Last time I tried ext3 copies, I hadn't installed the CF.)

For the tests, I used "dd" to copy between the My Video folder and the external filesystem on the USB drive (I used "dd" mostly because it prints out the I/O speed, but "cp" achieved the same speed), while the box was watching an SD channel; for each fs I did 2 runs of a 405MB TS file, and in all cases the results of both runs were fairly close.

Details of speeds achieved:
Copy to USB, MB/s:
ext2: 20.2
ext3: 17.9
FAT32: 14.1
exFAT: 9.4
NTFS: 4.6

Copy from USB, MB/s:
ext3: 14.8
ext2: 14.4
FAT32: 14.4
NTFS: 13.2
exFAT: 11.5

...and the Humax UI copied to the external ext3 filesystem at what speed? Would you believe, all of 3.5MB/s (!!). I wonder if they deliberately crippled it.

Given that I can't yet achieve full 100Mb/s Ethernet wire speed (of ~11 MB/s) because of my iffy powerline networking, I now have to choose which fs to use on my USB drive. NTFS is clearly an awful choice. Might go with exFAT (cos both linux and windows can read it, with some effort) but it's still not great. I haven't wanted to install ext2/3 drivers on windows but might have to revisit that. FAT32 would be fine if only it could handle big files...

I used "time" on the dd runs as well, to let me see how much user/system CPU time was consumed in each case. I expected that the FUSE cases might be worse than say ext3, but the trend wasn't very clear so I haven't bothered to include that data.

(Those megabytes are all binary btw; I just can't bring myself to use MiB/GiB yet. Maybe in another few years!)
 
neilski said:
I must say, the logic behind the whole encryption/decryption thing has always had me entirely baffled. Can't see the point in encrypting SD recordings, when the vanilla box just decrypts them if you ask it to copy to an external drive. (HD is another story of course.)
Click to expand...

You should be really happy Humax decided to encrypt all recordings on this box. As a result they needed to build in a process to allow decryption on copying to usb. Happily the decryption on copying turned out to be controlled by a single byte in the sidecar .hmt file, which allowed the box to decrypt HD as it does SD recordings. This allows previous recordings to be decrypted (even automatically and in situ).

Even more so as the same system was carried onto the later Freeview+ boxes.

The earlier Foxsat-HDR which also uses a very similar sidecar file arrangement records SD without encryption and HD without. As a result there is no known way of decrypting existing HD recordings (You can still copy to USB and play back from there, provided the original box that recorded it is playing back the content).

There are ways to record without encryption on HD recordings on this box even on a vanilla box (using the box as a generic fta box). There is also a patch available to the Humax settop software that turns off encryption for future recordings.

As a result old recordings can not be decrypted on this pvr.
 
grahamlthompson said:
As a result they needed to build in a process to allow decryption on copying to usb.
Click to expand...
Actually I guess that's part of what I don't understand.
Why would they want/need to decrypt when you copy to USB? If for some reason the stuff needs to be encrypted on the internal disk, why would it not need to be encrypted on an external disk?
(In that hypothetical world, someone would only be able to use the external disk to make backups of recordings for later playback on the same box if the internal disk died and had to be replaced...)
 
You must log in or register to reply here.
Back
Top