USB Mass Storage Drivers for Firmware Update

[Black Hole]

Is there anything "we" can discover about the mass storage drivers available to read the .hdf file pre-boot? Perhaps we can work out what sticks/drives are supported and which are not.

 

[bottletop]

I've not had any issues with USB drives for the HDR, but then I've only used bog standard ones for it.
I suspect the HDR prefers to use the first FAT32 partition to look for the firmware (.hdr) file.
I guess some USB drives that present themselves as multiple partitions (eg ones with CD/DVD partition, security partition, semi hidden partition) may confuse it.

 

[Black Hole]

What do you mean "bog standard"? If all typical UPDs are bog standard, why does my Win7 PC typically have to go through an "installing software" phase before presenting a plugged-in UPD in the file manager?

 

[bottletop]

What do you mean "bog standard"?

Eg a USB drive that has one partition of type FAT32, using MBR structure.

... If all typical UPDs are bog standard, why does my Win7 PC typically have to go through an "installing software" phase before presenting a plugged-in UPD in the file manager?

I fail to see what that has to do with a Humax HDR - it is not a PC or use Windows. (That is just the way Windows works.)

 

[Black Hole]

I fail to see what that has to do with a Humax HDR

I postulate that the reason some UPDs work for firmware updating and others don't, is because there is limited driver support in the mini-boot firmware update environment. I don't believe all the problems which have been reported can be put down to format. The hypothesis has never been countered.

That is just the way Windows works

What – if all UPDs can be accessed with a single generic driver, it still needs to load a new driver every time? Really?

 

[bottletop]

What – if all UPDs can be accessed with a single generic driver, it still needs to load a new driver every time? Really?

Yes! There are only, maybe, 3 main generic mass storage drivers for Windows. The USB3 one supersedes the USB2 one which supersedes the USB1.

(That is just the way Windows works.)

To elaborate on that, the moving icon you see when Windows reports it is installing the driver for a USB storage device is just masking the way Windows works. It's probably not downloading/installing another mass storage driver! (Unless it requires a newer one.)
Eg in WIndows 7, place a new USB drive into a USB port, then watch it install the driver. Now disconnect from the internet and safely remove the drive and insert it into the next free USB port. Repeat this until you run out of USB ports. So how many drivers did it install? It will usually report once for the first time a port sees a new drive. It is just the way Windows works - it's probably just enumerating the drive to port details etc.

 

[Black Hole]

It takes an awfully long time to do that.

 

[prpr]

It takes an awfully long time to do that.

This is one of the reasons some people hate Window$.
You want to see how much crud it accumulates in "Device Manager" when set to show hidden devices (and you need an extra environment variable set to expose the full horror, before starting it).

 

[/df]

I think Windows "Installing software" at this point means "configuring new device with an existing driver" as @bottletop suggests. Once upon a time the required driver could not be relied upon to be available in the system.

The original question raises the further question of where to find the low-level bootstrap code. Would it be enough to disassemble some part of one of the boot-loader updates, or is there some IO code that doesn't get updated?

 

[Black Hole]

As doubts have been raised, it is my intention to survey the UPDs I have at my disposal – to find out the truth of the matter. I usually confine myself to a known reliable one. Nonetheless, any insights you can provide will be most welcome.

 

[Black Hole]

Results so far: easy wins with 2GB, 4GB, and 8GB UPDs (I'm using a loader update .hdf for rapidity).

However, a 64GB "integral" UPD is proving problematic. It was identified (in Linux Mint) as file system type "fuse" (which I take to mean the Linux NTFS driver), so I used Mint to reformat it FAT32, but when plugged into the HDR-FOX it gets stuck on "SYSTEM START". I am currently trying to format it using Windows (which of course doesn't offer FAT32 as an option) command line format /FS:FAT32 J:, but it's taking a long time.

 

[prpr]

I am currently trying to format it using Windows (which of course doesn't offer FAT32 as an option) command line format /FS:FAT32 J:, but it's taking a long time.

Why didn't you use /Q ?

 

[Black Hole]

Why didn't you use /Q ?

I wanted to be sure any inconsistencies were completely flushed. 87%...

 

[MikeSh]

Why didn't you use /Q ?

Bond or Star Trek variant?

 

[Black Hole]

This is odd. It's finished saying the volume is too big for FAT32, and the file manager won't mount it. That's way below the theoretical limit for FAT32 (at least 2TiB). Windows happily formats it exFAT. I'll try once more (with /Q!).

...and this time it came back with "too big" immediately.

 

[bottletop]

This is odd. It's finished saying the volume is too big for FAT32, and the file manager won't mount it. That's way below the theoretical limit for FAT32 (at least 2TiB). Windows happily formats it exFAT. I'll try once more (with /Q!).

...and this time it came back with "too big" immediately.

That's because there is an artificial partition size limit of maybe 32GB imposed by (some versions?) MS Windows.
Try either

• creating 2 partitions of the USB drive, 32GB + 32GB, then format the two partitions as FAT32
• or a FAT32 formatting utility that will allow up FAT32 partitions greater than 32GB

 

[Black Hole]

The "HP USB Disk Storage Format Tool, V2.1.8" (which has rescued uncooperative UPDs in the past) ran a quick format to FAT32 in the blink of an eye, no problem. However, the HDR-FOX still would not recognise it for firmware update (although this time it did not cause a hang).

But it seems the 64GB UPD can be accessed post-boot, and list media files on it.

So here is a possible differentiation: there's something different about UPDs larger than 32GiB which requires special handling.

or a FAT32 formatting utility that will allow up FAT32 partitions greater than 32GB

See above.

creating 2 partitions of the USB drive, 32GB + 32GB, then format the two partitions as FAT32

I'll give that a go another day.

 

[bottletop]

@BH, next time you're in Linux Mint, try dmesg and sudo fdisk -l /dev/sdX (change X to suit) to verify/check the troublesome flash drive.

I tried some tests

• Before testing - I did HDR FW update to official FHTCP 1.03.13 to ensure it uses standard kernel and drivers
• To verify results, I I did HDR FW downgrade to official FHTCP 1.02.20 to ensure it uses standard kernel and drivers
• Used loader bin file to mimic @BH test
• Either Partition Type Id (hex) 0B or 0C works fine
• Setting the bootable flag makes no difference
• In general it works when the HDR FW named hdr_fox_t2_upgrade.hdf (not case sensitive) file is in the first primary partition and it should be FAT32
• To set up partitions and check results I used Linux Mint 20.1 Cinnamon (terminal CLI and GParted for GUI partitioning duties)
• Tried using a single EXT2 partiton but it fails - HDR just hangs on initial loader splash screen
• Given that 64GB USB flash drives weren't readily available circa 2010 when the HDR came out - I think the range of suitable drives is very reasonable
• It is possible to use portable hard drive - but it needs to be powered up before the HDR startup (ie tricky unless you have a self powered drive)
• It is possible to use portable SSD - but timing may be tricky (unless you power up the drive first)
• I did most of my tests using the front USB port, although I assume the rear port will work just as well
• post #25 https://hummy.tv/forum/threads/usb-mass-storage-drivers-for-firmware-update.10259/post-156170 suggests the rear port takes priority.

Spoiler: Works - Sandisk 64GB USB drive with single FAT32 partition

#dmesg
[ 378.770138] usb 2-1.2: new high-speed USB device number 6 using ehci-pci
[ 378.880209] usb 2-1.2: New USB device found, idVendor=0781, idProduct=****, bcdDevice= 0.10
[ 378.880216] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 378.880220] usb 2-1.2: Product: ********
[ 378.880223] usb 2-1.2: Manufacturer: SanDisk
[ 378.880227] usb 2-1.2: SerialNumber: *********************
[ 378.880791] usb-storage 2-1.2:1.0: USB Mass Storage device detected
[ 378.881115] scsi host6: usb-storage 2-1.2:1.0
[ 379.907541] scsi 6:0:0:0: Direct-Access SanDisk ******* 0001 PQ: 0 ANSI: 6
[ 379.908271] sd 6:0:0:0: Attached scsi generic sg3 type 0
[ 379.908984] sd 6:0:0:0: [sdc] 122544516 512-byte logical blocks: (62.7 GB/58.4 GiB)
[ 379.910026] sd 6:0:0:0: [sdc] Write Protect is off
[ 379.910034] sd 6:0:0:0: [sdc] Mode Sense: 53 00 00 08
[ 379.911105] sd 6:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 379.947459] sdc: sdc1
[ 379.951826] sd 6:0:0:0: [sdc] Attached SCSI removable disk

#sudo fdisk -l /dev/sdc
Disk /dev/sdc: 58.44 GiB, 62742792192 bytes, 122544516 sectors
Disk model: ********
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start       End   Sectors  Size Id Type
/dev/sdc1          32 122544515 122544484 58.4G  c W95 FAT32 (LBA)

Spoiler: Works - Integral 64GB USB drive with two FAT32 partitions

#dmesg
[ 39.924475] usb 2-1.1: new high-speed USB device number 5 using ehci-pci
[ 40.031953] usb 2-1.1: New USB device found, idVendor=26bd, idProduct=****, bcdDevice= 1.10
[ 40.031960] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 40.031964] usb 2-1.1: Product: USB DISK 3.0
[ 40.031968] usb 2-1.1: Manufacturer:
[ 40.031971] usb 2-1.1: SerialNumber: *********
[ 40.032534] usb-storage 2-1.1:1.0: USB Mass Storage device detected
[ 40.032951] scsi host6: usb-storage 2-1.1:1.0
[ 41.016887] scsi 6:0:0:0: Direct-Access USB DISK 3.0 PMAP PQ: 0 ANSI: 6
[ 41.017538] sd 6:0:0:0: Attached scsi generic sg3 type 0
[ 41.018384] sd 6:0:0:0: [sdc] 120933888 512-byte logical blocks: (61.9 GB/57.7 GiB)
[ 41.019121] sd 6:0:0:0: [sdc] Write Protect is off
[ 41.019132] sd 6:0:0:0: [sdc] Mode Sense: 2b 00 00 08
[ 41.019858] sd 6:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 42.329381] sdc: sdc1 sdc2
[ 42.333603] sd 6:0:0:0: [sdc] Attached SCSI removable disk
[ 42.675956] FAT-fs (sdc1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 42.693602] FAT-fs (sdc2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

#sudo fdisk -l /dev/sdc
Disk /dev/sdc: 57.68 GiB, 61918150656 bytes, 120933888 sectors
Disk model: USB DISK 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot    Start       End  Sectors  Size Id Type
/dev/sdc1  *        2048  67110911 67108864   32G  c W95 FAT32 (LBA)
/dev/sdc2       67110912 120933887 53822976 25.7G  c W95 FAT32 (LBA)

Spoiler: Works - Transcend Jetflash 512MB USB drive with single FAT32 partition

#dmesg
[ 227.580439] usb 2-1.2: new high-speed USB device number 6 using ehci-pci
[ 227.785992] usb 2-1.2: New USB device found, idVendor=0c76, idProduct=****, bcdDevice= 1.00
[ 227.785999] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 227.786004] usb 2-1.2: Product: TS512MJF2A
[ 227.786008] usb 2-1.2: Manufacturer: JetFlash
[ 227.786011] usb 2-1.2: SerialNumber: *************
[ 227.797851] usb-storage 2-1.2:1.0: USB Mass Storage device detected
[ 227.798260] scsi host7: usb-storage 2-1.2:1.0
[ 228.828694] scsi 7:0:0:0: Direct-Access JetFlash TS512MJF2A 1.00 PQ: 0 ANSI: 2
[ 228.829451] sd 7:0:0:0: Attached scsi generic sg3 type 0
[ 228.836929] sd 7:0:0:0: [sdc] 1024000 512-byte logical blocks: (524 MB/500 MiB)
[ 228.838162] sd 7:0:0:0: [sdc] Write Protect is off
[ 228.838169] sd 7:0:0:0: [sdc] Mode Sense: 37 00 00 08
[ 228.839357] sd 7:0:0:0: [sdc] No Caching mode page found
[ 228.839366] sd 7:0:0:0: [sdc] Assuming drive cache: write through
[ 228.867997] sdc: sdc1
[ 228.879875] sd 7:0:0:0: [sdc] Attached SCSI removable disk

#sudo fdisk -l /dev/sdc
Disk /dev/sdc: 500 MiB, 524288000 bytes, 1024000 sectors
Disk model: TS512MJF2A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start     End Sectors  Size Id Type
/dev/sdc1  *       32 1023999 1023968  500M  b W95 FAT32

Spoiler: Works - misc

64MB SD card attached via budget USB card reader works - single FAT16 partition filling up the card
16GB micro SD card attached via budget USB card reader works - single FAT32 partition filling up the card
128GB micro SD card attached via budget USB card reader works - single FAT32 partition filling up the card
2.5 inch 512GB USB 3.0 portable hard drive that is self powered (Buffalo HDW-PU3) via USB cable

Spoiler: May work - misc

2.5 inch 128GB SSD in USB 3.0 enclosure can work, but it may depend upon timing - sometimes it requires a few tries. Alternatively power it up first before the HDR.

Spoiler: Foolproof method of using 1TB 2.5inch drive to perform firmware update on HDR

• Place HDR onto standby.
• You need to power the drive before the HDR starts (or taken off standby).
• So, if it a 2.5inch laptop drive, use this type of USB 'Y' cable - www.amazon.com/Converter-Adapter-Compatible-Seagate-Kingston/dp/B07QPYDW6T
• Power up the drive by connecting the offshoot USB plug (i.e. furthest from SATA connector, it's only for power) to a USB power point (i.e. USB charge plug or medium size power bank)
• Connect the main USB plug (i.e. the middle USB plug for data or data&power) to the HDR.
• Switch on the HDR.

Spoiler: Not working - misc

2.5 inch 512GB USB 2.0 portable hard drive doesn't work for FW update, but works fine for storage after HDR boots up normally
3.5 inch 1TB desktop drive Seagate Expansion - doesn't work despite it has a separate power supply. This is due to the drive powering down when the usb data cable disconnects and takes a while to start up.

Summary - for HDR firmware flashing:
I've successfully managed to perform the firmware update using various media and sizes.
It worked when I tested it with various USB devices from 64MB(FAT16) to 1TB(FAT32)
It should work for 2TB FAT32 partitions as mass storage driver was introduced in Kernel 2.4.x and revised in Kernel 2.6.x (HDR uses 2.6.18-7.1)
The usable media/device include SD card, micro SD card, USB flash drive, USB hard drive.
For best results, use simple USB flash drives.
Devices that have these may give trouble - encrypted drives , encrypted or secure zones . password protection zones, embedded optical drive emulation, etc.
Place the upgrade file hdr_fox_t2_upgrade.hdf (not case sensitive) into the root of the drive - ie not in sub directory.
So, anyone struggling to get their drive to work, should consider

• how they're creating the drive/partition - use the standard OS method or switch to an app/utility to do it
• if their drive has hardware or filesystem errors or the drive takes longer than,roughly, 8-9 seconds to be recognised by the OS
• the file name and location - hdr_fox_t2_upgrade.hdf (not case sensitive) is in the root of the drive - ie not in sub directory
• just use another drive
• it'll be easier to use a simple flash drive or (micro) SD card with a card reader

Edits:
Added details of 512MB drive
Added details of 16GB Micro SD card with adapter
Added details of 512GB 2.5 inch USB 2.0 portable hard drive
Added details of 128GB 2.5 inch SSD in USB 3.0 enclosure
Added details of 128GB Micro SD card with adapter
Added summary
Revised details for SSD in USB 3.0 enclosure
Added details of 512GB 2.5 inch USB 2.0 self powered portable hard drive
Added Foolproof method of using 1TB 2.5 inch drive to perform firmware update on HDR
Revised summary
Added details of 64MB SD (FAT16) card with adapter

 

[MartinLiddle]

I have a suspicion that some of the reported failures are due to people using large capacity drives that have been formatted with GPT style partitions which certainly won't work.

 

[bottletop]

I have a suspicion that some of the reported failures are due to people using large capacity drives that have been formatted with GPT style partitions which certainly won't work.

I think so too, but there may be some other possibilities like

• filesystem errors delaying or preventing filesystem mount - so HDR can't access the file
• slow or delayed hardware device - not reporting back to HDR within a certain time frame
• hardware error like bad blocks
• some people use to have trouble before Windows defaulted to producing GPT style USB drives
• encrypted drives or drives that have extra partition(s) to emulate optical drive (or security zones)

But, in general, it is just easier to tell the affected person to simply reformat or use another (simple) USB drive.