• The forum software that supports hummy.tv will be upgraded to XenForo 2.3 on Wednesday the 20th of November 2024 starting at 7pm

    There will be some periods where the forum is unavailable, please bear with us. More details can be found in the upgrade thread.

Beta NTFS-3g

Status
Not open for further replies.
X

xyz321

Well-Known Member
A update to the ntfs-3g package (version 2013.1.13-3) should be on its way to the beta repository. This should enable gpt partitioned USB drives to be mounted. It now uses virtual-disk2 as a dependency

I have not been able to test it with a large drive (>2TB) so it would be useful if someone could try this out.
 
xyz321 said:
A update to the ntfs-3g package (version 2013.1.13-3) should be on its way to the beta repository. This should enable gpt partitioned USB drives to be mounted. It now uses virtual-disk2 as a dependency

I have not been able to test it with a large drive (>2TB) so it would be useful if someone could try this out.
Click to expand...

I am going to install this today and test a little later. I have GPT NTFS 4TB drive hooked up to my HDR Fox T2 so ideal for test purposes, might even hook a powered hub up and see if it'll mount two drives later (I'm at work so all this is being tested remotely for now, thanks TeamViewer)
 
Right I have the beta of NTFS-3G installed and running. Finds the 4TB drive and mounts it as "gpt-drive1", tested shares over SMB only as I haven't configured NFS correctly yet and it does stream properly. Seems to solve the intermittant crashing issue I had with the previous version NTFS-3G manually mounting GPT NTFS shares... so far, as the box has only been up an hour or two since installing and rebooting. If it's going to be used as a media server it would need to be on all the time so I can keep an eye on it remotely using the webif. Might help to get more verbose logs to feed back to you? Let me know.

This is excellent work from my early testing! If it's stable enough it will save me the fairly tedious task of transferring 9TB of content from two NTFS drives to EXT3, i.e. buying new drives, prepping and copying. I have a 5TB drive in a different enclosure I can test as well, and then hook both up to the rear USB port using a powered hub to test that whole 9TB so I can give you some more notes as I trial a few things.

Side note, what surprises me is these shares seem to start faster on a HDR Fox T2 sharing via SMB over a 100Mb/s ethernet port than from a HTPC where one drive was hooked up by eSATA, another by USB2 and shared via NFS over a 1Gb/s ethernet port. On a Freeview box that was never made to do this by the manufacturer which cost me £70 from eBay. Pretty impressive hardware.

Only thing I'd query so far is we're seeing a few empty directories mounted, so there's "drive1" and "gpt-drive2". I'd guess NTFS-3G is still mounting the msftr Microsoft Reserved partition as "gpt-drive2", not sure what the other one is for.
 
zekepliskin said:
Only thing I'd query so far is we're seeing a few empty directories mounted, so there's "drive1" and "gpt-drive2". I'd guess NTFS-3G is still mounting the msftr Microsoft Reserved partition as "gpt-drive2", not sure what the other one is for.
Click to expand...
Glad to hear that it seems to be working.

I suspect that 'drive1' is the mount point for the virtual USB drive. The additional gpt-drive directory may well be the reserved partition but it is very unlikely to be mounted. The disk I used for testing did not have a Microsoft reseerved partition.

If you could connect to the box via telnet and cut and paste the output of the 'mount' command it should become clearer

PS. The log file 'rag.log' provides useful information about how the partitions are being detected and mounted..
 
prpr said:
It's the work that's gone in to the software that is impressive, not the hardware.
Click to expand...

I agree, the adding of Linux utils is amazingly well done. Even the telnet interface is friendly to use. Great software, utilising the potential of the impressive hardware. It was never meant to be a media server and yet it's so good at it with the Custom Firmware. This NTFS-3G beta is further proof of that.

Anyway, let's paste some console and log output shall we? I've now hooked up my 5TB Seagate drive to test. Little temperamental - same enclosure used, same USB port (rear one under the ethernet port) however the HDR Fox T2 was having problems keeping it mounted, or the gpt-drive number would change. Turns out this drive has very aggressive (and noisy, ironically!) power management built in, causing it to dismount requiring a reboot to get it going again. We're fortunate to have busybox so I could simply do this :-

Code: 
hdparm -B 255 -S 0 /dev/sda

And now it will stay mounted (mostly; again this looks like a problem with the drive here as the 4TB HGST brand one doesn't do it), happily SMB sharing content to Kodi on Fire TV or playing old Foxsat recorded episodes on the Fox T2 itself. Talking of which, here's the current output of that command :-

Code: 
rootfs on / type rootfs (rw)

/dev/root on / type squashfs (ro)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

usbfs on /proc/bus/usb type usbfs (rw)

devpts on /dev/pts type devpts (rw)

tmpfs on /tmp type tmpfs (rw)

tmpfs on /media type tmpfs (rw)

/dev/mtdblock1 on /var/lib/humaxtv type jffs2 (rw)

/dev/mtdblock2 on /var/lib/humaxtv_backup type jffs2 (rw)

/dev/mtdblock2 on /etc/opkg type jffs2 (rw)

/dev/sdb1 on /media/drive1 type ext2 (rw)

/dev/sdc1 on /mnt/hd1 type ext3 (rw,data=ordered)

/dev/sdc2 on /mnt/hd2 type ext3 (rw,data=ordered)

/dev/sdc3 on /mnt/hd3 type ext3 (rw,data=ordered)

/dev/sdc2 on /opt/share/images/blue type ext3 (rw,data=ordered)

/dev/sdc2 on /mnt/hd2/My\040Video/[ModSettings] type ext3 (rw,data=ordered)

/dev/sdb1 on /mnt/hd2/My\040Video/usb-drive1 type ext2 (rw)

/dev/sdc2 on /media/drive1 type ext3 (rw,data=ordered)

[B]/dev/sda2 on /media/gpt-drive2 type fuseblk (rw,nosuid,nodev,user_id=0,group_id=0,allow_other)[/B]

The rag.log is definitely a job for pastebin... here's the full version https://pastebin.com/1Wh17Dhg
 
Last edited:
As suspected, 'drive1' is the virtual disk and 'gpt-drive1' is the Microsoft reserved partition (not mounted). Eventually it might be good to have the CF remove the unwanted directory. A quicker solution might be to create a new package which detects the Microsoft reserved partition and deletes the orphaned directory.

The usb-drive1 mount under 'My Video' has probably been created by the mvdisks package and is a link to the virual disk lwhich is not very useful). I will look into updating mvdisks over the coming weekend.
 
Thanks for the update xyz, nice to see people still developing for this surprisingly powerful box.

What I wouldn't mind doing is getting NTFS drives to mount by UUID to consistent mount points, for the purposes of adding things to library in Kodi etc. And if there are two drives mounted so each gets a distinct name, not a variable gpt-driveX assignment. Is there an easy way to do this? I'm happy to attempt it by trial and error on the command line as I'm no Linux expert by any means but some way to test it and then automate it at boot time would be very useful, as well as setting correct power management settings for all drives (in my case all off, drives constantly spinning).
 
I know that with Seagate drives you can download software from the manufacturer's website that allows you to change the standby time. Doing a quick search suggests that this function is available within the Seagate Dashboard software.
 
Hmmm. Well after several hours of messing around learning ways to do this I'm kinda stuck. I now have a script I can telnet into the box and use to set the NTFS drive to make the mount point directory, unmount the drive and mount it to where I want while also disabling the power management to stop it dismounting, however I can't get this script to run at boot time via crontab so it's done automatically. I'm using :-

Code: 
@reboot sleep 60 && /mod/ntfs.sh

The sleep line was inserted as I thought it would then allow for ntfs-3g mounting to a gpt-driveX number share after boot then putting the drive back where I want it, but with or without it the script won't run, still have to manually telnet in and run it. Using crontab would be useful so the script can automatically be run again at 5am after the OTA update disabler as well, for example.

I wanted to try and mount by UUID by entering the drive details into the /etc/fstab as well, however as that's squashfs read only - I'm guessing on the box's flash memory - that's not possible based on my limited understating.

Any tips here would be appreciated because it's driving me to distraction! :)
 
I quite like the idea of renaming the mount point to match either the volume label or the UUID mapped to a user friendly name. Whiilst the implemetation should be reasonably straight forward for GPT drives and /or NTFS drives, there is an iissue with exte2/3 partitions on drives using MBR. The Humax app displays the volume label on-screen for these latter formats. If the partition is unmounted and then remounted with a new mount point name the disk space indication no longer works. Some would say that it is already broken and I would have to agree with them.

Here is a patch against the beta version which renames the mount point to match the volume label. It is still work in progress, there could be some situations which break it. It could easily be modified to use the UUID instead of th evolume label but that would require a map file or database to map the UUID to user friendly names. The WebIF would then have to be modified to allow modification of the mapping file. This might be easier than changing the volume label which requires unmounting the partition first.

On the Humax /etc/fstab only contains flash partitions and temporary RAM based filesystems. The internal hard disk and USB drives are mounted via hotplug events. When a drive is iinserted or removed (or at boot up when all drives appear) the kernel triggers a hotplug event for each partition which then runs /etc/mdev/run-and-gun. This in turn runs the scripts in /mod/etc/mdev/, The combined output from these produces the rag.log file. At boot up many of these events can occur close together so you will get multiple executions of the same script at the same time. Some will be waiting for drives to become ready etc. This is why rag.log.contains a mixture of entries from different instances of the same script.

If one of the above schemes is to be implemented for ext2/3, then the changes should be made in the CF (i.e. to the run-and-gun script) since that is where ext2./3 mounts are handled. Similarly any changes to remove directories associated with failed mounts (e.g. Microsoft reserved partions) should also occur in that script IMO.

I was under the impression that the drive idle time could be saved to its internal flash memory, perhaps this is not the case.

Edit: Patch deleted - it has been superseded by a new version in the beta repository
 
Last edited:
Updates to ntfs-3g and mvdisks are on their way to the beta repository.

The main changes to ntfs-3g are a replacement for the patch above which fixes some failed mounts and now supports spaces in volume names.

mvdisks has been rewritten in order to support the new scheme.
 
Black Hole said:
This tells you how to run a script at boot time:

https://hummy.tv/forum/threads/symbolic-link.3926/#post-47862
Click to expand...

This was very useful, thank you. Mounting using a SXXexecutablescriptname in the /mod/etc/init.d folder was one way to get a USB drive mounted at boot time with a consistent path based on the mount point being the same, i.e. disc is sda. Might change if I have two NTFS drives connected and they don't always get seen by the Humax in the same order, but a good stopgap for now.

xyz321 said:
Updates to ntfs-3g and mvdisks are on their way to the beta repository.

The main changes to ntfs-3g are a replacement for the patch above which fixes some failed mounts and now supports spaces in volume names.

mvdisks has been rewritten in order to support the new scheme.
Click to expand...

Ah. Are you saying that the patch in the (code) tags above will be implemented in the forthcoming betas, meaning it's just a case of waiting for the package updates rather than manually applying the script above?

xyz321 said:
I was under the impression that the drive idle time could be saved to its internal flash memory, perhaps this is not the case.
Click to expand...

Yes I was too, but apparently Seagate chooses not to do that, or let you modify and save hdparm type values in their Dashboard software if drives are taken out of one enclosure and put into another, for example. Why would I take a drive out of the enclosure it came in? Weird choices to do with the control unit in the Seagate provided enclosure. I remember looking at the 5TB when it was hooked up to a desktop PC by SATA-II after I got it and it was partitioned very strangely, to the point where the test data I wrote on it wasn't accessible. So if the control unit got fried you couldn't just hook the drive up to another enclosure or internally to a desktop. Dumb. If I put that 5TB drive back in the enclosure it came in it would format it and use it's own weird partitioning scheme without warning. I know because I dropped a spare 2TB Hitachi in the Seagate USB enclosure and it did the very same thing. I know they're doing it to get around the problem older systems have with MBR/GPT - the very reason you're doing these beta updates to ntfs-3g on the Fox T2 here - but there are better ways. Do it in Windows/Linux/OS X software, not on the controller in hardware.

The spin down on my 5TB drive is ridiculous, literally a matter of seconds idle and it goes into standby, also dumb. The noise when it spins the platters back up is horrendous, it actually sounds like the heads are colliding with them. It isn't otherwise the drive would have been toast ages ago but because it sounds so hard on the mechanism I'm careful to disable it. I'd rather set a more reasonable 15 minute or so standby (it's a noisy drive too), however I've noticed even on a Windows PC some types of enclosure cause the drive to disappear from being mounted after going into standby. Disabling it is just easier. Definitely prefer HGST or WD drives now, far less grief and do run quieter, especially any of the WD AV-GP drives with the EURX in the model number (specifically made for DVR type uses). But anyway... enough rambling about Seagate.

xyz321 said:
Here is a patch against the beta version which renames the mount point to match the volume label. It is still work in progress, there could be some situations which break it. It could easily be modified to use the UUID instead of th evolume label but that would require a map file or database to map the UUID to user friendly names. The WebIF would then have to be modified to allow modification of the mapping file. This might be easier than changing the volume label which requires unmounting the partition first.
Click to expand...

I personally don't mind just seeing the UUID on the Storage menu, or manually coming up with some sort of "if UUID = this then change name to that" script with a little help. As long as the mount point is consistent it's not hard to tell from the directory structure of the drives which is which. As much as I've struggled to get my head around some of the rudiments of Linux and do simple things like this - well, simple when you know how - it's been an interesting learning curve.

---

EDIT: That reminds me, editing the /mod/etc/exports file was a MUCH easier way to configure NFS, the whole thing with the [ModSettings] folder was just confusing. Hand entering the mount points to share RW was much easier, and gets around the SMB server using an old protocol that won't mount on OS X (well it will via Kodi because it has SMB written into it - impressive work actually - but on the Finder it won't). NFS is better anyway... seems to buffer quicker because of the smaller overheads and Kodi also has that built in.
 
Last edited:
zekepliskin said:
Ah. Are you saying that the patch in the (code) tags above will be implemented in the forthcoming betas, meaning it's just a case of waiting for the package updates rather than manually applying the script above?
Click to expand...
The patch is now superseded by the new version which is now in the beta repository. I will delete it from the above message to avoid any further confusion.
 
xyz321 said:
The patch is now superseded by the new version which is now in the beta repository. I will delete it from the above message to avoid any further confusion.
Click to expand...

Got it. Installed and upgraded... works beautifully. It actually looks from the HDR Fox T2 like it was always meant to mount large NTFS drives by name because from the interface it's seamless! Quick edit of the /mod/etc/exports file and removal of my little script that runs from init.d at startup to add by name NFS shares and all's good thanks :)
 
Quick update on this since I've been using it for a month...

There do seem to be the odd few problems with streaming via NFS/SMB sharing using NTFS-3G mounted drives. I assumed it was when recording with detectads enabled because that's sometimes where the issues would start, as if the box was a little CPU overloaded but it seems to do it regardless, even after limiting detectads to only 40% CPU maximum usage.

Namely there's video corruption and it doesn't seem to buffer correctly when streaming to Kodi from either of the NTFS drives I had mounted, only after the HDR Fox T2 has been on for a while, requiring a reboot in order to be streaming smoothly again. Not sure what would cause it, happy to pull some log files if it'll help though.

I'm going to continue to test leaving the box on and streaming things across from only the My Video partition via SMB on the installed 4TB drive (i.e. downloaded files or programs the box itself has recorded; I disabled NFS a few days ago to try and eliminate that as the culprit but still seems to be an issue) but at this point I think it's related to mounting large NTFS drives.
 
I have been having some problems with ntfs-3g (2013.1.13-4) recently. I have two HDR-FOX units which are left with NTFS-formatted drives attached (MBR). With the above version of ntfs-3g, I keep finding that the drives are unmounted. When I rescan the drives (Webif eject button) they fail to mount and the designation changes from sda to sdc. If I uninstall ntfs-3g, the drives mount as read-only (Humax driver) without problems, and if I revert to version 2013.1.13-2, they mount with full read-write access using the CFW ntfs-3g driver. This makes me think that the problem relates to something in version 2013.1.13-4.
 
MontysEvilTwin said:
I have been having some problems with ntfs-3g (2013.1.13-4) recently. I have two HDR-FOX units which are left with NTFS-formatted drives attached (MBR). With the above version of ntfs-3g, I keep finding that the drives are unmounted. When I rescan the drives (Webif eject button) they fail to mount and the designation changes from sda to sdc. If I uninstall ntfs-3g, the drives mount as read-only (Humax driver) without problems, and if I revert to version 2013.1.13-2, they mount with full read-write access using the CFW ntfs-3g driver. This makes me think that the problem relates to something in version 2013.1.13-4.
Click to expand...

Yeah something was definitely b0rked... after many hours invested 18 months ago to try and keep NTFS drives under stable mountpoints, not spinning down etc and finding that when the box was on for 12 hours or more it wouldn't network share properly via NFS or SMB (video corruption) I gave up and just reconnected them to my HTPC and shared them that way as I had done before.
 
Status
Not open for further replies.
Back
Top