CFW 3.11 hard drive disappears

MontysEvilTwin

MontysEvilTwin

Well-Known Member
I am having a problem with CFW 3.11 on the HD-FOX. I occasionally run e2fsck to check the USB hard drive. There can be problems with lack of memory if the disk has many faults, but it mostly works fine. After upgrading to CFW 3.11, and running e2fsck the hard drive is no longer recognised. I think this is because the new version of e2fsck in CFW 3.11 has set the flag 'inline_data' on one or more inodes. I have downgraded to CFW 3.10 and tried to run the previous version of e2fsck (1.42.13) but it refuses to run, giving the following message:
Code: 
/dev/sda1 has unsupported feature(s): inline_data     
e2fsck: Get a newer version of e2fsck!
Is there a way I can make the older version ignore this and run anyway, clearing these 'inline_data' flags?
 
As far as I know, inline_data is an EXT4 flag and can't be set on an EXT3 filesystem. This is odd and needs more investigation.. 3.11 does have newer e2fsck since it's generally a good idea to keep the tools up-to-date to catch as many problems as possible.
 
Just tried tune2fs on one of my drives and it doesn't report that flag in the filesystem features..

Code: 
humax# tune2fs -l /dev/sda2
tune2fs 1.43.3 (04-Sep-2016)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          37d4a0d0-3016-4d68-bf30-94349068d0fe
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
 
You can try:
Code: 
debugfs -w /dev/sda1 -R 'features ^inline_data'
debugfs -w /dev/sda2 -R 'features ^inline_data'
debugfs -w /dev/sda3 -R 'features ^inline_data'

Edit: works here
 
Last edited:
af123 said:
You can try:
Code: 
debugfs -w /dev/sda1 -R 'features ^inline_data'
debugfs -w /dev/sda2 -R 'features ^inline_data'
debugfs -w /dev/sda3 -R 'features ^inline_data'

Edit: works here
Click to expand...
Code: 
humax# debugfs -w /dev/sda1 -R 'features ^inline_data'                    
debugfs 1.42.13 (17-May-2015)                                            
/dev/sda1: Filesystem has unsupported feature(s) while opening filesystem
features: Filesystem not open
I will reinstall CFW 3.11 and try again.
 
If the newer filesystem check tools are setting this flag (perhaps only in certain circumstances where there are small files that could reside within an inode) then I need to revert them!
I've run many filesystem checks on all my boxes with 3.11 and never seen this but MontysEvilTwin and The Greek have both seen a disk disappear following a check.
 
af123 said:
If the newer filesystem check tools are setting this flag (perhaps only in certain circumstances where there are small files that could reside within an inode) then I need to revert them!
I've run many filesystem checks on all my boxes with 3.11 and never seen this but MontysEvilTwin and The Greek have both seen a disk disappear following a check.
Click to expand...
After upgrading to CFW 3.11 I ran the following as per your instructions:
Code: 
debugfs -w /dev/sda1 -R 'features ^inline_data'
It ran correctly, and after a reboot the drive became available again. I reran e2fsck from maintenance mode again without the '-y' switch:
Code: 
HDFox# umount /dev/sda1                                                           
HDFox# e2fsck -f -C 0 /dev/sda1                                                   
e2fsck 1.43.3 (04-Sep-2016)                                                       
Pass 1: Checking inodes, blocks, and sizes                                         
Inode 16516 has inline data, but superblock is missing INLINE_DATA feature         
Clear<y>? no                                                                       
Inode 16516 has INLINE_DATA_FL flag on filesystem without inline data support.     
Clear<y>? no                                                                       
Inode 16545 has inline data, but superblock is missing INLINE_DATA feature         
Clear<y>? no                                                                       
Inode 16545 has INLINE_DATA_FL flag on filesystem without inline data support.     
Clear<y>? no                                                                       
Inode 16563 has inline data, but superblock is missing INLINE_DATA feature         
Clear<y>? no                                                                       
Inode 16563 has INLINE_DATA_FL flag on filesystem without inline data support.     
Clear<y>? no                                                                       
Inode 16659 has inline data, but superblock is missing INLINE_DATA feature         
Clear<y>? no                                                                       
Inode 16659 has INLINE_DATA_FL flag on filesystem without inline data support.     
Clear<y>? no
If you answer yes to one of the prompts the drive disappears again. Running the following confirmed that the 'inline_data' feature had been activated again.
Code: 
tune2fs -l /dev/sda1
It is the new version of e2fsck that is the culprit.
 
MontysEvilTwin said:
It is the new version of e2fsck that is the culprit.
Click to expand...
Right, thanks. I will withdraw CFW 3.11 and replace it with 3.12 and older e2fsck tools. This never came up in testing and shouldn't be happening - especially not with EXT3 as it's an EXT4 feature!

Your filesystem must have some tiny files on it (under 128 bytes) for them to fit within an inode. Are there just those 8 on sda1?
What about the others - sda2 & 3?
You can safely reformat sda1 if that's the only one but it would be interesting to see what they are.
Something like:
Code: 
humax# debugfs -R 'ncheck 16516' /dev/sda1
would find the file associated with the first inode in your log above.
 
Last edited:
af123 said:
Right, thanks. I will withdraw CFW 3.11 and replace it with 3.12 and older e2fsck tools. This never came up in testing and shouldn't be happening - especially not with EXT3 as it's an EXT4 feature!

Your filesystem must have some tiny files on it (under 128 bytes) for them to fit within an inode. Are there just those 8 on sda1?
What about the others - sda2 & 3?
You can safely reformat sda1 if that's the only one but it would be interesting to see what they are.
Something like:
Code: 
humax# debugfs -R 'ncheck 16516' /dev/sda1
would find the file associated with the first inode in your log above.
Click to expand...
There were more inodes indicated but I just pasted the first block. This is a HD-FOX so there is only one partition. I will run the command you suggested later tonight when I am back at home.
 
MontysEvilTwin said:
This is a HD-FOX so there is only one partition
Click to expand...
Ok. If you can find the inline files and remove them (or move them to another drive) then you should be able to unset the flag and be back in business. Otherwise I'm afraid you may have to archive the files off and reformat : (
Thanks for picking this up though - let's see what the files are and take it from there.
In the meantime, CFW 3.12 is available with e2fsprogs 1.42 which doesn't have the inline_data feature at all.
 
For what it's worth, I've spent some time trying to replicate this on one of my HDRs and I can't make e2fsck set the inline_data flag. I will be able to try it on my HD shortly.
I'd be interested in the full output of tune2fs -l
 
Running
Code: 
debugfs -w /dev/sda1 -R 'features ^inline_data'
unsets the file system attribute and makes the drive available again. Running e2fsck afterwards still flags up inodes with the 'inline_data' field set. If you say no to clearing the flags (running without the -y switch) the drive remains accessible, but if you allow it to clear a flag it is this which seems to cause 'inline_data' to be set as a file system attribute. I wonder if it is erroneously detecting inodes as flagged?
 
MontysEvilTwin said:
I wonder if it is erroneously detecting inodes as flagged?
Click to expand...
That's possible and then I suppose it would flag the whole filesystem.
I can't replicate it on either of my HD boxes I'm afraid.
Was this disk originally formatted on the Humax itself?

Use debugfs to track down one of the files which is flagged and we'll get a better idea.
 
Code: 
HDFox# tune2fs -l /dev/sda1                                                            
tune2fs 1.43.3 (04-Sep-2016)                                                          
Filesystem volume name:   Red Drive                                                    
Last mounted on:          <not available>                                              
Filesystem UUID:          87e0f82d-38d0-41a6-b7f2-96755c77bf44                        
Filesystem magic number:  0xEF53                                                      
Filesystem revision #:    1 (dynamic)                                                  
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype n      
eeds_recovery sparse_super large_file                                                  
Filesystem flags:         signed_directory_hash                                        
Default mount options:    (none)                                                      
Filesystem state:         not clean                                                    
Errors behavior:          Continue                                                    
Filesystem OS type:       Linux                                                        
Inode count:              30523392                                                    
Block count:              122087973                                                    
Reserved block count:     6104398                                                      
Free blocks:              117879828                                                    
Free inodes:              30506479                                            
First block:              0                                                    
Block size:               4096                                                  
Fragment size:            4096                                                  
Reserved GDT blocks:      994                                                  
Blocks per group:         32768                                                
Fragments per group:      32768                                                
Inodes per group:         8192                                                  
Inode blocks per group:   512                                                  
Filesystem created:       Wed Jul  9 18:58:42 2014                              
Last mount time:          Thu Mar 23 23:43:26 2017                              
Last write time:          Thu Mar 23 23:43:26 2017                              
Mount count:              4                                                    
Maximum mount count:      23                                                    
Last checked:             Thu Mar 23 19:15:22 2017                              
Check interval:           15552000 (6 months)                                  
Next check after:         Tue Sep 19 20:15:22 2017                              
Reserved blocks uid:      0 (user root)                                        
Reserved blocks gid:      0 (group root)                                        
First inode:              11                                                
Inode size:               256                                                    
Required extra isize:     28                                                      
Desired extra isize:      28                                                      
Journal inode:            8                                                      
Default directory hash:   tea                                                    
Directory Hash Seed:      dfbf8cab-fe37-4990-bcb8-461fe960c0b4                    
Journal backup:           inode blocks

The following gave no file path:
Code: 
HDFox# debugfs -R 'ncheck 16516' /dev/sda1
debugfs 1.43.3 (04-Sep-2016)            
Inode   Pathname                        
HDFox#
 
I ran e2fsck again:
Code: 
HDFox# umount /dev/sda1                                                     
HDFox# e2fsck -f -y -C 0 /dev/sda1                                           
e2fsck 1.43.3 (04-Sep-2016)                                                 
Pass 1: Checking inodes, blocks, and sizes                                   
Inode 16516 has inline data, but superblock is missing INLINE_DATA feature   
Clear? yes                                                                   
                                                                            
Pass 2: Checking directory structure                                         
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts                                           
Pass 5: Checking group summary information                                   
                                                                            
Red_Drive: ***** FILE SYSTEM WAS MODIFIED *****                             
Red_Drive: 16914/30523392 files (1.4% non-contiguous), 4208146/122087973 blocks
HDFox#
See also:
Code: 
HDFox# tune2fs -l /dev/sda1                                                   
tune2fs 1.43.3 (04-Sep-2016)                                                 
Filesystem volume name:   Red Drive                                           
Last mounted on:          <not available>                                     
Filesystem UUID:          87e0f82d-38d0-41a6-b7f2-96755c77bf44               
Filesystem magic number:  0xEF53                                             
Filesystem revision #:    1 (dynamic)                                         
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype i
nline_data sparse_super large_file                                           
Filesystem flags:         signed_directory_hash                               
Default mount options:    (none)                                             
Filesystem state:         clean                                               
Errors behavior:          Continue                                           
Filesystem OS type:       Linux                                               
Inode count:              30523392                                           
Block count:              122087973                                           
Reserved block count:     6104398                                             
Free blocks:              117879827                                           
Free inodes:              30506478                                           
First block:              0                                                   
Block size:               4096                                               
Fragment size:            4096                                               
Reserved GDT blocks:      994                                                 
Blocks per group:         32768
Fragments per group:      32768                                     
Inodes per group:         8192                                     
Inode blocks per group:   512                                       
Filesystem created:       Wed Jul  9 18:58:42 2014                 
Last mount time:          Thu Mar 23 23:43:26 2017                 
Last write time:          Fri Mar 24 00:29:53 2017                 
Mount count:              0                                         
Maximum mount count:      23                                       
Last checked:             Fri Mar 24 00:29:53 2017                 
Check interval:           15552000 (6 months)                       
Next check after:         Wed Sep 20 01:29:53 2017                 
Reserved blocks uid:      0 (user root)                             
Reserved blocks gid:      0 (group root)                           
First inode:              11                                       
Inode size:               256                                       
Required extra isize:     28                                       
Desired extra isize:      28                                       
Journal inode:            8                                         
Default directory hash:   tea                                       
Directory Hash Seed:      dfbf8cab-fe37-4990-bcb8-461fe960c0b4     
Journal backup:           inode blocks
'inline_data' has become a filesystem feature again.

EDIT. I removed the 'inline_data' filesystem feature ('debugfs -w /dev/sda1 -R 'features ^inline_data') and ran e2fsck again (options as above) and it gave the same message about inode 16516. It has also restored the 'inline_data' feature yet again. I have now removed this, again. The disk is accessible and can be recorded to. As I can not find a file attached to this inode, I will install CFW 3.12 and run the older version of e2fsck in the hope that it will clear the inode.
 
Last edited:
This is where the filesystem flag will be set by e2fsck:

https://github.com/tytso/e2fsprogs/blob/master/e2fsck/pass1.c#L1391

Looking at that block, if you answer "yes" (fix_problem() returns true) then it will enable the file-system feature otherwise it will ask if you want to clear the inode.

So, what you want to do to fix this is answer no then yes for each inode (assuming you have a backup!)

Code: 
Inode 16516 has inline data, but superblock is missing INLINE_DATA feature         
Clear<y>? no                                                                       
Inode 16516 has INLINE_DATA_FL flag on filesystem without inline data support.     
Clear<y>? yes
 
I manually corrupted one of my inodes by setting the flag and went through the check process which was able to repair the file-system without setting the feature flag, see below.

Code: 
gpttest# e2fsck -f /dev/sda1
e2fsck 1.43.3 (04-Sep-2016)
Pass 1: Checking inodes, blocks, and sizes
Inode 13 has inline data, but superblock is missing INLINE_DATA feature
Clear<y>? no
Inode 13 has INLINE_DATA_FL flag on filesystem without inline data support.
Clear<y>? yes
Pass 2: Checking directory structure
Entry 'small' in / (2) has deleted/unused inode 13.  Clear<y>? yes
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -14336
Fix<y>? yes
Free blocks count wrong for group #0 (30314, counted=30315).
Fix<y>? yes
Free blocks count wrong (246184, counted=246185).
Fix<y>? yes
Inode bitmap differences:  -13
Fix<y>? yes
Free inodes count wrong for group #0 (8177, counted=8178).
Fix<y>? yes
Free inodes count wrong (65519, counted=65520).
Fix<y>? yes

/dev/sda1: ***** FILE SYSTEM WAS MODIFIED *****

/dev/sda1: ********** WARNING: Filesystem still has errors **********

/dev/sda1: 16/65536 files (12.5% non-contiguous), 15959/262144 blocks

gpttest# e2fsck -f /dev/sda1
e2fsck 1.43.3 (04-Sep-2016)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sda1: 16/65536 files (12.5% non-contiguous), 15959/262144 blocks

gpttest# mount /dev/sda1 /mnt/hd1
 
I've created a small utility to scan and check a partition for these inline data flags - it's safe to run in normal mode (with filesystems mounted) and will print out the details of any inodes which have the flag set. I'll package it up tonight but for now you can download and run it by hand:

Code: 
humax# cd /tmp
humax# wget http://hpkg.tv/misc/scaninline
...
2017-03-24 10:41:03 (217 KB/s) - `scaninline' saved [19012/19012]

humax# chmod u+x scaninline
humax# ./scaninline /dev/sda1
Opened /dev/sda1 ''
Scanning inodes...
Found bad inode 13 = 0x10000000.
Scanned 10 in-use inodes.

... repeat for /dev/sda3 and /dev/sda2 on HDR
 
Last edited:
MontysEvilTwin - install the new e2recover package and that will give you the 1.43 versions of some utilities to make this easier.
They'll be on the disk but you can copy to /tmp to run while it is unmounted.

Code: 
humax# opkg files e2recover
Package e2recover (1.43.4) is installed on root and has the following files:
/mod/sbin/43.dumpe2fs
/mod/sbin/43.debugfs
/mod/sbin/43.tune2fs
/mod/sbin/scaninline
/mod/sbin/43.e2fsck
 
af123 said:
MontysEvilTwin - install the new e2recover package and that will give you the 1.43 versions of some utilities to make this easier.
They'll be on the disk but you can copy to /tmp to run while it is unmounted.

Code: 
humax# opkg files e2recover
Package e2recover (1.43.4) is installed on root and has the following files:
/mod/sbin/43.dumpe2fs
/mod/sbin/43.debugfs
/mod/sbin/43.tune2fs
/mod/sbin/scaninline
/mod/sbin/43.e2fsck
Click to expand...
Will this still work on the HD-FOX?
Late last night I ended up installing CFW 3.12 and ran e2fsck:
Code: 
HDFox# umount /dev/sda1                                                   
HDFox# e2fsck -f -y -C 0 /dev/sda1                                                 
e2fsck 1.42.13 (17-May-2015)                                                       
Pass 1: Checking inodes, blocks, and sizes                                         
Pass 2: Checking directory structure                                               
Pass 3: Checking directory connectivity                                           
Pass 4: Checking reference counts                                                 
Pass 5: Checking group summary information                                         
Red_Drive: 16914/30523392 files (1.3% non-contiguous), 4208148/122087973 blocks   
HDFox#
Thr filesystem appears error free now although this older version seems to have made no changes. Regarding the new version, since the 'inline_data' feature is EXT4 only, shouldn't e2fsck simply refuse to set it on EXT3 filesystems? Also, if you have to answer 'no' then 'yes' to get the correct outcome, using the -y switch with e2fsck directly, or fixdisk on the HDR-FOX, would lead to the 'inline_data' feature being set on the filesystem making the drive inaccessible to the Humax unit?
 
You must log in or register to reply here.
Back
Top