Crash Loop - what to do

[valve90210]

Hi All

My Hummy was fine yesterday evening though was running low on disk space so I spent a good while deleting loads of old stuff that had been watched.

When I went to bed last night I switched off as usual.

This morning I went to watch tv and my hummy came on as usual then after a minute or two crashed and restarted and is doing so continually.

I'm trying to get into maintenance mode to run a disk check but not having any luck so far. I can telnet in and get to the teknet menu but selecting maintenance mode just closes the telnet window and then nothing seems to happen.

My software details from Webif:
Web interface version: 1.0.8-2
Custom firmware version: 2.22 (build 1905)
Humax Version: 1.03.12

Can anyone give me a hand in getting my hummy up and running again please?

 

[Black Hole]

selecting maintenance mode just closes the telnet window and then nothing seems to happen.

The HDR-FOX doesn't reboot??

The reason the Telnet session closes is because the HDR has rebooted. When it reboots it should then be inactive (as a PVR) with "Maintenance" showing on the front panel display. Then you have to connect through Telnet again, and the menu has the maintenance mode options.

However, your crash symptoms might well be due to something other than the HDD file system. Have you tried a factory reset (Menu >> Settings >> Installation >> Factory Default)?

 

[valve90210]

I switched the box of at the back and upon switching back on it seemed to stay on for a bit longer and I have managed to get into maintenance mode and am currently running fix disk to see if that helps.

I've not done a factory reset as yet but will probably do so once the disk check is finished.

Wish I had chosen a more interesting channel when I had a chance, I've got the audio from bbc news talking about the same stories over and over while the disk is checking lol

 

[Bazza1024]

Oh this sounds very much like what my box is currently doing.

This is my replaced box
Custom firmware version: 2.22

was working really well, turned it on today and it just starts up gets to the point where it tells me it is running custom firmware 2.22 then it goes black, the drive sounds like it is starting to spin up and it says crash, rebooting in 10, 9, 8 .... and the loop begins again.

I thought i'd prevented the automatic updates.

 

[MontysEvilTwin]

@Bazza1024. Have you introduced a NAS into your network recently? In any case disconnect your dongle or Ethernet cable and see if it still crashes. If it does not crash now, turn off content share and reconnect to your network. If it is still stable it is probably something on your network crashing the DLNA server (e.g. a NAS running an incompatible version of Twonky). If it is still crashing you probably need to get into maintenance mode and run fix disk. It sounds like your system is not stable enough to do this though. If so, af123 may be able to supply a modified firmware version to get the unit booted into maintenance mode.
There are no OTA firmware updates running at the moment, so this is not the issue. In addition, the fact that you still get the 'custom firmware 2.22' banner means that your unit has not undergone an unexpected update. You could also try installing the latest custom firmware (e.g. 1.03.12/ CFW 2.23) to see if this makes a difference. In any case it will give you the latest Telnet menus and fixdisk version, if you can get that far.

 

[Bazza1024]

Thanks, I should have said the first thing I tried was unplugging the LAN cable and then the aerial feed, it was doing the same thing. I did introduced a dlna server a couple of weeks ago it didn't cause a problem at the time, but I see on here there are some conflicts there so I'll look out for that.

I reconnected it to the LAN this morning, got into maintenance mode and checked the disk and repair
/---------------------------------------------\
| M A I N T E N A N C E M O D E M E N U |
\---------------------------------------------/

[ Humax HDR-Fox T2 (humax) 1.03.12/2.22 ]

Note: Some areas of the hard disk are not mounted.
Not all functions will be available.
(normal if you have just run a disk check)

1 - Check and repair hard disk (fix-disk).
2 - Run short hard-disk self test.
3 - Run long hard-disk self test.
4 - Check self-test progress.
epg - Clear persistent EPG data.
x - Leave maintenance mode (Humax will restart).
diag - Run a diagnostic.
cli - System command line (advanced users).

Please select option: 1
Any additional options (or press return for none):
Are you sure you wish to run the hard disk checker? [Y/N] y
Running /bin/fix-disk
Custom firmware version 2.22


Checking disk sda

Partition /dev/sda1 is already unmounted
Partition /dev/sda2 is already unmounted
Partition /dev/sda3 is already unmounted

Running short disk self test

No pending sectors found - skipping sector repair
Using superblock 0 on sda1
Using superblock 0 on sda2
Using superblock 0 on sda3


Checking partition /dev/sda3...
e2fsck 1.41.14 (22-Dec-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found. Create? yes
.......
It found some illegal blocks, blocks claimed by more than one inode, etc

fixed all that but still does the same thing.

I'll try to upgrade to 1.03.12/ CFW 2.23 as you suggest.

I assume it must be possible to export recordings from this mode in case I have to do a factory reset.

 

[Ezra Pound]

A factory reset will not remove your recordings, there is a format Y/N option

 

[Black Hole]

I'll try to upgrade to 1.03.12/ CFW 2.23 as you suggest.

That is unlikely to achieve anything useful.

I assume it must be possible to export recordings from this mode in case I have to do a factory reset.

As EP says, Menu >> Settings >> Installation >> Factory Default is no danger to existing recordings unless you choose the format disk option. It will however mean reconfiguring all your settings, retuning, and clearing out the recording schedule (which should be automatically restored by the WebIF, presuming a user has CF installed). The point is that it has been known for corruption in the databases to cause problems, and a factory reset should clear them all out and rebuild, and has a relatively minor impact.

With this continuous crashing type of problem, the sequence should be:

1. Ensure the unit is tuned to a "safe" channel - eg BBC1: some services carry data which is incompatible with the HDR-FOX;
2. Disconnect the network - if it still reboots it's an internal problem;
3. Put the system in Maintenance Mode - if it still reboots it's a hardware or OS problem (skip this list - see notes);
4. While in maintenance mode, might as well run a fix-disk - this should cure any file system problems but not eliminate rogue files;
5. Perform a factory reset - if it still reboots it's probably not a database problem;
6. Re-flash the standard firmware - this is clutching at straws, but if somehow the firmware has become corrupted then re-flashing is a potential cure with no drawbacks;
7. Flash a previous version of the standard firmware in case the problem is a bug in the latest firmware;
8. Physically disconnect the HDD - if it then still reboots there is something serious wrong.

Presuming disconnecting the HDD stopped the reboots in normal mode (but a fix-disk didn't), either there is something physically wrong with the HDD or there is something wrong with the contents of the HDD. My recommendation then would be to connect the HDD to a PC and rescue anything important (even if it is encrypted - in which case it is vital to save at least the .hmt file in addition to the .ts file for each recording, and I would say save the whole set of sidecar files), and then reformat the HDD. Presuming this cures the crashing, the saved recordings can be put back and if the crashing resumes you know there is a rogue file in there somewhere - but past experience indicates these are more likely to stop the Humax going into standby properly than actually crash (although firmware updates may alter this behaviour).

When the system is in the CF Maintenance Mode (access via the Telnet menu), the normal operating firmware for the system is not operational, so there will be no opportunity for databases, disk files, or network interactions to cause crashes. If the system continues to crash one is looking at PSU glitches, processor/memory faults, or peripheral faults (eg a physical HDD problem) - or maybe, just maybe, the solid state memory that contains the operating system has become corrupted and a firmware re-flash might fix it. The HDD can be disconnected, and the PSU can be bypassed (by a competent person) in efforts to isolate the problem.

 

[MontysEvilTwin]

You could also try installing the latest custom firmware (e.g. 1.03.12/ CFW 2.23) to see if this makes a difference. In any case it will give you the latest Telnet menus and fixdisk version

I'll try to upgrade to 1.03.12/ CFW 2.23 as you suggest.

That is unlikely to achieve anything useful.

@Black Hole. If you read the custom firmware release notes here you will see that the update to CFW 2.23 includes 'Upgrade embedded filesystem check and repair utilities to latest version'. From the tone of my original message (quoted above) you will see that I was not expecting the upgrade CFW to 2.23 to necessarily fix the problem, but it does provide the latest fixdisk version. This seems like a valid suggestion to me. If you like, I can forensically pick through your troubleshooting list and highlight those which I think are unlikely to be of any use.

 

[Black Hole]

I'm not trying to start an argument, I don't understand your response.

It seems to me highly unlikely that these crashes are anything to do with a file system problem that has not been found in the CF2.22 diagnostics but will be found in CF2.23 (although of course if you are going to go through the potentially tedious process of running fix-disk you might as well be using the latest version). They are clearly not to do with corruption in the CF, so there is nothing to be gained by re-installing the CF from that point of view. The factory reset is the best course of action, followed by re-installation of the standard firmware in case that is corrupted (maybe even installation of a previous standard firmware in case it is a bug in 1.03.12).

f you like, I can forensically pick through your troubleshooting list and highlight those which I think are unlikely to be of any use.

My list and notes are for general reference, not the specific case here in particular. You are welcome to point out any errors or omission there may be in my list, I am committed to process improvement, but I do expect your points to be accompanied by convincing reasoning.

 

[MontysEvilTwin]

General advice for troubleshooting the problem, that I have read on this forum, includes running fixdisk. It probably is a long shot, but then some of the other steps recommended are too. I was just suggesting an update to the latest version of custom firmware first before running fixdisk.
I'm not trying to start an argument, I'm just standing up for myself. I don't understand why you had to comment and dismiss a perfectly reasonable suggestion. Why not just present your troubleshooting steps and leave it at that?

EDIT: since posting this message in response to the one above, Black Hole has made significant additions to his message which (of course) are not responded to here.

 

[af123]

With this continuous crashing type of problem, the sequence should be:

Excellent list - we should probably put that on the Wiki somewhere or as a reference thread.
There's one new (and recent) thing I would add to it which is that if the box is stuck in a reboot loop such that there is no opportunity to do a factory reset through the standard on-TV menus then you can load the Factory Reset Update File from http://wiki.hummy.tv/wiki/Firmware_Downloads which will do a full factory reset via the firmware update procedure (actually more thorough than the one through the on-TV menus). There is a side effect of this in that, if you have custom firmware installed, then you will also reset most of the settings for that too.

For the general diagnostic process, it's also worth considering just disabling the DLNA server (Content Sharing) as a step after disconnecting the network. I've seen crashes from a corrupt DLNA database which would still occur without network connectivity. Also, the DLNA database is not AFAIK cleared on a factory reset so the CFW Webif diagnostics page method for resetting that is worthwhile if disabling content sharing solves the reboots.


@Bazza1024 - it sounds like you're at the factory reset stage - if you have long enough to get into the menus. If not you can try loading the reset file I mention above.

 

[af123]

It seems to me highly unlikely that these crashes are anything to do with a file system problem that has not been found in the CF2.22 diagnostics but will be found in CF2.23

You're right although there could be edge cases - three years of development on those utilities must have fixed /some/ bugs!
I think it's good practice to upgrade to the latest version of CFW as part of the troubleshooting process in order to have the latest tools and supporting scripts like fix-disk.

 

[Black Hole]

Why not just present your troubleshooting steps and leave it at that?

My post 8 could have prevented the OP following a blind alley when there was a more likely path, which is why I commented.

Regardless of previous general advice, there is always potentially better advice as thinking develops. I appreciate you or others are trying to be helpful by echoing previous advice given, but you must acknowledge that it can become out of date or irrelevant under specific circumstances.

 

[Black Hole]

@af123 - I will digest and incorporate. The post is easy enough to link to for reference.

 

[Bazza1024]

Thanks for your help, I'll give this Factory Reset Update File a go. Currently the box doesn't get to a point where I can change channel or display any menu so a normal Factory Reset via menu is out of the question.

I assume this will still prompt if I want to format the drive at the end?

Is the recording schedule stored any place readable while in maintenance mode. I don't mind setting it all again via the menus but remembering what I had scheduled to record is a challenge, particularity with things that aren't currently airing.

 

[Black Hole]

I assume this will still prompt if I want to format the drive at the end?

I think it just flushes things as if you had done a factory reset, no user interaction - and no disk format. When you then start up normally you will probably be faced with the installation wizard.

Is the recording schedule stored any place readable while in maintenance mode. I don't mind setting it all again via the menus but remembering what I had scheduled to record is a challenge, particularity with things that aren't currently airing.

No, but the WebIF can restore it.

Schedules do not persist in the list, if they have had no "hits" in three months they get erased. If you want persistent search-and-schedule use the RS.

 

[MontysEvilTwin]

@Black Hole. While my post (#5) does not include a comprehensive troubleshooting guide, there is nothing incorrect in there. It puts forward a fair starting point. I fail to see how I might have lead Bazza1024 down a 'blind alley'?

With this continuous crashing type of problem, the sequence should be:

1. Ensure the unit is tuned to a "safe" channel - eg BBC1: some services carry data which is incompatible with the HDR-FOX;
2. Disconnect the network - if it still reboots it's an internal problem;
3. Put the system in Maintenance Mode - if it still reboots it's a hardware or OS problem (skip this list - see notes);
4. While in maintenance mode, might as well run a fix-disk - this should cure any file system problems but not eliminate rogue files;
5. Perform a factory reset - if it still reboots it's probably not a database problem;
6. Re-flash the standard firmware - this is clutching at straws, but if somehow the firmware has become corrupted then re-flashing is a potential cure with no drawbacks;
7. Flash a previous version of the standard firmware in case the problem is a bug in the latest firmware;
8. Physically disconnect the HDD - if it then still reboots there is something serious wrong.

Regarding your troubleshooting procedure (see above). In addition to af123's suggestions, I have the following comments corresponding to your list:

3. Recommend upgrading to the latest custom firmware version, if possible, before entering maintenance mode and running fix disk. This will probably become even more pertinent in the near future for users on 2.xx versions of custom firmware, when 3.00 (or later is available).
6. As you mention, this is clutching at straws: is it relevant?
7. Has anyone ever fixed a crash/ reboot problem by downgrading the official firmware version? If not is this suggestion useful? Perhaps delete steps 6 and 7: the procedure will be longer after incorporating af123's suggestions, so might it be fortuitous to prune back some of the dead wood?

 

[MontysEvilTwin]

Thanks for your help, I'll give this Factory Reset Update File a go. Currently the box doesn't get to a point where I can change channel or display any menu so a normal Factory Reset via menu is out of the question.

I assume this will still prompt if I want to format the drive at the end?

Is the recording schedule stored any place readable while in maintenance mode. I don't mind setting it all again via the menus but remembering what I had scheduled to record is a challenge, particularity with things that aren't currently airing.

I doubt it would wipe the hard drive without your confirmation, but it would be safer to wait for af123 to comment. His message (#12) said that this procedure deletes most of the custom firmware settings so I would not be confident that the schedule backup is retained without him confirming it. The backup is essentially just a text file though so you should be able to read/ copy it with Telnet access and knowledge of the necessary commands.

 

[Ezra Pound]

The recording schedule and favourites list are held in *.rbk files, e.g. /mod/var/backup/auto-2014-Jun-29-03:35.rbk The favourites are readable but the schedule is a bit more difficult :-

event  0      0      30      0      3      458876  1800    1404559800      0      FP.BBC.CO.UK/S2QV0V    1FP.BBC.CO.UK/21EYJD|  Click  1FP.BBC.CO.UK/20MZGI|1FP.BBC.CO.UK/20LEV5|1FP.BBC.CO.UK/20B0O4|1FP.BBC.CO.UK/1ZZZRC|1FP.BBC.CO.UK/1ZKSNA|      BBC NEWS HD    i7Click 00000000000000  50      0      4      0      0      0      0      0      22      0      65402  107    38961
event  0      0      30      0      3      65537  1800    1399591500      0      BDS.TV/26708    1BDS.TV/26708/007|      Spaced  1BDS.TV/26708/006|1BDS.TV/26708/005|1BDS.TV/26708/004|1BDS.TV/26708/003|1BDS.TV/26708/002|      4Music  i7Spaced        00000000000000  50      0      4      0      0      0      0      0      5      0      16675  18      20884
event  0      0      30      0      3      524416  2700    1403906100      0      FP.BBC.CO.UK/TJZBFI    1FP.BBC.CO.UK/1RKGKH|  The Graham Norton Show  1FP.BBC.CO.UK/1RKH8E|1FP.BBC.CO.UK/1RKH8D|1FP.BBC.CO.UK/1RKGF6|1FP.BBC.CO.UK/1RKGF5|1FP.BBC.CO.UK/1RKGF4|      BBC ONE HD      The Graham Norton Show  00000000000000  50      0      4      0      0      0      0      0      17      0      61100  101    62415
event  0      0      0      1      7      0      0      1404099000      0                                                      20140630043000  0      0      0      0      0      0      0      0      6      0      0              0
fav    1      1      BBC ONE W Mid  725
fav    1      1      BBC ONE HD      726
fav    1      1      BBC TWO HD      727
fav    1      1      BBC THREE HD    728
fav    1      1      BBC FOUR HD    729