• The forum software that supports hummy.tv has been upgraded to XenForo 2.3!

    Please bear with us as we continue to tweak things, and feel free to post any questions, issues or suggestions in the upgrade thread.

Thumbnails - any configuration?

raydon said:
If you are able to convert your video to native T2 format using AV2HDR-T2
Click to expand...

I've never had any luck with this tool.

I have just recorded 30 or so seconds of BBC1 on the Humax, decrypted, converted to .MPG and then tried to run that file through AV2HDR-T2.

The output is just not usable - the .MPG input file is about 16.5MB; the output .NTS file is 0 bytes; the output .TS file is about 1.6MB; the output .HMT looks about right at 3K.

Is there any chance you can take a look and see what might be wrong please?

I've dumped the whole lot here <removed>.
 
Black Hole said:
... elided ...

This is a typical problem with conflicting asynchronous processes, something I have had to consider almost all my career, and I am trying to offer my engineering experience to complement the software skills exhibited by others.

OK, carry on. Generate the auto-thumbnail asynchronously. The things I mention may never happen.
Click to expand...

You're either over or under-thinking it : )

The webif will generate the new thumbnail into a temporary file on the same filesystem as the recording and then use rename() to atomically switch it to the real filename.

On a UNIX system, the rename() system call is atomic (assuming the OS does not crash). That is, it cannot be split by any other filesystem operation. There is also no concept of locks other than ones implemented in a co-operative manner. Finally, there is nothing wrong with unlink()ing an open file. The file remains open but no longer present in the filesystem namespace.

If the Humax software is in the process of generating a thumbnail itself, then the .thm file will be unlink()ed as part of the rename() operation. The Humax process won't notice and will continue writing to the file descriptor until it's finished and then close the file. The data that it has written will then be discarded by the VFS layer.

If the Humax software has already decided to generate a .thm before the rename operation takes place, then it will most likely overwrite the just-generated on with its own (unless it opens the .thm file without the O_CREAT flag). Either way you end up with something consistent.

It /is/ more complicated if you want to use a flag in the .hmt file to track whether a custom thumbnail exists. I'm going to use your extra byte idea so that it's easy to tell if a thumbnail is automatic or custom.
 
Good. A proper analysis to address the problem instead of just keeping fingers crossed. I'm a bit amazed that a process can happily think it is writing data to a file that then gets discarded, but there it is. Would you have thought to do it using rename otherwise? My fear is that humaxtv might not be well behaved, being that it expects no competition.

ChrisDaniels said:
Your arrogance is rubbing people up the wrong way.
Click to expand...

I resent that. Project management is about anticipating problems and ensuring they are addressed, which is all I have been guilty of.
 
Black Hole said:
I'm a bit amazed that a process can happily think it is writing data to a file that then gets discarded, but there it is.
Click to expand...

It's intentional behaviour. A filesystem name is just one reference to a file.

From The Single UNIX ® Specification, Version 2

When the file's link count becomes 0 and no process has the file open, the space occupied by the file will be freed and the file will no longer be accessible. If one or more processes have the file open when the last link is removed, the link will be removed before unlink() returns, but the removal of the file contents will be postponed until all references to the file are closed.
Click to expand...

The new .thm file following the rename, although it has the same name as the previous one, is a separate entity with a different inode number.

Would you have thought to do it using rename otherwise?
Click to expand...

Yes : ) Although the current beta of the webif thumbnail code doesn't so this my development copy does.
 
I've been playing with something to automate new default thumbnails.
It generates a new thumbnail if there isn't one present or the 4th byte of the current one is 0xFF.
My linux scripting is very raw so I'm sure it could be done much better than this but it may be some use.

Some notes,
  • this works on all ts files in a single directory (might want to do it recursively from My Video)
  • it tries to generate a thm even for audio files (which obviously fails) - not sure if that could be detected
  • one of my test files wouldn't work with any offset other than 0 seconds (well I tried 1, 10, 20, 200, 400) - it may need a fallback behaviour if the wanted offset fails
  • the fourth byte of the new thumbnail file gets set to 0xFE after the file is generated
  • I'm using a seek time of 400 secs as a default (6 mins 40 seconds) - I like round numbers! This should be a setting ideally.

Code: 
for file in *.ts
do
        fileRoot=${file%.ts}
        process="0"
 
        if [ -f "$fileRoot.thm" ]
        then
                checkByte=`hexdump -n1 -s3 -e'"" 1/1 "%02xf"' "$fileRoot.thm"`
                if [ $checkByte == "ff" ]
                then
                        process="1"
                fi
        else
                process="1"
        fi
 
        if [ $process == "1" ]
        then
                ffmpeg -ss 400 -i "$file" -frames 1 -pix_fmt bgra -vf vflip -s 140x78 TempThumb.bmp
                if [ -f TempThumb.bmp ]
                then
                    dd bs=54 skip=1 if=TempThumb.bmp of="$fileRoot.thm"
                    rm TempThumb.bmp
                    echo -ne \\xFE | dd conv=notrunc bs=1 count=1 seek=3 of="$fileRoot.thm"
                fi
        fi
done
 
Black Hole said:
I resent that. Project management is about anticipating problems and ensuring they are addressed, which is all I have been guilty of.
Click to expand...

Who said it was your project to manage? These people don't work for you, they are not part of your engineering team
 
Never said it was, never said they did, but if nothing else this topic exemplifies a collaborative effort better than any other I can recall. Project management is a team role no more or less important than any other. Who said it was your place to comment? You do because you can.
 
Black Hole said:
This is a race condition and the consequences are undefined. We don't know what humaxtv will do if it finds a file it was expecting to write to is locked, in the worst case it could crash. If it gets in after the WebIF process has created the file, it might over-write the auto-generated file with nothing to show this has happened (apart from it being the wrong thumbnail). Various tricks can minimise the time window during which a conflict can exist, but never entirely eliminate it. The race condition can be avoided completely by the WebIF background process refusing to run until the humaxtv thumbnail has been generated.
Click to expand...

If you are going to be pedantic here if the humaxtv process did use the fact that the file wasn't there to process a new one then surely technically it could also check between the web-if deleting the old one and putting a new one in it's place and decide to add one too...

I guess we will find out at some point if the humaxtv process DID cause an issue with web-if producing a thumbnail but it sounds pretty remote, and I imagine more important assumptions and educated guesses must go on more than this!? How about for a start the fact that you can easily delete a video file from the web-if or move it elsewhere while the TV is showing it in Media etc?

Anyway I don't know about the inner workings of the processes obviously but who would have thought some comments about setting a thumbnail could cause all this, I was more interested in the decoding of the files than changing the pictures it decides to add, I suppose vaguely useful for a film or the like but fairly pointless for most series etc?
 
dragon-it said:
If you are going to be pedantic here if the humaxtv process did use the fact that the file wasn't there to process a new one then surely technically it could also check between the web-if deleting the old one and putting a new one in it's place and decide to add one too...
Click to expand...
This is the kind of thing I have been talking about, and the concern is less pedantic if it resulted in users finding mysterious crashes. However, we are assured that by creating the file under a different name, renaming the file is effectively an instantaneous way of creating the new thumbnail file or replacing an existing one. It is still possible for the standard thumbnail file to usurp the custom one, but by carrying a flag actually in the file this will be detectable.

dragon-it said:
it sounds pretty remote
Click to expand...
Any one individual has little chance of winning the lottery, but somebody wins every week.

dragon-it said:
How about for a start the fact that you can easily delete a video file from the web-if or move it elsewhere while the TV is showing it in Media etc?
Click to expand...
Have you tried it? The TV continues to show it in Media until you refresh the listing. The lists are not "live", they are snapshots. These problems are not an issue when the basic operation is a read, it's when two processes try to write to the same destination that it requires more thinking about.

dragon-it said:
I suppose vaguely useful for a film or the like but fairly pointless for most series etc?
Click to expand...
I agree, but the enquiry was made (not for the first time) and it caught the mood of the moment to do something about it. The next version looks the dog's doo-dahs.
 
Black Hole said:
Project management is a team role no more or less important than any other.
Click to expand...
Why do you think the guys that actually write the code need to be team lead by anybody, they are all more skilled at what they do than you are. You seem to be very keen to tell everybody what you used to be in Engineering, as if every other person who comments here works on a cheese counter at their local Deli, they are all Engineers, they just don't feel the need to tell everybody every 5 minutes
 
Black Hole said:
Have you tried it? The TV continues to show it in Media until you refresh the listing. The lists are not "live", they are snapshots. These problems are not an issue when the basic operation is a read, it's when two processes try to write to the same destination that it requires more thinking about.
Click to expand...

I haven't and don't suppose it causes any issues as you say, and what af123 etc. have done is amazing. Things aren't 100% known, I assume, from having seen the Humax source code, or to know what the authors at Humax may put in the next update but are best-guess from observations about how it handles each situation.

Anyway I'll shut up as extending the thread for no gain...
 
Black Hole said:
Never said it was, never said they did, but if nothing else this topic exemplifies a collaborative effort better than any other I can recall. Project management is a team role no more or less important than any other. Who said it was your place to comment? You do because you can.
Click to expand...
You didn't ?? From your post #25.:rolleyes:
The other matters I may not be able to help with (other than as team leader)
Click to expand...
 
prpr said:
I've never had any luck with this tool.
I have just recorded 30 or so seconds of BBC1 on the Humax, decrypted, converted to .MPG and then tried to run that file through AV2HDR-T2.
The output is just not usable - the .MPG input file is about 16.5MB; the output .NTS file is 0 bytes; the output .TS file is about 1.6MB; the output .HMT looks about right at 3K.
Is there any chance you can take a look and see what might be wrong please?
I've dumped the whole lot here.
Click to expand...
Hi prpr, Your query is a bit off-topic here so I've posted a response here in the AV2HDR-T2 thread.
 
Works very well, and more accurate than the bookmark only method
WiKi Notes HERE
I particularly like the feature that if there is a bookmark set, that the thumbnails centre on it, It's the best of both methods
 
It would be useful to be able to enter a time with one place of decimal e.g. 1.2 seconds. It would then make it possible to capture something which is fast moving.
 
Forgot to post the screenshot that I'd prepared!

setthm.png
 
You must log in or register to reply here.
Back
Top