WIP/Help please: add VTXv4 file format support

[girst]

VTXv4 is used by the dvbtext tool, which fetches pages from live DVB-{T,S,C} teletext.

hi, I've implemented rudimentary support for a format I've got thousands of teletext pages for. There are some problems I've yet to overcome, these are:

• how does Setm_SubPage work? each file contains only 1 subpage; they are differentiated by their file name (PPP_SS.vtx)
• how do i need to pad the very first line? mine is only 32 chars wide (left pad? right pad?)
• black-on-white text is missing (probably an issue with not setting text colour?)
• my pages are in german; all the umlauts are displayed wrongly. where/how can I set which language the pages are in?

I'd love to get some help with this, if you've got time, @peterkvt80. I can privately share with you my teletext archive, if you need to. Thanks!

[girst]

also pinging @ZXGuesser!

[ZXGuesser]

FileMonitor and LoadPageList assume one file per page, so if you have subpages in separate files you will just get multiple conflicting pages loaded, each with one subpage.

The very first line (row zero) will be replaced by the inserter generated header anyway.

Alpha Black is code 0x00 which suggests you have a problem handling nulls (or your TV doesn't support it?)

The National Option Set bits are are in m_pagestatus. You can set it with p->SetPageStatus(). For German you would OR 0x200 with any other status bits (see ttxpage.h)

[girst]

FileMonitor and LoadPageList assume one file per page, so if you have subpages in separate files you will just get multiple conflicting pages loaded, each with one subpage.

That seems like what is happening. is there a way around that? I'm not yet very familiar with the code (it's quite large), but can you in some way append subpages to loaded pages? I'd happily code that if you can tell me where to look for.

The very first line (row zero) will be replaced by the inserter generated header anyway.

Interesting. So is there a reason why all the other decoders are parsing it still?

Alpha Black is code 0x00 which suggests you have a problem handling nulls (or your TV doesn't support it?)

D'oh! Casting to strings with a null byte in the front probably won't help. (my tv definetly supports it; i can see the 'upstream teletext' from which i generate my archive from just fine)

The National Option Set bits are are in m_pagestatus. You can set it with p->SetPageStatus(). For German you would OR 0x200 with any other status bits (see ttxpage.h)

I'll have a look, thanks! Are the language bits exposed to the user somewhere, so you can manually override it? (dvbtext -- once again -- does not encode them)

[ZXGuesser]

That seems like what is happening. is there a way around that? I'm not yet very familiar with the code (it's quite large), but can you in some way append subpages to loaded pages? I'd happily code that if you can tell me where to look for.

Very difficult as it's sort of the opposite way round to how all the existing code works. The file monitor works by the pages having the filename they were loaded from.

Interesting. So is there a reason why all the other decoders are parsing it still?

Chunks of the code were copied over from https://github.com/peterkvt80/wxted and don't do anything but have never been removed to tidy up the source. That's why all the VTX and EP1 import code is present but not functional.

I'll have a look, thanks! Are the language bits exposed to the user somewhere, so you can manually override it? (dvbtext -- once again -- does not encode them)

The page status is read from the .TTI files so there's no facility to override it. If the status is missing it defaults to 0x8000 (which is an English page with no flags set) but I suppose a config file setting could be added to change that default.

[girst]

thanks! those insights are very helpful. can you elaborate a bit how Setm_SubPage works exactly? for now, i'm just ignoring any subpages other than 1.

small status update: black-on-white is working now (std::string(buf) -> std::string(buf, 40))

if there is interest (and some test pages) I could get the other file importers working again.

If the status is missing it defaults to 0x8000 ... but I suppose a config file setting could be added to change that default.

that would be exactly what i need!

[ZXGuesser]

thanks! those insights are very helpful. can you elaborate a bit how Setm_SubPage works exactly? for now, i'm just ignoring any subpages other than 1.

Setm_SubPage sets the subpage number, but currently GetPageCount() gets called for carousels after they're loaded to forcibly renumber them. Presumably just to deal with malformed TTI files @peterkvt80?

if there is interest (and some test pages) I could get the other file importers working again.

EP1 is broken anyway, it's based on an incomplete understanding of the format so it would better to start again. I've been meaning to look into that at some point because wxTED needs the same attention.

If the status is missing it defaults to 0x8000 ... but I suppose a config file setting could be added to change that default.

that would be exactly what i need!

The most awkward part is getting the configuration to the TTXPage object. I think by passing it to TTXPageStream and from there into new the TTXPage.

[girst]

Thanks! I've found out that the charset replacement is encoded!

regarding the subpages: what do you think of this idea?

0. let p = this;
1. read filemagic and header, parse pagenumber
2. if pagenumber already exists:
    a. get (preexisting) ttxpage object, loop until we reach nullpointer
    b. p = end of linked list
3. parse page into p

this moves the question to: "can i access already parsed pages by their pagenum?"

[ZXGuesser]

this moves the question to: "can i access already parsed pages by their pagenum?"

This is exactly the problem :)
Currently you'd have to iterate over the page list to find it, which you'd end up doing for every single file, every loop of the filemonitor.