MythTV QAM users: what format works best for you?

rkulagow
SD Staff
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

MythTV QAM users: what format works best for you?

Post by rkulagow » Fri Oct 17, 2014 2:08 pm

If you're a MythTV user, and you have a QAM lineup, how do you want your channels to appear? I've had some reports that I'm putting in the virtual channel number in the wrong place, making things not as good as they could be.

bonelifer
Posts: 24
Joined: Wed Sep 07, 2011 5:43 am

Re: MythTV QAM users: what format works best for you?

Post by bonelifer » Fri Oct 17, 2014 2:41 pm

I'm going to quote what I've said in email, then provide a SQL export of what it looks like for me when using the current system.
About what we talked about earlier. Using my WGN entry from the QAM export

{"channel":"1008_22","virtualChannel":"5","callsign":"WGN","name":"WGN","mplexID":"1","stationID":"","serviceID":"10"}

On my real TV, it's 22 literally. In mythtv's tables "channum" is just the display number(I could change it to 999 and it wouldn't matter to the backend for recording). But you've added it as 5-10, which looking at the info above seems like like "virtualChannel-serviceID". So I assume we'll still be required to do our own SQL updates to make our display channels right after our providers do a lineup change and we provide you with the new QAM extract. Currently I have to do this anyway when scanning for channels as such(except in the current case I'm adding the xmltvid and updating channel icons instead of fixing the display channel number(channum)):

example:
update channel set xmltvid=17098,icon="/home/william/.mythtv/channels/wgn_cw9_chicago.jpg" where callsign='WGN' AND channum='22';

Would it be possible to assume that any "channel" from the dump that has a "_" underscore and has more than say 3 digits could have it and the preceding data striped and it stored as the digits after the underscore. That would take into account examples like:

{"channel":"2_1","virtualChannel":"23","callsign":"AETN-HD","name":"AETN-HD","mplexID":"2","stationID":"","serviceID":"1"}

Which in my case on a real tv is 2-1.
SQL dump:

Code: Select all

INSERT INTO `channel` (`chanid`, `channum`, `freqid`, `sourceid`, `callsign`, `name`, `icon`, `finetune`, `videofilters`, `xmltvid`, `recpriority`, `contrast`, `brightness`, `colour`, `hue`, `tvformat`, `visible`, `outputfilters`, `useonairguide`, `mplexid`, `serviceid`, `tmoffset`, `atsc_major_chan`, `atsc_minor_chan`, `last_record`, `default_authority`, `commmethod`, `iptvid`) VALUES
(11102, '22', '5', 1, 'WGNAMER', 'WGN America', '/home/william/.mythtv/channels/wgn_cw9_chicago.jpg', 0, '', '17098', 0, 32768, 32768, 32768, 32768, 'ATSC', 1, '', 0, 1, 10, 0, 1008, 22, '2014-10-17 20:56:00', '', -1, NULL),
From my WGN example again. You can see that the 5 from your 5-10 gets put into "freqid", while the serviceid 10 is put in the "serviceid" field. It puts 1008_22 in "channum" which I use an SQL statement to remove "1008_" from all the channum fields. It also seems to put 22 into "atsc_minor_chan" and 1008 into "atsc_major_chan". The thing to remember is that the "channum" is the PRETTY number we type into mythtv to get to the desired channel. It has nothing to do with how mythtv backend actually tunes as it is only a vanity value to make it easier than to have to say type 1008_22 via remote. I've also added my SQL export of the channel table. Again this was exported from the current SD lineup(ie not the JSON lineup).
Attachments
channel.txt
(20.06 KiB) Downloaded 312 times

faginbagin
Posts: 8
Joined: Sun Mar 16, 2014 11:22 am

Re: MythTV QAM users: what format works best for you?

Post by faginbagin » Mon Oct 20, 2014 10:30 pm

Some background:

I have set up a test system that is a clean install of mythbuntu 14.04 and mythtv 0.27, configured as a stand alone master backend and frontend, and upgraded through Sept 2014. It has been configured to use one tuner on an HDHR2 device. I have done a channel scan using mythtv-setup. I have not imported the database from my "production" system.

The cable service provider is WideOpenWest, aka WOW, aka wowway.com. My cable service is what WOW calls "Basic Cable". I only get clear QAM channels, nothing encrypted. The only HD channels I get are the local broadcast channels. I get many basic cable channels, but they are all standard definition.

Following are two email exchanges with Robert K.
On 10/14/2014 11:13 AM, Robert Kulagowski wrote:
> On Mon, Oct 13, 2014 at 10:52 PM, faginbagin <mythtv@hbuus.com> wrote:
>> Here you go.
>
> I've compared your extract with this page:
>
> http://www.wowway.com/our-products/cabl ... eups/north
>
> Does that page accurately reflect what you get?
>
> Also, I will fixup the channel numbers; you've got 1008_131 for
> "Movies!", but on the web page that's 24-131. Any idea where
> "1008_131" came from?
>
> Thanks.

I'm not exactly sure what the 1008 represents, but I do know mythtv stores that value in the column atsc_major_chan for any channel that has the value tvformat = 'ATSC'. For each of these channels, mythtv assigns them a channum = 1008_<atsc_minor_chan>. And it assigns them a chanid = <sourceid>*1000 + <atsc_major_chan>*10 + <atsc_minor_chan>.

For those channels that aren't marked with tvformat=ATSC, you'll see that atsc_major_chan and atsc_minor_chan are zero, have tvformat='', have channum = <freqid>-<serviceid> (note the dash vs the underscore for ATSC channels), and have chanid = <sourceid>*1000 + <freqid>*10 + <serviceid>.

I'm not sure on what web page you see Movies! given the channel 24-131. The wowway channel line-up just shows 131. But I can tell you that 24-131 conforms to the <freqid>-<serviceid> format.

As for what wowway says I should get vs what I do get, I get everything marked as "Basic Cable" using the "Channel" numbers and I get some of the "HD Channel" numbers:

- I get broadcast channels in both SD and HD, like ABC on 7 and 200. The highest HD channel I get is 208.

- I get basic cable channels, but not their HD versions. For example, I get A&E in SD resolution on channel 46, but I don't get channel 218. Those HD channels are encrypted and I don't have a cable card tuner. Someone with a cable card tuner might get those, as well as the premium channels marked as "Basic with Digital TV or Ultra TV" and maybe "Signature Cable". Or maybe someone with an HD-PVR can get them because of the so-called the "analog hole".

The only channel I can tune that wowway doesn't mark as "Basic Cable" is "OnDemand - WOW" channel 1. All I get on it are previews.
On 10/14/2014 1:52 PM, Robert Kulagowski wrote:
> Do you have a QAM-capable TV? If yes, when you run a QAM scan, does it
> show channels in the "24-131" format?

I have three HDTVs with QAM/ATSC tuners.

In general, all three TVs tend to assign channel numbers that match the atsc_minor_chan number (as found by mythtv's channel scan), which is sometimes the same as the serviceid number, but not always.

For those channels that don't have an atsc_minor_chan number, the TVs assign channels of the form <freqid>.<serviceid>, where freqid is the analog cable channel number.

Clearly, both my HDTVs and mythtv use the atsc_minor_chan information at least during a channel scan, and possibly when tuning channels. Otherwise, why would mythtv store the info in the channel table? May I ask why you didn't preserve it (and atsc_major_chan)?
How do I want my channels to appear?

The simple answer: I want them to appear the same as they are shown on my cable service provider's web site:
http://www.wowway.com/our-products/cabl ... eups/north

The long answer:
I have accomplished this on my "production" mythtv system by editing the channum fields. For those marked by mythtv as ATSC, it was pretty easy. mythtv had initialized the channums to <atsc_major_chan>_<atsc_minor_chan>. What I wanted was <atsc_minor_chan>. So all I had to do was strip the leading <atsc_major_chan> and underscore, which in my case was stripping the string prefix, "1008_ ".

For example, here are the channel records for a couple of channels after a first time mythtv-setup channel scan:

Code: Select all

INSERT INTO `channel` (`chanid`, `channum`, `freqid`, `sourceid`, `callsign`, `name`, `icon`, `finetune`, `videofilters`, `xmltvid`, `recpriority`, `contrast`, `brightness`, `colour`, `hue`, `tvformat`, `visible`, `outputfilters`, `useonairguide`, `mplexid`, `serviceid`, `tmoffset`, `atsc_major_chan`, `atsc_minor_chan`, `last_record`, `default_authority`, `commmethod`, `iptvid`) VALUES
(11211,'1008_131','24',1,'MVIES','MVIES','',NULL,'','',0,32768,32768,32768,32768,'ATSC',1,'',1,1,131,0,1008,131,'0000-00-00 00:00:00','',-1,NULL),
(11112,'1008_32','32',1,'FSND','FSND','',NULL,'','',0,32768,32768,32768,32768,'ATSC',1,'',1,4,1,0,1008,32,'2014-09-30 09:04:00','',-1,NULL);
I want the first channel, MVIES, to have channum = 131 and the second, FSND, to have channum = 32.
I chose the first channel record, MVIES, as an example because it's the one that Robert asked about in the second email exchange. For this channel, it just so happens its serviceid, 131, equals its atsc_minor_chan. But the second record shows that's not always the case. For the FSND channel, the serviceid is 1, but the atsc_minor_chan is 32. This channel record shows one reason why I think its important for sd-utility.php --extract to capture the atsc_minor_chan values, something it wasn't doing as of Oct 13, 2014.

I can tune a number of cable channels that are not ATSC channels and for which mythtv-setup sets atsc_major/minor_chan to zero. Here's an example:

Code: Select all

INSERT INTO `channel` (`chanid`, `channum`, `freqid`, `sourceid`, `callsign`, `name`, `icon`, `finetune`, `videofilters`, `xmltvid`, `recpriority`, `contrast`, `brightness`, `colour`, `hue`, `tvformat`, `visible`, `outputfilters`, `useonairguide`, `mplexid`, `serviceid`, `tmoffset`, `atsc_major_chan`, `atsc_minor_chan`, `last_record`, `default_authority`, `commmethod`, `iptvid`) VALUES
(1661,'65-11','65',1,'1661','','',NULL,'','',0,32768,32768,32768,32768,'',1,'',0,11,11,0,0,0,'2014-09-30 09:18:00','',-1,NULL);
This happens to be INSP. My HDTVs show it as 65.11, and I have changed the channum on my "production" system to 11, which matches the channel that WOW has assigned to INSP. 11 just happens to be the serviceid, but I wouldn't expect that all non-ATSC channels for other cable users would have serviceids that equal the cable provider's channel number. I think it would be reasonable for SD to assign this channel the channum of 65.11, to match what most HDTVs would use.

Users who record OTA (Over the Air) broadcasts (like my sister), will almost certainly want their channel numbers to be of the form <atsc_major_chan>.<atsc_minor_chan>. That's because the <atsc_major_chan> value will be the broadcaster's original analog channel and the <atsc_minor_chan> will be 1 for their HD channel and 2 ... for their lower resolution channels. For example, in the Metro Detroit area, an OTA user will see these channels on their HDTVs for the ABC affiliate channels:
7.1 ABC-HD
7.2 LIVE
7.3 Bounce
I believe mythtv-setup will set the channum columns for the above channels to 7_1, 7_2 and 7_3.

To summarize, I think you need to start extracting atsc_major_chan and atsc_minor_chan columns from mythtv's channel tables and to use them when they are non-zero. When they are zero, use the analog channel number and serviceid, which in a fresh mythtv scan would be <freqid>.<serviceid>.

I'd also like to strongly encourage you to preserve mythtv's chanids. I think you're going to get a lot of resistance from mythtv users if you change the chanids. If they truncate their channel tables and then update to the channel table you're providing at present, they will lose the channel details associated with all the recordings they have made before changing to the JSON service. I believe the mythtv algorithm for chanids is something like the following:

Code: Select all

if atsc_major_chan > 0
	chanid = sourceid*1000 + atsc_major_chan*10 + atsc_minor_chan
else
	chanid = sourceid*1000 + analog_channel*10 + serviceid
Regards,
Helen

bonelifer
Posts: 24
Joined: Wed Sep 07, 2011 5:43 am

Re: MythTV QAM users: what format works best for you?

Post by bonelifer » Fri Nov 07, 2014 11:32 am

Maybe add a command line argument like "extqam", then:

Code: Select all

./sd-utility.php -extqam
E (extract)
Current output is:

Code: Select all

{"channel":"1008_22","virtualChannel":"5","callsign":"WGN","name":"WGN","mplexID":"1","stationID":"","serviceID":"10"}
extended output from the "extqam" parameter

Code: Select all

{"actChan":"","channel":"1008_22","virtualChannel":"5","callsign":"WGN","name":"WGN","mplexID":"1","stationID":"","serviceID":"10"}
All the values would be blank and advanced users could add the channel numbers as they actually appear on their TV(ie what they expect to see). If this value is filled in then use the given value. For channels like in my lineup that are like 2-1(2.1 depending on your TV set), you could limit the separator to "-" to be consistent.

So in my example, I'd fill in the value like this:

Code: Select all

{"actChan":"22","channel":"1008_22","virtualChannel":"5","callsign":"WGN","name":"WGN","mplexID":"1","stationID":"","serviceID":"10"}

rkulagow
SD Staff
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: MythTV QAM users: what format works best for you?

Post by rkulagow » Tue Nov 18, 2014 9:04 am

faginbagin wrote:Some background:

I have set up a test system that is a clean install of mythbuntu 14.04 and mythtv 0.27, configured as a stand alone master backend and frontend, and upgraded through Sept 2014. It has been configured to use one tuner on an HDHR2 device. I have done a channel scan using mythtv-setup. I have not imported the database from my "production" system.

The cable service provider is WideOpenWest, aka WOW, aka wowway.com. My cable service is what WOW calls "Basic Cable". I only get clear QAM channels, nothing encrypted. The only HD channels I get are the local broadcast channels. I get many basic cable channels, but they are all standard definition.

How do I want my channels to appear?

The simple answer: I want them to appear the same as they are shown on my cable service provider's web site:
http://www.wowway.com/our-products/cabl ... eups/north
Sorry it's taken a while to get back to this; the transition to self-hosting has brought a lot of new challenges...

Here's the generated QAM right now for your lineup based on the link you sent me; the first one is the Movies! (WJBK) channel.

Code: Select all

{
    "qamMappings": [
        "1"
    ],
    "map": {
        "1": [
            {
                "channel": "24-131",
                "virtualChannel": "131",
                "modulation": "qam_256",
                "frequency": "225000000",
                "serviceID": "131",
                "stationID": "35451"
            },

Since I don't have clear QAM anymore on my system it's all a matter of reverse-engineering.

Lets start with the basics; does the above accurately represent what you're seeing on your system for that station?

modulation, frequency and serviceID are used to create the appropriate tables in dtv_multiplex, so the issue probably isn't there, correct?

With the remaining fields, is it just that I'm stuffing the wrong thing into the channel table? Would additional data need to be in the extract?

Thanks.

faginbagin
Posts: 8
Joined: Sun Mar 16, 2014 11:22 am

Re: MythTV QAM users: what format works best for you?

Post by faginbagin » Wed Nov 19, 2014 1:37 pm

rkulagow wrote: Sorry it's taken a while to get back to this; the transition to self-hosting has brought a lot of new challenges...
No problem.
rkulagow wrote: Here's the generated QAM right now for your lineup based on the link you sent me; the first one is the Movies! (WJBK) channel.

Code: Select all

{
    "qamMappings": [
        "1"
    ],
    "map": {
        "1": [
            {
                "channel": "24-131",
                "virtualChannel": "131",
                "modulation": "qam_256",
                "frequency": "225000000",
                "serviceID": "131",
                "stationID": "35451"
            },

Since I don't have clear QAM anymore on my system it's all a matter of reverse-engineering.

Lets start with the basics; does the above accurately represent what you're seeing on your system for that station?
Yes.
rkulagow wrote: modulation, frequency and serviceID are used to create the appropriate tables in dtv_multiplex, so the issue probably isn't there, correct?
Correct.
rkulagow wrote: With the remaining fields, is it just that I'm stuffing the wrong thing into the channel table? Would additional data need to be in the extract?
Yes, you are missing the atsc_major/minor_chan columns. And, if the atsc_minor_chan is non-zero it would be a good value for channum.

There is also the issue of the algorithm used to generate chanids. You are using an algorithm that will make existing chanids in other tables point nowhere. That's going to be a problem for all but brand new mythtv users. As I said before, I believe the algorithm looks something like this:

Code: Select all

if atsc_major_chan > 0
   chanid = sourceid*1000 + atsc_major_chan*10 + atsc_minor_chan
else
   chanid = sourceid*1000 + analog_channel*10 + serviceid

rkulagow
SD Staff
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: MythTV QAM users: what format works best for you?

Post by rkulagow » Thu Nov 20, 2014 1:32 pm

I've created a new version of the extractor. Can you run a git pull and then send me the update please?

rkulagow
SD Staff
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: MythTV QAM users: what format works best for you?

Post by rkulagow » Sun Nov 23, 2014 4:52 pm

OK, I'm re-thinking the approach here. Rather than stuffing various tables, I may take the approach of "match what the user scanned based on the provider's callsign".

That is certainly less intrusive because we're not trying to re-invent the wheel so-to-speak, and the extract needs to include the scanned callsign and all the rest, but I can add another field to the QAM map called "providerCallsign". Therefore, after a QAM scan in MythTV, the utility function just needs to match on providerCallsign and leave all the other fields alone, other than updating the stationID, so that we know which data to download.

Does that sound more workable?

rkulagow
SD Staff
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: MythTV QAM users: what format works best for you?

Post by rkulagow » Tue Nov 25, 2014 10:43 am

OK, I've implemented code which returns the following:

Code: Select all

{
    "qamMappings": [
        "1"
    ],
    "map": {
        "1": [
            {
                "channel": "24-131",
                "virtualChannel": "131",
                "modulation": "qam_256",
                "frequency": "225000000",
                "serviceID": "131",
                "stationID": "35451",
                "atscMajor": "1008",
                "atscMinor": "131",
                "providerCallsign": "MVIES"
            },
            {
                "channel": "24-202",
                "virtualChannel": "202",
                "modulation": "qam_256",
                "frequency": "225000000",
                "serviceID": "202",
                "stationID": "19600",
                "atscMajor": "1008",
                "atscMinor": "202",
                "providerCallsign": "FOXHD"
            },
            {
                "channel": "24-203",
                "virtualChannel": "203",
                "modulation": "qam_256",
                "frequency": "225000000",
                "serviceID": "203",
                "stationID": "20359",
                "atscMajor": "1008",
                "atscMinor": "203",
                "providerCallsign": "NBCHD"
            },
And then in the sd-utility program I'll create a QAM option where we update the channel table based on the provider callsign. This will preserve all the scanned values from mythtv-setup, and then update the xmltvid based on the provider callsign.

The "station" JSON that comes later could then update the callsign to the "official" Gracenote callsign, but I'm still thinking about that.

Matching based on provider callsign will let me make the routine generic enough to work with DVB and FTA scans.

rkulagow
SD Staff
Posts: 915
Joined: Tue Aug 14, 2007 3:15 pm

Re: MythTV QAM users: what format works best for you?

Post by rkulagow » Mon Dec 01, 2014 9:01 am

As more QAM lineups come in, I can see that I will need to offer multiple match types, so that's what I'm working on now. For faginbagin, it looks like a match based on provider call sign would work, because your provider has that information. I just got a scan where the provider has nothing, there are no atsc_major and minor values, so it was an old "eyeball correlation". In that case I'll have a match based on the channel "76-76".

Post Reply