tv_grab_zz_sdjson not including new episode info?

[rykr]

Runniing tvheadend and using the tv_grab_zz_sdjson grabber with my sd account. Was getting unexpected results on new episodes so ran the grabber myself to get the data and I see zero info on a show that I know to be new to indicate that it is new.

I'm wondering what I'm missing. Is this a problem with the grabber?

[rmeden]

XMLTV has a <previously shown> element.

From: notes on "new" vs "repeat"

tv_grab_zz_sdjson may do something similar to tv_grab_na_dd

Old versions of TV_GRAB_NA_DD simply created an empty previously-shown tag to flag a repeat. Now we place "original-air-date" as a start attribute. <previously-shown start=original-air-date>. Applications are encouraged to use previously-shown/start to augment repeat detection. In TV_CHECK, I consider "new" any episode less than 2 weeks old.

[rykr]

XMLTV has a <previously shown> element.

From: notes on "new" vs "repeat"

tv_grab_zz_sdjson may do something similar to tv_grab_na_dd

Old versions of TV_GRAB_NA_DD simply created an empty previously-shown tag to flag a repeat. Now we place "original-air-date" as a start attribute. <previously-shown start=original-air-date>. Applications are encouraged to use previously-shown/start to augment repeat detection. In TV_CHECK, I consider "new" any episode less than 2 weeks old.

Thanks for the reply. It seems tv_grab_zz_sdjson is not doing that.

[rpcameron]

If I remember correctly, the problem is Tvheadend does not treat the program as new if there is a previously-shown element. Also, because of the way that dates/times are ambiguously defined in the XMLTV DTD, sometimes you may end up with a previously-shown date that occurs after the program's air date.

This is compounded by XMLTV treating the "new" element as meaning "this program has been aired on a different channel, and this is the first time it is being aired on this channel". However, Tvheadend treats the new element as one might expect: this is the first broadcast of this program on this channel.

Over at the Tvheadend forums I've posted patches for both tv_grab_zz_sdjson and tv_grab_zz_sdjson_sqlite that fixes these issues. What I've done is suppressed the generation of the previously-shown element if that airing of the program is marked "new" in th schedule.

I also filed a bug report about this with the sqlite version of the grabber, but the author has indicated that he is not going to change it because he wishes to closely follow the XMLTV DTD. While I understand his perspective, it makes it difficult to use with Tvheadend.

The forum posting with my patches is here: tv_grab_zz_sdjson patches. I also have an updated patch for the current version of the sqlite grabber that I have yet to post on that thread, but will do so later today.

[rpcameron]

Here are my tv_grab_zz_sdjson and tv_grab_zz_sdjson_sqlite patches. The first is for the version of the grabber that is included with the xmltv-utils package. The second is for the current (21 July 2017, v1.33) version from the author's GitHub repo.

tv_grab_zz_sdjson.patch:

Code: Select all

--- a/tv_grab_zz_sdjson   2017-02-03 15:19:59.000000000 -0800
+++ b/tv_grab_zz_sdjson   2017-05-16 11:55:37.426905237 -0700
@@ -1339,9 +1339,14 @@
 # The xmltv docs state this field is "When and where the programme was last shown".
 # However mythtv expects the original air date to be in this field.
 sub get_program_previously_shown {
-   my ($details) = @_;
+   my ($program, $details) = @_;
    my %previously_shown;
 
+   my $new = $program->{'new'};
+   if($new) {
+      return undef;
+   }
+
    my $date = $details->{'originalAirDate'};
    if($date) {
       my $dt = parse_original_airdate($date);
@@ -1457,7 +1462,7 @@
          'episode-num'      => get_program_episode($program, $details),
          'video'            => get_program_video($program),
          'audio'            => get_program_audio($program),
-         'previously-shown' => get_program_previously_shown($details),
+         'previously-shown' => get_program_previously_shown($program, $details),
          'premiere'         => get_program_premiere($program),
 #         'last-chance'      => undef,
          'new'              => get_program_new($program),


tv_grab_zz_sdjson_sqlite.patch:

Code: Select all

--- a/tv_grab_zz_sdjson_sqlite   2017-10-03 09:57:08.576347223 -0700
+++ b/tv_grab_zz_sdjson_sqlite   2017-09-22 13:22:00.512171274 -0700
@@ -1777,7 +1777,13 @@
 
         # XMLTV uses their standardized dates, while Schedules
         # Direct uses YYYY-MM-DD
-        if (defined($programDetails->{'originalAirDate'}))
+        # If "new", then do not generate previously-shown
+        if (defined($scheduleDetails->{'new'}))
+          {
+            my $new = $scheduleDetails->{'new'};
+            $w->emptyTag('new');
+          }
+        elsif (defined($programDetails->{'originalAirDate'}))
           {
             my $originalAirDate = $programDetails->{'originalAirDate'};
             my $offset = ' +0000';


[rkulagow]

You may want to post XMLTV-specific patches / code / discussion on the various XMLTV lists.

https://sourceforge.net/p/xmltv/mailman ... nth=201710

Would be a good starting point.

[rpcameron]

You may want to post XMLTV-specific patches / code / discussion on the various XMLTV lists.

https://sourceforge.net/p/xmltv/mailman ... nth=201710

Would be a good starting point.

The problem is that there isn't a good 1:1 mapping of of the data to XMLTV elements; also, the use of the elements that seem to match up is different in some cases. The problem isn't necessarily with the XMLTV, but that presently it is the only way to get SD data into Tvheadend, and in doing so things get "lost in translation".

I've posted them on the Tvheadend forum where they are more relevant, as the grabber author did not feel this was a real issue as it is slightly contrary to the XMLTV spec.