m3u-epg-editor

An m3u / epg file optimizer script written in python

m3u-epg-editor enables download of m3u / epg files from a remote (or local) server and enables the channel groups and channels within these files to be trimmed, optimized and reordered

These features might be useful where:

1. You want to achieve a filtered list and an improved custom sort order of TV channels
2. You have an underpowered device where SPMC / KODI / some other app running on that device struggles to download and process very large m3u / epg files
3. Your service provider supplies you with a url that returns an m3u file containing both live TV channels and VOD content in that one file and you want to filter it to contain only live TV channels

This script has been tested with the following IPTV providers:

1. VaderStreams
2. FabIPTV
3. NecroIPTV

VaderStreams do did support a unique feature to enable filtering groups within their m3u in the HTTP GET request/response via a filterCategory query string parameter. However there are some issues that are common to all IPTV service providers where there is no obvious, easy or free solution:

1. There is no method to remove specific channels within categories / groups
2. There is no method to re-order / sort channels within categories / groups to achieve a desired custom sort order
3. There is no method to reduce the volume of data within the epg to include only those channels that are required
4. There is no method to reduce the time window of data within the epg

There are commercially available online services that can solve these problems for a monthly / yearly subscription free.

m3u-epg-editor solves these problems for free on your own network / computer(s).

---

dependencies:

python

• m3u-epg-editor-py3.py is intended for use with Python v3.x. m3u-epg-editor-py3.py is currently being maintained.
• m3u-epg-editor-py2.py is / was intended for use with Python v2.7. Official support for Python v2.7 ended on 01/01/2020, ref: python/devguide#344 and https://www.python.org/dev/peps/pep-0373/. m3u-epg-editor-py2.py is no longer being maintained in this repo, please do not request any further maintenance changes to m3u-epg-editor-py2.py.

Python installers can be downloaded from the official python website: https://www.python.org/downloads/. In linux, Python can also be installed from a package repository with a package manager i.e. apt, yum etc or a software manager i.e. synaptic

python modules used by this script:

import sys
import os
import argparse
import json
import ast
import requests
import io
import re
import shutil
import gzip
from lxml.etree import Element, SubElement, parse, XMLParser
from xml.etree.ElementTree import tostring
import datetime
import dateutil.parser
import tzlocal
from urllib.request import url2pathname
from traceback import format_exception

installing required modules:

The majority of the modules mentioned above will be installed and available by default because they are a core part of Python

Python modules that are not included as part of a Python installation that do need to be installed are:

• requests
• lxml
• python-dateutil
• tzlocal

pip install -r requirements.txt / pip install requests lxml python-dateutil tzlocal

---

command line options:

$ python ./m3u-epg-editor-py3.py --help
usage: m3u-epg-editor-py3.py [-h] [--json_cfg [JSON_CFG]] [--m3uurl [M3UURL]] [--epgurl [EPGURL]]
                             [--request_headers [REQUEST_HEADERS]]
                             [--groups [GROUPS]] [--groupmode [GROUPMODE]]
                             [--discard_channels [DISCARD_CHANNELS]]
                             [--include_channels [INCLUDE_CHANNELS]]
                             [--discard_urls [DISCARD_URLS]]
                             [--include_urls [INCLUDE_URLS]]
                             [--id_transforms [ID_TRANSFORMS]]
                             [--group_transforms [GROUP_TRANSFORMS]]
                             [--channel_transforms [CHANNEL_TRANSFORMS]] [--range [RANGE]]
                             [--sortchannels [SORTCHANNELS]] [--xml_sort_type [XML_SORT_TYPE]]
                             [--tvh_start [TVH_START]] [--tvh_offset [TVH_OFFSET]] [--no_tvg_id]
                             [--no_epg] [--force_epg] [--no_sort] [--http_for_images]
                             [--preserve_case] [--outdirectory [OUTDIRECTORY]]
                             [--outfilename [OUTFILENAME]] [--log_enabled]

download and optimize m3u/epg files retrieved from a remote web server

optional arguments:
  -h, --help            show this help message and exit
  --json_cfg [JSON_CFG], -j [JSON_CFG]
                        A json input configuration file containing argument values. (default: None)
  --m3uurl [M3UURL], -m [M3UURL]
                        The url to pull the m3u file from. Both http:// and file:// protocols are
                        supported. (default: None)
  --epgurl [EPGURL], -e [EPGURL]
                        The url to pull the epg file from. Both http:// and file:// protocols are
                        supported. (default: None)
  --request_headers [REQUEST_HEADERS], -rh [REQUEST_HEADERS]
                        An optional json array of key value pairs representing any required HTTP
                        header values to be sent in m3u and epg HTTP requests (default: [])
  --groups [GROUPS], -g [GROUPS]
                        Channel groups in the m3u to keep or discard. The default mode is to keep
                        the specified groups, switch to discard mode with the -gm / --groupmode
                        argument (default: None)
  --groupmode [GROUPMODE], -gm [GROUPMODE]
                        Specify "keep" or "discard" to control how the -g / --group argument should
                        work. When not specified, the -g / --group argument behaviour will default
                        to keeping the specified groups (default: keep)
  --discard_channels [DISCARD_CHANNELS], -dc [DISCARD_CHANNELS]
                        Channels in the m3u to discard. Regex pattern matching is supported
                        (default: None)
  --include_channels [INCLUDE_CHANNELS], -ic [INCLUDE_CHANNELS]
                        Channels in the m3u to keep. Regex pattern matching is supported. Channels 
                        matched in this argument will always be kept, effectively overriding of 
                        any other group or channel or url exclusion configuration. (default: None)                        
  --discard_urls [DISCARD_URLS], -du [DISCARD_URLS]
                        Urls in the m3u to discard. Regex pattern matching is supported
                        (default: None)
  --include_urls [INCLUDE_URLS], -iu [INCLUDE_URLS]
                        Urls in the m3u to keep. Regex pattern matching is supported. Urls 
                        matched in this argument will always be kept, effectively overriding of 
                        any other group or channel or url exclusion configuration. (default: None)                                           
  --id_transforms [ID_TRANSFORMS], -it [ID_TRANSFORMS]
                        A json array of key value pairs representing source channel name values to
                        target tvg-id values to be transformed at processing time (default: [])
  --group_transforms [GROUP_TRANSFORMS], -gt [GROUP_TRANSFORMS]
                        A json array of key value pairs representing source group names to target
                        groups names to be transformed at processing time (default: [])
  --channel_transforms [CHANNEL_TRANSFORMS], -ct [CHANNEL_TRANSFORMS]
                        A json array of key value pairs representing source channel names to target
                        channel names to be transformed at processing time (default: [])
  --range [RANGE], -r [RANGE]
                        An optional range window hours value used to validate programmes added to
                        the newly generated epg xml (default: 168)
  --sortchannels [SORTCHANNELS], -s [SORTCHANNELS]
                        The optional desired sort order for channels in the generated m3u (default:
                        None)
  --xml_sort_type [XML_SORT_TYPE], -xs [XML_SORT_TYPE]
                        Specify "alpha" or "m3u" to control how channel elements within the
                        resulting EPG xml will be sorted. When not specified channel element sort
                        order will follow the original source xml sort order (default: none)
  --tvh_start [TVH_START], -ts [TVH_START]
                        Optionally specify a start value to initialise the absolute start of
                        numbering for tvh-chnum attribute values (default: None)
  --tvh_offset [TVH_OFFSET], -t [TVH_OFFSET]
                        An optional offset value applied to the Tvheadend tvh-chnum attribute
                        within each channel group (default: None)
  --no_tvg_id, -nt      Optionally allow channels with no tvg-id attribute to be considered as
                        valid channels (default: False)
  --no_epg, -ne         Optionally prevent the download of and the creation of any EPG xml data
                        (default: False)
  --force_epg, -fe      Works in tandem with no_tvg_id and no_epg. When EPG processing is enabled
                        and when this option is specified as true, the generated EPG file will be
                        populated with elements for channels in the m3u file that normally would
                        have no EPG data (default: False)
  --no_sort, -ns        Optionally disable all channel sorting functionality (default: False)
  --http_for_images, -hi
                        Optionally prevent image attributes being populated where the source
                        contains anything other than a http url i.e. data:image uri content
                        (default: False)
  --preserve_case, -pc  Optionally preserve the original case sensitivity of tvg-id and channel
                        attributes as supplied in the original M3U and EPG file data through to the
                        target newly generated M3U and EPG files (default: False)
  --outdirectory [OUTDIRECTORY], -d [OUTDIRECTORY]
                        The output folder where retrieved and generated file are to be stored
                        (default: None)
  --outfilename [OUTFILENAME], -f [OUTFILENAME]
                        The output filename for the generated files (default: None)
  --log_enabled, -l     Optionally log script output to process.log (default: False)

Supplying arguments via a JSON configuration file:

All runtime arguments can be supplied via the file described by the --json_cfg / -j path. There is a sample JSON configuration file here. When arguments are supplied via a JSON configuration file in the --json_cfg / -j argument, all arguments are expected to passed in that JSON configuration file and any arguments that are passed via the cli will be ignored. In other words, use --json_cfg / -j for all arguments or pass all individual arguments via the cli but do not attempt to use both.

sample usage calls (urls intentionally incomplete):

VaderStreams:

$ python ./m3u-epg-editor-py3.py -m="http://xxx.xxx.xxx/vget?username=<USERNAME>&password=<PASSWORD>&format=ts" -e="http://xxx.xxx/p2.xml.gz" -g="'sports','premium movies'" -ic="'willow hd','bein sports espanol hd'" -s="'sky cinema drama hd','sky cinema comedy hd','sky cinema villians hd','sky cinema premiere hd','sky cinema sci-fi & horror hd','sky cinema thriller hd','sky cinema select hd','sky cinema family hd','sky cinema disney hd'" -r=12 -d="/home/target_directory" -f="output_file"

FabIPTV:

$ python ./m3u-epg-editor-py3.py -m="http://xxx.xxx:8080/get.php?username=<USERNAME>&password=<PASSWORD>&type=m3u_plus&output=ts" -e="http://xxx.xxx:8080/xmltv.php?username=<USERNAME>&password=<PASSWORD>" -g="'uk + 1 channels','uk bt sport','uk documentaries','uk entertainment','uk movies','uk other sports','uk sky sports'" -ic="'dave hd'" -s="'bbc one +1','bbc two +1','itv +1','itv 2 +1','itv 3 +1','itv encore +1','itv4 +1','itvbe +1','channel 4 +1','channel 5 +1','sky living +1','sky1 +1'" -r=12 -d="/home/target_directory" -f="output_file"

---

files created by this script:

Each time this script is run, the following files will be created / overwritten in the specified --outdirectory / -d path:

• original.m3u

  This is the original unmodified m3u file downloaded from the specified --m3uurl / -m server

• original.channels.txt

  This is a raw text file log that contains an unfiltered list of all channel names from the original m3u

• original.gz

  This is the original unmodified epg gzip file downloaded from the specified --epgurl / -e server. No EPG processing is performed when the optional --no_epg / -ne argument is supplied.

• original.xml

  This is the original unmodified epg xml file extracted from the original epg gzip file. No EPG processing is performed when the optional --no_epg / -ne argument is supplied.

• [--outfilename].m3u

  This is the new rewritten m3u file created from the original m3u file. This will contain all of the channels that you've decided to keep. Channels are optionally sorted according to the sort order specified in --sortchannels / -s. If --no_sort / -ns is specified, anything supplied in --sortchannels / -s is ignored.

• [--outfilename].channels.txt

  This is basically a raw text file log containing the list of channel names from the original m3u that you've decided to keep. This can be useful for constructing a desired --sortchannels / -s sort order.

• [--outfilename].xml

  This is the new rewritten epg file created from the original epg file that contains epg data for all of the channels that you've decided to keep. If --range / -r was specified, epg data will be filtered to only include entries that fall within the range window of range_start <= programme_start <= range_end. No EPG file will be created when the optional --no_epg / -ne argument is supplied.

• no_epg_channels.txt

  This is a log of any channels that you decided to keep that were subsequently found to have no epg data available. This can be useful to help to construct a list of --channels / -c channels to exclude if i.e. you only want to keep those channels where epg data is available