View source for Mythexternrecorder v4l2 vlc

== Introduction ==

It is now understood that MythTV no longer supports the use of uncompressed video capture cards, if it indeed ever did. This page sets out a way of getting those signals into your recorded programs list using the recently introduced "mythexternrecorder" tool.

== Background ==

The MythTV wiki recommends against the use of uncompressed video capture cards because of the processor overhead involved in using the main CPU to compress the signal, rather than having a card that has hardware compression on board. However, if you have a powerful machine without too much load they will work perfectly well and are cheaper than the compressing cards.

The actual card used in this build was a [https://www.mythtv.org/wiki/Hauppauge_WinTV_HVR5525 Hauppauge WinTV-HVR5525] but a [https://www.mythtv.org/wiki/Hauppauge_ImpactVCB-e Hauppauge ImpactVCB-e] would work just as well.

== Setup ==

The standard mode of operation for the mythbackend service is that it runs as an unprivileged user, quite often the user "mythtv". This page will reference that user, so if your setup uses a different username, please adjust accordingly. You will be using this user for most of these operations. If you need to run a particular section as root it will be noted.

This procedure places the relevant operational files in the ".mythtv" subdirectory of the mythtv user (~mythtv/.mythtv or /var/lib/mythtv/.mythtv), again this is not vital, but the examples will include this path. Please adjust if you place them somewhere else.

A full and functioning MythTV setup is a prerequisite for this guide. It is more than adequately documented elsewhere on this wiki.

If you don't already have them installed, you will need the V4L Utilities. Under RH distros this is the package "v4l-utils". This supplies the tool "v4l2-ctl".

Please ensure that the card(s) is installed and that the relevant logs appear after a cold reboot, that the firmware is loading properly and that the following validation test succeeds.

== Validate your card ==

You should see something like the following in the boot log once you have cold rebooted the machine. ie not a reboot or a resume, but a boot from zero.

<pre>
... kernel: cx23885: CORE cx23885[1]: subsystem: 0070:7137, board: Hauppauge ImpactVCB-e [card=43,autodetected]
... kernel: tveeprom: Hauppauge model 71200, rev H4  , serial# 14123685 
... kernel: tveeprom: tuner model is None (idx 0, type 4)
... kernel: tveeprom: TV standards none (eeprom 0x00)
... kernel: tveeprom: audio processor is CX23888 (idx 40)
... kernel: tveeprom: decoder processor is CX23888 (idx 34)
... kernel: tveeprom: has no radio
... kernel: cx23885: cx23885[1]: warning: unknown hauppauge model #71200
... kernel: cx23885: cx23885[1]: hauppauge eeprom: model=71200
... kernel: cx25840 16-0044: cx23888 A/V decoder found @ 0x88 (cx23885[1]) 
... kernel: cx25840 16-0044: loaded v4l-cx23885-avcore-01.fw firmware (16382 bytes)
... kernel: cx23885: cx23885[1]: registered device video1 [v4l2]
... kernel: cx23885: cx23885[1]: registered device vbi1
... kernel: cx23885: cx23885[1]: alsa: registered ALSA audio device
... kernel: cx23885: cx23885_dev_checkrevision() Hardware revision = 0xd0
... kernel: cx23885: cx23885[1]/0: found at 0000:06:00.0, rev: 4, irq: 24, latency: 0, mmio: 0xf5200000
</pre>

The above example is for the Hauppauge ImpactVCB-e, which is a simple card without any other functionality on board. If you are using a more complex card where the analogue functionality is a small part of the whole, you will see more entries than this.

As the "mythtv" user perform the following additional test:

<pre>
[mythtv@mediabox ~]$ v4l2-ctl -d /dev/video1 --all
Driver Info:
	Driver name      : cx23885
	Card type        : Hauppauge WinTV-HVR5525
	Bus info         : PCIe:0000:05:00.0
	Driver version   : 5.8.8
	Capabilities     : 0x85230011
		Video Capture
		VBI Capture
		Tuner
		Audio
		Read/Write
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x05230001
		Video Capture
		Tuner
		Audio
		Read/Write
		Streaming
		Extended Pix Format
Priority: 2
Frequency for tuner 0: 0 (0.000000 MHz)
Tuner 0:
	Name                 : Television
	Type                 : Analog TV
	Capabilities         : 62.5 kHz stereo lang1 lang2 freq-bands 
	Frequency range      : 0.000 MHz - 0.000 MHz
	Signal strength/AFC  : 0%/0
	Current audio mode   : lang1
	Available subchannels: mono lang2 
Video input : 0 (Television: no signal)
Audio input : 2 (TV)
Video Standard = 0x00001000
	NTSC-M
Format Video Capture:
	Width/Height      : 720/480
	Pixel Format      : 'YUYV' (YUYV 4:2:2)
	Field             : Interlaced
	Bytes per Line    : 1440
	Size Image        : 691200
	Colorspace        : SMPTE 170M
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
Crop Capability Video Capture:
	Bounds      : Left 0, Top 0, Width 720, Height 480
	Default     : Left 0, Top 0, Width 720, Height 480
	Pixel Aspect: 11/10
Selection Video Capture: crop_default, Left 0, Top 0, Width 720, Height 480, Flags: 
Selection Video Capture: crop_bounds, Left 0, Top 0, Width 720, Height 480, Flags: 
Streaming Parameters Video Capture:
	Frames per second: 29.970 (30000/1001)
	Read buffers     : 2

User Controls

                     brightness 0x00980900 (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                       contrast 0x00980901 (int)    : min=0 max=127 step=1 default=64 value=64 flags=slider
                     saturation 0x00980902 (int)    : min=0 max=127 step=1 default=64 value=64 flags=slider
                            hue 0x00980903 (int)    : min=-128 max=127 step=1 default=0 value=0 flags=slider
                         volume 0x00980905 (int)    : min=0 max=65535 step=655 default=65024 value=65024 flags=slider
                        balance 0x00980906 (int)    : min=0 max=65535 step=655 default=32768 value=32768 flags=slider
                           bass 0x00980907 (int)    : min=0 max=65535 step=655 default=32768 value=32768 flags=slider
                         treble 0x00980908 (int)    : min=0 max=65535 step=655 default=32768 value=32768 flags=slider
                           mute 0x00980909 (bool)   : default=0 value=0
</pre>

If you get something like the above then you are ready to start setting up the mythexternrecorder tool

== Set up mythexternrecorder ==

=== Overview ===

The mythexternrecorder tool is one of a family of utilities provided in the standard MythTV install that respond to the "External Blackbox Recorder" type in mythtv-setup. They are run on demand by mythbackend when a channel is to be used either for LiveTV or for recordings. The backend will execute the command that is entered as part of the card entry creation in mythtv-setup. This is turn has a configuration file which includes another command that is the actual line driver.

The mythexternrecorder tool takes textual commands from mythbackend via its standard in and provides responses and debug via standard error and, when called on to do so by a command from mythbackend will fire up a second utility that reads from the video device, compresses the video and sound, multiplexes the data as a "Transport Stream" and sends the result out via its own stdout through a pipe back to mythexternrecorder, which then places the recording into your recordings directory. Any suitable utilities can be used for the pipe driver, but this example uses "vlc" to do this work. There are other examples in the MythTV wiki of "ffmpeg" being used instead.

=== Configuration ===

This section is run on the machine on which you have installed the card.

==== Create /usr/local/bin/MythExternRecorder ====

As user root, using your preferred text editor, create the following file:

/usr/local/bin/MythExternRecorder
<pre>
#!/bin/bash

## echo "`date`: /usr/bin/mythexternrecorder" "${@}" >> /var/log/mythexternrecorder.log

/usr/bin/mythexternrecorder -v all:debug --exec --conf /var/lib/mythtv/.mythtv/mythexternal.conf "${@}"
</pre>

NB the line starting "##" is for debug purposes and is not needed during normal operations. If you enable it you will need to make sure that /var/log/mythexternrecorder.log can be written by user "mythtv".

Once created make sure it is executable.

eg "chmod 755 /usr/local/bin/MythExternRecorder"

==== Create /var/lib/mythtv/.mythtv/mythexternal-channels.conf ====

Create a channels file for some of the channels you will be tuning using this card. Do at least two so that you have the ability to do a channel change for testing purposes. Your favourite EPG source can be used by mythfilldatabase to create and configure the rest of the channels later.

As user mythtv, using your preferred text editor create the following file:

/var/lib/mythtv/.mythtv/mythexternal-channels.conf
<pre>
[101]
NAME="A channel"
CALLSIGN=ACHANNEL
XMLTVID=12345
URL="101"
ICON="https://schedulesdirect-api20141201.s3.us-east-1.amazonaws.com/assets/stationLogos/s12345_dark_360w_270h.png"

[102]
NAME="Nother Channel"
CALLSIGN="NOTHCH"
XMLTVID=23456
URL="102"
ICON="https://schedulesdirect-api20141201.s3.us-east-1.amazonaws.com/assets/stationLogos/s23456_dark_360w_270h.png"
</pre>

This is obviously a bogus file as yor channels will differ from the one in this build, but the things to note are:
* The URL line needs to be present and needs to be the same as the INI section header (eg "[101]")
* The XMLTVID needs to be the correct name/number for that channel

==== Create /var/lib/mythtv/.mythtv/mythexternal.conf ====

As user mythtv, using your preferred text editor create the following file:

/var/lib/mythtv/.mythtv/mythexternal.conf
<pre>
[RECORDER]
# The recorder command to execute.  %URL% is optional, and
# will be replaced with the channel's "URL" as defined in the
# [TUNER/channels] (channel conf) configuration file
command="cvlc --no-disable-screensaver -I dummy \"v4l2://:dev=/dev/video0:input=1\" --input-slave=alsa://plughw:2,0 --sout \"#transcode{vcodec=mpeg1,acodec=mp4a,vb=3000,ab=256,width=920,height=576,venc=ffmpeg{keyint=80,hurry-up,vt=800000},deinterlace}:file{mux=ts,dst=-}\""

# Optional cleanup command which is executed when this recorder is shut down
#cleanup="/home/mythtv/bin/input-finished.sh"
cleanup="killall vlc"

# Used in logging events, %ARG% are replaced from the channel info
desc=cvlc \"%URL%\" \"%CHANNUM%\" \"%CHANNAME%\" \"%CALLSIGN%\"

[TUNER]
# An optional CONF file which provides channel details.
channels=/var/lib/mythtv/.mythtv/mythexternal-channels.conf

# If [TUNER/command] is provided, it will be executed to "tune" the
# channel. A %URL% parameter will be substituted with the "URL" as
# defined in the [TUNER/channels] configuration file. %CHANNUM% will be
# replaced with the channum provided by mythbackend
#command="/home/mythtv/bin/tune-channel.sh %URL% %CHANNUM%"

# If [TUNER/ondatastart] is provided, it will be executed as soon
# as data is seen from the [RECORDER/command] stdout. This can be
# used to do any final cleanup of the tunning operation.
#ondatastart="/home/mythtv/bin/control.py --leftplay 2"

# if [TUNER/newepisodecommand] is provided, it will be executed
# if a new episoded is starting up on the channel that is already
# tuned. In other words, if [TUNER/command] is skipped because
# the current channel is already correct, this will be executed
# instead.
#newepisodecommand="/home/mythtv/bin/control.py --touch %CHANNUM%"


# Timeout for changing channels in msecs
timeout=20000


[SCANNER]
# When MythTV scans for channels, The contents of the [TUNER/channels]
# config file are used to populate MythTV's channel information.
# If a command is provided here, it will be executed first, so it can
# populate the [TUNER/channels] config file
#command=/home/myth/bin/scan.sh "%CHANCONF%"

# Timeout for scan command in msecs
#timeout=60000
</pre>

NB
* If you are not using the video device "/dev/video0" then substitute the correct device above.
* If you are not using the alsa device "hw2,0" then substitute the correct device above
* If you are unsure which ALSA device you need then you will be able to tell by running up "tvtime", immediately exiting and noting the console output

eg:

<pre>
[mythtv@mediabox ~]$ tvtime
Running tvtime 1.0.10.
Reading configuration from /etc/tvtime/tvtime.xml
Reading configuration from /var/lib/mythtv/.tvtime/tvtime.xml
videoinput-auto: using analog TV device /dev/video0
Alsa devices: cap: hw:2,0 (/dev/video0), out: default

Thank you for using tvtime.
</pre>

==== Video Source ====

You will need to have a suitable EPG listing source already set up for the next step. In this example it is to be called "Blackbox". There are a variety of different sources for this information. For this build the services of "Schedules Direct" were employed. The advantage of using this project is that irrespective of the program source (DVB-T, Satelite, Set top box etc) the listings for the actual programs will be consistent across all sources, so when the backend is scheduling the recording timings, it is comparing like with like and you won't end up with several recordings of the same thing, with subtly different titles and descriptions, which would be a waste of disk space, and could potentially mean that something else gets bumped as there are insufficient channels available. It does cost, but it's not much and the money is for paying the licensing fees on some of the data they acquire. They are a non-profit. For further details see [https://www.schedulesdirect.org/aboutus Schedules Direct].

==== External channel changer ====

If you are pulling the programming in from an external device via composite or S-Video, you will likely need an external program to change the channels. This again is not part of this HOWTO as it varies from device to device. In this build the script is called "/usr/local/bin/zapper" and takes a single argument of the channel to be tuned.

eg /usr/local/bin/zapper 101

Please note that there is a place for this command in the mythexternrecorder config file, but it was found that putting it there leads to problems. It is better to put it in the device's "Input connection" in mythtv-setup and leave the command commented out in the mythexternrecorder config file.

=== mythtv-setup ===

This next section is within the mythtv-setup utility. Run this on the machine the card is in, as the user mythtv.

==== Create MythTV device ====

* Select "2. Capture Cards"
* Select "New capture card"
* Move to the new entry and select
* Select card type "External (black box) recorder" (right at the bottom)
* Press the right arrow to configure the card
* Under command path enter "/usr/local/bin/MythExternRecorder"
* Ensure that "File Info" acknowledges the file
* press left arrow to back out and ensure that you allow it to "Save and Exit"

==== Connect device to it's video source ====

* Select "5. Input Connections"
* Select the line for your new device. It should read "[EXTERNAL : /usr/local/bin/MythExternRecorder ]"
* Fill is as follows:
{| border="1" cellspacing="0" cellpadding="5" style="border-collapse:collapse; border-color:#8eabd0; background:#e7edf5"
|- style="background: lightsteelblue"
|+
!Input
!Value
|-
|Input name:
|MPEGTS
|-
|Display name:
|Blackbox
|-
|Video source:
|Blackbox
|-
|External channel change command:
|/usr/local/bin/zapper
|-
|Preset tuner to channel:
|101
|}

Once these are filled in you can use the "Scan for channels" option to fill in the channels table.

NB
* The channel chosen as the preset above is dependant on what channels you have available.
* The Display name is arbitrary. It was decided to name it after the video source for this build but you can call it what you like.
* The "Scan for channels" step also does something clever with the pidcache table in the database, so it is adviced that this is used at least once.

[TODO] If anyone has a clearer idea of what the "Scan for channels" step does to the pidcache, do please update this page, or insert external reference.

== Populate the EPG ==

You should now be ready to pull the program data down from your favourite EPG source using mythfilldatabase. This process is not documented here as it varies considerably by EPG provider.

== Start watching your programs ==

If you now go to the "Program Guide" in mythfrontend you should see your new channels with program data. Select one of the new channels and then select "Watch this channel", you should see the onscreen information go through the stages of unlocked, partial lock, lock and then the program you selected should play.

== Notes ==

=== The "vlc" command ===

The file /var/lib/mythtv/.mythtv/mythexternal.conf contains a line in the [RECORDER] section entitled "command". In the above example it reads:
<pre>
command="cvlc --no-disable-screensaver -I dummy \"v4l2://:dev=/dev/video0:input=1\" --input-slave=alsa://plughw:2,0 --sout \"#transcode{vcodec=mpeg1,acodec=mp4a,vb=3000,ab=256,width=920,height=576,venc=ffmpeg{keyint=80,hurry-up,vt=800000},deinterlace}:file{mux=ts,dst=-}\""
</pre>

==== screensaver ====

This flag is to prevent vlc from trying to disable the screensaver. As it will not be run in an environment with "DISPLAY" set, it won't work and produces a warning.

==== dummy ====

This flag and argument prevent vlc from running up the full UI, just the pipline.

==== v4l2.. ====

Ensure that the video device is the correct one and that the input section is set to the correct input number. In the card this build uses the TV input was 0, the Composite input was 1 and S-Video was 2.

==== input-slave ====

Ensure that the alsa device is set in line with the ALSA device created for this particular card. See the reference to "tvtime" above.

==== sout ====

This section is the real guts of the operation. It's what the rest of this process is built to support.

It is likely that other vcodec and acodec lines could be used, but these ones work, so it is advised that you at least start from here as a "known good" and experiment from there.

Do not change the mux (ts) or the destination (-) as both of these are necessary and the only option that will work.

Bear in mind that in order to be viewed live or recorded without buffer overflows, the compression has to run in real time so if you want anything trick done to the payable file, do this as a post recording job with mythtranscode or equivalent.

==== alternatives to vlc ====

There are other examples in this wiki and elsewhere that show ffmpeg being used instead of vlc. So if using vlc is a problem, this is an alternative.

==== test before use ====

It is advised that before tuning to a channel using this setup, the vlc line is tested in isolation.

* If possible connect the incoming analogue cable to an alternative known good display (ie directly into the back of a TV that can display analogue S-Video/Composite signals) to make sure that there is indeed a signal on the line and that the channel change command is doing what it should.
* Connect it back up to the card and execute the following as user mythtv

NB Copy the command from the mythexternrecorder config file and remove the backslash characters before the quote marks.

<pre>
[mythtv@mediabox ~]$ cvlc --no-disable-screensaver -I dummy "v4l2://:dev=/dev/video0:input=1" --input-slave=alsa://plughw:2,0 --sout "#transcode{vcodec=mpeg1,acodec=mp4a,vb=3000,ab=256,width=920,height=576,venc=ffmpeg{keyint=80,hurry-up,vt=800000},deinterlace}:file{mux=ts,dst=-}" > test.ts
</pre>

* Let the process run for 10 seconds or so then use <ctrl>-c to stop it
* Check the content of "test.ts" using ffmpeg

<pre>
[mythtv@mediabox ~]$ ffmpeg -i test.ts
ffmpeg version 4.2.4 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 10 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' --extra-cflags=' ' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librsvg --enable-libsrt --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzimg --enable-libzvbi --enable-avfilter --enable-avresample --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-libmfx --enable-runtime-cpudetect
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[mpegts @ 0x56317bfbdb40] PES packet size mismatch
Input #0, mpegts, from 'test.ts':
  Duration: 00:00:05.24, start: 0.400000, bitrate: 2779 kb/s
  Program 1 
    Metadata:
      service_name    : 
      service_provider: 
    Stream #0:0[0x64]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, bt470bg/smpte170m/bt709, progressive), 920x576 [SAR 128:115 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0xc8]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 232 kb/s
At least one output file must be specified
</pre>

* Check that the output you see for "Stream #0:0" and "Stream #0:1" look like the above. Other stream types will also work, but the above is known to work and will be a relatively light load on the CPU.

Good Luck :-)