Thursday, October 18, 2018

A quick (and probably incomplete) guide to getting kiwirecorder and kiwiwspr (now wsprdaemon) running on Linux

What follows is likely to be quite incomplete - but it contains the major steps in installing the kiwirecorder and kiwiwspr packages on a Linux machine.  I did my installation on Ubuntu, but similar may be done on other distros, such as Debian - including on the Raspberry Pi platform.

I don't claim to be an expert in such things and it is likely that I missed a few steps that may be specific to some distros/platforms - but the phrase "Google is your friend" is applicable if you run into trouble as it's likely that I can't help in your specific case.

What is "kiwirecorder"?

The kiwirecorder package works in conjunction with a "KiwiSDR" (see KiwiSDR.com for more information) - a relatively inexpensive (approx. $300) stand-alone, Linux based HF receiver capable of tuning from a few kHz to over 30 MHz.  Intended to be remotely accessed, it sports a Web-based user interface that provides both audio and a waterfall display along with the capability of demodulating various modes (SSB, CW, AM, FM) and decoding several different types of transmissions, including CW, FAX, FSK and WSPR.  Depending on configuration it is possible to have up to eight "users" connected, via the Internet, to a single KiwiSDR, each using their own frequency and mode settings.

While capable, it may be that this interface isn't quite what you need:  Perhaps you need an audio stream to feed somewhere else - maybe, to another computer that is decoding a mode that is not "built in" to the KiwiSDR or to record the goings-on on a certain frequency:  This is where the "kiwirecorder" comes in.

The kiwirecorder script makes a minimalist connection the the KiwiSDR, not requiring the user interface (display, waterfall, etc.) providing only a streaming audio connection with the frequency and mode specified in the connecting URL.  This stream may then be processed by another computer connected to it (locally and/or via the Internet) to do what it is that you wish to do.

Exactly how this works and its options are beyond the scope of this article, but the source code and documentation may be found here:

https://github.com/hcab14/kiwiclient

What is "kiwiwspr"? 

Note:  The "kiwiwspr" package has been renamed "wsprdaemon" as it is no longer specific only to the kiwirecorder package - see the addendum at the bottom of this posting.

While the KiwiSDR itself is capable of decoding WSPR transmissions, doing so takes a lot of its processing power and if there are many WSPR transmissions to be decoded within a passband - such as 20 or 30 meters during band openings - it may "miss" many opportunities if there isn't enough processing power to go through all of the latent signals.  Additionally, in a multi-user environment, having several WSPR decoders running simultaneously causes the normal user interface to slow noticeably - particularly with respect to the update rate of the waterfall display.

By "piping" audio to another, more capable computer, more effective WSPR decoding may be done by recording about 110 seconds of audio starting at the beginning of an even-numbered minute and then passing that file to the "wsprd" (wspr decoder) program which is part of the WSJT-X suite.  Using an outboard computer for this task minimizes the load on the KiwiSDR itself and with the faster, better hardware of the remote computer, more "in depth" processing may be done to allow the recover of more weak signals than may be possible with the KiwiSDR's built-in decoder - and be able to do so more quickly, with the latest version of the software.

Installing kiwirecorder and kiwiwspr 

The following assumes that you have a running Ubuntu Linux computer running on a network that has access to the KiwiSDR.  Ideally, such a computer would be placed on a local network, but this is not required.
  • Be sure that the computer in question is reasonably up-to-date and has installed on it Python.  If Python is not already installed, the steps on doing so are readily found on the web. 
  • The computer in question must have access to means of keeping the clock within +/- 2 seconds of UTC.  This is usually done by via NTP or similar.  If the computer's clock is not held to within a couple seconds of UTC, WSPR decoding will fail.
  • It is highly recommended that the KiwiSDR's GPS antenna be placed outside in a location with a clear view of the sky and that its GPS lock is verified.  By supplying the KiwiSDR a valid GPS signal both its frequency and time may be held to good accuracy.
The steps:
  • Log into the computer as the user that you wish to run the kiwirecorder and related software.  It is generally a good idea not to be logged in as a user with root access when you are running scripts.
  • Install "kiwiclient" and change to its directory after install:
    • git clone https://github.com/jks-prv/kiwiclient
    • cd kiwiclient
  •  Install Python dependencies for the scripts:
    • sudo apt-get install python-pip
    • sudo pip install numpy scipy
    • (There are other ways to install "numpy" and "scipy", but the above works fine)
  • Install wsprd (WSPR decoder for linux):
    • sudo add-apt-repository -y ppk:ki7mt/wsjtx-next
    • sudo apt-get update
    • sudo apt-get install wsjtx
    • Important: 
      • If the version that you need isn't available via the repository and/or a new version of WSPR has recently released, it may not be available via the repository as an installation package.  To manually install a newer/different version, do the following:
        • Download the most current binary that is appropriate for your system.  Binaries for the most common distros are typically found here:  https://physics.princeton.edu/pulsar/k1jt/wsjtx.html
        • In the example this package is called "wsjtx_2.0.0-rc3_amd64.deb", but you would choose the appropriate binary for your system.
          • As of the time of this writing, only "1.9" is available via an install package and version 2.0 - which offers slightly better performance - has to be manually isntalled.
        • Copy the binary to a directory on the target machine where you have access.
        • In the location where you copied the file do:  sudo dpkg -i wsjtx_2.0.0-rc3_amd64.deb  (You would substitute the name of the file that you downloaded and copied to the Linux machine)
        • You may get an error saying that "libgfortran3" is not installed:  This package may be installed thusly:  sudo apt-get install libgfortran3
        • After you install libdfortran3, the wsjtx package may automatically complete installation, but if not (or if in doubt) try reinstalling it using the "sudo dpkg..." step, above
  • Install bc:
    • sudo apt-get install bc
  •  Go into "tools" subdirectory (kiwiclient/tools)
    • cd tools
  • Find the newest "kiwiwspr" script and copy it to the "tools" director.
    • While a version of the "kiwiwspr" script is included with kiwiclient, it may not be the newest.
      • Go to this discussion on the ValentFX.com site and look near the end for the newest kiwiwspr script.   This same discussion also provides clues and help for troubleshooting.
    • Make sure to CHMOD the new kiwiwspr script (if you updated it) to 774 so that it may be executed.
This should complete the installation.  The directory of interest will always be "kiwiclient/tools".

Let us assume for what follows that the kiwiwspr script is called (or has been renamed to) "kiwiwspr.sh"
  • Execute the script once to get the "default" configuration file:
    •  ./kiwiwspr.sh
    • If all goes right, you will get a message stating that a "default" configuration file has been created.
  • Copy the resulting file ("kiwiwspr.conf") and rename it as something else (e.g. "kiwiwspr.conf.original) so that you can refer to it later.
  • Edit the file to include one's own specific KIWI information.  Each of the items below is required!
    • At the top under "KIWI_LIST", enter the required information, separated by at least ONE space:
      • "OurID" - Name of the KIWI (no spaces).  It is this name that will be used to refer to this particular KiwiSDR in the schedule described below.
      • "IP:PORT" IP address (numerical) and the KiwiSDR's web interface port port (typically 8073)
      • "Mycall" - the call by which WSPR decodes are reported:  This is the "reporter" callsign on wsprnet.org
      • "MyGrid" - the 4 or 6 character Maidenhead grid location.
      • "KiwiPassword" - This is the password, if any, to access the Kiwi if it is not public.  You must put the word "NULL" here if there is no password - it cannot be left blank.
      • Note:  Several different KiwiSDRs may be defined in this table:  Each KiwiSDR's definition is as above, surrounded by quotes, on its own line.
      • As an example, one might have entered, on its own line, the following:
        • "my_kiwi 192.168.1.234:8073 w7xyz-1 cn01dc NULL"  This means that a receiver called "my_kiwi" can be reached via the IP address of 192.168.1.234, port 8073 and it will report to wsprnet using the callsign of "w7xyz-1" using the grid square of "cn01dc" and will not require a password.
    • There may be a table labeled "CAPTURE_JOBS":  This is no longer used and it may be ignored.  (It's recommended that you leave it as-is for now if it is there.)
    • There is a table called "WSPR_SCHEDULE" that defines on what frequencies, with which KiwiSDR instance and when the WSPR decoding "jobs" are to be scheduled in the following format, each on its own line with quotes:
      • "time kiwi_name,first_band kiwi_name,second_band kiwi_name, third_band"
      • Or, using the example above:
        • "15:00 my_kiwi,80 my_kiwi,40 my_kiwi,30 my_kiwi,20"
          • This would cause "my_kiwi" to start decoding WSPR on the 80, 40, 30 and 20 meter bands at 15:00 local time.
        • You can also set the schedule relative to you sunrise and sunet time, the location of the sunrise/sunset being determined by the grid square that you entered.  This can be useful if you have only a limited number of audio instances available and you wish to favor the bands that are best in the daytime or nighttime.  Example of this are:
          • "sunset-01:15 my_kiwi,80 my_kiwi,40"  - This would start WSPR decoding sessions on 80 and 40 meters at 1 hour and 15 minutes (e.g. 01:15) before local sunset as indicated by the "minus" sign after "sunset".
          • "sunrise+2:30 my_kiwi,20 my_kiwi,15" - This would start the WSPR decoding sessions on 20 and 15 meters at 2 hours and 30 minutes after local sunset as indicated by the "plus" sign after "sunrise".
        • If you wish, you may also specify UTC (GMT) instead of local time, in which case the you would add ",UDT" after the numerical time as in:
          • "01:00,UDT my_kiwi,80 my_kiwi,40" - This would start the WSPR sessions on 80 and 40 meters at 0100 UTC.  Times relative to Sunrise/sunset would not apply here.
Options:

As of version 1-1g there is an optional variable that can be specified called "KIWIRECORDER_CLIENT_NAME".  One would include a link in the kiwiwspr.conf" file like this:

declare KIWIRECORDER_CLIENT_NAME=(
    "My_Kiwirecorder"
)

The above will cause the text "My_Kiwirecorder" to show up as the name of the use on the KiwiSDR instead of "kiwirecorder.py".  Do note that this needs to be placed at or near the top of the kiwiwspr.conf file or else it may not have an effect.

Bands/Frequencies:

Valid bands, their base RF frequencies and receive passbands are:
  • BAND  DIAL FREQ  RF (carrier) Frequency range
  • 2200     136.0kHz         (137.3-137.7 kHz)
  • 630       474.2 kHz        (475.5-475.9 kHz)
  • 160       1836.6 kHz      (1837.9-1838.3 kHz)
  • 80         3568.6 kHz      (3569.9-3670.3 kHz)
  • 80eu     3592.6 kHz      (3593.9-3594.3 kHz)  Deprecated - Usage to be discontinued.
  • 60         5287.2 kHz      (5288.5-5288.9 kHz)  Not on a U.S. channel.
  • 60eu     5364.7 kHz      (5366.0-5366.4 kHz)  Not on a U.S. channel.
  • 40         7038.6 kHz      (7039.9-7040.3 kHz)
  • 30         10138.7 kHz    (10140.0-10140.4 kHz)
  • 20         14095.6 kHz    (14096.9-14097.3 kHz)
  • 17         18104.6 kHz    (18105.9-18106.3 kHz)
  • 15         21094.6 kHz    (21095.9-21096.3 kHz)
  • 12         24924.6 kHz    (24925.9-24926.3 kHz)
  • 10         28124.6 kHz    (28125.9-28126.3 kHz)
Notes about the bands/frequencies listed above:
  •  The "80eu" band is still used in some parts of Europe and by other amateurs who have yet to change:  Use of this frequency is deprecated and will eventually cease.
  • The "60" meter band reflects the available of that in much of the world while the "60eu" band is better-suited for use in many European countries.  Because 60 meter frequencies/channels differ, you should pick the frequency that reflects the geographical area of interest when it comes to receiving WSPR transmissions.  (Note that none of the listed 60 meter frequencies correlate with the channels available to U.S. amateurs.)
  • The "Dial Freq" is that to which one would tune using USB while the "RF" frequency range is that which this script will use to "look" for WSPR signals.
  • The audio band 1300-1700 Hz is used for decoding and offset should be applied to determine the actual transmitted/received carrier frequencies.  (e.g. for 40 meters, RF frequencies of 7039.9-7040.3 are in the passband.) 
  • The detection bandwidth is 400 Hz instead of the nominal 200 Hz to accommodate "out of band" WSPR operation, frequency drift of your receiver and others' transmitters and the effects of the audio filters applied by the Kiwi.
If you get errors:
  • If a package didn't install, you'll get errors indicating such.  Because there are many different possible scenarios with your operating system, I will not cover them here.  Remember:  Google is your friend when it comes to helping solve specific errors.
  • If, when you run the "kiwiwspr" script, you get errors about missing elements, etc., these are typically due to data that is expected (like band) but is missing.  Re-check to make sure that:
    • ALL of the required fields are entered in the "kiwiwspr.conf" file.
    • That there are NO spaces within names or times.
    • That there are NO spaces after commas
    • That the line with the definition starts and ends with a double quote (e.g. " ) - even if the definition itself spans several lines.

Starting the script:

After installation, you should be in the "kiwiclient/tools" directory.

Assuming that the script is called, to start ALL jobs, do:

./kiwiwspr.sh -j a    - Start ALL jobs
./kiwiwspr.sh -j z    - Stop ALL jobs
./kiwiwspr.sh -j s    - Show running jobs

At this point there should be no errors and you should see "users" called "kiwirecorder.py" appearing on the KiwiSDR on the bands you specify for the most recent time before the current time.  For example, if it is 11:00 and you have two jobs - one scheduled for 09:00 and another for 17:00, the 09:00 schedule will be executed.

If all goes well, you should then start the watchdog, which will allow the schedule to be executed.  If you do not start the watchdog, jobs that were running at the time you started them will continue to run and they will not change according to the schedule.

The watchdog is started/checked/stopped with the following commands:

./kiwiwspr.sh -z a      - Start all watchdog jobs
./kiwiwspr.sh -z z      - Kill all watchdog jobs
./kiwiwspr.sh -z s      - Show watchdog jobs

Use "kiwiwspr.sh" with no arguments for more information about the above commands - and for more information.

If all goes well you will, after 5-10 minutes - assuming that your KiwiSDR is connected to a working antenna and there are signals on the (bands) that you have selected and you have Internet connectivity - start seeing WSPR reports on "wsprnet.org" using the callsign specified in the "MyCall" field in the "kiwiwspr.conf" file in the "KIWI_LIST".

Remember:  A WSPR receiving interval starts at the beginning of each even minute and runs for almost 2 minutes.  It will then take another minute or so for wsprd to decode the WSPR transmissions and forward them to wsprnet.org where they may be displayed.

Final comments:


Remember that the above steps are for an up-to-date Ubuntu Linux distro running on a PC platform and there may be differences in how to install it on other distros/hardware platforms.  If you encounter difficulties, I probably won't be able to help:  I'd just do what you could do in that situation, and that is to refer to Google for a solution to the same/similar problem.

I would like to thank Rob Robinett for his hard work in producing the kiwiwspr script along with John Seamons for his very hard work on the KiwiSDR itself and making the "kiwirecorder" package available.

Best of luck!

* * * * * * * * * * *
[8 February, 2019]

Update:  "kiwiwspr" is now "wsprdaemon"

The "kiwiwspr" package has since been renamed the "wsprdaemon" package as its usefulness has been expanded beyond simply using a KiwiSDR.  As of the time of this update (8 February) its "new" capabilities include:
  • Elimination of "duplicate" reception reports from "dirty" transmitters or from modulation caused by noise blankers:  Only the strongest report is used.
  • Allowing the use of multiple receivers sharing the same frequency.  This is useful if one has several antennas and wish to submit only the report of the "best" signal of the lot.
  • Using audio from a receiver via a sound card.
Even more features may have been added since this page was updated.
 
The configuration of the "multi receiver" aspect is documented in the "wsprdaemon.conf" that accompanies the script while the use of an audio input is beyond the scope of this document.

The modification has required some changes to the way the wsprdaemon package is installed as compared to old "kiwiwspr":
  • The "~/wsprdaemon" directory is created and the wsprdaemon .sh and .conf files are put into it
  • A "/dir/tmp/wspr-captures" directory is created.  If using a Raspberry Pi, this directory should use tmpfs file system or else you will "wear out" the SD card.  If using a computer with an SSD, you may still consider doing this.
  • Use CHMOD to make sure that the wsprdaemon.sh file may be executed
  • The configuration from the old "kiwiwspr.conf" file may be copied to, "wsprdaemon.conf" but the array "KIWI_LIST" should be renamed "RECEIVER_LIST".
    • The newer "wsprdaemon.conf" includes a the ability to define in RECEIVER_LIST a virtual receiver consisting of several receivers.  If more than one receiver decodes the same transmission, only the reception with the best report is used.  (Even if a single receiver is used, this eliminates multiple reports as noted above.)
    • For example, if you have defined "RX1" and "RX2" you can define "MERGED_RX1" as using "RX1" and "RX2".  One can use that virtual receiver name (e.g. "MERGED_RX1") in the scheduling and band definition.
  • If you have previously been using "kiwirecorder", copy its directory tree into the "~/wsprdaemon" directory.
    • If necessary, rename it from "~/wsprdaemon/kiwirecorder" to "~/wsprdaemon/kiwirecorder-jks-v0.1"
  • To start wsprdaemon, use "./wsprdaemon -a"
  • To stop wsprdaemon, use  "./wsprdaemon -z"

This page stolen from ka7oei.blogspot.com

[End]


2 comments:

  1. Thank you for your excellent installation instructions. Can I refer to them in that forum thread?
    Also, I want to encourage kiwiwspr users to install wsjt-x v2.0rc3. kiwiwspr-1-1g.sh and newer exploit the sensititity of the wsppd v2.0 to increase the number of spots by 10% or more.

    ReplyDelete
    Replies
    1. Hi Rob - thanks again for your hard work in putting together this script - and feel free to refer to it and offer suggestions/updates - particularly with respect to other Linux variants.

      I looked for wsprd v2.0rc3 a few days ago but it wasn't (yet) available at ki7mt - and I wasn't ready to go through an explanation of how to manually install something when I wrote it. If you know of somewhere else to get the newest version with "get", I'd be interested.

      I do need to update it slightly with your recent changes, but I'll do that when I get a bit more time.

      73,
      Clint
      KA7OEI

      Delete





PLEASE NOTE:


While I DO appreciate comments, those comments that are just vehicles to other web sites without substantial content in their own right WILL NOT be posted!

If you include a link in your comment that simply points to advertisements or a commercial web page, it WILL be rejected as SPAM!