Multi-band transmitter and monitoring system for Eclipse monitoring (Part 1)

It should not have escaped your attention - at least if you live in North America - there there have been/will be two significant solar eclipses occurring in recent/near times:  One that occurred on October 14, 2023 and another eclipse that will happen during April, 2024.  The path of "totality" of the October eclipse happened to pass through Utah (where I live) so it is no surprise that I went out of my way to see it - just as I did back in 2012:  You can read my blog entry about that here.

 Figure 1:
The eclipse in progress - a few minutes
before "annularity".
(Photo by C. L. Turner)

I will shortly produce a blog entry related to my activities around the October 14, 2023 eclipse as well.

The October eclipse was of the "annular" type meaning that the moon is near-ish apogee meaning that the subtended angle of its disk is insufficient to completely block the sun owing to the moon's greater-than-average distance from Earth:  Unlike a solar eclipse, there is no time during the eclipse where it is safe to look at the sun/moon directly, without eye protection.

The sun will be mostly blocked, however, meaning that those in the path of "totality" experienced a rather eerie local twilight with shadows casting images of the solar disk:  Around the periphery of the moon it was be possible to make out the outline of lunar mountains - and those unfortunate to stare at the sun during this time will receive a ring-shaped burn to their retina.

From the aspect of a radio amateur, however, the effects of a total and annular solar eclipse are largely identical:  The diminution of the "D" layer and partial recombination of the "F" layers of the ionosphere causing what are essentially nighttime propagation conditions during the daytime - geographically limited to those areas under the lunar shadow.

In an effort to help study these sort of effects - and to (hopefully) better-understand the propagation effects, a number of amateurs went (and are) going out into the field - in or near the path of "totality" - and setting up simultaneous, multi-band transmitters.

Producing usable data

Having "Eclipse QSO Parties" where amateur radio operators make contacts during the eclipse likely goes back nearly a century - the rarity of a solar eclipse making the event even more enigmatic.  In more recent years amateurs have been involved in "citizen science" where they make observations by monitoring signals - or facilitate the making of observations by transmitting them - and this happened during the October eclipse and should also happen during the April event as well.

While doing this sort of thing is just plain "fun", a subset of this group is of the metrological sort (that's "metrology", no "meteorology"!) and endeavor to impart on their transmissions - and observations of received signals - additional constraints that are intended to make this data useful in a scientific sense - specifically:

• Stable transmit frequencies.  During the event, the perturbations of the ionosphere will impart on propagated signals Doppler shift and spread:  Being able to measure this with accuracy and precision (which are NOT the same thing!) adds another layer of extractable information to the observations.
  

• Stable receivers.  As with the transmitters, having a stable receiver is imperative to allow accurate measurement of the Doppler shift and spread.  Additionally, being able to monitor the amplitude of a received signal can provide clues as to the nature of the changing conditions.
  

• Monitoring/transmitting at multiple frequencies.  As the ionospheric conditions change, its effects at different frequencies also changes.  In general, the loss of ionization (caused by darkness) reduces propagation at higher frequencies (e.g. >10 MHz) and with lessened "D" layer absorption lower frequencies (<10 MHz) the propagation at those frequencies is enhanced.  With the different effects at different frequencies, being able to simultaneously monitor multiple signals across the HF spectrum can provide additional insight as to the effects.
  

To this end, the transmission and monitoring of signals by this informal group have established the following:

• GPS-referenced transmitters.  The transmitters will be "locked" to GPS-referenced oscillators or atomic standards to keep the transmitted frequencies both stable, accurate - and known to within milliHertz.
  

• GPS referenced receivers.  As with the transmitters, the receivers will also be GPS-referenced or atomic-referenced to provide milliHertz accuracy and stability.

With this level of accuracy and precision the frequency uncertainties related to the receiver and transmitter can be removed from the Doppler data.  For generation of stable frequencies, a "GPS Disciplined Oscillator" is often used - but very good Rubidium-based references are also available, although unlike a GPS-based reference, the time-of-day cannot be obtained from them.

Why this is important:

Not to demean previous efforts in monitoring propagation - including that which occurs during an eclipse - but unless appropriate measures are taken, their contribution to "real" scientific analysis can be unwittingly diminished.  Here are a few points to consider:

• Receiver frequency stability.  One aspect of propagation on HF is that the signal paths between the receiver and transmitter change as the ionosphere itself changes.  These changes can be on the order of Hertz in some cases, but these changes are often measured in 10s of milliHertz.  Very few receivers have that sort of stability and the drift of such a receiver can make detection of these Doppler shifts impossible.
  

• Signal amplitude measurement.  HF signals change in amplitude constantly - and this can tell us something about the path.  Pretty much all modern receivers have some form of AGC (Automatic Gain Control) whose job it is to make sure that the speaker output is constant.  If you are trying to infer signal strength, however, making a recording with AGC active renders meaningful measurements of signal strength pretty much impossible.  Not often considered is the fact that such changes in propagation also affect the background noise - which is also important to be able to measure - and this, too, is impossible with AGC active.
  

• Time-stamping recordings.  Knowing when a recording starts and stops with precision allows correlation with other's efforts.  Fortunately this is likely the easiest aspect to manage as a computer with an accurate clock can automatically do so (provided that one takes care to preserve the time stamps of the file, or has file names that contain such information) - and it is particularly easy if one happens to be recording a time station like WWV, WWVH, WWVB or CHU.

In other words, the act of "holding a microphone up to a speaker" or simply recording the output of a receiver to a .wav file with little/no additional context makes for a curious keepsake, but it makes the challenge of gleaning useful data from it more difficult.

One of our challenges as "citizen scientists" is to make the data as useful as possible to us and others - and this task has been made far easier with inexpensive and very good hardware than it ever has been - provided we take care to do so.  What follows in this article - and subsequent parts - are my reflections on some possible ways to do this:  These are certainly not the only ways - or even the best ways - and even those considerations will change over time as more/different resources and gear become available to the average citizen scientist. 

* * *

How this is done - Receiver:

The frequency stability and accuracy of MOST amateur transceivers is nowhere near good enough to provide usable observations of Doppler shift on such signals - even if the transceiver is equipped with a TCXO or other high-stability oscillator:  Of the few radios that can do this "out of the box" are some of the Flex transceivers equipped with a GPS disciplined oscillator.

To a certain degree, an out-of-the-box KiwiSDR can do this if properly set-up:  With a good, reliable GPS signals and when placed within a temperature-stable environment (e.g. temperature change of 1 degree C or so during the time of the observation) they can be stable enough to provide useful data - but there is no guarantee of such.

To remove such uncertainty a GPS-based frequency reference is often applied to the KiwiSDR - often in the form of the Leo Bodnar GPS reference, producing a frequency of precisely 66.660 MHz.  This combination produces both stable and accurate results.  Unfortunately, if you don't already have a KiwiSDR, you probably aren't going to get one as the original version was discontinued in 2022:  A "KiwiSDR 2" is in the works, but there' no guarantee that it will make it into production, let alone be available in time for the April, 2024 eclipse. 

Figure 2:
The RX-888 (Mk2) - a simple and relatively inexpensive
box that is capable of "inhaling" all of HF at once.
Click on the image for a larger version.

The RX-888 (Mk2)

A suitable work-around has been found to be the RX-888 (Mk2) - a simple direct-sampling SDR - available for about $160 shipped (if you look around).  This device has the capability of accepting an external 27 MHz clock (if you add an external cable/connector to the internal U.FL connector provided for this purpose) in which it can become as stable and accurate as the external reference.

This SDR - unlike the KiwiSDR, the Red Pitaya and others - has no onboard processing capability as it is simply an analog-to-digital coupled with a USB3 interface so it takes a fairly powerful computer and special processing software to be able to handle a full-spectrum acquisition of HF frequencies.

Software that is particularly well-suited to this task is KA9Q-Radio (link).  Using the "overlap and save" technique, it is extraordinarily efficient in processing the 65 Megasamples-per-second of data needed to "inhale" the entire HF spectrum.  This software is efficient enough that a modest quad-core Intel i5 or i7 is more than up to the task - and such PCs can be had for well under $200 on the used market.

KA9Q-Radio can produce hundreds of simultaneous virtual receivers of arbitrary modes and bandwidths which means that one such virtual receiver can be produced for each WSPR frequency band:  Similar virtual receivers could be established for FT-8, FT-4, WWV/H and CHU frequencies.  The outputs of these receivers - which could be a simple, single-channel stream or a pair of audio in I/Q configuration - can be recorded for later analysis and/or sent to another program (such as the WSJT-X suite) for analysis.

Additionally, using the WSPRDaemon software, the multi-frequency capability of KA9Q-Radio can be further-leveraged to produce not only decodes of WSPR and FST4W data, but also make rotating, archival I/Q recordings around the WSPR frequency segments - or any other frequency segments (such as WWV, CHU, Mediumwave or Shortwave broadcast, etc.) that you wish.

Comment:  I have written about the RX-888 in previous blog posts:

• Improving the thermal management of the RX-888 (Mk 2) - link 
• Measuring signal dynamics of the RX-888 (Mk 2) - link
  

Full-Spectrum recording

Yet another capability possible with the RX-888 (Mk2) is the ability to make a "full spectrum" recording - that is, write the full sample rate (typically 64.8 Msps) to a storage device.  The result are files of about 7.7 gigabytes per minute of recording that contain everything that was received by the RX-888, with the same frequency accuracy and precision as the GPS reference used to clock the sample rate of the '888.  

What this means is that there is the potential that these recordings can be analyzed later to further divine aspects of the propagation changes that occurred during, before and after the eclipse - especially by observing signals or aspects of the RF environment itself that one may not have initially thought to consider:  This also can allow the monitoring of the overall background noise across the HF spectrum to see what changes during the eclipse, potentially filling in details that might have been missed on the narrowband recordings.

Because such a recording contains the recordings of time stations (WWV, WWVH, CHU and even WWVB) it may be possible to divine changes in propagation delay between those transmit sites and the receive sites.  If a similar GPS-based signal is injected locally, this, too, can form another data point - not only for the purposes of comparison of off-air signals, but also to help synchronize and validate the recording itself.

By observing such a local signal it would be possible to time the recording to within a few 10s of nanoseconds of GPS time - and it would also be practical to determine if the recording itself was "damaged" in some way (e.g. missed samples from the receiver):  Even if a recording is "flawed" in some way, knowing the precise location an duration of the missing data allows this to be taken into account and to a large extent, permit the data "around" it to still be useful.

Actually doing it:

Up to this point there has been a lot of "it's possible to" and "we have the capability of" mentioned - but pretty much everything mentioned so far was used during the October, 2023 eclipse.  To a degree, this eclipse is considered to be a rehearsal for the April 2024 event in that we would be using the same techniques - refined, of course, based on our experiences.

While this blog will mostly refer to my efforts (because I was there!) there were a number of similarly-equipped parties out in the fields and at home/fixed stations transmitting and receiving and it is the cumulative effort - and especially the discussions of what worked and what did not - that will be valuable in preparation for the April event.  Not to be overlooked, this also gives us valuable experience with propagation monitoring overall - an ongoing effort using WSPRDaemon - where we have been looking for/using other hardware/software to augment/improve our capabilities.

In Part 2 I'll talk about the receive hardware and techniques in more detail.

Stolen from ka7oei.blogspot.com

[END]

Using the jt9 executable to receive FST4W signals

Note: 

Since originally posted, WSJT-X v2.3.0-rc2 was released, adding a feature to the "JT9" executable that simplifies this process (the "-F" parameter) as described below.  Note that most of this post was originally written soon after "rc1" had become available.

* * *

As a heavy user of K1JT's WSPR and operating on the 2200 and 630 meter bands, I have noted with interest the introduction of the "FST4W" mode in the recent (v2.3.0-rc1) wsjt-x release.  Operating using the same detection bandwidth as WSPR (when FST4W is operated in the 120 second mode) it offers a theoretical 1.4dB improvement in detection sensitivity.

Being involved with wsprdaemon (link to that project here ) - an open-source project that automates and optimizes reception of WSPR signals on all bands, particularly if multiple receivers/antennas are used - we have been watching this development with interest, particularly since FST4W has the likelihood of supplanting conventional WSPR operation, especially on the lowest amateur bands (2200, 630 and possibly 160 meters) where minimal of Doppler shift is expected.

Internally, WSJT-X  uses the subordinate wsprd program as the decoding (and encoding) engine.  As a stand-alone program, the wsprd executable code may be invoked with a command line to decode signals contained within a .wav file that was captured during the standard two minute interval - aligned with even UTC minutes - and produce a text file containing the decoded signals.

Why use the executable rather than the entire wsjt-x suite?  The fact is that the use of the wsjt-x suite does not lend itself easily to script-driven, bare-minimum, lightweight implementations where further processing of the decoded data (to remove duplicate decodes from multiple receivers, antennas and to use this same data for further analysis of signal/noise) is desired.

The "jt9" executable:

After a bit of digging about, it was "discovered" that FST4W - being an offshoot of the JT9 protocol - was handled not by the wsprd executable, but the jt9 executable.  Simply executing this program with no arguments will yield a list of command-line arguments which, on the face of it, made it appear that updating the wsprdaemon to include the decoding of FST4W signals would be a relatively simple matter.

Except that it didn't work.

Initial testing with strong, off-air FST4W signals that was known to be decodable (because farther-flung stations were able to decode the very same transmissions) yielded no results when the .wav file was applied to the jt9 program - but automatic execution over many hours yielded the occasional off-air decode.  Confused by this, I sought help on the WSJT-X groups.io forum.  Fortunately, Joe Taylor and several of the developers offered a clue:  The "-f" parameter of the jt9 executable, described minimally as "Receive Frequency Offset".

Apparently, the default center frequency of the jt9 executable - at least when in FST4W mode (and maybe others) is 1500 Hz - a fact implied when one gets the display of command-line arguments.  What is not so clear - and only alluded to in the available documentation - is that the apparent bandwidth of the decoding, at least in the 120 second mode, is on the order of 40 Hz (+/- 20 Hz).  Addendum:  This issue was fixed with the "-F" parameter - see below.

At a quick glance through the source code (file "jt9.f90"), this bandwidth setting appears to be hard-coded into a shared variable (apparently accessible by other programs in the WSJT-X suite) called "ntol" (likely a number referring to the "frequency tolerance" setting in the GUI) that is not available via the jt9 command line - at least, not without modification of the source code.  (The possibility of directly accessing these shared variables exists - but this would be platform-specific, a bit messy and somewhat dangerous!)

Unfortunately, this fixed +/-20Hz bandwidth does not appear to be compatible with the way that the FST4W mode has (already!) found use on 2200 and 630 meters where it is used along-side the WSPR mode in the 200 Hz subbands.

A hell of a kludge:

Update - kludge no longer needed:

As of version 2.3.0-rc2 it appears that a new parameter "-F" was added to allow something other than a +/-20Hz bandwidth (referred to as "tolerance") to be used, likely eliminating the need for multiple decodes, below.  A possible command-line for this would be:

jt9 -W -p 120 -f 1500 -F 200 <wav file to be processed> 

With the center frequency (-f) being the center of the passband (1500 Hz) and the "-F" parameter referred to as"tolerance" (e.g. detection bandwidth) being 200 Hz. 

Initial testing indicates that the -F parameter does what it's supposed to do and the kludge below is now longer required.

This fact implies that in order to use something other than the GUI version of the wsjt-x software, a work-around must be invoked.  The following is a bare-minimum example of how one might do this via the command line:

jt9 -W -p 120 -f 1420 <wav file to be processed> 

jt9 -W -p 120 -f 1460 <wav file to be processed>

jt9 -W -p 120 -f 1500 <wav file to be processed>

jt9 -W -p 120 -f 1540 <wav file to be processed>

jt9 -W -p 120 -f 1580 <wav file to be processed>

(One might include the -H, -L and -d parameters in actual practice.)

In other words, in order to cover the entire 200 Hz WSPR subband, the JT9 executable (v2.3.0-rc1) must be executed - processing the same .wav file - at least five times:  The results of the decoding will, in each case, be found in the file "decoded.txt".  If one wishes to implement an equivalent of the -w parameter of the wsprd executable (e.g. +/- 150 Hz "wideband" mode), you will need even more invocations than above.

The result from the above mess will be five different decoding results, each of which must be saved (e.g. renamed) between subsequent executions to prevent overwriting by the previous instance.  After this, the five results would be concatenated to yield a single file - but there is a catch:  It is likely - particularly if the signal is strong - that the same signal will be decoded more than once.  Apparently, the "+/- 20Hz" limit isn't the result of a "brick-wall" filter:  Signals beyond this frequency range may be decoded, but the reported S/N values will likely be reduced as distance of the received signal from the specified center frequency increases.  In short, this means that the results of the concatenated version of the "decoded" file(s) must be sorted and all but the single, "strongest" decode (e.g. best SNR) for each station must be discarded.

Comment:

It would appear that just five iterations to cover the 200 Hz bandwidth is not enough:  I received correspondence from a reader of this blog that observed that a frequency variation of less than 10 Hz from that defined by the "-f" parameter can affect the S/N reading by about 1 dB.  Make of that what you will!

* * * * * * * * *

If one wishes to integrate the FST4W decodes into the existing WSPR captures for processing, yet another step must be undertaken:  "Fixing" the formatting.  Not surprisingly, the output in the "decoded.txt" is not formatted the same as the results of the decoding from the wsprd executable meaning that one will need to do a few things, after the fact, to "fix" them - particularly if you wish to forward them to wsprnet.org, including:

• Supply the date.  The "decoded.txt" includes the time - but not the date.  Because date of the .wav file may not be the same as the system date (e.g. later processing of the .wav files - or the interval being processed occurred just before the new day) - one must use the actual date of the recording.  The obvious place to obtain this is from the name of the .wav file being processed.
  

• Frequency offset.  The information that one might send to wsprnet.org must include the carrier frequency of the received signal, but the output in the "decoded" file has only the audio frequency:  One must obtain the LO frequency of the receiver being used from "somewhere else" and calculate this on the fly.
  

• Supply missing information.  The "decoded.txt" file does not have all of the same information fields that one might supply when uploading WSPR spots, so this information must be added as necessary.
  

• Arrange the fields in the proper order.  Once the needed information is applied, one will probably want to use "awk" or similar to produce the same order as the wsprd data - assuming this wasn't already done in the process.
  

* * *

There are two outputs from the jt9 executable - one directly from the program itself to the standard console and that output to the file "decoded.txt" - and the latter is the most useful. 

Console output:

0416 -24  0.7 1515 `  KA7OEI DN40 17                                 
<DecodeFinished>   0   1

The fields are:  <time UTC> <SNR in dB> <DT?> <Audio frequency in Hz> <always "`"> <Callsign received> <Grid of received station> <Reported TX power in dBm>

From the "decoded.txt" file:

0416   0  -24   0.7   1515.   0   KA7OEI DN40 17                        FST4

 The fields are:  <time UTC> <unknown - possibly drift in Hz> <SNR in dB> <DT?> <Audio frequency in Hz> <unknown> <Callsign received> <Grid of received station> <Reported TX power in dBm> <Always "FST4">

* * *

There you have it:  The germ of what would be needed if one wishes to supplement the existing WSPR decodes with the newer FST4W mode using just the bare executables.  If one wishes to decode other than the 120 second FST4W mode, things get even more complicated!

Sample audio file:

An audio file containing both FST4W-120 and WSPR transmissions may be found HERE - right-click to download.  This file contains an FST4W-120 transmission by KA7OEI from about 116km distant and (at least) two WSPR transmissions.

* * * 

 

Note:  It appears that the "-F" parameter, above, modifies the default ntol setting as described above.

P.S.:  While it would be pretty trivial tweak the code to allow modification of the ntol variable via command line, this would complicate the ongoing maintenance of the wsprdaemon code.  We can only hope that the current authors see fit to include a means by which the entire wspr subband can be monitored with a single invocation of the jt9 executable.

 

This page stolen from ka7oei.blogspot.com

[End]

About the WSPRDaemon noise graphs - and repurposing for use in monitoring WWV(B) signal levels over time.

What is WSPRDaemon?

The WSPRDaemon program is largely a work of Rob, AI6VN and it exists for the purpose of facilitating the reception of WSPR signals off air, processing the data, and making that data available publicly - not only to wsprnet.org, but also via the wsprdaemon.org web site.  The WSPRDaemon script runs on a Linux computer - typically a Raspberry Pi - and typically takes data from a web-connected receiver (typically a KiwiSDR) - but it can also use the ubquitous RTL-SDR dongle or even the "raw" audio input from a receiver via a sound card.

For each of the (approximately) 2 minute receive "cycles" of WSPR transmissions (which are, worldwide, scheduled to start precisely at the beginning of each even-numbered minute) an audio (.wav) file is recorded for each receiver - and this audio is then processed using the "WSJT-X" program to decode the WSPR transmissions found within that audio stream.

Features not found in WSJT-X that are made available in the wsprdaemon script include the scheduling of receivers (e.g. switching a limited number receivers to most favorable bands) and the "merging" of several receivers - possibly using different antennas for diversity - that are listening to the same band so that only the best of the WSPR "spots" is reported, avoiding duplicate reports of a given transmission.

• More information about wsprdaemon may be found on the wsprdaemon Github page.
• For more about WSPR, see the WSPR Wikipedia page and the other pages to which it links - especially K1JT's page.

What it does:

While not precisely calibrated, having long-term, disparate records of WSPR signal reception from all over the world can provide useful information to amateur radio operators and researchers alike, providing a glimpse into the propagation of LF, HF and even VHF (and higher bands) which can help divine when propagation is occurring between two stations and when it occurred.  Having a "live" and past database of these events can help validate/tweak models of the Earth's geomagnetic field and its interaction with the sun on the ionosphere - and to satisfy the intellectual curiosity of anyone who wishes to study this by themselves.

Accumulation of "noise" data:

In addition to the accumulation of WSPR data, the wsprdaemon software is able to measure the apparent noise floor on the specific frequencies associated with WSPR transmissions on the amateur bands and with some hardware it is possible to calibrate this measurement in absolute terms of intercepted noise power.

If receivers taking these readings are found at disparate "RF quiet" sites around the world, this data can be informative of the natural, background noise which can be indicative of the state of the ionosphere and the Earth/Space environment:  In some cases, it is possible to observe the rising/setting of strong radio noise sources such as Sagittarius A and, occasionally, "noisy" planets in our solar system such as Jupiter not to mention the tremendous noise that can result when the Earth's magnetic field is being abused by our local star.

The software makes two separate noise measurements:

• The noise "floor" within the passband.  This reading (the "RMS" level, in Red on the wsprdaemon.org graphs) is the calculated noise floor level and its processing attempts to "remove" the effects of other signals within the detection passband.

• The signal power within the passband.  This reading (the "FFT" level, in Blue on the wsprdaemon.org graphs) is the total power within the passband.  Unlike the RMS reading, this is indicative of the cumulative power intercepted and will always be higher.

Even though the apparent detection bandwidth of these receivers is on the order of 400 Hz, all readings are scaled (by approximately 26 dB) such that the power readings reported are relative to a 1 Hertz detection bandwidth.

A typical graph may be seen below:

Figure 1:
A typical graph showing the last 24 hours of noise on the 40 meter amateur band a receiver at the Northern Utah WebSDR.

The displayed time, in UTC, shows the noise level rising at around local nighttime and then dropping off at night.  This graph is "skewed" somewhat by the overnight presence of strong thunderstorms in the Eastern United States, the intensity of which gradually tapered off overnight and into the day and by the fact that the antenna used is a log periodic beam has significant gain that is pointed in the direction of those storms.

Almost the "inverse" of this is the noise graph from another receiver at the Northern Utah WebSDR site:

Figure 2:
This graph depicts a 24 hour plot of the noise floor on the 20 meter band.

The noise floor can be seen to increase during the daylight hours, but drop to the floor during the night when propagation and ionospheric stimulation by the sun effectively ceased.  As expected - particularly during the period of low sun activity during which this is being written - the noise floor and signals decrease during local nighttime.

In theory, careful analysis of the noise data used to produce the above graphs can provide the opportunity to analyze HF propagation modes and the effects of Earth-Space environment.

Monitoring signals from consistent sources:

It occurred to me that the wsprdaemon script also afforded the opportunity for something else for which it was originally intended - the monitoring of consistent signal sources of known transmitter power and location.  From the Utah location, one source of signals - those from the NIST in the form of the WWV and WWVB  transmitters - was obvious.

A quick modification of the wsprdaemon script allowed the addition of additional frequencies:  While there would clearly be no WSPRnet reporting on these non-amateur channels, the noise measurements would still be posted as the graph below depicts:

Figure 3:
 A noise graph of the WWVB transmission at 60 kHz in Fort Collins, Colorado.

The red line shows the received signal level from WWVB - the same signal used to set many automatic clocks - on 60 kHz.  During daylight hours the signal level is pretty consistent at the "-100dBm" mark while during the night, signal levels vary a bit - particularly during sunrise/sunset where ionospheric perturbations are evident.

The "Blue" line is largely influenced by the 17dB amplitude modulation of the WWVB carrier used to convey time and date information, but it is also prone to being "diluted" by peaks in the background noise as can be seen during the nighttime hours (from about 0400-1100 UTC) where propagated lightning static is evident.

From the same site, on a frequency that is orders of magnitudes higher we get this plot:

Figure 4:
 A noise graph of the signal and noise levels on the 10 MHz WWV/H frequency.

The graph of Figure 4 is a bit more cluttered as one might expect.  The Red "line" shows the wildly varying signal of the signal on 10 MHz - which could be from either WWV in Colorado OR WWVH in Hawaii.  As is the nature of HF, these signals can vary significantly - not only between day and night, but also from one moment to the next.  The blue line generally depicts the noise level at 10 MHz, but this may not be truly representative as it may be being affected by the ever-present modulation on the WWV/H carrier - primarily in the form of the 100 Hz time code modulation.

The utility of the graph in Figure 4 may be debatable as there is not one, single signal source, but it does provide a general perception of the signal levels that one might expect - and how the time of day affects them.

Final comments:

In addition to the general monitoring of the noise floor on the HF bands where WSPR monitoring is taking place, the wpsrdaemon script can also be used to monitor signals from known transmitters.  To be sure, this wasn't the intended use of this software and if such data is useful, it's likely that the utility and accuracy of such measurements could be improved.

* * * * * * * * * * * * * * * * *

Modifications to the wsprdeamon script for version 2.8a:

In the "wsprdaemon.sh" file, one need only add a few lines to the code to produce the "new" bands in the array "WSPR_BAND_LIST", as in:

"WWVB        58.5"
"WWV_2_5  2498.5"
"WWV_5      4998.5"
"WWV_10     9998.5"
"WWV_15    14998.5"
"WWV_20    19998.5"
"WWV_25    24998.5"
"CHU_3       3328.5"
"CHU_7       7848.5"
"CHU_14     14658.5"

Each new "band" is named by the first entry on the line (e.g. "WWVB") and the frequency of the carrier that one wishes to monitor is defined in kHz in the second entry.  Note that the frequency used here is 1.5 kHz lower than the actual carrier frequency to be monitored.

In the "wsprdeamon.conf" file where the receiver and its use is to be defined, these "bands" - defined in wsprdaemon.sh - are used in exactly the same way as any other band in the list.  For example, one might schedule the start a hypothetical receiver called "KIWI_1" on the WWVB signal at 0000 local time as follows:

declare WSPR_SCHEDULE=(
"00:00  KIWI_1,WWVB"
)

Comment:  At present, wsprdaemon will dutifully try to process the audio file for WSPR spots - but it will fail to do so.  It should be possible to modify the code to add an argument that will prevent this from happening to reduce processor loading.

* * * * * * * * * * * * * * * * *

This page stolen from ka7oei.blogspot.com

[END]

Setting/calibrating signal and noise levels for signal/noise level monitoring using the "wsprdaemon" script

The WSPRDaemon script, a work largely of Rob, AI6VN, is designed to automate WSPRNet reporting of signals transmitted using the WSPR protocol - (The script may be found on GitHub, here).  While the WSJT-X program, itself, can report the signals that it hears in terms of callsign, Maidenhead grid locator and apparent signal-noise ratio, the WSPRDaemon script leverages the multi-receiver capability that can exist in various hardware configurations - including the KiwiSDR - which, with the appropriate platform, can receive as many as 15 signals on 15 different frequencies simultaneously.

Comments:

While this discussion is focused on methods and measurements exclusively using the KiwiSDR as that is the hardware with which I am most familiar, where similar capabilities exist, these same methods can be applied to other receive platforms.

Similarly, the methods discussed on this page need not be specific to the software mentioned and may be applied any time receiver signal level measurement is needed.

Care has been taken such that at the Northern Utah WebSDR, all of the WebSDR's receivers' S-meters are calibrated to within a dB or so in the center of their target band using the methods described below.

HF noise floor measurements:

In addition to facilitating the decoding and reporting of WSPR signals, the WSPRDaemon script can also divine the apparent noise floor within the receive bandwidth on some of these platforms.  In the case of the KiwiSDR - where an "S-Meter" reading is available via the remote interface - this reading can be used to derive absolute measurements and such data, taken from multiple locations over disparate geographical locations, can be used to glean information about the noise on the HF bands over time.

In practice, accurate, absolute signal level readings are quite difficult to obtain owing to the issue of meaningful calibration - and the more complicated the antenna system is, the more difficult it may be to get such measurements as one must consider things like:

• Cabling losses
• Splitter losses
• Filter losses
• Amplification
• Calibration of the receiver
• Gain of the antenna (e.g. dBi, dBd)

Fortunately, all but the last on the list may be accounted with a single measurement using a known-accurate signal source.

The "short" version:

The ultimate goal of this exercise is quite simple:

Input a signal of known amplitude into your antenna system - say, -60 dBm - and have the noise logger report that same signal level, +/- your known antenna gain.

The steps below are included in the event you may not be familiar with the methods/techniques involved.

An example receive system:

As an example, the KiwiSDR-based portion of the system in use at the Northern Utah WebSDR (a browser-enabled remote HF receiver found at sdrutah.org) will be discussed and a simplified block diagram of the signal path may be seen below:

Figure 1:
Simplified KiwiSDR Signal Path used at the Northern Utah WebSDR showing several gain blocks and filters.
As noted, the directional coupler is connected in "reverse" as it is being used to inject rather than extract a signal from the signal path.
Click on the image for a larger version.

The components of the receive system depicted above in detail:

As with any antenna system, it begins with an antenna - but for the moment, we will ignore any intrinsic gain - or loss - that the antenna may have.

• Directional coupler:  At the Northern Utah WebSDR, the Mini-Circuit Labs ZFDC-20-3 is used at the main antenna input.  This device has minimal insertion loss (<0.25dB) on the "through" line and there is a separate port (labeled "CPL") that is coupled at 20dB from the through line.  This sort of device is typically used to extract a small sample present on the line while minimally affecting it and "in" and "out" ports are labeled for such purposes, but here it is being used in reverse to inject a signal into the receive signal path.

This device is convenient in that it has the advantage that it may be left in line all of the time with negligible effects at HF and it can put test signals into the receive signal path without affecting off-air performance.

This device is optional in that the same measurements may be obtained without it by connecting the signal source directly to the antenna port - the obvious disadvantage being that the signal path is interrupted - something that we try to avoid on a busy, multi-user system such as a WebSDR.

• Calibrated signal source:   It should be no surprise that having a known-accurate signal source is an absolute necessity when it comes to calibrating the signal level readings with available levels in the range of -60 to -30dBm being the most useful.  Having access to such devices (e.g. Communications Test Set or RF Service Monitor) is helpful - but other means of obtaining a usefully-accurate signal include:

• An "inexpensive" calibrated signal source like the Elecraft XG-3 - the >$300 price tag (when fully equipped) being the reason for the quotes.

• A "transfer" measurement from a signal source with a consistent level.  Some antenna analyzers - including the popular "NanoVNA" - can output a level that is fairly consistent.  Coupled with outboard attenuators (a switchable step attenuator and/or an assortment of fixed attenuators) one can use known-accurate test equipment (an RF power meter, signal level meter, spectrum analyzer) to obtain a table of signal power readings for each of the HF amateur bands.

• Because of the harmonic content of the NanoVNA, it is preferred that a selective means of measuring RF power like a spectrum analyzer be used rather than radiometric detector like a power meter - but the difference is unlikely to be significant.

• Important note:  The amplitude of the NanoVNA's output port is too high to be directly used as a calibration signal - especially if amplification is used in the signal path.  Expect to need attenuation in the range of 20 to 50 dB to provide a usable signal level:  It's recommended that one use a switchable step attenuator and/or a series of fixed attenuators to get reasonable signal levels.

• A transfer measurement from a piece of equipment that is likely to be "pretty close" - such as the KiwiSDR itself:  With the default "factory" S-meter calibration level of -13dB the KiwiSDR to which I have access have typically been within a couple of dB when comparisons were made.
• For calibrating, a signal source (e.g. NanoVNA) would be connected directly to the input of the receiver - via the attenuator(s) - and the S-meter levels noted for different settings on each of the amateur bands.

• AM BCB Filtering:  In many cases, strong AM broadcast band (mediumwave) signals will be present on the antenna port.  Even if these signals are not strong enough to overload the RF amplifier that follows, such signals can overload the receiving device downstream.  As mentioned later, there are advantages to strategically attenuating strong signals to minimize the total power impinging on the A/D converter of any direct-sampling receiver.
  

The AM broadcast-band filter used at the Northern Utah WebSDR is a rather complicated affair (and is described in this article:  Managing HF Signal Dyanmics and preventing overload... (link)).  Its main purpose is to reduce the amplitude of several very strong (50kW) local transmitters to prevent the "wasting of amplifier power and A/D bits" on very few signals.

• Amplification (post AM/BCB filtering):  In general, a 15 dB system noise figure is sufficient to allow the detection of a 10 meter signal above even the lowest expected level of background noise found at this frequency.  For this reason it is suggested that an RF amplifier be placed after the AM BCB filter (to minimize the probability of overload and intermodulation/mixing products).  Remember:  Once loss appears in a receive system due to loss/attenuation, no amount of amplification can restore the system noise figure beyond that point.

Such an amplifier should also be placed prior to any RF splitting as much as is practical as any loss - which can be from the feedline, in filters or in a splitter - will directly contribute to the the system noise figure.  In cases where very strong signals may be present on the antenna system (local AM BCB, strong Shortwave Broadcast signals - when the bands open) it is preferable to have an amplifier at this point that has only modest gain - in the area of 12-15 dB:  A high-gain amplifier (>20dB) may be more prone to overload in such conditions and the typical means of preventing this (addition of an attenuator in front of the amplifier) is not recommended as this will increase system noise figure.

• Limited attenuation high-pass filter:  Any receive system is capable of handling only a certain amount of total signal input power - and the direct-sampling receiver of the KiwiSDR is no exception.  The overload level of the KiwiSDR's 14 bit A/D converter is approximately -13dBm, so the total power arriving at its antenna port should be kept below that level and strategic, selective attenuation of local, strong carriers - often those of AM/mediumwave broadcast transmitters - should be considered - but there is another problem:

The sensitivity of a KiwiSDR is approximately -155dBm (in a 1 Hz bandwidth) over its frequency range which means that in a unity-gain HF antenna system that at 10 meters, the KiwiSDR will be "short" by at least 6-8dB from hearing the textbook "rural quiet" noise floor on that band, requiring 10-15 dB of overall amplification to place the 10 meter noise floor comfortably above that of the receiver's in that environment.  This amplifications has a cost:  With the addition of a 12 dB RF amplifier, the KiwiSDR will now overload at -25dBm.  It is not uncommon for the absolute power level of a single high-power shortwave broadcast station to exceed this level under good band conditions - and there will be many such signals on the bands - not including the wide-band energy from lightning static - that can combine in overall signal power and exceed this overload level.

What system noise figure is required for HF reception? This is a tricky question to answer because conditions on the HF band vary so much - most strongly on a per-band basis. Assuming quiet (e.g. "dead"), bands the lack of storm static. the absence of solar noise and the receiver being located in an area completely absent of man-made noise, noise figures equal to or lower should be attained for the receive system to have a "natural noise" limit: 1.8 MHz:  45dB NF 3.5 MHz:  37dB NF 7 MHz:  30dB NF 14 MHz:  24dB NF 21 MHz:  20dB NF 28 MHz:  15dB NF 50 MHz:  9dB NF 144 MHz:  2dB NF These numbers are approximate, but are a general guideline for system design.

Were the KiwiSDR being used over a very narrow range of frequencies - such as the case for most commercial direct-sampling receivers like the Icom IC-7300 and IC-7610 - this could be managed by gain adjustment in the signal path, but if the KiwiSDR is to be used over the entire HF spectrum, this cannot readily be done.  Instead, one can selectively insert attenuation where the signals+noise are likely to be the highest, namely below approximately 10 MHz and adjust calibration accordingly.

The use of this "limited attenuation" high-pass filter leaves signals above 12 MHz alone but offers significant attenuation (about 12 dB - around 2 "S" units) to signals below 8 MHz.  Because of the relatively high noise levels in the lower HF bands, additional attenuation - and the commensurate increase in system noise figure - can be tolerated without the loss of useful sensitivity.

Note:  This same issue of overload will occur with similar devices - such as the Red Pitaya - if operated as wideband HF receivers, without input band-pass filtering.  Simple math will indicate that even if the KiwiSDR had a 16 bit A/D converter - which would theoretically yield another 12 dB of dynamic range - one would still need to take these same precautions.  It is for this reason why all well-designed direct-sampling receive gear has narrowband filtering that precedes its A/D converter.

Unfortunately, the "limited attenuation" filter fits a niche requirement as is not a commercially-available device, but one may be built without too much difficulty:  Two versions are described here:

• A limited-attenuation high-pass filter for the KiwiSDR - link. This filter starts to attenuate below 12 MHz, offering about 12 dB below 8 MHz.

• Revisiting the limited-attenuation high-pass filter - link.  This version is similar to that above, but its attenuation pass range is limited  to the range of approximately 1700 kHz to 12 MHz, allowing the passage of signals in the AM broadcast band and below for the reasons discussed in the article.

• Because of the simplicity of these filters they exhibit a low return loss (high reflectance) at lower frequencies and  it is recommended that they be preceded with an amplifier to present the input feedline with a more consistent "50 ohm" impedance.

• Amplification (post high-pass filter):  Where high signal levels are likely to exist it is prudent to distribute the signal gain across several blocks rather than have a single high-gain amplifier near the "front" of the signal path.  An amplifier placed after the "limited attenuation" high-pass filter will see lower signal levels from the very strong lower HF-band signals and be less subject to overload.

• Splitter:  A 4-way splitter is depicted in figure 1.  A real-world 4-way splitter will have 6.5-7.5 dB loss between the input and any of its output ports which means that for the higher-frequency bands it is particularly important to have established the system gain and noise figure prior to the splitter.  Like any other block of gain or loss, the impact of such a device must be considered when evaluating the system overall.

Analyzing your receive system:

The procedure following this point assumes the availability of a known-accurate signal source:  See the heading "Calibrated Signal Source", above, for more information.

Again, we are characterizing the signal path beyond the antenna and ignoring any gain/loss of the antenna itself for the moment.

A procedure for obtaining the calibration of the receive signal path is as follows:

• Input a known signal level at the input of the receive signal path.  This signal would be input at the antenna port - or via the "coupling" port of the directional coupler.  This signal should be input at a frequency in the band of interest as calibration values should be obtained for every amateur band to be used.  For WSPR operation it is desirable that one generates test signals within the WSPR passband.

• Tune in the signal and note the signal level on the receiver.  A known signal level of at least 20dB (30dB is better) above the receiver noise floor and other signals in the receiver passband is suggested to minimize effects of the noise and signals - but the level should be no higher than necessary to avoid overload/compression of any amplifiers - or the receiver itself.

• If using a directional coupler as suggested, make sure that it is oriented correctly by comparing the signal levels when the generator output is fed directly into the receive signal path and again with signals fed via the coupler:  If its "in/out" is reversed the coupling level will be inaccurate. (Again, for a coupler used for extracting a signal from a coax, the connection will be reversed.)

• In the case of the 20 dB coupler suggested above, the signal levels reaching the receiver will be 20 dB lower due to the coupling loss than that if the signal generator coupled directly and this should be remembered when taking measurements.

• If the levels of the test signal are low (less than 20dB S/N) and cannot be increased, narrowing of the receive passband can minimize the effects of noise - just observe the S-meter while making such adjustments to verify that the test signal remains within the passband.

• Because we are measuring the amplitude of a coherent signal within a finite bandwidth rather than trying to measure the noise power in a specific bandwidth, we must make sure our test signal is strong enough that the power of the background noise is a small percentage of the total.

• While it is necessary to compensate for detection bandwidth when measuring the noise within that bandwidth, one does not do so for coherent signals as all of the power of that signal is (theoretically) all on a single frequency.  It is for this reason that the measurement obtained of the coherent signal should match the actual power of that signal.

• Knowing the signal level being input to the signal path, observe the S-meter reading on the KiwiSDR on a per-band basis.  It is not important that the S-meter read correctly - except for the benefit of users connected to the KiwiSDR and looking at the meter - as the levels reported by wsprdaemon will be adjusted in its calibration file.

• If you are running wsprdaemon and are producing a signal within the WSPR passband, the signal will show up on the graphs after several minutes as a constant level - and this level can be used to verify calibration.

• The level displayed on the graph will be that of the KiwiSDR plus the amount of attenuation/amplification specified in the wsprdaemon.conf file for that band.  Use either raw data or the interactive Grafana data to read the signal levels precisely.

• At the Northern Utah WebSDR, the KiwiSDR S-meter calibration setting has been adjusted so that for bands above the roll-off of the high-pass filter (e.g. 20-10 meters) the S-meter read the same as the input signal level.  At lower HF bands, the S-meter reads low by the amount of attenuation of the filter.  Since there is currently only one S-meter calibration point that applies to all frequencies, there is nothing to be done to correct this.

Any adjustment of the KiwiSDR's S-meter calibration should be done before completing the steps below.  If it is changed, the final per-band calibration values will need to be adjusted accordingly.

• Record, for each band, the amount of RF you are applying at the antenna port (taking into account the coupler, if used) and the S-meter reading on the KiwiSDR.

• The difference between these two readings will be the amount of gain - or loss - in your signal path.  Again, we are not concerned about the absolute gain of the antenna itself at this point.

• If the reading on the KiwiSDR is above that of the absolute input level, signal gain is indicated at that frequency.

• If the reading on the KiwiSDR is below that of the absolute input level, signal loss is indicated at that frequency.

• We have now characterized the gain/loss of the signal path.  If our antenna had unity (0 dB) gain we would use the numbers that we'd just obtained as an offset in the wsprdaemon.conf file.  For example:

• If -50dBm were present at the input of the signal path and the KiwiSDR read -38dBm on 10 meters, we would know that we had 12 dB gain at that frequency.  To compensate we would put -12 db in the entry for 10 meters in the wsprdaemon.conf file.

• If -50dBm were present at the input of the signal path and the KiwiSDR read -52 dBm on 80 meters we would know that we had 2dB attenuation at that frequency.  To compensate we would put 2 dB in the entry for 80 meters in the wsprdaemon.conf file.

• Again, to verify our calculations we can place a carrier of known signal level within the WSPR passband for the band being tested.  If the measurements have been done correctly the levels on the graph should match those of the level of the RF at the input of the signal path.

Compensating for coax cable losses:

The next step would be to include the losses of any coaxial cable in the system before the point in the signal path where we injected our test signal.  This can be directly measured with a wattmeter and dummy load being used to measure the power lost over its length or - presuming a fairly low VSWR - can be estimated using the manufacturer's loss values on a per-band basis.

These loss values would be added to the per-band numbers in the wsprdaemon.conf file.

Compensating for antenna gain:

With simple equipment, calibrating the signal path for an absolute level is pretty easy - but the gain of the receive antenna at a given frequency is more difficult to determine as very few people have access to a known-accurate antenna range - so we will have to make an educated guess.  Because many HF antennas have definite patterns of of lobes and nulls one will have to decide if the peak gain of the pattern will be used, or some sort of averaged gain value.

Passive antennas:

For simple wire antennas, the gain can be estimated with reasonable (+/- a few dB) accuracy.  For example, a half-wave dipole above typical ground will likely be in the area of 1-2 dBi gain peak while the gain of a commercially made vertical or Yagi antenna might be available from the manufacturer's data sheet.

For multi-wavelength wire antennas such as long wires or rhombics, antenna simulation programs are suggested and may be the only reasonable means of estimating gain - but again, one will have to make a decision on what gain value to apply in light of the likely existence of lobes and nulls.

Active HF antennas:

If the feed from a conventional antenna (beam, dipole) includes an amplifier, the steps mentioned above may be used to estimate its gain and one need only add the loss of the feedline and gain of an amplifier to obtain the desired offset for the wsprdaemon.conf file.

Electrically-short active antennas such as the PA0RDT mini-whip pose a particular problem.  They are essentially a capacitive plate (or wire) that couples RF from the "aether" - but the amount of coupling (which correlates with the apparent gain of the antenna) can vary widely, the determining factors including:

• Height above "ground".  In general, the higher above the ground - earth or metal - the more signal such an antenna will intercept.

• Type of "ground".  The ground above which the antenna is mounted could be earth (dirt. soil) which is lossy or it could be a metal roof (much preferred!) to which the feedline's ground is bonded.

• Size of the "antenna".  The traditional PA0RDT antenna uses a PC board plate for coupling - but there are many variations on what size of coupling conductor - wire or plate - is used, all of which can affect the apparent gain.

• Gain response of the amplifier with frequency.  In most - if not all - implementations of an E-field whip are subject to gain drop-off with higher frequency and this is not easy to measure directly and determining a suitable number is almost entirely guesswork.

• Unfortunately, such drop-off conspires with the high RF levels at low-medium HF frequencies.  With a broadband, direct-sampling receiver like the KiwiSDR, overload will surely occur if enough gain is added to the system to attain the natural noise floor at the higher HF bands (e.g. 10 meters) unless strong compensation (e.g. something like the high-pass filter described above) is used.

However one might obtain the antenna gain, this value would be applied to the wsprdaemon.conf file:  Positive gain would mean a reduction of this number by the dB value while negative gain would mean a commensurate increase in this value.

What to expect:

In theory, the noise floor one sees should generally reflect that predicted in the ITU-R data, reproduced in figure 2, below:

Figure 2:  "Typical" noise floor for various radio environments.  Because the above chart is based on a 500 Hz bandwidth, one would subtract 27dB from its vertical axis to scale to an equivalent 1 Hz noise bandwidth when measuring noise power.
Click on the image for a larger version.

To be sure, the values depicted in Figure 2 are very generalized - and one would be lucky to approach the values of "quiet rural".  In general, these value depicted above would be most representative when the respective frequency band is "dead" (e.g. daytime for the lower bands, nighttime for the higher bands) with a "quiet sun".  While usable only as a general guideline, one should be suspect of their calibration settings if the "quiet band" noise floor values they measure on their system is markedly lower than those of the chart above.

Cross-checking noise level measurements with the receiver:

It is possible to check the noise level measurement provided by wsprdaemon using the receiver in question, but to do this two conditions must be satisfied:

• You must know the receive bandwidth.  For most SDRs this is pretty easy:  The "bandwidth" setting - which is typically at the -3dB or -6dB points - is "fairly close".

• The passband must contain ONLY noise and no other signals.  On the WSPR frequencies this may not be possible, but measuring on a "nearby" frequency (up to a few 10s of kHz away, provided that such measurements aren't affected by narrow band-pass filters that might be present).
• Wider (SSB) bandwidths are fine, but it may be necessary to user narrow bandwidths to avoid signals within the passband.

 The method of making this measurement is approximately thus:

• Note the average signal level reading (in dBm).
• Since we are looking for what amounts to a rolling average over time, the readings of occasional noise bursts must be considered which means that a "fast" AGC should be used and one should not be looking at any "peak" readings.
• Since we are likely doing a "sanity check" simply staring at the signal meter and guesstimating the average.
• It is best to do this during "quiet" conditions on a given band - that is, when it is dead:  Trying it on 80 meters during the night where there may be a lot of summer static is not recommended.

•  Note the receive bandwidth.
• On an SDR, this is likely to be the "bandwidth" figure which is probably taken from either the -3dB or -6dB points - which will be "close enough.

• Calculate the bandwidth noise power ratio.
• Because wsprdaemon's noise power measurements are scaled for 1 Hz detection bandwidth, the reading obtained on any receiver with a wider noise bandwidth than that will be much higher and the reading (in dB) must be offset by that amount.
• As an example:  If you are using a 500 Hz receiver bandwidth (e.g. a CW filter) you need to account for a 500:1 power ratio which is:
• 10 * log10(500) = 26.99 = 27 dB - that is, one would subtract 27 dB from the average level that you observed.
• In other words, to scale to the 1 Hz wsprdaemon bandwidth:
• Your signal level - (10 * log10(RX bandwidth in Hz))

Conclusion:

While it is unlikely that the typical amateur will be able to be absolutely confident of the accuracy of their absolute HF noise floor measurements, the steps above will greatly limit the uncertainty of the system's measurement to the the properties of the antenna itself.

[End]

 This page stolen from ka7oei.blogspot.com

A quick (and probably incomplete) guide to getting kiwirecorder and kiwiwspr (now wsprdaemon) running on Linux

What follows is likely to be quite incomplete - but it contains the major steps in installing the kiwirecorder and kiwiwspr packages on a Linux machine.  I did my installation on Ubuntu, but similar may be done on other distros, such as Debian - including on the Raspberry Pi platform.

I don't claim to be an expert in such things and it is likely that I missed a few steps that may be specific to some distros/platforms - but the phrase "Google is your friend" is applicable if you run into trouble as it's likely that I can't help in your specific case.

What is "kiwirecorder"?

The kiwirecorder package works in conjunction with a "KiwiSDR" (see KiwiSDR.com for more information) - a relatively inexpensive (approx. $300) stand-alone, Linux based HF receiver capable of tuning from a few kHz to over 30 MHz.  Intended to be remotely accessed, it sports a Web-based user interface that provides both audio and a waterfall display along with the capability of demodulating various modes (SSB, CW, AM, FM) and decoding several different types of transmissions, including CW, FAX, FSK and WSPR.  Depending on configuration it is possible to have up to eight "users" connected, via the Internet, to a single KiwiSDR, each using their own frequency and mode settings.

While capable, it may be that this interface isn't quite what you need:  Perhaps you need an audio stream to feed somewhere else - maybe, to another computer that is decoding a mode that is not "built in" to the KiwiSDR or to record the goings-on on a certain frequency:  This is where the "kiwirecorder" comes in.

The kiwirecorder script makes a minimalist connection the the KiwiSDR, not requiring the user interface (display, waterfall, etc.) providing only a streaming audio connection with the frequency and mode specified in the connecting URL.  This stream may then be processed by another computer connected to it (locally and/or via the Internet) to do what it is that you wish to do.

Exactly how this works and its options are beyond the scope of this article, but the source code and documentation may be found here:

https://github.com/hcab14/kiwiclient

What is "kiwiwspr"? 

Note:  The "kiwiwspr" package has been renamed "wsprdaemon" as it is no longer specific only to the kiwirecorder package - see the addendum at the bottom of this posting.

While the KiwiSDR itself is capable of decoding WSPR transmissions, doing so takes a lot of its processing power and if there are many WSPR transmissions to be decoded within a passband - such as 20 or 30 meters during band openings - it may "miss" many opportunities if there isn't enough processing power to go through all of the latent signals.  Additionally, in a multi-user environment, having several WSPR decoders running simultaneously causes the normal user interface to slow noticeably - particularly with respect to the update rate of the waterfall display.

By "piping" audio to another, more capable computer, more effective WSPR decoding may be done by recording about 110 seconds of audio starting at the beginning of an even-numbered minute and then passing that file to the "wsprd" (wspr decoder) program which is part of the WSJT-X suite.  Using an outboard computer for this task minimizes the load on the KiwiSDR itself and with the faster, better hardware of the remote computer, more "in depth" processing may be done to allow the recover of more weak signals than may be possible with the KiwiSDR's built-in decoder - and be able to do so more quickly, with the latest version of the software.

Installing kiwirecorder and kiwiwspr 

The following assumes that you have a running Ubuntu Linux computer running on a network that has access to the KiwiSDR.  Ideally, such a computer would be placed on a local network, but this is not required.

• Be sure that the computer in question is reasonably up-to-date and has installed on it Python.  If Python is not already installed, the steps on doing so are readily found on the web. 

• The computer in question must have access to means of keeping the clock within +/- 2 seconds of UTC.  This is usually done by via NTP or similar.  If the computer's clock is not held to within a couple seconds of UTC, WSPR decoding will fail.

• It is highly recommended that the KiwiSDR's GPS antenna be placed outside in a location with a clear view of the sky and that its GPS lock is verified.  By supplying the KiwiSDR a valid GPS signal both its frequency and time may be held to good accuracy.

The steps:

• Log into the computer as the user that you wish to run the kiwirecorder and related software.  It is generally a good idea not to be logged in as a user with root access when you are running scripts.

• Install "kiwiclient" and change to its directory after install:
• git clone https://github.com/jks-prv/kiwiclient
• cd kiwiclient

•  Install Python dependencies for the scripts:
• sudo apt-get install python-pip
• sudo pip install numpy scipy
• (There are other ways to install "numpy" and "scipy", but the above works fine)

• Install wsprd (WSPR decoder for linux):
• sudo add-apt-repository -y ppk:ki7mt/wsjtx-next
• sudo apt-get update
• sudo apt-get install wsjtx
• Important: 
• If the version that you need isn't available via the repository and/or a new version of WSPR has recently released, it may not be available via the repository as an installation package.  To manually install a newer/different version, do the following:
• Download the most current binary that is appropriate for your system.  Binaries for the most common distros are typically found here:  https://physics.princeton.edu/pulsar/k1jt/wsjtx.html
• In the example this package is called "wsjtx_2.0.0-rc3_amd64.deb", but you would choose the appropriate binary for your system.
• As of the time of this writing, only "1.9" is available via an install package and version 2.0 - which offers slightly better performance - has to be manually isntalled.
• Copy the binary to a directory on the target machine where you have access.
• In the location where you copied the file do:  sudo dpkg -i wsjtx_2.0.0-rc3_amd64.deb  (You would substitute the name of the file that you downloaded and copied to the Linux machine)
• You may get an error saying that "libgfortran3" is not installed:  This package may be installed thusly:  sudo apt-get install libgfortran3
• After you install libdfortran3, the wsjtx package may automatically complete installation, but if not (or if in doubt) try reinstalling it using the "sudo dpkg..." step, above

• Install bc:
• sudo apt-get install bc

•  Go into "tools" subdirectory (kiwiclient/tools)
• cd tools

• Find the newest "kiwiwspr" script and copy it to the "tools" director.
• While a version of the "kiwiwspr" script is included with kiwiclient, it may not be the newest.
• Go to this discussion on the ValentFX.com site and look near the end for the newest kiwiwspr script.   This same discussion also provides clues and help for troubleshooting.
• Make sure to CHMOD the new kiwiwspr script (if you updated it) to 774 so that it may be executed.

This should complete the installation.  The directory of interest will always be "kiwiclient/tools".

Let us assume for what follows that the kiwiwspr script is called (or has been renamed to) "kiwiwspr.sh"

• Execute the script once to get the "default" configuration file:
•  ./kiwiwspr.sh
• If all goes right, you will get a message stating that a "default" configuration file has been created.

• Copy the resulting file ("kiwiwspr.conf") and rename it as something else (e.g. "kiwiwspr.conf.original) so that you can refer to it later.

• Edit the file to include one's own specific KIWI information.  Each of the items below is required!
• At the top under "KIWI_LIST", enter the required information, separated by at least ONE space:
• "OurID" - Name of the KIWI (no spaces).  It is this name that will be used to refer to this particular KiwiSDR in the schedule described below.
• "IP:PORT" IP address (numerical) and the KiwiSDR's web interface port port (typically 8073)
• "Mycall" - the call by which WSPR decodes are reported:  This is the "reporter" callsign on wsprnet.org
• "MyGrid" - the 4 or 6 character Maidenhead grid location.
• "KiwiPassword" - This is the password, if any, to access the Kiwi if it is not public.  You must put the word "NULL" here if there is no password - it cannot be left blank.
• Note:  Several different KiwiSDRs may be defined in this table:  Each KiwiSDR's definition is as above, surrounded by quotes, on its own line.
• As an example, one might have entered, on its own line, the following:
• "my_kiwi 192.168.1.234:8073 w7xyz-1 cn01dc NULL"  This means that a receiver called "my_kiwi" can be reached via the IP address of 192.168.1.234, port 8073 and it will report to wsprnet using the callsign of "w7xyz-1" using the grid square of "cn01dc" and will not require a password.

• There may be a table labeled "CAPTURE_JOBS":  This is no longer used and it may be ignored.  (It's recommended that you leave it as-is for now if it is there.)

• There is a table called "WSPR_SCHEDULE" that defines on what frequencies, with which KiwiSDR instance and when the WSPR decoding "jobs" are to be scheduled in the following format, each on its own line with quotes:
• "time kiwi_name,first_band kiwi_name,second_band kiwi_name, third_band"
• Or, using the example above:
• "15:00 my_kiwi,80 my_kiwi,40 my_kiwi,30 my_kiwi,20"
• This would cause "my_kiwi" to start decoding WSPR on the 80, 40, 30 and 20 meter bands at 15:00 local time.
• You can also set the schedule relative to you sunrise and sunet time, the location of the sunrise/sunset being determined by the grid square that you entered.  This can be useful if you have only a limited number of audio instances available and you wish to favor the bands that are best in the daytime or nighttime.  Example of this are:
• "sunset-01:15 my_kiwi,80 my_kiwi,40"  - This would start WSPR decoding sessions on 80 and 40 meters at 1 hour and 15 minutes (e.g. 01:15) before local sunset as indicated by the "minus" sign after "sunset".
• "sunrise+2:30 my_kiwi,20 my_kiwi,15" - This would start the WSPR decoding sessions on 20 and 15 meters at 2 hours and 30 minutes after local sunset as indicated by the "plus" sign after "sunrise".
• If you wish, you may also specify UTC (GMT) instead of local time, in which case the you would add ",UDT" after the numerical time as in:
• "01:00,UDT my_kiwi,80 my_kiwi,40" - This would start the WSPR sessions on 80 and 40 meters at 0100 UTC.  Times relative to Sunrise/sunset would not apply here.

Options:

As of version 1-1g there is an optional variable that can be specified called "KIWIRECORDER_CLIENT_NAME".  One would include a link in the kiwiwspr.conf" file like this:

declare KIWIRECORDER_CLIENT_NAME=(
    "My_Kiwirecorder"
)

The above will cause the text "My_Kiwirecorder" to show up as the name of the use on the KiwiSDR instead of "kiwirecorder.py".  Do note that this needs to be placed at or near the top of the kiwiwspr.conf file or else it may not have an effect.

Bands/Frequencies:

Valid bands, their base RF frequencies and receive passbands are:

• BAND  DIAL FREQ  RF (carrier) Frequency range
• 2200     136.0kHz         (137.3-137.7 kHz)
• 630       474.2 kHz        (475.5-475.9 kHz)
• 160       1836.6 kHz      (1837.9-1838.3 kHz)
• 80         3568.6 kHz      (3569.9-3670.3 kHz)
• 80eu     3592.6 kHz      (3593.9-3594.3 kHz)  Deprecated - Usage to be discontinued.
• 60         5287.2 kHz      (5288.5-5288.9 kHz)  Not on a U.S. channel.
• 60eu     5364.7 kHz      (5366.0-5366.4 kHz)  Not on a U.S. channel.
• 40         7038.6 kHz      (7039.9-7040.3 kHz)
• 30         10138.7 kHz    (10140.0-10140.4 kHz)
• 20         14095.6 kHz    (14096.9-14097.3 kHz)
• 17         18104.6 kHz    (18105.9-18106.3 kHz)
• 15         21094.6 kHz    (21095.9-21096.3 kHz)
• 12         24924.6 kHz    (24925.9-24926.3 kHz)
• 10         28124.6 kHz    (28125.9-28126.3 kHz)

Notes about the bands/frequencies listed above:

•  The "80eu" band is still used in some parts of Europe and by other amateurs who have yet to change:  Use of this frequency is deprecated and will eventually cease.
• The "60" meter band reflects the available of that in much of the world while the "60eu" band is better-suited for use in many European countries.  Because 60 meter frequencies/channels differ, you should pick the frequency that reflects the geographical area of interest when it comes to receiving WSPR transmissions.  (Note that none of the listed 60 meter frequencies correlate with the channels available to U.S. amateurs.)
• The "Dial Freq" is that to which one would tune using USB while the "RF" frequency range is that which this script will use to "look" for WSPR signals.
• The audio band 1300-1700 Hz is used for decoding and offset should be applied to determine the actual transmitted/received carrier frequencies.  (e.g. for 40 meters, RF frequencies of 7039.9-7040.3 are in the passband.) 
• The detection bandwidth is 400 Hz instead of the nominal 200 Hz to accommodate "out of band" WSPR operation, frequency drift of your receiver and others' transmitters and the effects of the audio filters applied by the Kiwi.

If you get errors:

• If a package didn't install, you'll get errors indicating such.  Because there are many different possible scenarios with your operating system, I will not cover them here.  Remember:  Google is your friend when it comes to helping solve specific errors.
• If, when you run the "kiwiwspr" script, you get errors about missing elements, etc., these are typically due to data that is expected (like band) but is missing.  Re-check to make sure that:
• ALL of the required fields are entered in the "kiwiwspr.conf" file.
• That there are NO spaces within names or times.
• That there are NO spaces after commas
• That the line with the definition starts and ends with a double quote (e.g. " ) - even if the definition itself spans several lines.

Starting the script:

After installation, you should be in the "kiwiclient/tools" directory.

Assuming that the script is called, to start ALL jobs, do:

./kiwiwspr.sh -j a    - Start ALL jobs
./kiwiwspr.sh -j z    - Stop ALL jobs
./kiwiwspr.sh -j s    - Show running jobs

At this point there should be no errors and you should see "users" called "kiwirecorder.py" appearing on the KiwiSDR on the bands you specify for the most recent time before the current time.  For example, if it is 11:00 and you have two jobs - one scheduled for 09:00 and another for 17:00, the 09:00 schedule will be executed.

If all goes well, you should then start the watchdog, which will allow the schedule to be executed.  If you do not start the watchdog, jobs that were running at the time you started them will continue to run and they will not change according to the schedule.

The watchdog is started/checked/stopped with the following commands:

./kiwiwspr.sh -z a      - Start all watchdog jobs
./kiwiwspr.sh -z z      - Kill all watchdog jobs
./kiwiwspr.sh -z s      - Show watchdog jobs

Use "kiwiwspr.sh" with no arguments for more information about the above commands - and for more information.

If all goes well you will, after 5-10 minutes - assuming that your KiwiSDR is connected to a working antenna and there are signals on the (bands) that you have selected and you have Internet connectivity - start seeing WSPR reports on "wsprnet.org" using the callsign specified in the "MyCall" field in the "kiwiwspr.conf" file in the "KIWI_LIST".

Remember:  A WSPR receiving interval starts at the beginning of each even minute and runs for almost 2 minutes.  It will then take another minute or so for wsprd to decode the WSPR transmissions and forward them to wsprnet.org where they may be displayed.

Final comments:


Remember that the above steps are for an up-to-date Ubuntu Linux distro running on a PC platform and there may be differences in how to install it on other distros/hardware platforms.  If you encounter difficulties, I probably won't be able to help:  I'd just do what you could do in that situation, and that is to refer to Google for a solution to the same/similar problem.

I would like to thank Rob Robinett for his hard work in producing the kiwiwspr script along with John Seamons for his very hard work on the KiwiSDR itself and making the "kiwirecorder" package available.

Best of luck!

* * * * * * * * * * *

[8 February, 2019]

Update:  "kiwiwspr" is now "wsprdaemon"

The "kiwiwspr" package has since been renamed the "wsprdaemon" package as its usefulness has been expanded beyond simply using a KiwiSDR.  As of the time of this update (8 February) its "new" capabilities include:

• Elimination of "duplicate" reception reports from "dirty" transmitters or from modulation caused by noise blankers:  Only the strongest report is used.
• Allowing the use of multiple receivers sharing the same frequency.  This is useful if one has several antennas and wish to submit only the report of the "best" signal of the lot.
• Using audio from a receiver via a sound card.

Even more features may have been added since this page was updated.
 
The configuration of the "multi receiver" aspect is documented in the "wsprdaemon.conf" that accompanies the script while the use of an audio input is beyond the scope of this document.

The modification has required some changes to the way the wsprdaemon package is installed as compared to old "kiwiwspr":

• The "~/wsprdaemon" directory is created and the wsprdaemon .sh and .conf files are put into it
• A "/dir/tmp/wspr-captures" directory is created.  If using a Raspberry Pi, this directory should use tmpfs file system or else you will "wear out" the SD card.  If using a computer with an SSD, you may still consider doing this.
• Use CHMOD to make sure that the wsprdaemon.sh file may be executed
• The configuration from the old "kiwiwspr.conf" file may be copied to, "wsprdaemon.conf" but the array "KIWI_LIST" should be renamed "RECEIVER_LIST".
• The newer "wsprdaemon.conf" includes a the ability to define in RECEIVER_LIST a virtual receiver consisting of several receivers.  If more than one receiver decodes the same transmission, only the reception with the best report is used.  (Even if a single receiver is used, this eliminates multiple reports as noted above.)
• For example, if you have defined "RX1" and "RX2" you can define "MERGED_RX1" as using "RX1" and "RX2".  One can use that virtual receiver name (e.g. "MERGED_RX1") in the scheduling and band definition.
• If you have previously been using "kiwirecorder", copy its directory tree into the "~/wsprdaemon" directory.
• If necessary, rename it from "~/wsprdaemon/kiwirecorder" to "~/wsprdaemon/kiwirecorder-jks-v0.1"
• To start wsprdaemon, use "./wsprdaemon -a"
• To stop wsprdaemon, use  "./wsprdaemon -z"

This page stolen from ka7oei.blogspot.com

[End]