Showing posts with label fst4w. Show all posts
Showing posts with label fst4w. Show all posts

Friday, October 20, 2023

Multi-band transmitter and monitoring system for Eclipse monitoring (Part 1)

It should not have escaped your attention - at least if you live in North America - there there have been/will be two significant solar eclipses occurring in recent/near times:  One that occurred on October 14, 2023 and another eclipse that will happen during April, 2024.  The path of "totality" of the October eclipse happened to pass through Utah (where I live) so it is no surprise that I went out of my way to see it - just as I did back in 2012:  You can read my blog entry about that here.

 Figure 1:
The eclipse in progress - a few minutes
before "annularity".
(Photo by C. L. Turner)
I will shortly produce a blog entry related to my activities around the October 14, 2023 eclipse as well.

The October eclipse was of the "annular" type meaning that the moon is near-ish apogee meaning that the subtended angle of its disk is insufficient to completely block the sun owing to the moon's greater-than-average distance from Earth:  Unlike a solar eclipse, there is no time during the eclipse where it is safe to look at the sun/moon directly, without eye protection.

The sun will be mostly blocked, however, meaning that those in the path of "totality" experienced a rather eerie local twilight with shadows casting images of the solar disk:  Around the periphery of the moon it was be possible to make out the outline of lunar mountains - and those unfortunate to stare at the sun during this time will receive a ring-shaped burn to their retina.

From the aspect of a radio amateur, however, the effects of a total and annular solar eclipse are largely identical:  The diminution of the "D" layer and partial recombination of the "F" layers of the ionosphere causing what are essentially nighttime propagation conditions during the daytime - geographically limited to those areas under the lunar shadow.

In an effort to help study these sort of effects - and to (hopefully) better-understand the propagation effects, a number of amateurs went (and are) going out into the field - in or near the path of "totality" - and setting up simultaneous, multi-band transmitters.

Producing usable data

Having "Eclipse QSO Parties" where amateur radio operators make contacts during the eclipse likely goes back nearly a century - the rarity of a solar eclipse making the event even more enigmatic.  In more recent years amateurs have been involved in "citizen science" where they make observations by monitoring signals - or facilitate the making of observations by transmitting them - and this happened during the October eclipse and should also happen during the April event as well.

While doing this sort of thing is just plain "fun", a subset of this group is of the metrological sort (that's "metrology", no "meteorology"!) and endeavor to impart on their transmissions - and observations of received signals - additional constraints that are intended to make this data useful in a scientific sense - specifically:

  • Stable transmit frequencies.  During the event, the perturbations of the ionosphere will impart on propagated signals Doppler shift and spread:  Being able to measure this with accuracy and precision (which are NOT the same thing!) adds another layer of extractable information to the observations.
  • Stable receivers.  As with the transmitters, having a stable receiver is imperative to allow accurate measurement of the Doppler shift and spread.  Additionally, being able to monitor the amplitude of a received signal can provide clues as to the nature of the changing conditions.
  • Monitoring/transmitting at multiple frequencies.  As the ionospheric conditions change, its effects at different frequencies also changes.  In general, the loss of ionization (caused by darkness) reduces propagation at higher frequencies (e.g. >10 MHz) and with lessened "D" layer absorption lower frequencies (<10 MHz) the propagation at those frequencies is enhanced.  With the different effects at different frequencies, being able to simultaneously monitor multiple signals across the HF spectrum can provide additional insight as to the effects.

To this end, the transmission and monitoring of signals by this informal group have established the following:

  • GPS-referenced transmitters.  The transmitters will be "locked" to GPS-referenced oscillators or atomic standards to keep the transmitted frequencies both stable, accurate - and known to within milliHertz.
  • GPS referenced receivers.  As with the transmitters, the receivers will also be GPS-referenced or atomic-referenced to provide milliHertz accuracy and stability.

With this level of accuracy and precision the frequency uncertainties related to the receiver and transmitter can be removed from the Doppler data.  For generation of stable frequencies, a "GPS Disciplined Oscillator" is often used - but very good Rubidium-based references are also available, although unlike a GPS-based reference, the time-of-day cannot be obtained from them.

Why this is important:

Not to demean previous efforts in monitoring propagation - including that which occurs during an eclipse - but unless appropriate measures are taken, their contribution to "real" scientific analysis can be unwittingly diminished.  Here are a few points to consider:

  • Receiver frequency stability.  One aspect of propagation on HF is that the signal paths between the receiver and transmitter change as the ionosphere itself changes.  These changes can be on the order of Hertz in some cases, but these changes are often measured in 10s of milliHertz.  Very few receivers have that sort of stability and the drift of such a receiver can make detection of these Doppler shifts impossible.
  • Signal amplitude measurement.  HF signals change in amplitude constantly - and this can tell us something about the path.  Pretty much all modern receivers have some form of AGC (Automatic Gain Control) whose job it is to make sure that the speaker output is constant.  If you are trying to infer signal strength, however, making a recording with AGC active renders meaningful measurements of signal strength pretty much impossible.  Not often considered is the fact that such changes in propagation also affect the background noise - which is also important to be able to measure - and this, too, is impossible with AGC active.
  • Time-stamping recordings.  Knowing when a recording starts and stops with precision allows correlation with other's efforts.  Fortunately this is likely the easiest aspect to manage as a computer with an accurate clock can automatically do so (provided that one takes care to preserve the time stamps of the file, or has file names that contain such information) - and it is particularly easy if one happens to be recording a time station like WWV, WWVH, WWVB or CHU.

In other words, the act of "holding a microphone up to a speaker" or simply recording the output of a receiver to a .wav file with little/no additional context makes for a curious keepsake, but it makes the challenge of gleaning useful data from it more difficult.

One of our challenges as "citizen scientists" is to make the data as useful as possible to us and others - and this task has been made far easier with inexpensive and very good hardware than it ever has been - provided we take care to do so.  What follows in this article - and subsequent parts - are my reflections on some possible ways to do this:  These are certainly not the only ways - or even the best ways - and even those considerations will change over time as more/different resources and gear become available to the average citizen scientist. 

* * *

How this is done - Receiver:

The frequency stability and accuracy of MOST amateur transceivers is nowhere near good enough to provide usable observations of Doppler shift on such signals - even if the transceiver is equipped with a TCXO or other high-stability oscillator:  Of the few radios that can do this "out of the box" are some of the Flex transceivers equipped with a GPS disciplined oscillator.

To a certain degree, an out-of-the-box KiwiSDR can do this if properly set-up:  With a good, reliable GPS signals and when placed within a temperature-stable environment (e.g. temperature change of 1 degree C or so during the time of the observation) they can be stable enough to provide useful data - but there is no guarantee of such.

To remove such uncertainty a GPS-based frequency reference is often applied to the KiwiSDR - often in the form of the Leo Bodnar GPS reference, producing a frequency of precisely 66.660 MHz.  This combination produces both stable and accurate results.  Unfortunately, if you don't already have a KiwiSDR, you probably aren't going to get one as the original version was discontinued in 2022:  A "KiwiSDR 2" is in the works, but there' no guarantee that it will make it into production, let alone be available in time for the April, 2024 eclipse. 

Figure 2:
The RX-888 (Mk2) - a simple and relatively inexpensive
box that is capable of "inhaling" all of HF at once.
Click on the image for a larger version.

The RX-888 (Mk2)

A suitable work-around has been found to be the RX-888 (Mk2) - a simple direct-sampling SDR - available for about $160 shipped (if you look around).  This device has the capability of accepting an external 27 MHz clock (if you add an external cable/connector to the internal U.FL connector provided for this purpose) in which it can become as stable and accurate as the external reference.

This SDR - unlike the KiwiSDR, the Red Pitaya and others - has no onboard processing capability as it is simply an analog-to-digital coupled with a USB3 interface so it takes a fairly powerful computer and special processing software to be able to handle a full-spectrum acquisition of HF frequencies.

Software that is particularly well-suited to this task is KA9Q-Radio (link).  Using the "overlap and save" technique, it is extraordinarily efficient in processing the 65 Megasamples-per-second of data needed to "inhale" the entire HF spectrum.  This software is efficient enough that a modest quad-core Intel i5 or i7 is more than up to the task - and such PCs can be had for well under $200 on the used market.

KA9Q-Radio can produce hundreds of simultaneous virtual receivers of arbitrary modes and bandwidths which means that one such virtual receiver can be produced for each WSPR frequency band:  Similar virtual receivers could be established for FT-8, FT-4, WWV/H and CHU frequencies.  The outputs of these receivers - which could be a simple, single-channel stream or a pair of audio in I/Q configuration - can be recorded for later analysis and/or sent to another program (such as the WSJT-X suite) for analysis.

Additionally, using the WSPRDaemon software, the multi-frequency capability of KA9Q-Radio can be further-leveraged to produce not only decodes of WSPR and FST4W data, but also make rotating, archival I/Q recordings around the WSPR frequency segments - or any other frequency segments (such as WWV, CHU, Mediumwave or Shortwave broadcast, etc.) that you wish.

Comment:  I have written about the RX-888 in previous blog posts:

  • Improving the thermal management of the RX-888 (Mk 2) - link 
  • Measuring signal dynamics of the RX-888 (Mk 2) - link

Full-Spectrum recording

Yet another capability possible with the RX-888 (Mk2) is the ability to make a "full spectrum" recording - that is, write the full sample rate (typically 64.8 Msps) to a storage device.  The result are files of about 7.7 gigabytes per minute of recording that contain everything that was received by the RX-888, with the same frequency accuracy and precision as the GPS reference used to clock the sample rate of the '888.  

What this means is that there is the potential that these recordings can be analyzed later to further divine aspects of the propagation changes that occurred during, before and after the eclipse - especially by observing signals or aspects of the RF environment itself that one may not have initially thought to consider:  This also can allow the monitoring of the overall background noise across the HF spectrum to see what changes during the eclipse, potentially filling in details that might have been missed on the narrowband recordings.

Because such a recording contains the recordings of time stations (WWV, WWVH, CHU and even WWVB) it may be possible to divine changes in propagation delay between those transmit sites and the receive sites.  If a similar GPS-based signal is injected locally, this, too, can form another data point - not only for the purposes of comparison of off-air signals, but also to help synchronize and validate the recording itself.

By observing such a local signal it would be possible to time the recording to within a few 10s of nanoseconds of GPS time - and it would also be practical to determine if the recording itself was "damaged" in some way (e.g. missed samples from the receiver):  Even if a recording is "flawed" in some way, knowing the precise location an duration of the missing data allows this to be taken into account and to a large extent, permit the data "around" it to still be useful.

Actually doing it:

Up to this point there has been a lot of "it's possible to" and "we have the capability of" mentioned - but pretty much everything mentioned so far was used during the October, 2023 eclipse.  To a degree, this eclipse is considered to be a rehearsal for the April 2024 event in that we would be using the same techniques - refined, of course, based on our experiences.

While this blog will mostly refer to my efforts (because I was there!) there were a number of similarly-equipped parties out in the fields and at home/fixed stations transmitting and receiving and it is the cumulative effort - and especially the discussions of what worked and what did not - that will be valuable in preparation for the April event.  Not to be overlooked, this also gives us valuable experience with propagation monitoring overall - an ongoing effort using WSPRDaemon - where we have been looking for/using other hardware/software to augment/improve our capabilities.

In Part 2 I'll talk about the receive hardware and techniques in more detail.


Stolen from ka7oei.blogspot.com

[END]



Thursday, October 22, 2020

Using the jt9 executable to receive FST4W signals

Note: 

Since originally posted, WSJT-X v2.3.0-rc2 was released, adding a feature to the "JT9" executable that simplifies this process (the "-F" parameter) as described below.  Note that most of this post was originally written soon after "rc1" had become available.

* * *

As a heavy user of K1JT's WSPR and operating on the 2200 and 630 meter bands, I have noted with interest the introduction of the "FST4W" mode in the recent (v2.3.0-rc1) wsjt-x release.  Operating using the same detection bandwidth as WSPR (when FST4W is operated in the 120 second mode) it offers a theoretical 1.4dB improvement in detection sensitivity.

Being involved with wsprdaemon (link to that project here ) - an open-source project that automates and optimizes reception of WSPR signals on all bands, particularly if multiple receivers/antennas are used - we have been watching this development with interest, particularly since FST4W has the likelihood of supplanting conventional WSPR operation, especially on the lowest amateur bands (2200, 630 and possibly 160 meters) where minimal of Doppler shift is expected.

Internally, WSJT-X  uses the subordinate wsprd program as the decoding (and encoding) engine.  As a stand-alone program, the wsprd executable code may be invoked with a command line to decode signals contained within a .wav file that was captured during the standard two minute interval - aligned with even UTC minutes - and produce a text file containing the decoded signals.

Why use the executable rather than the entire wsjt-x suite?  The fact is that the use of the wsjt-x suite does not lend itself easily to script-driven, bare-minimum, lightweight implementations where further processing of the decoded data (to remove duplicate decodes from multiple receivers, antennas and to use this same data for further analysis of signal/noise) is desired.

The "jt9" executable:

After a bit of digging about, it was "discovered" that FST4W - being an offshoot of the JT9 protocol - was handled not by the wsprd executable, but the jt9 executable.  Simply executing this program with no arguments will yield a list of command-line arguments which, on the face of it, made it appear that updating the wsprdaemon to include the decoding of FST4W signals would be a relatively simple matter.

Except that it didn't work.

Initial testing with strong, off-air FST4W signals that was known to be decodable (because farther-flung stations were able to decode the very same transmissions) yielded no results when the .wav file was applied to the jt9 program - but automatic execution over many hours yielded the occasional off-air decode.  Confused by this, I sought help on the WSJT-X groups.io forum.  Fortunately, Joe Taylor and several of the developers offered a clue:  The "-f" parameter of the jt9 executable, described minimally as "Receive Frequency Offset".

Apparently, the default center frequency of the jt9 executable - at least when in FST4W mode (and maybe others) is 1500 Hz - a fact implied when one gets the display of command-line arguments.  What is not so clear - and only alluded to in the available documentation - is that the apparent bandwidth of the decoding, at least in the 120 second mode, is on the order of 40 Hz (+/- 20 Hz)Addendum:  This issue was fixed with the "-F" parameter - see below.

At a quick glance through the source code (file "jt9.f90"), this bandwidth setting appears to be hard-coded into a shared variable (apparently accessible by other programs in the WSJT-X suite) called "ntol" (likely a number referring to the "frequency tolerance" setting in the GUI) that is not available via the jt9 command line - at least, not without modification of the source code.  (The possibility of directly accessing these shared variables exists - but this would be platform-specific, a bit messy and somewhat dangerous!)

Unfortunately, this fixed +/-20Hz bandwidth does not appear to be compatible with the way that the FST4W mode has (already!) found use on 2200 and 630 meters where it is used along-side the WSPR mode in the 200 Hz subbands.

A hell of a kludge:

Update - kludge no longer needed:

As of version 2.3.0-rc2 it appears that a new parameter "-F" was added to allow something other than a +/-20Hz bandwidth (referred to as "tolerance") to be used, likely eliminating the need for multiple decodes, below.  A possible command-line for this would be:
jt9 -W -p 120 -f 1500 -F 200 <wav file to be processed> 
With the center frequency (-f) being the center of the passband (1500 Hz) and the "-F" parameter referred to as"tolerance" (e.g. detection bandwidth) being 200 Hz. 
Initial testing indicates that the -F parameter does what it's supposed to do and the kludge below is now longer required.

This fact implies that in order to use something other than the GUI version of the wsjt-x software, a work-around must be invoked.  The following is a bare-minimum example of how one might do this via the command line:

jt9 -W -p 120 -f 1420 <wav file to be processed> 

jt9 -W -p 120 -f 1460 <wav file to be processed>

jt9 -W -p 120 -f 1500 <wav file to be processed>

jt9 -W -p 120 -f 1540 <wav file to be processed>

jt9 -W -p 120 -f 1580 <wav file to be processed>

(One might include the -H, -L and -d parameters in actual practice.)

In other words, in order to cover the entire 200 Hz WSPR subband, the JT9 executable (v2.3.0-rc1) must be executed - processing the same .wav file - at least five times:  The results of the decoding will, in each case, be found in the file "decoded.txt".  If one wishes to implement an equivalent of the -w parameter of the wsprd executable (e.g. +/- 150 Hz "wideband" mode), you will need even more invocations than above.

The result from the above mess will be five different decoding results, each of which must be saved (e.g. renamed) between subsequent executions to prevent overwriting by the previous instance.  After this, the five results would be concatenated to yield a single file - but there is a catch:  It is likely - particularly if the signal is strong - that the same signal will be decoded more than once.  Apparently, the "+/- 20Hz" limit isn't the result of a "brick-wall" filter:  Signals beyond this frequency range may be decoded, but the reported S/N values will likely be reduced as distance of the received signal from the specified center frequency increases.  In short, this means that the results of the concatenated version of the "decoded" file(s) must be sorted and all but the single, "strongest" decode (e.g. best SNR) for each station must be discarded.

Comment:

It would appear that just five iterations to cover the 200 Hz bandwidth is not enough:  I received correspondence from a reader of this blog that observed that a frequency variation of less than 10 Hz from that defined by the "-f" parameter can affect the S/N reading by about 1 dBMake of that what you will!

* * * * * * * * *

If one wishes to integrate the FST4W decodes into the existing WSPR captures for processing, yet another step must be undertaken:  "Fixing" the formatting.  Not surprisingly, the output in the "decoded.txt" is not formatted the same as the results of the decoding from the wsprd executable meaning that one will need to do a few things, after the fact, to "fix" them - particularly if you wish to forward them to wsprnet.org, including:

  • Supply the date.  The "decoded.txt" includes the time - but not the date.  Because date of the .wav file may not be the same as the system date (e.g. later processing of the .wav files - or the interval being processed occurred just before the new day) - one must use the actual date of the recording.  The obvious place to obtain this is from the name of the .wav file being processed.
  • Frequency offset.  The information that one might send to wsprnet.org must include the carrier frequency of the received signal, but the output in the "decoded" file has only the audio frequency:  One must obtain the LO frequency of the receiver being used from "somewhere else" and calculate this on the fly.
  • Supply missing information.  The "decoded.txt" file does not have all of the same information fields that one might supply when uploading WSPR spots, so this information must be added as necessary.
  • Arrange the fields in the proper order.  Once the needed information is applied, one will probably want to use "awk" or similar to produce the same order as the wsprd data - assuming this wasn't already done in the process.

* * *

There are two outputs from the jt9 executable - one directly from the program itself to the standard console and that output to the file "decoded.txt" - and the latter is the most useful. 

Console output:

0416 -24  0.7 1515 `  KA7OEI DN40 17                                 
<DecodeFinished>   0   1

The fields are:  <time UTC> <SNR in dB> <DT?> <Audio frequency in Hz> <always "`"> <Callsign received> <Grid of received station> <Reported TX power in dBm>

From the "decoded.txt" file:

0416   0  -24   0.7   1515.   0   KA7OEI DN40 17                        FST4

 The fields are:  <time UTC> <unknown - possibly drift in Hz> <SNR in dB> <DT?> <Audio frequency in Hz> <unknown> <Callsign received> <Grid of received station> <Reported TX power in dBm> <Always "FST4">

* * *

There you have it:  The germ of what would be needed if one wishes to supplement the existing WSPR decodes with the newer FST4W mode using just the bare executables.  If one wishes to decode other than the 120 second FST4W mode, things get even more complicated!

Sample audio file:

An audio file containing both FST4W-120 and WSPR transmissions may be found HERE - right-click to download.  This file contains an FST4W-120 transmission by KA7OEI from about 116km distant and (at least) two WSPR transmissions.

* * * 
 
Note:  It appears that the "-F" parameter, above, modifies the default ntol setting as described above.

P.S.:  While it would be pretty trivial tweak the code to allow modification of the ntol variable via command line, this would complicate the ongoing maintenance of the wsprdaemon code.  We can only hope that the current authors see fit to include a means by which the entire wspr subband can be monitored with a single invocation of the jt9 executable.

 

This page stolen from ka7oei.blogspot.com

[End]