Analyzing audio servers and streams

Working on the fmstream.org radio stream directory portal got me into the neccessity to do some research about the audio servers and their behaviour. The aim was nothing less than to compile the most comprehensive list of URLs that are compatible with the HTML5 player.

The situation:

• There is no central directory. The big platforms shoutcast.com and xiph.org (for Icecast servers) only offer a large but limited portion of stations in their yellow pages (YP) belonging to their standard. Shoutcast.com YP access requires a licence.
• Many stream-URLs are not public because they want you to use their embedded players to increase traffic on their websites. Unfortunately many of these players do not work on any device
• Server header responses are not standardized and often incorrect
• You easily end up in a mess dealing with real streams, playlists/container formats (pls,m3u,m3u8,asx,asf,ra,ram...), redirects, IPs and text-based URLs
• Even if you have the stream-URL geoblocking still persists
• The stream lists (listen.pls,stream.m3u etc) provided by the servers often contain incorrect links
• For WMA you still need an external player, but this format seems to disappear (only 0.01% of all streams detected)
• Since 2020 many browsers have started to block cross-origin-access (CORS) from secure https sites to 'insecure' http streams. The vast majority of stream servers is not ready for this transition yet.

The good news:

• Most streams are MP3, AAC or OGG. Firefox and Chrome support them all with the standard HTML5-player.
• You can easily find out the stream URL with developer tools in most browsers (F12 - network - media)
• Even Flash (becoming out-of-fashion these days) is just a container with one of the playable formats inside (the same with other container formats and players)
• Protocols other than http(s) do not play a role (very few rtsp and mms left)
• fmstream.org offers a large audio format and server statistics

Checking formats, bit rates, sample rates and channels

This is important to judge the audio quality and required bandwidth.

You can obtain this information by using...

ffprobe

ffprobe is based on the ffmpeg project and delivers the most accurate results. It is a command-line executable tool and can be integrated into all sorts of software environments. It seems that ffprobe is analyzing the deep stream data instead of just checking stream-packet-headers or http-headers.

Packet headers

The first bytes of a packet (also called frames sometimes) can obtain reliant coding information. MP3 packet headers contain bitrate and samplerate. You can find them within the stream, capturing some KBytes with the fread-command in php or other methods. They start with ff f. (12 bits of 1) both in MP3 and AAC streams. It is a reoccurring sequence of 4 (aac) or 6 (mp3) bytes. But AAC streams are more complicated. I am working on a script trying to extract the coding information: PHP audio codec info to check bitrate and samplerate.

AAC headers

• They use ADTS structure
• AAC comes in 3 major versions:
  • LC (Low complexity)
  • HE-AAC(v1) adds SBR (Spectral Band Replication)
  • HE-AACv2 also adds PS (parametric stereo). HE-AAC is also called aacPlus.
• The sample rate refers to the core data as used in LC. When SBR is used as extension in HE-AAC, the output sample rate will be doubled.
• The channel number also refers to the aac core. It is only correct if no PS (parametric stereo) from HE-AACv2 is used - then mono will be shown and output will be stereo.
• The bitrate has to be calculated by counting the number of packets within a specific file size.
  • The AAC packets (also called frames sometimes) have 1024 samples (not bytes) by definition. So if you have a sample rate (core) of 22050 Hz, that means means 22050 samples per second, then it will be 22050 / 1024 = 20 packets per second.
  • If you have 200 Bytes per packet (5 headers per KB), it will be 200 * 20 = 4000 Bytes per second.
  • One byte is 8 bits. So it is 4 Kbyte/s * 8 = 32 Kbit/s.
• Structure information for SBR and PS can be found here in the ETSI specification for DAB+ (digital radio). Still I do not understand where exactly to find the he_aac_super_frame_header within a stream. It should start with a 0-byte according to the document.

HTTP headers

The HTTP header is something totally different. It is not part of the stream but comes from a server after a request. The information is not always accurate. Icecast has some tags in it like icy-br for bitrate, sample rate and channels.

Server index pages

Icecast gives bitrate and mime type but no information about AAC extensions (HE-AAC/SBR/PS). Shoutcast gives bitrate and samplerate. Both samplerates tend to relate to the AAC core.

Some remarkable things

• Shoutcast does NOT use standard shoutcast servers for the roughly 30.000 streams they host themselves
• the 1800 _SC-suffixed streams from IHeart are not Shoutcast either
• the 'Notice2: Shoutcast Server...' in header-responses means nothing. It can be Icecast as well.

Domains

• many stream servers run on a specific IP and port like http://1.2.3.4:8000
• but be aware: servers can be configured by using host request-header: radio1.someurl.com:8000 can be different from radio2.someurl.com:8000 though DNS-servers will point to the same IP (not observed with Shoutcast servers yet)

Server types

Shoutcast

1.2.3.4:8000 response will depend on the user-agent:

• a html page (/index.html) if the user-agent in the request-header is a browser (Mozilla, Chrome, IE)
• the stream if the user-agent in the request-header is a player ('WAFA' for Winamp)
• the first stream can always be accessed with /; by the browser

Shoutcast V2

• can host more than one stream
• index.html indicates which streams are active (no xml or similar available for non-admins)
• /listen.pls?sid=[n] will produce a file (text format) with a list of URLs for stream [n] (starting with 1)
• standard stream access URL: /stream/[n]/;
• if any non-active sid is chosen, the pls will point to this URL anyway
• streams can get an individual /streampath. Numbers still work anyway.
• pls also contains the official stream name ("Title1"). Often the radio station's name. But this can be anything.
• /stats?sid=[n] shows an xml with basic data for each stream including bitrate, homepage, stream name and server version
• stream names can be anything, sometimes containing station name or bitrate or both or anything
• Title1 in the pls and streampath in stats are often identical but not always
• 70% of playlists (pls/m3u/asx) contain only one main link. When there are several ones, they often refer to the plain IP. Sometimes they lead to different programs.
• Other playlist formats (listen.m3u,listen.asx) and song info can be accessed via the Public Pages API

Shoutcast V1

• is outdated but still present
• does not support /stats, /listen.asx and multistream
• listen.pls and listen.m3u are supported but ?sid=[n] is not allowed
• index.html (or blank) will be redirected to index.html?sid=1 via the header
• SC servers do not seem to check the host in the request header. In the cases I checked multiple domains with the same IP will lead to the same result.

Icecast

• IP:port response will stream the first resource or return a web interdface depending on version (and user-agent?)
• Basic info about automated server info (does not work on all versions and configurations)
• multiple streams with their own /streampath
• /stream.m3u (playlist), status.xsl, status-json.xsl (status and stream info) and /stream.xspf (stream info) should be available since version 2.4.1. but are often not configured

Broadcaster Solutions

There is a wide range of free and commercial software packages for setting up your stream and organizing it. This list is small but it is meant to grow :-) Many solutions are based on Icecast with a few modifications and more restrictions - metadata cannot be retrieved that easy anymore.

Free Software

• Liquidsoap