Iceshrimp.NET RSS Bot

Real ‘Shrimp Syndication

Turning RSS feeds into an Iceshrimp.NET bot driven timeline.

This is a much simpler reworking of the Mastodon rss-to-toot scripts.

Why ?

Mastodon has a low character count per toot.

(I am aware that character count is instance specific; don’t contact me to correct me.)

Also, I had to create several variants of the script to allow toots to fit this character limit.

Then, choose the correct variant to use, to prevent toots from exceeding the character limit.

Doing this meant that some toots were sent with, say, the RSS ‘Description’ excluded.

My Iceshrimp.NET instance character limit per Note allows for much longer posts.

So, no variants needed.

Feeding the Bot

The overall workflow concept is quite straightforward.

The python code takes an RSS feed.

It takes the latest RSS post in the feed.

If that RSS post is different to the last Note sent, it crafts a new Iceshrimp.NET Note announcing the new RSS content, and updates the Iceshrimp.NET timeline.

A cron job then runs every 15 minutes to loop through this workflow, picking up and sending new additions to the RSS feed to the Iceshrimp.NET timeline.

Prerequisites to installing the Iceshrimp.NET RSS Bot

Create a new account

Firstly, and while it may be stating the obvious, there needs to be an account to send the Notes FROM.

Obviously you can skip this part if you intend to use an existing account.

Turn Registrations ON:

nano iceshrimp.net/Iceshrimp.Backend/configuration.ini

;; Whether to allow instance registrations
;; Options: [Closed, Invite, Open]
#Registrations = Closed
Registrations = Open

Then, in a Terminal send a POST request:

curl -X 'POST' \
	'http://localhost:3000/api/iceshrimp/auth/register' \
	-H 'accept: application/json' \
	-H 'Content-Type: application/json' \
	-H 'Host: shrimp.example.org' \
	-d '{
	"username": "account_name_goes_here",
	"password": "super_long_password_goes_here"
	}'

The output should be something like this:

{"status":"authenticated","isAdmin":false,"isModerator":false,"token":"this_is_the_TOKEN","user":{"id":"9fqw78arpr8rv21m","username":"account_name_goes_here","host":null,"displayName":null,"avatarUrl":"https://shrimp.example.org/identicon/9fqw78arpr8rv21m","bannerUrl":null,"instanceName":"consummatetinkerer.net","instanceIconUrl":null}}

Make a note of the Authentication TOKEN

Secondly, make a note of the TOKEN; it will be difficult to recreate this Terminal output.

This bit is the Authentication TOKEN:

"token":"this_is_the_TOKEN"

Then, turn the Registrations back OFF:

nano iceshrimp.net/Iceshrimp.Backend/configuration.ini

;; Whether to allow instance registrations
;; Options: [Closed, Invite, Open]
Registrations = Closed
#Registrations = Open

Test the TOKEN works

Finally, test to see that it all works.

In a Terminal post something to the instance using the TOKEN provided:

curl -X 'POST' \
	'https://shrimp.example.org/api/iceshrimp/notes' \
	-H 'accept: application/json' \
	-H 'Content-Type: application/json' \
	-H "Authorization: Bearer this_is_the_TOKEN" \
	-d '{
	"text": "Test Post",
	"visibility": "public"
	}'

A couple of notes.

Firstly, the ‘text‘ parameter is required; this is the ‘body’ of the Note.

Secondly, the ‘visibility‘ parameter is required; the Note will fail to post without it.

Files for running the Iceshrimp.NET RSS Bot

Keep it secret, keep it safe

The Python file that contains the access TOKEN and other needed connection info:

secrets.py

ICESHRIMP_URL = 'https://shrimp.example.org'
ICESHRIMP_TOKEN = 'this_is_the_TOKEN'
ICESHRIMP_NOTE = ICESHRIMP_URL + '/api/iceshrimp/notes'
FEED_URL = 'rss_feed_url_goes_here'
GAME_HASHTAGS = 'game_hashtags_go_here'
NOTE_HASHTAGS = '#hashtag #hashtag #hashtag'
FILE_PATH = '/home/foo/full_filepath_goes_here/'

Sending a single Iceshrimp.NET Note

This will populate an Iceshrimp timeline with a single item of RSS content.

The script below will post only the latest item in the RSS feed:

bot_no_image.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# A bot that takes an RSS feed and posts to iceshrimp.NET
# Parses the latest entry in the RSS feed and if it doesn't match the last note, sends the latest entry

from secrets import *
from pathlib import Path
import requests
import feedparser

RSS_FEED = feedparser.parse(FEED_URL)
RSS_ENTRY = RSS_FEED.entries[0]

# Function to build the note, send it and update the 'last_note.txt' file
def send_note():

    # Build the headers including the ICESHRIMP_TOKEN
    headers = {
        'accept': 'application/json',
        'Content-Type': 'application/json',
        'Authorization' : f'Bearer {ICESHRIMP_TOKEN}',
    }

    # Build the text part of the note from the chosen 'RSS_ENTRY'
    note_str = ''
    note_str += f"{RSS_ENTRY['title']}\n\n"
    note_str += f"{RSS_ENTRY['description']}\n"
    note_str += f"\n\n{RSS_ENTRY['link']}\n\n"
    note_str += f"\n\nThis is an automated post from the {GAME_HASHTAGS} RSS feed dated {RSS_ENTRY['published']}\n\n"
    note_str += f"\n\n{NOTE_HASHTAGS}\n\n"

    # Build the payload
    payload = {
        'text': note_str,
        'visibility': 'public',
    }

    # POST the payload to the ICESHRIMP_URL
    response = requests.post(ICESHRIMP_NOTE, headers=headers, json=payload)

    # Update the 'last_note.txt' file with 'latest_entry'
    last_note = open(Path(FILE_PATH) / "last_note.txt","w")
    last_note.write(latest_entry)
    last_note.close()

    # Update the 'last_rss.txt' file with 'published' date
    last_rss = open(Path(FILE_PATH) / "last_rss.txt","w")
    last_rss.write(published)
    last_rss.close()

    # Output a response
    print("New note sent from the " + GAME_HASHTAGS + " RSS feed!")

# Find the latest RSS entry
latest_entry = RSS_ENTRY.id

# Find the latest published date
published = RSS_ENTRY.published

# Find the 'last_note' sent
with open(Path(FILE_PATH) / "last_note.txt") as last_note:
    last_note = last_note.read()

# If 'latest_entry' does not equal 'last_note' then send a new note based on 'latest_entry'
if latest_entry != last_note:
    send_note()

elif latest_entry == last_note:
    # Output a response
    print("Nothing new to add from the " + GAME_HASHTAGS + " RSS feed!")

Sending a stream of Notes

What if we want to populate an Iceshrimp timeline with a whole stream of RSS content?

The following works in a similar way to the ‘Sending a single Iceshrimp Note’ approach, except it starts at the bottom of the RSS feed.

And then loops upwards through the feed items.

That way, the latest RSS post will never be the latest Note, until the Python loop has reached the start of the RSS content.

The script below will cycle through all of the items in the RSS feed:

bot_all_no_image.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# A bot that takes an RSS feed and posts to iceshrimp.NET
# Parses the latest entry in the RSS feed and if it doesn't match the last note, sends the latest entry

from secrets import *
from pathlib import Path
import requests
import feedparser
import time

RSS_FEED = feedparser.parse(FEED_URL)

# Function to build the note, send it and update the 'last_note.txt' file
def send_note():

    # Build the headers including the ICESHRIMP_TOKEN
    headers = {
        'accept': 'application/json',
        'Content-Type': 'application/json',
        'Authorization' : f'Bearer {ICESHRIMP_TOKEN}',
    }

    # Build the text part of the note from the chosen 'RSS_ENTRY'
    note_str = ''
    note_str += f"{RSS_ENTRY['title']}\n\n"
    note_str += f"{RSS_ENTRY['description']}\n"
    note_str += f"\n\n{RSS_ENTRY['link']}\n\n"
    note_str += f"\n\nThis is an automated post from the {GAME_HASHTAGS} RSS feed dated {RSS_ENTRY['published']}\n\n"
    note_str += f"\n\n{NOTE_HASHTAGS}\n\n"

    # Build the payload
    payload = {
        'text': note_str,
        'visibility': 'public',
    }

    # POST the payload to the ICESHRIMP_URL
    response = requests.post(ICESHRIMP_NOTE, headers=headers, json=payload)

    # Update the 'last_note.txt' file with 'latest_entry'
    last_note = open(Path(FILE_PATH) / "last_note.txt","w")
    last_note.write(latest_entry)
    last_note.close()

    # Update the 'last_rss.txt' file with 'published' date
    last_rss = open(Path(FILE_PATH) / "last_rss.txt","w")
    last_rss.write(published)
    last_rss.close()

# How many RSS entries are there (-1 as the indexing starts at 0)
count = len(feedparser.parse(FEED_URL)['entries'])-1

# Loop through all of the RSS entries (until it is less than 0, where the indexing started)
print("There are", count+1, "RSS entries to send")
while count != -1:
    latest_entry = RSS_FEED.entries[count].id
    # Find the published date
    published = RSS_FEED.entries[count].published
    RSS_ENTRY = RSS_FEED.entries[count]
    #print("Sending - " + RSS_FEED.entries[count].title, count, "notes remaining")
    print("Sending - " + RSS_FEED.entries[count].title + " -", count, "notes remaining")
    send_note()
    count = count -1
    time.sleep(1)

# Output a response
print("All done!")

RSS items and Note logging

A couple of files are needed to keep track of:

What the last item in an RSS feed is
What the last Note sent was

It is by comparing these contents of these two files that the script decides whether create and send a new Note.

The text file that contains the ID of the last RSS entry:

last_rss.txt

This file can start empy, it will be written to

The text file that contains the ID of the last RSS entry that was posted as a Note:

last_note.txt

This file can start empy, it will be written to

Deploying the Iceshrimp.NET RSS Bot

I prefer to have all of the files for a given feed in the same directory.

However there is nothing to stop you having multiple bot folders, each with their own RSS feed and TOKEN.

I have more than one Python script set up and they are all called from a bash script every 15 minutes using cron.

cron job

This is what calling one of those jobs looks like.

# Run the Iceshrimp bot that turn RSS feeds into notes every N minutes
*/5 * * * * python /home/foo/bot_with_image.py &

However, these individual commands could potentially be contained in a single bash script and have cron execute that instead.

Finally, you can find this bot by searching for:

@GamesBot@ConsummateTinkerer.net

Using the Fedi App of your choice.

Code examples are on Forgejo .

Real ‘Shrimp Syndication#

Feeding the Bot#

Prerequisites to installing the Iceshrimp.NET RSS Bot#

Create a new account#

Make a note of the Authentication TOKEN#

Test the TOKEN works#

Files for running the Iceshrimp.NET RSS Bot#

Keep it secret, keep it safe#