Posting to Mastodon and Twitter with a Python Program

Plans and Data Design

As described on the first page in this series, my need to learn some Python plus the existence of the excellent Twython Python library for the Twitter API led to my choice to program this in Python. I found an earlier implementation, which was written in the rather old Python 2 and had some logical problems and lacked some desired functionality. But it gave me a good start.

I would use text data files, one for each identity, each file with one record per line. While the auto-posting program would be in Python, I would sometimes do some testing and analysis with Bash utilities or a Bash script.

And so, the field delimiter for each record needed to be a character with no special meaning in either language. Not "+" because of Python, not "$" because of Bash, and so on. I chose "@", which does have special meaning within both Mastodon and Twitter content, but that caused no problem.

A record is formatted as:

date time@attachment@message content...

The attachment would be optional, and seldom used, and so most of the lines in my data files look like:

date time@@message content...

Empty lines and comment lines starting "#" will be ignored.

The date and time format is: MM-DD HH:MM

The message can contain "@" and "#". Everything from the second "@" to the end of the line is the message text string, including any following "@" and "#" characters.

The result looks like this:

$ more ~/twitter/scheduled-toots-ToiletGuru
# Technical updates
02-20 09:04@@The #Python #programming language was first released OTD in 1991 https://cromwell-intl.com/open-source/python-social-media-automation/?s=tb
02-20 14:04@@The #Python #programming language was first released OTD in 1991 https://cromwell-intl.com/open-source/python-social-media-automation/mastodon-twitter-app.html?s=tb
02-20 20:04@@The #Python #programming language was first released OTD in 1991 https://cromwell-intl.com/open-source/python-social-media-automation/python-program.html?s=tb

# Timed promotion for travel pages
05-30 12:02@@Just one week until the anniversary of #DDay https://cromwell-intl.com/travel/france/normandy/?s=tb
12-24 20:03@@French resistance fighter Fernand Bonnier de La Chappelle assassinated Vichy French Admiral François Darlan OTD in 1942 after the invasion of #Morocco and Algeria https://cromwell-intl.com/travel/morocco/tangier/beat-generation.html?s=tb #history #travel

# Timed promotion for toilet-guru.com
09-21 12:02@@Just one week until Thomas Crapper's birthday! https://toilet-guru.com/thomas-crapper.html?s=tb
09-28 12:02@@It's Thomas Crapper's birthday! @ThomasCrapper https://toilet-guru.com/thomas-crapper.html?s=tb

$ more ~/twitter/scheduled-toots-Conan
01-22 08:41@robert-howard.jpg@By Crom, 'tis Robert E. Howard's birthday!
02-20 11:52@@Hoist a tankard in memory of #Conan IV, Duke of Brittany, who died on this day in 1171 https://cromwell-intl.com/travel/france/mont-saint-michel-saint-malo/?s=tb #travel #history
03-19 14:23@@On the morrow the Picts observe the #Equinox at their circles of #megaliths https://cromwell-intl.com/travel/uk/skara-brae/?s=tb #travel #prehistory
07-21 23:16@@Hoist a tankard in memory of Makoto "Mako" Iwamatsu, who played Akiro the wizard and the chronicler of my adventures, for he died OTD in 2006 https://cromwell-intl.com/travel/japan/tokyo-akihabara/?s=tb #history

Around the time that I started this project, both my parents' health declined. For about four months I lived at their place as resident cook, cleaner, laundryman, etc. Then I was there off and on up to a month at a time for the next three and a half years. Fortunately my work could be suspended for the initial four-month block, and then scheduled around the following shorter blocks. I teach short courses in Linux and networking and cybersecurity. Plus, there's an always expanding consulting project developing scripts to harden Linux and build custom media for automated installations.

I had spare time in between the valet and nursing aide work.

Wikipedia has a series of pages for every day of the year listing world events and the births and deaths of prominent figures. Each day's page ends with a list of holidays and observances, and that starts with a list of Roman Catholic feast days involving many saints in Italy, Spain, and Portugal. I have been to Italy but not really very much, while I have not been to Spain or Portugal at all.

But then there's a link to that day's Eastern Orthodox liturgical commemorations. Now we're talking, Orthodox saints from places I have been — Greece, Turkey, Bulgaria, Romania, Russia, Syria, Egypt, Estonia, Latvia, Lithuania, and so on. Many opportunities to promote travel pages.

I added the string ?s=tb to each URL. Everything from the server name to the first "?", if any, specifies the resource the client is requesting. After the "?" you can add an arbitrary list of variables:
?variable1=value1&variable2=value2...

For example, if you enter url format in the Google search box, you will be redirected to a page with a URL similar to:

https://www.google.com/search?q=url+format&oq=URL+format&aqs=chrome.0.0i512l10.3288j0j15&sourceid=chrome&ie=UTF-8

protocol:  https
server:    www.google.com
resource:  /search
variables: q=url+format&oq=URL+format&aqs=chrome.0.0i512l10.3288j0j15&sourceid=chrome&ie=UTF-8

My server ignores the added variable string, but it is logged. This way I can use fundamental UNIX-family commands to analyze how many page views are driven by posts by the two identities.

When I added automated posting to Mastodon in November 2022, I left the ?s=tb string on all URLs in both schedule files. But I added logic to the Python posting program to modify that string before posting, to indicate:
?s=tb Bob on Twitter
?s=tc Conan on Twitter
?s=mb Bob on Mastodon
?s=mc Conan on Mastodon

Install the Requests Library for Mastodon Posting

Install the latest Requests Python library.

FreeBSD:
  # pkg install py39-requests

Linux (Debian-derived):
  $ sudo apt install python3-requests

Linux (Red-Hat-derived):
  # dnf install python3-requests

Otherwise, see the project page for installation instructions:
https://docs.python-requests.org/en/latest/

These fragments of Python show how it's used:

$ cat tooter.py
#!/usr/bin/python
# On FreeBSD, use /usr/local/bin/python instead

import requests

[...]

AccessToken='fE+1Gqhq3airG9GTf8yVOh+rbRFUmKU+4wurAfejlwv'

data = { "status": 'I posted this with #Python!' }
url = "https://mastodon.world/api/v1/statuses"
r = requests.post(url,
                  data=data,
                  headers={'Authorization': 'Bearer '+AccessToken})

Install the Twython Library for Twitter Posting

The Twython library for Python provides use of the Twitter API.

Download Twython from GitHub and install it. Something like this:

% git clone https://github.com/ryanmcgrath/twython.git
% cd twython
% sudo python setup.py install 

These fragments of Python show how it's used:

$ cat tooter.py
#!/usr/bin/python
# On FreeBSD, use /usr/local/bin/python instead

from twython import Twython

[...]

AppKey='Tg4AlocM9RCObNZvaLVPTYjne'
AppSecret='HlVmmJuF0BgQc+Pior9Cogve6qWT8/cK63w9bHoz0gukU8dSn+'
OauthToken='+t3se024RdGWUU/3a7Wfxn9TGu8sLkOIBLpSuOdL/o8IFY9V+'
OauthTokenSecret='ZO4VK7AhVgpJLJ7SK1vznThA6WqH3g4Jq0Q9pd2G/gr'

twitter = Twython(AppKey, AppSecret, OauthToken, OauthTokenSecret)

twitter.update_status(status='I posted this with #Python!') 

Environment Variables and Locale

I tried running an early version of my program with a data file that contained non-ASCII data. Could it handle something like this:

Visit Sainte-Mère-Église and Λέρος and Новосибирск. #travel https://cromwell-intl.com/travel/?s=tb

No. It choked on the non-ASCII "è" and "É" before getting close to the Greek or the Cyrillic. It failed with variations of the following error message:

Traceback (most recent call last):
  File "./tooter.py", line 146, in <module>
    print('toot = '+toot)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 49: ordinal not in range(128)

It took a while to figure this out. The problem was that I actually had no environment variable LANG set to anything. The result is that things are done in ASCII. Despite an unset LANG and thus default ASCII interpretation, I could copy UTF-8 text from one window and paste it into another. So far, this situation hadn't been a problem for me.

Python was interpreting the lack of environment variable LANG as a directive to deal with data as ASCII. That fails as soon as it hits the first non-ASCII character.

I had fixed that, but after an update, things were suddenly broken. I eventually figured out that I needed another environment variable. PYTHONIOENCODING needed to be set to utf8.

For good measure, I added an entry to ~/.vimrc so I could safely copy and paste non-ASCII text from my web pages or Wikipedia pages into the schedule file:

set encoding=utf-8

The initial fix for interactive testing was to set the environment variables LANG, LC_ALL, and PYTHONIOENCODING appropriately in my shell startup file. Eventually, I also needed to set LANG and PYTHONIOENCODING when the script is run by the cron daemon.

I set LC_ALL within the program:

[...]
import locale
[...]
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
[...] 

For me the correct setting for environment variables LANG and LC_ALL was en_US.utf8, at least on Linux. And that led to...

Linux versus FreeBSD

I leave at least one Linux computer running all the time when I'm at home, but I shut things down if I'm going to be away for more than a day. So, I planned to run the scheduled script on my web server.

My server runs FreeBSD. It's a Google Compute Engine instance in a data center somewhere in the Pacific Northwest.

I have only noticed two differences between getting this code to work on both FreeBSD and Linux.

First, Python is added as a package on FreeBSD, so all the Python binaries are in /usr/local/bin/ and not in /usr/bin/ as on Linux.

Second, and this is a subtle one, the locale spelled en_US.utf8 on Linux is instead spelled en_US.UTF-8 on FreeBSD. Upper versus lower case, and add or delete a dash. Get that exactly right or else things will not work as expected!

Time Zones

My server keeps time in UTC. It resides three time zones away from my home, and sometimes I'm somewhere else, so why not UTC?

However, I want to specify posting times in my home time zone. It's the east coast of North America, so whether I'm looking to communicate with friends or draw people to a web site, that time zone makes sense to me. So:

import locale
import datetime
import time
[...]
# Server runs on UTC, specify posts in EST/EDT:
os.environ['TZ'] = 'America/New_York'
time.tzset() 

After the time.tzset() call, datetime.datetime.now() returns the time as it is in New York.

The Program

After importing the needed modules I define two functions. checkTime() compares the current time to a specified time. It returns True if they match and False otherwise.

tootIt() is passed the identity name, a string with the status text, and a string that either names an image file to attach or is empty. It then uses Twython to upload the data to Twitter.

The main code then directs output into a log file, specifies where to find input data files, and sets the locale and time zone. The main loop, as simplified pseudo code, is:

import the modules
define the functions
for each identity {
    read that identity's data file
    for each line {
        split the line into date/time, attachment, and text
        if checkTime() says date/time is right now {
            tootIt()
        }
    }
} 

Here is the full program. Notice that it contains authentication data. Make sure that your web server will not send this to a client. Don't store this within the web root.

#!/usr/local/bin/python
# Change that to /usr/bin/python on Linux.

import os
import sys
import locale
import datetime
import time
import re
import traceback
from twython import Twython

def tootIt(identity, tootText, tootAttach):

    # Yes, of course these have been replaced with random base64 data!
    if identity == 'ToiletGuru' :
	# for Twitter:
        AppKey='TW9uIEp1biAxMSAyMzowMTo1N'
        AppSecret='I9SfhXcINLF4XAp9ku1UMvAgqaDnL69Y1rvOPooXxtcPwBOnqn'
        OauthToken='2170513145-Bs4XDay16WfnRWxMkEZgx1Ap1Mg3caCqSfKiY7Z'
        OauthTokenSecret='MLGTHDBoFbVG521uUwbSiLRrwwns+R9c8aKXeaWFCAdYg'
        AccessToken='O2GDW1HodOC0XZkAtcZ+qvpYhZS2S0R9uYtR/AwhHJ8'
    elif identity == 'Conan':
	# for Twitter:
        AppKey='8ohHU22fHb297Y2iyI986ZSN1'
        AppSecret='0j7107KRDd9WZ7MsyVUQABt4RYshZYaWaL0pqsjZzbrLZ7bef6'
        OauthToken='157304025713067307-jpULUBKhy8jbj8bGoJiTlbhr1NJm2YQ'
        OauthTokenSecret='uEZt10giZJnN5ZGWQSRiRSvmgBLxv6I2ygt2xrfxpA0TB'
	# and for Mastodon:
        AccessToken='8G4Y/6EE7WjqjsQ+P0CJ2JkgwqwQhLyJF+eAEic7Hig'
    else:
        AppKey='bogus'
        AppSecret='bogus'
        OauthToken='bogus'
        OauthTokenSecret='bogus'
        AccessToken='bogus'

    # First, post to Mastodon.
    #
    # To post with an attachment we must first upload the image file to the
    # /api/vi/media URL and extract the media_id from the resulting JSON data.
    # Then we post to the /api/vi/statuses URL with the data being both the
    # text message string and the media_id.  To do up to the Mastodon limit
    # of 4 attachments, use a loop to post them and append their media_id
    # values into a sequence.
    #  -- Initially set:
    #       media_ids = []
    #  -- Replace the line:
    #       media_id = json_data['id']
    #     with:
    #       media_id = json_data['id']
    #       media_ids.append(media_id)
    #     and have that end the loop.
    #  -- Then replace the line:
    #       data = { "status": mastoText, "media_ids[]": media_id }
    #     with
    #       data = { "status": mastoText, "media_ids[]": media_ids }
    #
    # Modify the appended variable=value, "?s=tb" from the data file, to:
    #   tb = Twitter / Bob      (meaning "TweetBot" originally)
    #   tc = Twitter / Conan
    #   mb = Mastodon / Bob
    #   mb = Mastodon / Conan
    if identity == 'ToiletGuru' :
      mastoText = re.sub('s=tb', 's=mb', tootText)
      tweetText = tootText
    else:
      mastoText = re.sub('s=tb', 's=mc', tootText)
      tweetText = re.sub('s=tb', 's=tc', tootText)

    if tootAttach == '':
        data = { "status": mastoText }
    else:
        # Useful guidance found here:
        # https://roytang.net/2021/11/mastodon-api-python/
        image_file = attachDirectory+tootAttach
        data = { 'description': 'Test file ' + tootAttach }
        # Mode 'rb' = read, binary mode
        # Without b it tries to parse the data as UTF-8 text.
        f = open(image_file, 'rb')
        files = { 'file': (image_file, f, 'application/octet-stream') }
        if identity == 'ToiletGuru':
            url = "https://mastodon.world/api/v1/media"
        else:
            url = "https://mstdn.social/api/v1/media"
        r = requests.post(url,
                          files=files,
                          headers={'Authorization': 'Bearer '+AccessToken})
        json_data = r.json()
        media_id = json_data['id']
        data = { "status": mastoText, "media_ids[]": media_id }

    # Now that "data" has been set to text only or text plus image
    # references, (re)set "url" and post this.
    # We could print() r.json() to analyze what happened.
    if identity == 'ToiletGuru':
        url = "https://mastodon.world/api/v1/statuses"
    else:
        url = "https://mstdn.social/api/v1/statuses"
    r = requests.post(url,
                      data=data,
                      headers={'Authorization': 'Bearer '+AccessToken})

    # Also post to Twitter:
    twitter = Twython(AppKey, AppSecret, OauthToken, OauthTokenSecret)

    try:

	# Just text:
        if tootAttach == '':
            twitter.update_status(status=tweetText)
            print(' Successfully tweeted!')
            print('  '+tweetText)

	# Or with an attachment:
        else:
            # twitter.update_status() insists on a non-empty status.
            if tweetText == '':
                tweetText = ' '
            completeAttPath = attachDirectory+tweetAttach
            attachment = open(completeAttPath, 'rb')
            response = twitter.upload_media(media=attachment)
            twitter.update_status(status=tweetText, media_ids=[response['media_id']])
            attachment.close()
            print(' Successfully tweeted!')
            print('  content:  '+tweetText)
            print('  attached: '+tweetAttach)

    except:
        print('Error:  Could not be tweeted.')
        print(' content:  '+tweetText)
        print(' attached: '+tweetAttach)
        print('Traceback:\n'+traceback.format_exc())

    return


# Check time specification in the form "yyyy-mm-dd hh:mm"
def checkTime(nowTime, tootTime):

    # Does the current time match the specified time?
    timeIsRight = False
    targetDate, targetTime = tootTime.split(" ")

    # Is the request in the proper format?
    if re.match('^[0-9]{4}-[0-9]{2}-[0-9]{2}$',targetDate) and re.match('^[0-9]{1,2}:[0-9]{2}$',targetTime):
        # If so, is it the right day?
        if nowTime.strftime("%Y-%m-%d") == targetDate:
            # If so, is it the correct time?
            if nowTime.strftime("%H:%M") == targetTime:
                timeIsRight = True
    else:
        print(tootTime+' -> Wrong date/time format!')

    return timeIsRight


###############################################################################
# Main program starts here

# All cron job output is mailed to the owner unless redirected.
# Uncomment these two lines to append all output to a log file:
sys.stdout = open('/tmp/tooter.log', 'a')
sys.stderr = open('/tmp/tooter-error.log', 'a')
# Uncomment this line to discard standard output:
# sys.stdout = open('/dev/null', 'w')

# Data locations
dataDirectory   = '/home/cromwell/twitter/'
attachDirectory = '/home/cromwell/twitter/attach/'
# file will be this base plus identity name
tootFileBase = 'scheduled-toots-'

# locale = en_US.UTF-8  FreeBSD
#          en_US.utf8   Linux
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

# Server runs on UTC, specify posts in EST/EDT:
os.environ['TZ'] = 'America/New_York'
time.tzset()
nowTime = datetime.datetime.now()

for identity in [ "ToiletGuru", "Conan" ] :

    tootFile = tootFileBase+identity

    # Read list of scheduled posts
    with open(dataDirectory+tootFile, 'r') as tootList:

        for toot in tootList:

            # Remove any leading/trailing white space.
            toot = toot.strip()

	    # Blank lines result in empty strings, silently continue.
	    # Also skip any lines whose first non-blank character is "#".
            if toot != '' and not re.match('^#.*', toot) :

                try:
                    # Each line should be the following, attachment is optional:
                    #   time@attachment@content
                    # in format:
                    #   yyyy-mm-dd hh:mm@[attachment.jpg]@This message includes "@".
                    tootTime, tootAttach, tootText = re.split('@', toot, 2)
                    # Remove any leading/trailing white space.
                    tootTime = tootTime.strip()
                    tootAttach = tootAttach.strip()
                    tootText = tootText.strip()
                    if len(tootText) > 280:
                        print('Text is too long:')
                        print(' '+tootText)
                    elif tootText == '' and tootAttach == '':
                        print('Refusing to post empty toot.')
                    elif checkTime(nowTime, tootTime):
                            print('Trying to post for '+identity+' at '+tootTime)
                            tootIt(identity, tootText, tootAttach)

                except ValueError:
                    print('Incorrect line format.')
                    print(' Line needs two "@" delimiters in the line, more can be in text:')
                    print(' yyyy-mm-dd hh:mm@[attachment.jpg]@This message includes "@".')
                    print('I found:')
                    print(' '+toot)
                    pass

Scheduling With cron

I tell the cron daemon to run the program every minute. Note that you must set any needed environment variables within the crontab file.

% crontab -l
# Must set environment variable(s)
LANG = en_US.UTF-8
PYTHONIOENCODING = utf8
# min  hr  day-of-month  month  day-of-week  command
* * * * * /home/cromwell/mastodon/tooter.py

It Works!

Now you can easily schedule toots and tweets with a very simple interface. No need to struggle with the tedious point and click scheduling on TweetDeck.

🦣 Bob the Traveler   🦣 Conan the Sysadmin
🐦 ToiletGuru 🐦 Conan the Sysadmin

As for the Unicode:
🦣 = 🦣
🐦 = 🐦

I have found https://fedi.tips/ to be very helpful.

If you wonder "How do I install something like TweetDeck for Mastodon?"
Click Preferences,
then Enable advanced web interface,
then Save changes

I immediately found Mastodon to have much more interaction than Twitter.

As for how much added visibility Mastodon brings, definitely a lot but it's hard to quantify. When you post a link on a Mastodon server, every server that sees the posts connects to the URL to resolve metadata.

Links in Mastodon posts should appear accompanied by a preview, once the various instances have cached the preview data. When a post contains a URL, Mastodon instances read the referenced page to record the <title> string and, within the <head> section, the image in the Twitter and Open Graph metadata strings in the <head>...</head> section of the page:

<!DOCTYPE html>
<html lang="en" xml:lang="en">
    <head>
        <meta charset="UTF-8">
        [...]
        <meta name="twitter:image" content="...">
        <meta name="og:image" content="...">
        <meta name="og:site_name" content="...">
        [...]
    </head>
    <body>
        [...]
    </body>
</html> 

Then it shows those as a preview some of the time. Not always. See this discussion for more on link previews.