Posting to Twitter with a Python Program

Twitter and Python

I will use the excellent Twython wrapper for the Twitter API. It works with Python 2.6+ and Python 3.

I started by testing the tweet_schedule.py program. It doesn't run with Python 3, it does some things I don't particularly want, and it doesn't do some things that I do want. But it was a good starting point.

tweet_schedule.py

That program was useful for testing, and to figure out how Twython works. And it served as the model for what became my first real Python program!

I reimplemented the project in Python 3, with significant changes to the logic. My version supports multiple Twitter identities. In a single run it checks the to-do lists for all identities, posting plain text messages as well as images with accompanying text.

Installing the Twython Module

Twython

Download Twython from GitHub and install it. Something like this:

% mkdir ~/twitter
% cd ~/twitter
% git clone git://github.com/ryanmcgrath/twython.git
% cd twython
% sudo python setup.py install

These fragments of Python show how it's used:

from twython import Twython

[...]

twitter = Twython(AppKey, AppSecret, OauthToken, OauthTokenSecret)

twitter.update_status(status='I posted this with #Python!')

Multiple Identities

I am maintaining just two Twitter identities right now. My main Twitter identity is @ToiletGuru, a mix of travel and Linux and cybersecurity, plus experiments with promoting a site that's a mix of travel and plumbing. The other is @conan_sysadmin, more strictly limited to Linux and networking and cybersecurity.

My program simply reads multiple schedule files, one for each identity. If it's time for one of them to post a message, the program selects the appropriate authentication data for that identity.

The schedule files are one line per record with "@" as the delimiter: date and time, then the attached image if any, then the content. The content can itself contain an "@" as in this example. Blank lines are ignored, lines with "#" as their first non-whitespace character are ignored, and white space at the beginning and end of each field will be removed.

$ more ~/twitter/scheduled-tweets-ToiletGuru
# Technical updates
2018-06-02 21:34@@Working on a #Python program https://cromwell-intl.com/open-source/python-twitter-automation/python-program.html

# Travel promotion
2018-05-30 12:02@@Just one week until the anniversary of #DDay https://cromwell-intl.com/travel/france/normandy/

# Timed promotion for toilet-guru.com
2018-09-21 12:02@@Just one week until Thomas Crapper's birthday! https://toilet-guru.com/thomas-crapper.php
2018-09-28 12:02@@It's Thomas Crapper's birthday! @ThomasCrapper https://toilet-guru.com/thomas-crapper.php

$ more ~/twitter/scheduled-tweets-Conan
2018-01-22 08:41 @ robert-howard.jpg @ By Crom, 'tis Robert E. Howard's birthday!

Environment Variables and Locale

I tried running an early version of my program with a data file that contained non-ASCII data. Could it handle something like this:

Visit Sainte-Mère-Église and Новосибирск.

No. It choked on the non-ASCII "è" and "É" before getting close to the Cyrillic. It failed with variations of the following error message:

Traceback (most recent call last):
  File "./tweeter.py", line 146, in <module>
    print('tweet = '+tweet)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 49: ordinal not in range(128)

It took a while to figure this out. The problem was that I actually had no environment variable LANG set to anything. The result is that things are done in ASCII. Despite an unset LANG and thus default ASCII interpretation, I could copy UTF-8 text from one window and paste it into another. So far, this situation hadn't been a problem for me.

Python was interpreting the lack of environment variable LANG as a directive to deal with data as ASCII. That fails as soon as it hits the first non-ASCII character.

The fix was to set the environment variable LANG appropriately in my shell startup file. I also explicitly set LC_ALL, and for good measure, I added an entry to ~/.vimrc:

set encoding=utf-8

For me the correct setting for environment variables LANG and LC_ALL was en_US.utf8, at least on Linux. And that led to...

Linux versus FreeBSD

I leave at least one Linux computer running all the time when I'm at home, but I shut things down if I'm going to be away for more than a day. So, I planned to run the scheduled script on my web server.

My server runs FreeBSD. It's a Google Compute Engine instance in a data center somewhere in the Pacific Northwest. (Another page has a description of how to deploy FreeBSD in Google Cloud)

I have only noticed two differences between getting this code to work in both places.

First, Python is added as a package on FreeBSD, so all the Python binaries are in /usr/local/bin and not in /usr/bin as on Linux.

Second, and this is a subtle one, the locale spelled en_US.utf8 on Linux is en_US.UTF-8 on FreeBSD. Upper versus lower case, and add or delete a dash. Get that exactly right or else things will not work as expected!

Update — After an update, things were suddenly broken. I eventually figured out I needed another environment variable. PYTHONIOENCODING needed to be set to utf8.

I set that in my shell setup file and within the program. Here are some fragments of the Python program:

import locale
[...]
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

Time Zones

My server keeps time in UTC. It resides three time zones away from my home, and sometimes I'm somewhere else, so why not UTC?

However, I want to specify tweet times in my home time zone. It's the east coast of North America, so whether I'm looking to communicate with friends or draw people to a web site, that time zone makes sense to me. So:

import locale
import datetime
import time
[...]
# Server runs on UTC, specify posts in EST/EDT:
os.environ['TZ'] = 'America/New_York'
time.tzset()

After the time.tzset() call, datetime.datetime.now() returns the time as it is in New York.

The Program

After importing the needed modules I define two functions. checkTime() compares the current time to a specified time. It returns True if they match and False otherwise.

tweetIt() is passed the identity name, a string with the status text, and a string that either names an image file to attach or is empty. It then uses Twython to upload the data to Twitter.

The main code then directs output into a log file, specifies where to find input data files, and sets the locale and time zone. The main loop, as simplified pseudo code, is:

import the modules
define the functions
for each identity {
    read that identity's data file
    for each line {
        split the line into date/time, attachment, and text
        if checkTime() says date/time is right now {
            tweetIt()
        }
    }
}

Here is the full program.

#!/usr/local/bin/python3.6

import os
import sys
import locale
import datetime
import time
import re
import traceback
from twython import Twython

def tweetIt(identity, tweetText, tweetAttach):

    # Yes, of course these have been replaced with random data!
    if identity == 'ToiletGuru' :
        AppKey='TW9uIEp1biAxMSAyMzowMTo1N'
        AppSecret='I9SfhXcINLF4XAp9ku1UMvAgqaDnL69Y1rvOPooXxtcPwBOnqn'
        OauthToken='2170513145-Bs4XDay16WfnRWxMkEZgx1Ap1Mg3caCqSfKiY7Z'
        OauthTokenSecret='MLGTHDBoFbVG521uUwbSiLRrwwns+R9c8aKXeaWFCAdYg'
    elif identity == 'Conan':
        AppKey='8ohHU22fHb297Y2iyI986ZSN1'
        AppSecret='0j7107KRDd9WZ7MsyVUQABt4RYshZYaWaL0pqsjZzbrLZ7bef6'
        OauthToken='157304025713067307-jpULUBKhy8jbj8bGoJiTlbhr1NJm2YQ'
        OauthTokenSecret='uEZt10giZJnN5ZGWQSRiRSvmgBLxv6I2ygt2xrfxpA0TB'
    else:
        AppKey='bogus'
        AppSecret='bogus'
        OauthToken='bogus'
        OauthTokenSecret='bogus'

    twitter = Twython(AppKey, AppSecret, OauthToken, OauthTokenSecret)

    try:

	# Just text:
        if tweetAttach == '':
            twitter.update_status(status=tweetText)
            print(' Successfully tweeted!')
            print('  '+tweetText)

	# Or with an attachment:
        else:
            # twitter.update_status() insists on a non-empty status.
            if tweetText == '':
                tweetText = ' '
            completeAttPath = attachDirectory+tweetAttach
            attachment = open(completeAttPath, 'rb')
            response = twitter.upload_media(media=attachment)
            twitter.update_status(status=tweetText, media_ids=[response['media_id']])
            attachment.close()
            print(' Successfully tweeted!')
            print('  content:  '+tweetText)
            print('  attached: '+tweetAttach)

    except:
        print('Error:  Could not be tweeted.')
        print(' content:  '+tweetText)
        print(' attached: '+tweetAttach)
        print('Traceback:\n'+traceback.format_exc())

    return


# Check time specification in the form "yyyy-mm-dd hh:mm"
def checkTime(nowTime, tweetTime):

    # Does the current time match the specified time?
    timeIsRight = False
    targetDate, targetTime = tweetTime.split(" ")

    # Is the request in the proper format?
    if re.match('^[0-9]{4}-[0-9]{2}-[0-9]{2}$',targetDate) and re.match('^[0-9]{1,2}:[0-9]{2}$',targetTime):
        # Is it the right day?
        if nowTime.strftime("%Y-%m-%d") == targetDate:
            # If so, is it the correct time?
            if nowTime.strftime("%H:%M") == targetTime:
                timeIsRight = True
    else:
        print(tweetTime+' -> Wrong date/time format!')

    return timeIsRight


###############################################################################
# Main program starts here

# All cron job output is mailed to the owner unless redirected.
# Uncomment these two lines to append all output to a log file:
sys.stdout = open('/tmp/tweeter.log', 'a')
sys.stderr = open('/tmp/tweeter.log', 'a')
# Uncomment this line to discard standard output:
# sys.stdout = open('/dev/null', 'w')

# Data locations
dataDirectory   = '/home/cromwell/twitter/'
attachDirectory = '/home/cromwell/twitter/attach/'
# file will be this base plus Twitter identity name
tweetFileBase = 'scheduled-tweets-'

# locale = en_US.UTF-8  FreeBSD
#          en_US.utf8   Linux
locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')

# Server runs on UTC, specify posts in EST/EDT:
os.environ['TZ'] = 'America/New_York'
time.tzset()
nowTime = datetime.datetime.now()

for identity in [ "ToiletGuru", "Conan" ] :

    tweetFile = tweetFileBase+identity

    # Read list of scheduled tweets
    with open(dataDirectory+tweetFile, 'r') as tweetList:

        for tweet in tweetList:

            # Remove any leading/trailing white space.
            tweet = tweet.strip()

	    # Blank lines result in empty strings, silently continue.
	    # Also skip any lines whose first non-blank character is "#".
            if tweet != '' and not re.match('^#.*', tweet) :

                try:
                    # Each line should be the following, attachment is optional:
                    #   time@attachment@content
                    # in format:
                    #   yyyy-mm-dd hh:mm@[attachment.jpg]@This message includes "@".
                    tweetTime, tweetAttach, tweetText = re.split('@', tweet, 2)
                    # Remove any leading/trailing white space.
                    tweetTime = tweetTime.strip()
                    tweetAttach = tweetAttach.strip()
                    tweetText = tweetText.strip()
                    if len(tweetText) > 280:
                        print('Tweet is too long:')
                        print(' '+tweetText)
                    elif tweetText == '' and tweetAttach == '':
                        print('Refusing to post empty tweet.')
                    elif checkTime(nowTime, tweetTime):
                            print('Trying to post for '+identity+' at '+tweetTime)
                            tweetIt(identity, tweetText, tweetAttach)

                except ValueError:
                    print('Incorrect line format.')
                    print(' Line needs two "@" delimiters in the line, more can be in text:')
                    print(' yyyy-mm-dd hh:mm@[attachment.jpg]@This message includes "@".')
                    print('I found:')
                    print(' '+tweet)
                    pass

Scheduling With `cron`

I tell the cron daemon to run the program every minute. Note that you must set any needed environment variables within the crontab file.

% crontab -l
# Must set environment variable(s)
LANG = en_US.UTF-8
PYTHONIOENCODING = utf8
# min  hr  day-of-month  month  day-of-week  command
* * * * * /home/cromwell/twitter/tweeter.py