enter soundbot

I’ve recently found out about youtube-dl, a neat little command line tool for downloading videos off the web. Looking into it I quickly realized how big of a deal that simple premise was. Besides working with a whole bunch of websites, this little piece of software also came in the form of a Python package that could be easily hooked up to whatever else I felt like. Like, say, a bot.

That’s where I got the idea for Soundbot: a small chatbot assistant that would sit in my contact list, to whom I could forward cool public domain and creative commons media I happened to find online, to be downloaded to my personal music collection.

It would work as follows:

1. From my phone, while listening to a song or watching a music video that I like, I hit “share”, and pick Soundbot from my contact list.

2. The bot’s back-end receives the URL of the aforementioned piece of media, runs it through youtube-dl and performs any file processing or format conversion necessary to spit out a nice little mp3 track.

3. The resulting file is uploaded to a convenient cloud hosting service, such as Google Play Music or Dropbox.

4. Back on my phone, a cloud-enabled music player detects the change, then syncs and downloads the freshly ripped song.

the innards

Once this plan was laid out, I started messing around with youtube-dl immediately, and setting up the whole mp3 ripping part of the project proved surprisingly easy! Here’s how I do it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

from youtube_dl import YoutubeDL, postprocessor

def rip_audio(url):
    options = {
        "outtmpl": "out/%(title)s.%(ext)s",
        "noplaylist": True,
        "postprocessors": [
            {
                "key": "FFmpegExtractAudio",
                "preferredcodec": "mp3"
            }
        ]
    }

    with YoutubeDL(options) as downloader:
        downloader.download([url])

Pretty straightforward, huh? All this function does is create a dictionary of “options”, which describe how the YoutubeDL object should handle downloaded files. There are only three options in use here:

• outtmpl: the template string for the output filename. The value "out/%(title)s.%(ext)s" means the file will be stored in the out folder, its name will be the title of the original source material, and its extension will be whatever comes the other end, as defined by the other options, or youtube-dl’s default behavior.

• noplaylist: whether to download every file or just the first one when the source URL points to a playlist. True here means “only download the first thing you find”.

• postprocessors: now, this is the juicy part. Youtube-dl comes with a bunch of post-processing options for after the audio/video file has been downloaded. Here all I’m using is the FFmpegExtractAudio post-processor so that whatever comes in from the source URL is converted to an mp3 file, as per the preferredcodec option.

These options are then passed to the YoutubeDL object, which operates in a handy context manager syntax.

And just like that, in 16 lines of code, we’ve got ourselves a function that, when given an URL to any of the 1089 supported websites, will download the media file, extract its audio track, and save it in a folder. Problem solved!

“But wait!”, you might be saying. “That’s not what we wanted at all!”

And you would be right, my imaginary projection of a reader. The ideal scenario includes not only downloading the desired track, but also uploading it to a file hosting service from where we can retrieve it at a later date. Hmm, if only there was a straightforward way to add an extra step to the file download pipeline. Some way to further process it, a bit after the previous steps were done. If only…

the dbox post-processor

Youtube-dl post-processors are not only super cool and useful, they are also pretty extensible! So, by making use of the PostProcessor base class, and the Dropbox for Python SDK, we could write our very own post-processor that takes the output mp3 file and uploads it to an easily accessible cloud folder. This is how that looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import dropbox

from dropbox.files import WriteMode
from os.path import basename

from youtube_dl.postprocessor.common import PostProcessor

class DboxPP(PostProcessor):
    def __init__(self, downloader=None, token=None):
        self.dbx = dropbox.Dropbox(token)
        if downloader:
            downloader.to_screen("[dbox] token: " + token)
        super().__init__(downloader)

    def run(self, information):
        self.upload(information['filepath'])
        return super().run(information)

    def upload(self, filepath):
        if (self._downloader):
            self._downloader.to_screen("[dbox] uploading: " + filepath)
        with open(filepath, 'rb') as f:
            data = f.read()
            self.dbx.files_upload(
                data,
                "/soundbot/tracks/" + basename(filepath),
                WriteMode("overwrite")
            )

There’s quite a bit going on here, so let’s break it down. In this file I’m defining the DboxPP class, that extends the base PostProcessor class from youtube-dl. We can later use this much in the same way we did FFmpegExtractAudio, by plopping it into the postprocessors list in the downloader options.

First off there’s __init__(), the constructor. This method receives two parameters, downloader, which is the downloader object created in the with block in the previous file; and token, a custom parameter to be defined within the options dictionary. This token is our Dropbox API authentication token, needed to programmatically upload files. To get one of those, read up on how to create a Dropbox app.

Then there’s the run() method is what’s effectively called when a post-processor starts. It receives the information parameter: a dictionary of metadata from the downloaded file. It contains a bunch of useful, well, information, but for now the "filepath" is all that matters to us. This contains the path to the resulting file after all the previous post-processors have done their thing.

Finally, there’s upload(). This method takes a file path, reads its contents and passes them to the Dropbox.file_upload() method, overwriting any existing file, if necessary.

Now that we have our post-processor, it’s time to go back and make some changes to our rip_audio() function. Let’s see how that looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

from youtube_dl import YoutubeDL, postprocessor

from soundbot.postprocessor.dropbox_pp import DboxPP

def rip_audio(url):
    with open("config/dbox.token") as token_file:
        db_token =  token_file.read()

    options = {
        "outtmpl": "out/%(title)s.%(ext)s",
        "noplaylist": True,
        "postprocessors": [
            {
                "key": "FFmpegExtractAudio",
                "preferredcodec": "mp3"
            },
            {
                "key": "Dbox",
                "token": db_token
            }
        ]
    }

    postprocessor.DboxPP = DboxPP

    with YoutubeDL(options) as downloader:
        downloader.download([url])

First we have to add a couple lines in the beginning to read the Dropbox token from somewhere, no big deal. Now, this next part might look weird at first, and that’s because it really is. You see, the YoutubeDL object expects the list of post-processors to be formatted so that each entry contains a dictionary with a "key" item, containing the name of the post-processor’s class as a string, minus the “PP” suffix, plus items for any extra parameters the post-processor’s constructor receives, after the downloader param. Confusing enough?

On top of that, it then uses that string to look the post-processor up inside the youtube_dl.postprocessor namespace. So we have to inject our class into that by doing postprocessor.DboxPP = DboxPP.

And just like that, we’re done! Well, sorta. This isn’t a bot at all yet, it’s just a python script that I have to call by hand to download some music for me. But since this post is already long enough, I’ll go into the details of how I’m thinking about building the bot’s back-end and messenger integration later on.

In the meantime, feel free to browse Soundbot’s git repo to watch me struggle through this project in real time. See ya!