How to use Laravel with Python and the command line

Laravel is a fantastic framework for developing state-of-the-art web applications.

Python is terrific for data processing.

So how to leverage the best of both worlds?

Last week I have been experimenting with a Laravel sideproject performing sentiment analysis on a Twitter stream.

Turns out, the best of-the-shelf package for sentiment analysis of microtexts is VADER, a Python package.

According to VADER’s authors it “is a is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media”.

That’s a mouthful for saying that it takes a short text (like a tweet) and returns a sentiment score.

Options

So back to how to use Python with Laravel. There are a number of ways you could go about that:

  • look for a Laravel or PHP port of the Python package. I found some, but I was not impressed. They either didn’t work with my Laravel installation or were simply outdated. Also, I was concerned with performance – I rather offload the heavy lifting to Python.
  • write a PHP port of the package yourself. This would be a complex task, and I also wouldn’t benefit from future updates to the official VADER package.
  • wrap the Python script in a Python-powered REST API with Flask which can be called from the Laravel app. But I prefer to keep things as simple as possible. I.e. setting up Flask would require me to perform additional server configuration. More complexity = more things that can go wrong = more time spent on maintenance. (It would be an interesting approach for a sentiment-analysis microservice solution though :)).
  • let Laravel write the texts to storage such as a .txt file or a mySQL database. Periodically analyze unprocessed records with the Python script. This works great with massive datasets, but also means Laravel has little/no notion of what the Python script is doing.
  • interact with the Python script callable via the command line. And that’s what I ended up doing.

A few notes

This blog post focuses on the “Using Laravel with Python” part. If you want to know more about listening to a Twitter stream with Laravel, be sure to check out Spatie’s Laravel Twitter stream package. It works like a charm.

Also, make sure to sanitize the Tweet texts before processing them.

The simplest version

My initial solution passes a single sanitized string from Laravel to the Python script using the command line, and the Laravel app then parses the output from the command line.

The Python script:

# analyse_string.py
#!/usr/bin/python

import sys
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyser = SentimentIntensityAnalyzer()
print(str(analyser.polarity_scores(sys.argv[1])))

It’s not well-documented, but Laravel actually does include a Symphony Process component for running commands.

Laravel:

use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;

$text = 'The text you are desperate to analyze :)";
$process = new Process("python3 /Path/To/analyse_string.py \"{$text}\"");
$process->run();

// executes after the command finishes
if (!$process->isSuccessful()) {
    throw new ProcessFailedException($process);
}

echo $process->getOutput();
// Result (string): {'neg': 0.204, 'neu': 0.531, 'pos': 0.265, 'compound': 0.1779}

The improved version

There is room for improvement though. So I refined the flow a bit, and the script can now take an array of strings at once. But since it is impossible to pass objects as command line parameters, it is passed as a JSON encoded string.

Laravel:

// $json = an encoded JSON string
$process = new Process("python3 /Path/To/analyse_json.py {$json}");
$process->run();

// executes after the command finishes
if (!$process->isSuccessful()) {
    throw new ProcessFailedException($process);
}


dump(json_decode($process->getOutput(), true));

The Python script, now handling an encoded JSON string and outputting a new encoded JSON string:

# analyse.json.py
#!/usr/bin/python

import sys
import json
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

x=sys.argv[1]
data=json.loads(x)

analyser = SentimentIntensityAnalyzer()

for item in data:
    item['sentiment'] = analyser.polarity_scores(item['text'])
    item.pop('text', None)

print(json.dumps(data))

Another trick for reading command line output with Laravel

There is one other method I used for reading command line output back into Laravel: accessing the output buffer.

Let’s say I have a black-box Laravel/PHP function interacting with the command line. A function which I cannot or do not want to modify for system architecture considerations (i.e. separation of concerns).

I can get its command line output by wrapping it in ob_start() and ob_get_clean(), like this:

ob_start();
black_box_function_printing_to_command_line();
$output = ob_get_clean();

You can find more information on accessing the output buffer here.

Access the full source code

Get access to the full source code using the form below.

You’ll get the full Laravel project along with installation instructions.

The project leverages Laravel jobs, the queue worker, Eloquent/MySQL and console commands via artisan. A nice read ;).

Check it out now.

Leave a Reply

Your email address will not be published. Required fields are marked *