How to use Laravel with Python and the command line

Laravel is a fantastic framework for developing state-of-the-art web applications.

Python is terrific for data processing.

So how to leverage the best of both worlds?

Last week I have been experimenting with a Laravel sideproject performing sentiment analysis on a Twitter stream.

Turns out, the best of-the-shelf package for sentiment analysis of microtexts is VADER, a Python package.

According to VADER’s authors it “is a is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media”.

That’s a mouthful for saying that it takes a short text (like a tweet) and returns a sentiment score.

Options

So back to how to use Python with Laravel. There are a number of ways you could go about that:

  • look for a Laravel or PHP port of the Python package. I found some, but I was not impressed. They either didn’t work with my Laravel installation or were simply outdated. Also, I was concerned with performance – I rather offload the heavy lifting to Python.
  • write a PHP port of the package yourself. This would be a complex task, and I also wouldn’t benefit from future updates to the official VADER package.
  • wrap the Python script in a Python-powered REST API with Flask which can be called from the Laravel app. But I prefer to keep things as simple as possible. I.e. setting up Flask would require me to perform additional server configuration. More complexity = more things that can go wrong = more time spent on maintenance. (It would be an interesting approach for a sentiment-analysis microservice solution though :)).
  • let Laravel write the texts to storage such as a .txt file or a mySQL database. Periodically analyze unprocessed records with the Python script. This works great with massive datasets, but also means Laravel has little/no notion of what the Python script is doing.
  • interact with the Python script callable via the command line. And that’s what I ended up doing.

A few notes

This blog post focuses on the “Using Laravel with Python” part. If you want to know more about listening to a Twitter stream with Laravel, be sure to check out Spatie’s Laravel Twitter stream package. It works like a charm.

Also, make sure to sanitize the Tweet texts before processing them.

The simplest version

My initial solution passes a single sanitized string from Laravel to the Python script using the command line, and the Laravel app then parses the output from the command line.

The Python script:

# analyse_string.py
#!/usr/bin/python

import sys
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyser = SentimentIntensityAnalyzer()
print(str(analyser.polarity_scores(sys.argv[1])))

It’s not well-documented, but Laravel actually does include a Symphony Process component for running commands.

Laravel:

use Symfony\Component\Process\Process;
use Symfony\Component\Process\Exception\ProcessFailedException;

$text = 'The text you are desperate to analyze :)";
$process = new Process("python3 /Path/To/analyse_string.py \"{$text}\"");
$process->run();

// executes after the command finishes
if (!$process->isSuccessful()) {
    throw new ProcessFailedException($process);
}

echo $process->getOutput();
// Result (string): {'neg': 0.204, 'neu': 0.531, 'pos': 0.265, 'compound': 0.1779}

The improved version

There is room for improvement though. So I refined the flow a bit, and the script can now take an array of strings at once. But since it is impossible to pass objects as command line parameters, it is passed as a JSON encoded string.

Laravel:

// $json = an encoded JSON string
$process = new Process("python3 /Path/To/analyse_json.py {$json}");
$process->run();

// executes after the command finishes
if (!$process->isSuccessful()) {
    throw new ProcessFailedException($process);
}


dump(json_decode($process->getOutput(), true));

The Python script, now handling an encoded JSON string and outputting a new encoded JSON string:

# analyse.json.py
#!/usr/bin/python

import sys
import json
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

x=sys.argv[1]
data=json.loads(x)

analyser = SentimentIntensityAnalyzer()

for item in data:
    item['sentiment'] = analyser.polarity_scores(item['text'])
    item.pop('text', None)

print(json.dumps(data))

Another trick for reading command line output with Laravel

There is one other method I used for reading command line output back into Laravel: accessing the output buffer.

Let’s say I have a black-box Laravel/PHP function interacting with the command line. A function which I cannot or do not want to modify for system architecture considerations (i.e. separation of concerns).

I can get its command line output by wrapping it in ob_start() and ob_get_clean(), like this:

ob_start();
black_box_function_printing_to_command_line();
$output = ob_get_clean();

You can find more information on accessing the output buffer here.

Access the full source code

Get access to the full source code using the form below.

You’ll get the full Laravel project along with installation instructions.

The project leverages Laravel jobs, the queue worker, Eloquent/MySQL and console commands via artisan. A nice read ;).

Check it out now.

30 Replies to “How to use Laravel with Python and the command line”

  1. Hello sir, I have read your post, but I don’t know how to run your project, would you mind showing me how to run it, maybe a instruction video, or more details explanation about the way you run your project
    Really appreciate that, thank you.

      1. Hello Sander,

        Thank you for your reply, because my english is not very good at all, so I might not quite understand all what you say in the read me file. I’m doing my thesis, and my project is just same same like yours. So, do you mind or have time to make an instruction video for me, really appreciate if you do that. But in case you don’t have time, can you explain more about the instruction, step by step.

        Thanks again

  2. Hi Johnny Joe,

    I have no plans for recording instructions.

    This blog post and app are intended for intermediate and advanced Laravel developers. If you’re just starting Laravel development, be sure to check out the official docs at https://laravel.com.

    If you’re comfortable with the basics of setting up a new Laravel app… Your English seems good enough to interpret the readme.md file.
    I’ll include an excerpt for you and other readers below. If you get stuck trying this, let me know where precisely you get stuck

    Here it is:

    This project provides a few console commands:

    – php artisan sentiment:string {string}
    – php artisan sentiment:json {json_string}
    – php artisan twitter:listen {for}

    If you want to study the code, I’d recommend to start at the `app/Console/Commands` folder.

    ## Installation

    – Create a MySQL table for this project
    – Rename or copy `.env.example` into `.env`
    – Configure database connection in `.env`
    – Configure Twitter API keys in `.env`
    – Run `php artisan migrate`
    – Make sure `python3` is on your PATH
    – Install the [VADER python package](https://github.com/cjhutto/vaderSentiment) (via pip)

    1. Hi Sander

      Thanks for your reply. So in case you don’t have time for recording instruction for me, it’s okay man, I’ll try to understand your readme file and do as you say, if I get stuck in any step of it, I will ask you, hope you don’t mind if I ask too much :). By the way, can I have your email? It’s more convenience if we contact through the email, right? Here my official email: (blocked by Sander). Hope you reply me soon

      Thank you

        1. Hey Sander . i did all way to share and only i reiceved is python code … I don’t know what benefir tutorial you publish . Could you send full project and function for me . Thanks for your share !

          1. Sorry to hear the benefits of this tutorial are not clear to you.
            As the post title says, it’s about
            “How to use Laravel with Python and the command line”. It includes Laravel PHP and python code examples.

            I see you have signed up to the newsletter to receive a zip file (see your mail) with the project source code.

            The project has been crafted with care and took me about 20 hours of research and development. It has been used and tested by many other Laravel enthousiasts.

            If the zip does not contain a full source, could you please send me a list/ screenshot of the zip’s root folder?

    1. Hi Vo Nguyen,

      Thanks for sending the screenshots. I have looked at it and all seems fine, all necessary files are included. You forget to mention that there actually is a complete Laravel application included in the source files, so I was surprised by that. 🙂

      The main application is developed in Laravel, that’s why the other 98% are not Python files (.py). It only leverages two python (.py) scripts, which indeed are also detailed in this tutorial. But these two python scripts are all that’s needed, as they perform some quite heavy lifting (sentiment analysis) using the VADER library (make sure it’s installed – see the included readme file).

      If you’re not yet comfortable with Laravel, I suggest to study other Laravel projects first, as this tutorial is advanced level.

  3. Hello Sander,
    Thank you for tutorial.
    I encounter some issue : I set up a virtual environment for python where i put my script which call some pip lib in lib\site-packages.
    My script file works well when called from shell but when it is called by laravel Command it failed with ‘module not found’.
    I guess it is an issue with the import and sys.path as the script is not called from a folder in the virtual environment.
    I am a beginner on that : do you know how to solve that?
    Thanks

    1. I would like to clarify my post:
      My script file in the virtual environment call a library i downloaded with pip and import in the script file. This library ‘lamba’ imported causes an error, moduleNotFoundError : no module named ‘lambda’. This module is however present in the lib/site-packages and its parent folder is in the sys path. My script file is at the top of project folder and works well when called from python shell.

      1. Hi David,

        My Python skills are a bit rusty, but here are a few things you could check:
        1) make sure Laravel is executing the command in the intended virtual environment by executing “pip -V” from within Laravel (https://stackoverflow.com/questions/1871549/determine-if-python-is-running-inside-virtualenv).
        2) (related to the previous) make sure Laravel calls the intended Python binary by executing “which python” (or “which python3”) from Laravel. It should give the same output as calling it directly from the commandline.
        3) try setting the import statement as an absolute path instead of a relative one.

        Hope this helps!

    1. Thanks Yooper! At the time of writing that package’s VADER port seemed a bit outdated. I do see some recent activity at the github repo now though :).

  4. Hi Sander,

    Thanks for your great post! It helped me a lot on a project I’m currently working on. I’m able to run my python script from a Laravel command.

    My problem is that in my python script, I would like to save an object to a file. When I run the script from command line. It works just fine. But when I run it via the Laravel command, it prints the output of the python script, but does not create the file.

    I’m thinking that this is due to some security precaution in PHP. I’ve tried every possible PHP command (exec, system, shell_exec …) and the Symphony Process, to no avail.

    I also tried to pass the object as the output in my python script but I did not succeed. The object contains objects in itself.

    Ultimately my goal is to run this script via a RESTful API. right now if I run the command on a POST request, it gives me an error on “not finding dependencies”. But currently that’s the next step for me.

    Sorry for the long post and my ramblings, but I’m really stuck on this and would really appreciate any help.

    Thanks in advance!

    1. Hi Pouya,

      The error `not finding dependencies` may be related to the short discussion I had with David (see other comments in this post). See if that is helpful. If the dependencies are resolved incorrectly (due to wrong python binary / different environment), Python will have a hard time.

      Hope this helps,

      Sander

      1. Thanks for your quick reply!
        I have read all the comments for a clue before posting. Actually calling the command from an API will be my next problem.

        Currently I’m stuck getting the python script to write files from within the Laravel command. It’s like the script runs just fine, but it cannot persist the files. Like it’s run in a sandbox environment or something like that. Do you have any idea what can I do to make this happen?

        If I run the script from the command line, everything is just fine.

        I can send you my codes for you to take a look if it’s fine with you. My email is: pooya.khadem@gmail.com

        Looking forward to your reply,
        Pouya

        1. Hi Pouya,

          The Symfony Process Component is used in this tutorial.

          According to the Symfony docs there are some security measures in place for the underlying “exec, passthru, shell_exec and system” php functions.

          My best bet right now is to look into the php “safe_mode” and “safe_mode_exec_dir” settings.

          A workaround could be to pass the data back to Laravel (see tutorial) and store it from there.

          And finally, for troubleshooting, try the easiest steps possible:
          1. Call “touch /some_dir/testfile.txt” from Laravel. Check if file is created.
          2. Call a python script from Laravel that generates a similar testfile. Check if it is created.

          1. Thanks mate. I finally got to make it work.
            The trick was serializing the output of the python script, as you mentioned. From there I could make the file I wanted. Now laravel handles all the file management tasks, and python only focuses on the logic.

            But I still have the second problem. Whenever I call the Laravel command over HTTP, it does not recognize the imports in the python script.

            I’m suspecting two problems. First is that over HTTP it runs as www-data, and it may be a permission problem. Second, it may be a problem with paths or something like that.

            When I run the command from CLI everything works like a charm.

            Every possible suggestion would be really appreciated. I’ll keep you updated on the progress if you’re interested.

            Regards

          2. Hi Pouya,

            Glad you fixed it! I think you’re right on the possible causes via web. Were you able to fix this as well?

  5. Hi, Sander. Congrats for the very nice article.

    I’ve done something similar and it works fine in my localhost. My pythons scripts return the results to my Laravel controller perfectly.

    But, when I uploaded my application to my remote server I’m getting python exceptions caught by Symfony/process:

    “Error Output:\n
    ================\n
    Traceback (most recent call last):\n
    File “/usr/local/lib/python3.4/dist-packages/nltk/corpus/util.py”, line 80, in __load\n
    try: root = nltk.data.find(‘{}/{}’.format(self.subdir, zip_name))\n
    File “/usr/local/lib/python3.4/dist-packages/nltk/data.py”, line 675, in find\n
    raise LookupError(resource_not_found)\n
    LookupError: \n
    **********************************************************************\n
    Resource \e[93mstopwords\e[0m not found.\n
    Please use the NLTK Downloader to obtain the resource:\n
    \n
    \e[31m>>> import nltk\n
    >>> nltk.download(‘stopwords’)\n
    \e[0m\n
    Searched in:\n
    – ‘/var/www/nltk_data’\n
    – ‘/usr/share/nltk_data’\n
    – ‘/usr/local/share/nltk_data’\n
    – ‘/usr/lib/nltk_data’\n
    – ‘/usr/local/lib/nltk_data’\n
    – ‘/usr/nltk_data’\n
    – ‘/usr/share/nltk_data’\n
    – ‘/usr/lib/nltk_data’\n
    **********************************************************************\n”

    I’ve run the script directly in the terminal and it is working, so it is not an issue of missing dependencies.

    Am I forgetting to config something in my Apache/Laravel remove environment ?

    1. Hi Nigel,

      I think you’re right: it’s probably something in your server setup. These can be a pain to debug, but I’m afraid I cannot be of much help to you here.

      Be sure to check out the conversation I had with Pouya’s in the comments.

      1. I think this may come in handy when debugging a process from within your php application.

        According to the Symfony docs you can use

        $process->mustRun();

        instead of

        $process->run();

        This way you get a ProcessFailedException if the process fails somehow. On this exception, you can use getProcess()->getErrorOutput().

        Hope this helps.

    1. I’m not familiar with HolyC. But if it’s possible to interact via the terminal and handle string input/output, I think the same strategy could work. Alternatively you could use files to pass data.

  6. Hi Sandar,

    When I am calling python script from the laravel.It showing the below error.But when i call this in terminal its working fine.

    ProcessFailedException in pythonController.php line 25:
    The command “python C:\wamp64\www\iFrostLVL\app\Http\Controllers\pyexample.py” failed.

    Exit Code: 1(General error)

    Working directory: C:\wamp64\www\iFrostLVL\public

    Output:
    ================

    Error Output:
    ================
    ‘python’ is not recognized as an internal or external command,
    operable program or batch file.

Leave a Reply to Thomas Cancel reply

Your email address will not be published. Required fields are marked *