Archive for the ‘Programming’ category

Deploying circuits.web with Nginx/uwsgi

May 7th, 2019

Edit (May 2019): Modified some of the code so that it would be compatible with Python 3.x

I’m a very minimal person when it comes to frameworks, I don’t generally like something that needs to generate an entire application file structure like you’d see with Django. When I was searching around for various frameworks to get me started with python and web development, I investigated the usual; DJango, CherryPy, Web.Py. I fell in love with circuits due it’s ease and simplicity, yet it can be quite powerful. This article will show you how to get Nginx setup with uWSGI along with a sample circuits.web application.

Nginx, as of version 0.8.40, comes deployed with the uwsgi module unless otherwise excluded (ie: –without-http_uwsgi_module), the older legacy version 0.7.63 and above came with the module, but needed to be compiled into it. The syntax I use in this article conforms to 0.8.40 and newer and might not work with an older version of Nginx. (Currently I’m using this on Nginx 1.10.3, with Python 2.7.13 & 3.5.3)

This article assumes you already have Nginx 0.8.40+ and Python installed.

Installing the uWSGI server

As stated on their wiki, “uWSGI is a fast (pure C), self-healing, developer/sysadmin-friendly application container server.”, it utilizes the uwsgi protocol (notice the all-lowercase spelling), and supports WSGI applications served from it.

Once you’ve downloaded the source from the wiki, you can install it (a number of methods here). For my purpose I’ve moved the compiled uwsgi binary to my /usr/local/bin folder so that I could call it from anywhere on my system.

The python module for uwsgi should be installed as well.

# pip install uwsgi

And for Python 3

# pip3 install uwsgi

Other than FreeBSD I have not seen uWSGI readily available in most distribution’s package systems.

You can test your installation by moving out of the source folder, and calling the binary:

# uwsgi --version
uWSGI 2.0.18

Simple Hello world app without daemonizing uWSGI

Now we’re going to setup a very simple WSGI hello world application, and host it behind uWSGI and use Nginx to serve it. We’re not going to daemonize the uWSGI as such you’ll see it’s output in your terminal as connections are made. Also in Nginx we’ll simply send everything to the backend application for this demonstration.

Configure Nginx

server {
	server_name myapp.example.com;
 
	location / { 
		uwsgi_pass 127.0.0.1:3031;
		#You can also use sockets such as : uwsgi_pass unix:/tmp/uwsgi.sock;
		include uwsgi_params;
	}
}

If Nginx is currently running, either restart it, or you can reload the configuration with “nginx -s reload”.

Create a WSGI application
In your application folder create a python file, for example myapp.py:

def application(environ, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return ["Hello World!"]

For Python 3 a major change is that UWSGI requires responses in bytes instances and not strings. The fastest way is to type a string response as a byte, handy for constructed responses, otherwise you can encode it as UTF-8.

def application(environ, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return [b"Hello World!"]
    # or ["Hello World!".encode('utf-8')]

Deploy with uWSGI
Now we’ll want to deploy a simple single-worker uWSGI instance to serve up your application from your application folder:

# uwsgi -s 127.0.0.1:3031 -w myapp
*** Starting uWSGI 0.9.6.7 (64bit) on [Mon Jan 31 00:10:36 2011] ***
compiled with version: 4.4.5
Python version: 2.6.6 (r266:84292, Dec 26 2010, 22:48:11)  [GCC 4.4.5]
uWSGI running as root, you can use --uid/--gid/--chroot options
 *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
 *** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
allocated 640 bytes (0 KB) for 1 request's buffer.
binding on TCP port: 3031
your server socket listen backlog is limited to 100 connections
initializing hooks...done.
...getting the applications list from the 'myapp' module...
uwsgi.applications dictionary is not defined, trying with the "applications" one...
applications dictionary is not defined, trying with the "application" callable.
application 0 () ready
setting default application to 0
spawned uWSGI worker 1 (and the only) (pid: 20317)

Newer versions of UWSGI and Python 3 may require some additional flags:

# uwsgi -s 127.0.0.1:3031 --plugin python3 -w /home/youruser/www/app.domain.com/myapp.py
    ...

If you attempt to connect to the site, and all goes well you’ll see Hello World on the screen as well as some log outputs on your terminal. To shut down the uWSGI server use Ctrl+C on that screen.

Now for some fun with Circuits.web

Circuits is a Lightweight Event driven Framework for the Python Programming Language, with a strong Component Architecture. It’s also my favorite framework for deploying very simple web applications (but can be used for far more complicated needs, for example SahrisWiki).

For this article I’ll show you a few ways circuits.web can really simplify handling requests and URI parsing.

First thing we’ll need to do is get the circuits module installed into Python.

If you do not already have Python SetupTools, install them:

apt-get install python-setuptools

And then simply install via easy_install:

easy_install circuits

You can get the egg (2.6 only) or the source of Circuits 1.3.1 from PyPi.Python.org.

For Python 3 you can use pip3 (or easy_install)

pip3 install circuits

On to page 2…

Configuring Nginx for Nibbleblog 4.0.3

July 25th, 2014

This is a simple configuration example to replicate the .htaccess rules for NibbleBlog 4.0.3.

For those unfamiliar with NibbleBlog:

Nibbleblog is a powerful engine for creating blogs, all you need is PHP to work. Very simple to install and configure (only 1 step).

Nginx Configuration Example
Tested on Nginx 1.7.3

server {
    listen 0.0.0.0:80;
    root /path/to/public_html;
    server_name domain.com www.domain.com;
 
    # Directory indexing is disabled by default
    # So no need to disable it, unless you enabled it yourself
 
    # access log turned off for speed
    access_log off; #enable if you need it
    error_log /var/log/nginx/nib.error.log; #or path to your error log
 
    error_page 404 = /index.php?controller=page&action=404;
 
    # main location block
    location / {
        expires 7d;
        try_files $uri $uri/ @rewrites;
    }
 
    # rewrite rules if file/folder did not exist
    # based off rules for version 4.0.3's .htaccess
    location @rewrites {
        rewrite ^/category/([^/]+)page-([0-9+])$ /index.php?controller=blog&action=view&category=$1&number=$2 last;
        rewrite ^/category/([^/]+)/$ /index.php?controller=blog&action=view&category=$1&number=0 last;
        rewrite ^/tag/([^/]+)/page-([0-9]+)$ /index.php?controller=blog&action=view&tag=$1&number=$2 last;
        rewrite ^/tag/([^/]+)/$ /index.php?controller=blog&action=view&tag=$1&number=0 last;
        rewrite ^/page-([0-9]+)$ /index.php?controller=blog&action=view&number=$1 last;
        rewrite ^/post/([^/]+)/$ /index.php?controller=post&action=view&post=$1 last;
        rewrite ^/post-([0-9]+)/(.*)$ /index.php?controller=post&action=view&id_post=$1 last;
        rewrite ^/page/([^/]+)/$ /index.php?controller=page&action=view&page=$1 last;
        rewrite ^/feed/$ /feed.php last;
        rewrite ^/([^/]+)/$ /index.php?controller=page&action=$1 last;
 
        # if any of the redirects/urls above contain a *.php file in the url
        # then you may have to create a separate location ~ ^/post-([0-9]+)/, etc block
        # otherwise the php block will be invoked before this @rewrites block does
 
    }
 
    # cache control for static files
    location ~* \.(?:ico|css|js|gif|jpe?g|png)$ {
        expires max;
        add_header Pragma public;
        add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    }
 
    # files to block or handle differently
 
    # Don't log commonly requested files that may not always exist
    location = /robots.txt  { access_log off; log_not_found off; }
    location = /favicon.ico { access_log off; log_not_found off; }
 
    # block access to these files (shadow.php, keys.php, hidden/tmp files, xml files)
    location ~ /shadow\.php { access_log off; log_not_found off; deny all; } # block access to shadow.php
    location ~ /keys\.php   { access_log off; log_not_found off; deny all; } # block access to keys.php
    location ~ /\.          { access_log off; log_not_found off; deny all; } # block access to hidden files (starting with .)
    location ~ ~$           { access_log off; log_not_found off; deny all; } # block access to vim temp files (starting with ~)
    location ~ \.xml        { access_log off; log_not_found off; deny all; } # block access to xml files
 
    # typical PHP handling block
    # with a safeguard against passing a non-existent
    # php file to the PHP interpreter
 
    location ~ \.php$ {
        try_files $uri =404;
 
        fastcgi_param  QUERY_STRING       $query_string;
        fastcgi_param  REQUEST_METHOD     $request_method;
        fastcgi_param  CONTENT_TYPE       $content_type;
        fastcgi_param  CONTENT_LENGTH     $content_length;
 
        fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
        fastcgi_param  SCRIPT_FILENAME    $request_filename;
        fastcgi_param  REQUEST_URI        $request_uri;
        fastcgi_param  DOCUMENT_URI       $document_uri;
        fastcgi_param  DOCUMENT_ROOT      $document_root;
        fastcgi_param  SERVER_PROTOCOL    $server_protocol;
 
        fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
        fastcgi_param  SERVER_SOFTWARE    nginx;
 
        fastcgi_param  REMOTE_ADDR        $remote_addr;
        fastcgi_param  REMOTE_PORT        $remote_port;
        fastcgi_param  SERVER_ADDR        $server_addr;
        fastcgi_param  SERVER_PORT        $server_port;
        fastcgi_param  SERVER_NAME        $server_name;
 
        fastcgi_pass 127.0.0.2:9000;
 
        #fastcgi_pass unix:/tmp/php.sock; 
        #use the above line instead if running php-fpm on a socket
    }
 
}

This should work pretty well as long as the newest NibbleBlog does not utilize php file names in the friendly urls (from my testing, it no longer does that).

Also Friendly URLs are not enabled by default in NibbleBlog, you need to enable it under SEO tab under settings. (ignore the instructions regarding .htaccess since you’re not using apache).

Twitter Timeline Feed with PHP & JSON

April 14th, 2013

I have had my share of users needing help fixing broken implementations with twitter on their own website, such as pulling from an incorrect url, or using the incorrect twitter API format. Most of the time they are pulling the XML response, but twitter offers a much nicer format that works rather well with PHP; JSON.

Quick Example
A pull from your twitter timeline is as simple as:

<?php
 
$json = file_get_contents("https://api.twitter.com/1/statuses/user_timeline/karlblessing.json", TRUE);
 
$twitter_feed = json_decode($json, true);
 
// check for errors
if(isset($twitter_feed['errors'])) {
    foreach($twitter_feed['errors'] as $error) {
        echo "(".$error['code'].") ".$error['message']."<br>";
    }
} else {
    // Loop thru and spit out the text of each tweet returned
    foreach($twitter_feed as $tweet) {
        echo $tweet['text']."<br>";
    }
?>

I do not use Twitter often, so course my own feeds may be scarce.

Example utilizing an array

If you wish to have a bit more control over how the content is interpreted. For this we can utilize an array so the data may be used elsewhere on a page.

<?php
/* 
Configuration Array
 
explanation of each option can be seen here : https://dev.twitter.com/docs/api/1/get/statuses/user_timeline
 
user = the screen_name of the twitter user you wish to query
count = the "maximum" number of items to be returned
retweet = true or false to include retweets in the response
entities = true or false
exclude_replies = true or false to exclude replies
contributor_details = true or false
trim_user = true or false to trim extra user details
 
*/
 
$twitter = array(
	"user" => "karlblessing",
	"count" => "4",
	"retweet" => "true",
	"entities" => "true",
	"exclude_replies" => "true",
	"contributor_details" => "false",
	"trim_user" => "false"
);
 
// a small function to convert "created at" time to [blank] minutes/hours/days ago
 
function relativeTime($time)
{
    $delta = strtotime('+2 hours') - strtotime($time);
    if ($delta < 2 * MINUTE) {
        return "1 min ago";
    }
    if ($delta < 45 * MINUTE) {
        return floor($delta / MINUTE) . " min ago";
    }
    if ($delta < 90 * MINUTE) {
        return "1 hour ago";
    }
    if ($delta < 24 * HOUR) {
        return floor($delta / HOUR) . " hours ago";
    }
    if ($delta < 48 * HOUR) {
        return "yesterday";
    }
    if ($delta < 30 * DAY) {
        return floor($delta / DAY) . " days ago";
    }
    if ($delta < 12 * MONTH) {
        $months = floor($delta / DAY / 30);
        return $months <= 1 ? "1 month ago" : $months . " months ago";
    } else {
        $years = floor($delta / DAY / 365);
        return $years <= 1 ? "1 year ago" : $years . " years ago";
    }
}
 
// prepare the array
 
$twitter_feed = array();
 
// form the API url for the request
 
$api_url = "https://api.twitter.com/1/statuses/user_timeline/".$twitter['user'].
	".json?include_entities=".$twitter['entities'].
	"&include_rts=".$twitter['retweet'].
	"&exclude_replies=".$twitter['exclude_replies'].
	"&contributor_details=".$twitter['contributor_details'].
	"&trim_user=".$twitter['trim_user'].
	"&count=".$twitter['count'];
 
// obtain the results 
 
$json = file_get_contents($api_url, true);
 
// decode the json response as a PHP array
 
$decode = json_decode($json, true);
 
//check for error during the last decode
if(json_last_error != JSON_ERROR_NONE) {
	// http://www.php.net/manual/en/function.json-last-error.php
	$twitter_feed[] = array('error' => "Unable to decode response");
} elseif(isset($decode['errors'])) {
	// just grabbing the first error listed
	$twitter_feed[] = array('error' => $decode['errors'][0]['message']);
} else {
	// if no decode or twitter response errors then proceed.
 
	foreach($decode as $tweet) {
		// If you are including retweets, you may want to check the status
		// as the main text is truncated as opposed to the original tweet
 
		// If you used the trim_user option, the retweeted user screen name will not be avaialble
 
		if (isset($tweet['retweeted_status'])) {
			$tweet_text = "RT @{$tweet['retweeted_status']['user']['screen_name']}: 
			{$tweet['retweeted_status']['text']}";
		} else {
			$tweet_text = $tweet['text'];
		}
 
		$twitter_feed[] = array(
			'text' => $tweet_text, 
			'created_at' => relativeTime($tweet['created_at']),
			'link' => "http://twitter.com/".$twitter['user']."/status/".$tweet['id']
		);
 
		unset($tweet_text);
	}
}
 
unset($decode, $json, $tweet);
?>
 
<?php
 
// in a later portion of your code or page you can break down the array like so:
 
foreach($twitter_feed as $tweet) {
	echo "<a href=\"{$tweet['link']}\" target=\"_blank\">{$tweet['text']}</a><br>{$tweet['created_at']}<br><br>";
}
 
?>

The above would list out a maximum of 4 tweets to the screen as hyperlinks to the status ID, followed by the tweet’s creation date in a format such as “4 days ago”.

If there are a lot of retweets, and you did not include retweets in the response, there is a possibility that your response will be blank. This is because the count is a literal maximum which is not factored after the data has been filtered. So when not including retweets you will want to set your count high enough so that some normal tweets may be returned and then simply limit the number shown via the code.

The option for screen_name from the retweeted status will not be available if you use the user_trim option, so if you wish to use this information, be sure to set it to false.

Additional Information

Most of the information regarding user timelines via the Twitter API can be found at Their Documentation.

text, created_at, id are not the only options you can use. If you were to print_r() the decoded json response you will see all the extra information you may use.

Such as entities which will break down all the hashtags, users_mentioned and urls that were in the tweet. Or you can get the tweet information regarding location (geo, cordinates), or tweet source (such as iphone, application, etc).

Here’s an example of the JSON decoded into a PHP array (using print_r to display) for just one of my own tweets.

[1] => Array
        (
            [created_at] => Tue Oct 30 03:34:11 +0000 2012
            [id] => 263121594531061760
            [id_str] => 263121594531061760
            [text] => My Dropbox Referral Link. I'd be surprised if people didn't have an account yet. Super-handy if you have a smartphone http://t.co/OQ8IDYBc
            [source] => <a href="http://www.dropbox.com" rel="nofollow">Dropbox </a>
            [truncated] => 
            [in_reply_to_status_id] => 
            [in_reply_to_status_id_str] => 
            [in_reply_to_user_id] => 
            [in_reply_to_user_id_str] => 
            [in_reply_to_screen_name] => 
            [user] => Array
                (
                    [id] => 106994601
                    [id_str] => 106994601
                    [name] => Karl Blessing
                    [screen_name] => KarlBlessing
                    [location] => Grand Rapids, Mi
                    [url] => http://kbeezie.com
                    [description] => Webdeveloper
                    [protected] => 
                    [followers_count] => 40
                    [friends_count] => 52
                    [listed_count] => 1
                    [created_at] => Thu Jan 21 08:30:25 +0000 2010
                    [favourites_count] => 0
                    [utc_offset] => -18000
                    [time_zone] => Eastern Time (US & Canada)
                    [geo_enabled] => 1
                    [verified] => 
                    [statuses_count] => 95
                    [lang] => en
                    [contributors_enabled] => 
                    [is_translator] => 
                    [profile_background_color] => FFFFFF
                    [profile_background_image_url] => http://a0.twimg.com/profile_background_images/68539068/python-logo.png
                    [profile_background_image_url_https] => https://si0.twimg.com/profile_background_images/68539068/python-logo.png
                    [profile_background_tile] => 
                    [profile_image_url] => http://a0.twimg.com/profile_images/2549311960/ooc473od4tv58hlffxvy_normal.jpeg
                    [profile_image_url_https] => https://si0.twimg.com/profile_images/2549311960/ooc473od4tv58hlffxvy_normal.jpeg
                    [profile_link_color] => 0F6FFF
                    [profile_sidebar_border_color] => FFFFFF
                    [profile_sidebar_fill_color] => FFFFFF
                    [profile_text_color] => 333333
                    [profile_use_background_image] => 
                    [default_profile] => 
                    [default_profile_image] => 
                    [following] => 
                    [follow_request_sent] => 
                    [notifications] => 
                )
 
            [geo] => 
            [coordinates] => 
            [place] => 
            [contributors] => 
            [retweet_count] => 0
            [favorite_count] => 0
            [entities] => Array
                (
                    [hashtags] => Array
                        (
                        )
 
                    [urls] => Array
                        (
                            [0] => Array
                                (
                                    [url] => http://t.co/OQ8IDYBc
                                    [expanded_url] => http://db.tt/fJi2Poc
                                    [display_url] => db.tt/fJi2Poc
                                    [indices] => Array
                                        (
                                            [0] => 118
                                            [1] => 138
                                        )
 
                                )
 
                        )
 
                    [user_mentions] => Array
                        (
                        )
 
                )
 
            [favorited] => 
            [retweeted] => 
            [possibly_sensitive] => 
            [lang] => en
        )

If the above were a part of $twitter_feed and we wanted to pull the first URL used in the tweet.

<?php
echo $twitter_feed[1][entities][urls][0][expanded_url];
// would output : http://db.tt/fJi2Poc
?>

And there you have it.

Allowing secure WordPress Updates with SSH2

April 2nd, 2013

The Set Up

For the purpose of this guide:

  • PHP-FPM runs as an unprivileged user such as www-data or www (FreeBSD).
  • The owner of the web files is non-root such as “WebUser” belonging to the www/www-data group.
  • PECL-ssh2 has been installed for PHP
  • You currently use paired key authentication (How-To)

The Guild

If you are like myself, you may be running your wordpress-driven site on a VPS without a control panel, or even without a typical FTP server (ie: SSH/SCP only). I’ll show you how to set up wordpress to update itself via SSH, while only doing so when you allow it.

For those of you using Apache’s SuExec, this guide will not be of much use, as SuExec executes PHP and other processes as the same user that owns the files. In which case the permission setting at the very bottom of this page may be of more help to you, or you can use ‘direct’ mode instead of ssh2.

PECL-ssh2

First thing we need to do is make sure PHP has the SSH2 extension installed. This can be installed via PECL, or in the case of FreeBSD ports:

cd /usr/ports/www/security/pecl-ssh2
make install clean
service php-fpm restart

Once installed SSH2 will be visible on a php_info() output.

Paired Key Authentication for PHP

Now we need to provide PHP with a public/private key, for this purpose let us create a folder to store these files in. Because PHP runs as www, and the files are owned by WebUser (or whichever username you’ve given to manage your web files), PHP will not have free reign to any existing paired keys that user may already exist. Besides it is not advisable to use the same keys for your main Web User.

For my purposes, websites are stored in a path under the Web user’s home folder.
Example: /home/WebUser/www/domain_name.com/public_html

We can create a folder in “www” (outside of any of the website’s public documents), named .ssh:

mkdir /home/WebUser/www/.ssh
cd /home/WebUser/www/.ssh
ssh-keygen -b 4096 -C php-key -t rsa
** When asked use the path /home/WebUser/www/.ssh/id_rsa instead of the default
** Provide a passphrase, DO NOT leave it blank.

You do not need to create such a strong key using 4096 bits for local communication, nor do you need to store it in a folder named “.ssh”. The keys do not even need to be named id_rsa, so feel free to mix it up a bit, just remember your public key will have a pub extension. You can even create separate keys for each website so as long as you do not place them inside the publicly accessible root.

The “php-key” portion of the command is the comment, if you are using multiple keys, you can edit this comment to help keep organized.

Authorizing the new keys

As mentioned before, this guide assumes you are already using paired key authenication. As a result there should be an authorized_keys file placed in your User’s .ssh folder (typically /home/UserName/.ssh/authorized_keys).

In order for the new keys to be given permission to use the web user’s credentials, we need to add the content of id_rsa.pub to authorized_keys. You may do this either in an editor such as WinSCP, or via the command line, such as using the ‘cat’ command:

cat /home/WebUser/www/.ssh/id_rsa.pub >> /home/WebUser/.ssh/authorized_keys

Make sure there is a double arrow in the command above and not a single one, or you risk replacing the entire content of authorized_keys with just that one public key.

The purpose of the passphrase, which is not required but STRONGLY encouraged is to make it so that PHP cannot utilize this key unless you explicitly provide the passphrase. This would of course prevent a malicious script from acting as the main Web User, when you are not in the process of performing an update or installation (since your passphrase would only be provided at those times).

While you can also store the passphrase in the wp-config.php as the FTP_PASS, it is also strongly discouraged.

Setting Key Ownership and Permission

Because PHP in this configuration runs as www-data or www, it will not be able to access the newly created keys unless we change their ownership and permission. With the commands below we’re setting the ownership of the .ssh folder to www:www (or www-data:www-data) and changing the permissions so that only the owner of the file can read the private key, and owner+group can read the public key; Though only the owner really ever needs to see it, as the public key provided in the authorized_keys, but normally you will not be logged in as www, and may need to read the content of the file.

chown -R www:www /home/WebUser/www/.ssh
chmod 0400 /home/WebUser/www/.ssh/id_rsa
chmod 0440 /home/WebUser/www/.ssh/id_rsa.pub

Modifying wp-config.php

Below is the code segment that needs to be added to the wp-config.php file. First the method is defined as ssh2, then the absolute folder paths, then the paths to the public and private key, and finally the username and host SSH will attempt to connect to.

define('FS_METHOD', 'ssh2');
define('FTP_BASE', '/home/WebUser/www/YourDomain.com/public_html/WordPressRoot/');
define('FTP_CONTENT_DIR', '/home/WebUser/www/YourDomain.com/public_html/WordPressRoot/wp-content/');
define('FTP_PLUGIN_DIR ', '/home/WebUser/www/YourDomain.com/public_html/WordPressRoot/wp-content/plugins/');
define('FTP_PUBKEY', '/home/WebUser/www/.ssh/id_rsa.pub');
define('FTP_PRIKEY', '/home/WebUser/www/.ssh/id_rsa');
define('FTP_USER', 'WebUser');
define('FTP_HOST', 'YourDomain.com');
// If you are not using port 22, append :Port# to the domain, such as YourDomain:1212

With the following in place, you’ll be able to:

  • Install a Theme or Plugin from WordPress itself
  • Update a Theme or Plugin automatically in WordPress
  • Delete a Theme or Plugin from within WordPress
  • Upgrade WordPress Itself from the automatic update utility

Each time you perform any of the above tasks, you’ll be informed that the public and private keys are invalid, this error is only shown because without the passphrase it cannot continue. So provide the passphrase via the password field each time to perform the tasks. Make sure the “ssh2” radio button has been selected when you do this.

If you are uploading a zip file to install a theme/plugin

While the setting above will work for the most part with the automatic fetch-n-install, such as searching for plugin and then clicking install. It may not work when providing a zip file from your local storage.

If this becomes the case we just need to adjust the permissions of the upload directory. Assuming your files are owned by WebUser:www and PHP runs as www:www, we need to set the permissions of the /wp-content/uploads folder to 775 (read/write/execute for both the WebUser owner, and www group, but keep read/execute on ‘other’).

chmod 0775 /home/WebUser/www/YourDomain.com/public_html/wp-content/uploads/

If you have content already in there you may need to add on the recursive flag with -R before 0775.

chmod -R 0775 /home/WebUser/www/YourDomain.com/public_html/wp-content/uploads/

For the purpose of installations, this is only required in order for PHP to move the uploaded zip file to the uploads folder where it will be unpacked. From there the familiar SSH dialog will appear to continue the rest of the installation. After which the uploaded zip file will be removed from the uploads folder.

Securing your Uploads folder on Nginx

Because PHP is now capable of writing to the uploads folder, there is a chance that someone may attempt to upload a script into it and as such execute from it. The uploads folder should not host any executable scripts so to fix this we need to add some rules into the configuration for Nginx.

This location block should go before the PHP location block.

location ~* ^/wp-content/uploads/.*\.php$ {
    return 403;
}

Any attempts to call a PHP script from the uploads folder will now be met with a forbidden response code.

For further information on securing PHP and Nginx please refer to Securing Nginx & PHP

GeoIPLookup: Geolocation from the CLI

November 29th, 2012

In order to do a geolookup from the command line, we have to get the GeoIP binary and database installed.

FreeBSD

cd /usr/ports/net/GeoIP
make install clean

Debian/Ubuntu

apt-get install geoip-bin geoip-database

CentOS/RHEL

yum install GeoIP GeoIP-data

By default these install the free version of the GeoLite Country binary database (GeoIP.dat) usually in the /usr/local/share or /usr/share directory. You can either copy in a custom version of the database or edit your GeoIP.conf file to enter in your license information.

Performing a Lookup

To perform a lookup you would simply type geoiplookup followed by an IP address, for example let’s look up one of google’s IPs.

geoiplookup 74.125.225.33
GeoIP Country Edition: US, United States

Some Tricks with cat, grep, sed, and xargs

If you have a text file with a list of IP addresses you can instead pass it thru with xargs

cat ./ip_list.txt | xargs -n 1 geoiplookup { }
GeoIP Country Edition: US, United States
GeoIP Country Edition: CN, China
GeoIP Country Edition: JP, Japan
GeoIP Country Edition: BR, Brazil
GeoIP Country Edition: SG, Singapore
GeoIP Country Edition: SG, Singapore
GeoIP Country Edition: SG, Singapore

If you have duplicate IPs you can filter them out with sort and uniq:

cat ./ip_list.txt | sort | uniq | xargs -n 1 geoiplookup { }

But that by itself may not be too useful to you. Perhaps you want to get the total number of IPs in a file sorted by the least instances to the most instances of a single country.

cat ./ip_test.txt | sort | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort
   2 GeoIP Country Edition: JP, Japan
   3 GeoIP Country Edition: FR, France
   3 GeoIP Country Edition: GB, United Kingdom
   3 GeoIP Country Edition: IP Address not found
   3 GeoIP Country Edition: RU, Russian Federation
   3 GeoIP Country Edition: SG, Singapore
   3 GeoIP Country Edition: UA, Ukraine
   4 GeoIP Country Edition: TH, Thailand
   5 GeoIP Country Edition: KR, Korea, Republic of
   6 GeoIP Country Edition: DE, Germany
   7 GeoIP Country Edition: TR, Turkey
  27 GeoIP Country Edition: CN, China
  28 GeoIP Country Edition: US, United States

If you’re using pf (OpenBSD’s Packet Filter) with pflog, you can even get an output of the most countries that were blocked in your packet filter log with the following command:

tcpdump -ntr /var/log/pflog | awk '{print }' | sed -r 's/ \>.*//g' | \
grep -oE '[0-9]{1,}.[0-9]{1,}.[0-9]{1,}.[0-9]{1,}' | grep -v "0.0.0.0" | \
sort -n | uniq | xargs -n 1 geoiplookup { } | sort | uniq -c | sort -n | sed -r 's/ GeoIP Country Edition://g'
 
   1 AR, Argentina
   1 BB, Barbados
   1 BG, Bulgaria
   1 CA, Canada
   1 CZ, Czech Republic
   1 EU, Europe
   1 HR, Croatia
   1 ID, Indonesia
   1 IE, Ireland
   1 IS, Iceland
   1 IT, Italy
   1 LA, Lao People`s Democratic Republic
   1 LB, Lebanon
   1 PT, Portugal
   1 RO, Romania
   1 UY, Uruguay
   1 VN, Vietnam
   2 BR, Brazil
   2 ES, Spain
   2 IN, India
   2 JP, Japan
   3 FR, France
   3 GB, United Kingdom
   3 IP Address not found
   3 RU, Russian Federation
   3 SG, Singapore
   3 UA, Ukraine
   4 TH, Thailand
   6 DE, Germany
   6 KR, Korea, Republic of
   7 TR, Turkey
  27 CN, China
  29 US, United States

Securing Nginx and PHP

December 16th, 2011

Disclaimer
This write up is intended for single-user systems where you are the only user expected to log in via shell/terminal/sftp (or at least people you actually trust). This collection of tips does not cover situations where you may have multiple users home folders or a shared hosting situation utilizing nginx and php-fpm. Generally speaking if you have to read this post for security tips you probably should not be allowing access to any other user but yourself in the first place.

If you do not care to read this whole write up, just remember one thing: `777` is not a magical quick-fix; it’s an open invitation to having your system compromised. (Script-Kiddies… can you say jackpot?)

User/Groups

Generally speaking your server, which will most likely be a VPS running some fashion of linux will already have a web service user and group. This will sometimes be www, www-data, http, apache, (or even nginx if a package manager installed it for you). You can run the following command to get a list of users that are on your system.

cat /etc/passwd

Both Nginx and PHP-FPM should run as a web service, on a Debian squeeze this would be www-data:www-data, or on FreeBSD www:www.

If your server was set up with root being the main user you should create an unprivileged user for handling your web files. This will also make it easier to handle permissions when uploading your web files via SFTP. For example the following command on a debian system would create a user named kbeezie which has www-data as the primary group.

useradd -g 33 -m kbeezie

Group ID #33 is the id for www-data on Debian Squeeze (you can verify with id www-data). You may have to su into the new user and change the password (or usermod to change). This will also create a home folder in /home/kbeezie/ by default. You can log in via SFTP to this user and create a www folder if you wish. You’ll notice that the files will be owned by kbeezie:www-data, which will allow Nginx and PHP to read from, but also gives you group-level control over how the web services may treat those files.

This is ideal since you’re not giving nginx or php-fpm too much control over the user’s files and they can be controlled with the group flag. You could also create the user with it’s own group such as kbeezie:kbeezie and just change the group of the web files to www-data where appropriate.

SSH Options

It is usually advisable to disable Root access via /etc/ssh/sshd_config with the following line:

PermitRootLogin no

However make sure you can log in with your new unprivileged user via SSH, and also make sure that you can `su` into root permission. On a FreeBSD system only a user belonging to the wheel group can su into root, and only a user listed in the sudoers file can use the sudo command. However on Debian/etc the user could have www-data as it’s own group and still be able to su/sudo as long as the root password is valid. Your password should be at least 12 characters long and contain digits, symbols and at least one capital letter. Do not use the same password for root and your web user.

Once you’ve verified that you’re able to obtain root status with the new user you can safely disable root log in via sshd_config and restart the ssh deaemon.

You should also change your default SSH port, which is 22. While a port scanner could probably find the new SSH port it is usually best practice not to use the default port for any type of secure access. Like before make sure you can log into the new port (you can configure sshd_config to listen to both 22 and another port to test this out).

SSH – PubKey Authentication

If you are on OSX or another unix/linux operating system, like I am, setting up pub key authentication is fairly painless. Logged in as your web user on the server you can run the following command:

ssh-keygen

The above by default will ask for a passphrase for your private key as well as a destination to save both the id_rsa and id_rsa.pub files (which will normally be ~/.ssh/). You can then copy your own user’s public key to an authorized_key file with the following command.

cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

Then on your own computer you can run the ssh-keygen command as well, copy your own computer’s public key from the id_rsa.pub file and add it as another line to your server’s authorized_keys file.

If you have `PubkeyAuthentication yes` listed in your sshd_config file with the authorized key path being that of .ssh in your home folder the server should allow you to log in without prompting you for a password. Just remember that if you chose not to use a passphrase for your private key then it is possible for anyone who obtains your id_rsa* files to log into your server without being prompted for a password.

You can even turn off password authentication completely and rely solely on public key authentication by setting `PasswordAuthentication no` in your sshd_config file. However keep in mind, unless you have another means of getting into your server you might get locked out if you lose your public/private key or access to the machine you use to log in (also not every SFTP or Terminal application supports pub key authentication).

I actually have public key authentication set up with my iPad using iSSH for quick server access on the go (and you do not need a jailbroken iPad for this).

On the next page are some Nginx and PHP specific configurations to hardening your installation.

Nginx Flood Protection with Limit_req

April 9th, 2011

The Test

I’ll show you a very simple demonstration of Nginx’s Limit Request module and how it may be helpful to you in keeping your website up if you are hit by excessive connections or HTTP based denial-of-service attacks.

For this test I’m using a copy of my site’s about page saved as about.html, and the Blitz.io service (which is free at the moment) to test the limit_req directive.

First I test the page with the following command in Blitz, which will essentially ramp the number of concurrent connections from 1 to 1,075 over a period of 1 minute. The timeout has been set to 2 minutes, and the region set to California. I also set it to consider any response code other than 200 to be an error, otherwise even a 503 response will be considered a hit.

-p 1-1075:60 --status 200 -T 2000 -r california http://kbeezie.com/about.html

Not bad right? But what if that were a php document. That many users frequently might cause your server to show 502/504 errors as the php processes behind it keep crashing or timing out. This is especially true if you’re using a VPS or an inexpensive dedicated server without any additional protection. (Though if you use either, iptables or pf might be a good resource to look into)

You can of course use caching and other tools to help improve your website, such as using a wordpress caching plugin, which you should be using anyways. But sometimes that one lone person might decide to hit one of your php scripts directly. For those type of people we can use the limit request module.

Let’s create a zone in our http { } block in Nginx, we’ll call it blitz and set a rate of 5 request per second, and the zone to hold up to 10MB of data. I’ve used the $binary_remote_addr as the session variable since you can pack a lot more of those into 10MB of space than the human readible $remote_addr.

limit_req_zone $binary_remote_addr zone=blitz:10m rate=5r/s;

Then inside my server block I set a limit_req for the file I was testing above:

location = /about.html {
	limit_req zone=blitz nodelay;
}

So I reload Nginx’s configuration and I give Blitz another try:

You’ll notice that now only about 285 hits made it to the server, thats roughly 4.75 requests per second, just shy of the 5r/sec we set for the limit. The rest of the requests were hit with a 503 error. If you check out the access log for this you’ll see that a majority of the requests will be the 503 response with the spurts of 200 responses mixed in there.

Using this can be quite helpful if you want to limit the request to certain weak locations on your website. It can also be applied to all php requests.

Applying Limit Request to PHP

If you would like to limit all access to PHP requests you can do so by inserting the limit_req directive into your php block like so:

location ~ \.php {
	limit_req   zone=flood;
	include php_params.conf;
	fastcgi_pass unix:/tmp/php5-fpm.sock;
}

It may help to play with some settings like increasing or decreasing the rate, as well as using the burst or nodelay options. Those configuration options are detailed here: HttpLimitReqModule.

Note about Blitz.io

You may have noticed from the graphs above that the test were performed for up to 1,075 users. That is a bit misleading. All of the hits from the California region came from a single IP (50.18.0.223).

This is hardly realistic compared to an actual rush of traffic, or a DDOS (Distributed Denial of Service) attack. This of course also explains why the hits are consistent with the limiting of a single user as opposed to a higher number that reflecting a higher number of actual users or IP sources. Load Impact actually uses separate IPs for their testing regions. However with the free version you are limited to a max of 50 users and number of times you may perform the test. You would have to spend about $49/day to perform a test consisting up to 1,000 users to your site.

Testing from a single IP can be easily done from your own server or personal computer if you got enough ram and bandwidth. Such tools to do this from your own machine include: siege, ab, openload and so forth. You just don’t get all the pretty graphs or various locations to test from.

Also if you were testing this yourself, you have to remember to use the –status flag as Blitz will consider 5xx responses as a successful hit.

Better Alternatives

I won’t go into too much details, but if you are serious about protecting your site from the likes of an actual DDOS or multi-service attack it would be best to look into other tools such as iptables (linux), pf (packet filter for BSD) on the software side, or a hardware firewall if your host provides one. The limit request module above will only work for floods against your site over the HTTP protocol, it will not protect you from ping floods or various other exploits. For those it helps to turn off any unnecessary services and to avoid any services listening on a public port that you do not need.

For example on my server, the only public ports being listened on are HTTP/HTTPS and SSH. Services such as MySQL should be bound to a local connection. It may also help to use alternate ports for common services such as SSH, but keep in mind it doesn’t take much for a port sniffer to find (thats where iptables/pf come in handy).

Nginx and Codeigniter The Easy Way

March 5th, 2011

I personally don’t use Codeigniter (was never much of a fan of PHP frameworks), however I had a client approach me with this issue so I decided to take a stab at it.

Now normally you would look for any existing .htaccess that comes with a script package and attempt to convert the rewrite rules. But seriously though, why do we really want to go thru all that fuss when we can simply use try files:

server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;
    root /path/to/your/website.com/;
 
    location / {
        try_files $uri $uri/ /index.php;
    }
 
    # For more information on the next two lines see http://kbeezie.com/view/nginx-configuration-examples/
    include php.conf;
    include drop.conf;
}

In a nutshell this should normally work… but why does it not?

Quite simple really; CodeIgniter by default uses PATH_INFO which is a really antiquated method of parsing the uri. As a result we must tell CI to use REQUEST_URI instead. To do this open up your config.php under /system/application/config/ and find the $config[‘uri_protocol’] and set it to this:

$config['uri_protocol'] = "REQUEST_URI";

You could also choose to use AUTO, but since we know we’ll be dealing with request_uri, it is best to set it as such (though if you do run into problems, give AUTO a try).

If you have not already set the index page, you will want to blank it out in order for it to work with a rewrite method (request_uri, etc).

$config['index_page'] = "";

For known static files, take it a step further and capture common static requests so that you can cache accordingly:

	location ~* \.(ico|css|js|gif|jpe?g|png)$ {
		expires max;
		add_header Pragma public;
		add_header Cache-Control "public, must-revalidate, proxy-revalidate";
	}

Stop Bit.ly from Traversing your Redirects

November 12th, 2010

Bit.ly is one of my favorite URL Shortener, but it has one major flaw when it comes to link building; it tends to drill down to the final destination of a link and shortens the link at the end. This can make shortening a prosper redirect nearly impossible as the shortened url bypasses it completely.

To illustrate this observe the image below:

In a nutshell Bit.ly will keep following a redirect until it hits the end, and will use the final destination as the url that needs to be shortened. This also means even if you create several of your own redirects with different names, the shortened url will always be the same if they all point to the same destination.

The solution is rather simple, though it may be a tad inconvenient depending on your setup. If you use your own custom PHP redirect script you can simply place this at top:

if($_SERVER['HTTP_USER_AGENT'] == 'bitlybot') { echo 'Hi Bitly'; exit(); }

What this will do is halt the redirect if it detects bitlybot, so that the final destination happens to be the same one you pasted in the box, as such will leave your link logic safe (such as geo-targeting, IP blocks, etc)

Thats really all there is to it.

PS: I’ll be including that into the next version of KBLinker.

Mimic Apache mod_geoip in Nginx

November 12th, 2010

Maxmind makes a variety of APIs and tools to use their geolocation database and one such tool is the mod_geoip module for Apache. Using the GeoIP module at the apache level means that PHP can access the visitor’s country code simply by means of an environment variable such as this:

echo $_SERVER['GEOIP_COUNTRY_CODE'];

To setup Nginx with this capability we’ll need to recompile Nginx if you have not already used the –with-http_geoip_module compile option.

First we’ll need to install the GeoIP API system-wide:

CentOS (yum)

yum install GeoIP-devel

Debian/Ubuntu (aptitude)

apt-get install libgeoip-dev

Mac OS X 10.5+ (via HomeBrew)

brew install geoip
sudo brew link geoip

Once you have the GeoIP library installed you can then proceed to recompile Nginx, this is rather simple if you have installed Nginx from source (this assumes you already have the nginx source unpacked somewhere):

$ cd src/nginx-0.8.53
$ nginx -V 
nginx version: nginx/0.8.53
built by gcc 4.4.4 (Debian 4.4.4-6) 
TLS SNI support enabled
configure arguments: --prefix=/opt --with-pcre=/root/src/pcre-8.02 --with-md5=/usr/lib 
--with-sha1=/usr/lib --with-http_ssl_module --with-http_realip_module --with-http_gzip_static_module 
--with-openssl=/root/src/openssl-1.0.0a/ --without-mail_pop3_module
 --without-mail_imap_module --without-mail_smtp_module
$ ./configure --prefix=/opt --with-pcre=/root/src/pcre-8.02 --with-md5=/usr/lib --with-sha1=/usr/lib \
--with-http_ssl_module --with-http_realip_module --with-http_gzip_static_module \
--with-openssl=/root/src/openssl-1.0.0a/ --without-mail_pop3_module \
--without-mail_imap_module --without-mail_smtp_module --with-http_geoip_module
$ make && sudo make install
$ /etc/init.d/nginx restart

We’ll want to download the Maxmind Geolite Country Database some place Nginx can use it.

$ cd /opt/conf
$ wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
$ gunzip ./GeoIP.dat.gz

Then we need to tell Nginx where to find that file, so in your nginx.conf add this in your http { } block:

geoip_country  /opt/conf/GeoIP.dat;
#geoip_city     /opt/conf/GeoLiteCity.dat;
#Uncomment the above if you also wish to lookup cities

Once that is done you can restart Nginx, you’ll be able to use variables such as $geoip_country_code to obtain the visitor’s country code. Full details of this module can be found at HttpGeoIPModule.

We’re not done yet, now we need to make it so that PHP see’s these results in the same fashion it would with the apache module. In your fastcgi_params or where ever you are passing fastcgi_param values to PHP you’ll wish to add at least these two lines:

	fastcgi_param GEOIP_COUNTRY_CODE $geoip_country_code; 
	fastcgi_param GEOIP_COUNTRY_NAME $geoip_country_name;

Once that is done, restart Nginx. Now you’ll be able to access the visitor’s country code in PHP via $_SERVER[‘GEOIP_COUNTRY_CODE].

You can instead use the module to redirect visitors straight from nginx in the following fashion:

    server {
        server_name  mysite.com www.mysite.com;
	root /opt/html/mysite.com;
 
        location / {
                try_files /index_$geoip_country_code.html index.html;
        }
    }

If a visitor from Russia visits your site it will try to load index_RU.html, and if that is not found will fall back to index.html, likewise if you created a index_US.html a visitor from the United States will see that page’s content.