Archive for the ‘Python’ category

Deploying circuits.web with Nginx/uwsgi

January 31st, 2011

I’m a very minimal person when it comes to frameworks, I don’t generally like something that needs to generate an entire application file structure like you’d see with Django. When I was searching around for various frameworks to get me started with python and web development, I investigated the usual; DJango, CherryPy, Web.Py. I fell in love with circuits due it’s ease and simplicity, yet it can be quite powerful. This article will show you how to get Nginx setup with uWSGI along with a sample circuits.web application.

Nginx, as of version 0.8.40, comes deployed with the uwsgi module unless otherwise excluded (ie: –without-http_uwsgi_module), the older legacy version 0.7.63 and above came with the module, but needed to be compiled into it. The syntax I use in this article conforms to 0.8.40 and newer and might not work with an older version of Nginx. (Currently I’m using this on Nginx 0.9.4, with Python 2.6.6)

This article assumes you already have Nginx 0.8.40+ and Python installed.

Installing the uWSGI server

As stated on their wiki, “uWSGI is a fast (pure C), self-healing, developer/sysadmin-friendly application container server.”, it utilizes the uwsgi protocol (notice the all-lowercase spelling), and supports WSGI applications served from it.

Once you’ve downloaded the source from the wiki, you can install it (a number of methods here). For my purpose I’ve moved the compiled uwsgi binary to my /usr/local/bin folder so that I could call it from anywhere on my system.

Other than FreeBSD I have not seen uWSGI readily available in most distribution’s package systems.

You can test your installation by moving out of the source folder, and calling the binary:

# uwsgi --version
uWSGI 0.9.6.7

Simple Hello world app without daemonizing uWSGI

Now we’re going to setup a very simple WSGI hello world application, and host it behind uWSGI and use Nginx to serve it. We’re not going to daemonize the uWSGI as such you’ll see it’s output in your terminal as connections are made. Also in Nginx we’ll simply send everything to the backend application for this demonstration.

Configure Nginx

server {
	server_name myapp.example.com;
 
	location / { 
		uwsgi_pass 127.0.0.1:3031;
		#You can also use sockets such as : uwsgi_pass unix:/tmp/uwsgi.sock;
		include uwsgi_params;
	}
}

If Nginx is currently running, either restart it, or you can reload the configuration with “nginx -s reload”.

Create a WSGI application
In your application folder create a python file, for example myapp.py:

def application(environ, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return ["Hello World!"]

Deploy with uWSGI
Now we’ll want to deploy a simple single-worker uWSGI instance to serve up your application from your application folder:

# uwsgi -s 127.0.0.1:3031 -w myapp
*** Starting uWSGI 0.9.6.7 (64bit) on [Mon Jan 31 00:10:36 2011] ***
compiled with version: 4.4.5
Python version: 2.6.6 (r266:84292, Dec 26 2010, 22:48:11)  [GCC 4.4.5]
uWSGI running as root, you can use --uid/--gid/--chroot options
 *** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
 *** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
allocated 640 bytes (0 KB) for 1 request's buffer.
binding on TCP port: 3031
your server socket listen backlog is limited to 100 connections
initializing hooks...done.
...getting the applications list from the 'myapp' module...
uwsgi.applications dictionary is not defined, trying with the "applications" one...
applications dictionary is not defined, trying with the "application" callable.
application 0 () ready
setting default application to 0
spawned uWSGI worker 1 (and the only) (pid: 20317)

If you attempt to connect to the site, and all goes well you’ll see Hello World on the screen as well as some log outputs on your terminal. To shut down the uWSGI server use Ctrl+C on that screen.

Now for some fun with Circuits.web

Circuits is a Lightweight Event driven Framework for the Python Programming Language, with a strong Component Architecture. It’s also my favorite framework for deploying very simple web applications (but can be used for far more complicated needs, for example SahrisWiki).

For this article I’ll show you a few ways circuits.web can really simplify handling requests and URI parsing.

First thing we’ll need to do is get the circuits module installed into Python.

If you do not already have Python SetupTools, install them:

apt-get install python-setuptools

And then simply install via easy_install:

easy_install circuits

You can get the egg (2.6 only) or the source of Circuits 1.3.1 from PyPi.Python.org.

On to page 2…

Generating Ioncube Licenses

September 28th, 2010

Ioncube Encoder Pro or Cerberus is required to generate license files that can be distributed to your customers. Most of the nitty-gritty involved with the make_license executable can be found in the user guide (a pdf document) distributed with the encoder in section 4. The entry level version of Ioncube Pro cannot generate licenses, however much of the same restrictions can be applied to an encoded project (section 3 of the user guide) on a per-customer basis, pro/cerberus eliminates the hassle with having to re-encode and redistribute the project for each customer.

In this article we’ll talk about two of the most popular restrictions that can be applied to a license. But first we need to look at how to prepare a project to be used with a license. Since I am using Mac OSX there is no graphical user interface, as such these instructions will be based on the command line options.

Preparing Project for License Management

Let us say that we have our project in a folder called myproject, the simplest way to encode such would be to use the following command (ioncube_encoder5 is for PHP 5.x, ioncube_encoder is for PHP 4.x, version 7 of Ioncube will likely have a separate binary for PHP 5.3 encoding)

ioncube_encoder5 /projects/myproject --into /encoded-projects

The above will encode myproject into a new folder located under /encoded-projects, no licensing or restrictions has yet been applied. To encode the project so that a license file is required you add

ioncube_encoder5 --with-license key.php --passphrase yourpassphrasehere /projects/myproject --into /encoded-projects

Note there is a passphrase used, you must use a passphrase when specifying a license file to be searched, you will use the same passphrase when generating a license with make_license. With the options above the encoded files will search for a file called key.php, and will check the parent folder recursively until it finds it. If it is a valid license the script will execute normally. If it is not, you’ll get a simple error stating that the code has an invalid license.

However I’ve always preferred to show errors a bit more gracefully and more explanatory and to do so the script needs to be able to handle the validation. To do this we add the –license-check option like so.

ioncube_encoder5 --with-license key.php --passphrase yourpassphrasehere --license-check script \
/projects/myproject --into /encoded-projects

What this does is disables the automatic checking done by the ionCube loader, and allows you to handle the script’s response to an license issue using the Loader API. Below is a simple example how to check if the license is expired and respond accordingly.

<?php
 
if(ioncube_file_is_encoded() === false) { 
	/* if this is in a function you can return true to bypass license check if the file is not encoded,
	makes it easier to debug when you don't have to keep turning on and off the license check during
	coding and encoding. */
} else { 
	//Obtains the license properties
	$ic_prop = ioncube_license_properties();
 
	/*
	Always use === when matching boolean, since a non-boolean response could be interpreted as true 
	or false when it would in fact be an array or other value. For example the function above will return an 
	array of license properties, but if the file is not encoded or does not have a valid license it will 
	return FALSE. As such will never return TRUE but an array response comared with == could be
	interpreted as a FALSE response.
	*/
 
	if($ic_prop === FALSE)
	{
		//ioncube_license_properties returns false if a license file is not found or invalid/corupted
		echo "License File Not Found or Not Valid.";
		exit();
	}
 
	/* The two functions below will return FALSE if the file is 1) encoded, 2) requires a license and
	3) the server/time does not meet the license restrictions. */
 
	if(ioncube_license_matches_server() === FALSE)
	{
		//This will check to see if the current domain matches the license restriction
		echo "License is Not Valid for This Domain.";
		exit();
	}
 
	$expiry = ioncube_license_has_expired();
 
	if(ioncube_license_has_expired() === TRUE)
	{
		//The above function will return true if the license is expired
		echo "License Has Expired.";
		exit();
	}	
 
	/*
	Other checks such as property values, like encoding the user's transaction ID or feature restrictions 
	from the license file. License properties such as UserName can be grabed like so:
	$ic_prop['UserName']['value'] 
	*/
}

So there you have it a very brief explanation of how to encode a project to use a license, and how to check for the license yourself so that you may control the output (such as making your own branded page with the error as opposed to a simple text on white background) or alter the project’s features based on the license properties.

Generating a License with make_license

The ionCube encoder ships with a make_license binary, the Windows and Linux version of the encoder both ship with a linux version of the binary (plus a windows executable for Windows). The Mac OSX version of the encoder only ships with an OSX binary of make_license, which annoyed me quite a bit as I tried to use it on my linux-based hosting provider. For OSX users as of version 6.5 you have to request a linux binary of the make_license file from ionCube support.

In a nutshell this is how utilize the make_license binary manually on a linux server:

./make_license --passphrase yourpassphrasehere --header-line '<?php exit(0); ?>' --property "UserName='Chuck Norris'"

Note that you have to use the same passphrase used to encode your project. The above license has no restrictions yet, rather just a few header lines and encoding a property called UserName with the value of Chuck Norris within. The results of above will generate something like this:

<?php exit(0); ?>
 
 
------ LICENSE FILE DATA -------
9TS21X45EIPmmmjcWh+ZLlelwqJyMLD8
R2SpnyDbbMtdlpgO5bKYCwVI8wM1oqH1
SCLOc/tJ93duEBVt1BFX6//GL+UkLtXI
ZxYecENH0KR6sAoR5iEDSMcgicXpxcto
NR0VDOJETrkmWGHkc+KhpELoB4iF+RjB
ALizJ3gzQjL9HTfDZS+zGF0JVzy7IBrg
fNJ5pvxUJ+p/15hrhnYfWhOlkB5lhr/x
B+ommB3rAwVAxhRVf0nHGisRo+TFEKi=
--------------------------------

The reason for the header lines is because I like to generate my licenses as php files, such as key.php, this way the license in a public location cannot be viewed via the web since the PHP portion would exist the script before it reached the license data, and its easier than telling your customers to place the license data outside of the public_html folder, especially if you’re not using domain or hardware restrictions on the license file.

To add a domain restriction such as example.com (and www.example.com) we would add –allowed-server option like so.

/make_license --passphrase yourpassphrasehere --header-line '<?php exit(0); ?>' \
--property "UserName='Chuck Norris'" --allowed-server example.com,www.example.com

The above will make it so that the license is only valid on example.com and www.example.com, wildcards can also be used such as *.domain.com, or for a single character api?.domain.com (where ? can be no more than a single character), you can also use brackets to match a defined set such as [123].domain.com would match 1.domain.com 2.domain.com and 3.domain.com, likewise [!123].domain.com would match any domain as long as it wasn’t 1., 2. or 3. IP ranges can also be defined (Section 3.6.3 of the user manual for more details).

Time-based restrictions can be applied with –expire-in and –expire-on.

--expire-in 7d
--expire-in 8h
--expire-on 2012-12-21

The first two would expire in 7 days, or 8 hours, the last one would expire on December 21st 2012 (along with the rest of us).

On Page 2: Automating License Generation with PHP or Python

PayPal IPN Revised for Python

January 24th, 2010

This article adds onto the previous entry Paypal IPN with PHP, by showing you how to process an Instant Payment Notification from Paypal with Python. For more information about setting up your PayPal account or purchase code for Instant Payment Notifications, refer to the link above.

For any python app, you will need a way to launch it from the terminal. You can use the “shebang” below to make the app self serving. You will of course need to make sure it has executable privilages (chmod +x ./app.py on most unix/linux systems). If the instance of python you are using is not the systemwide instance such as python 2.6 installed to /opt, you will then then change it to use python2.6 instead.

#!/usr/bin/env python

You still launch the app via the interpreter directly, omitting the above shebang if you’d like.

With the exception of circuits.web, all of the modules used are built into Python (as of 2.6 to my knowledge).

from time import time
from urllib import urlencode
from sqlite3 import connect, Error as sqerr
from urllib2 import urlopen, Request
from circuits.web import Controller, Server

I have created a separate method for verification to keep the rest of the code cleaner. This is pretty much the same process PayPal provides developers, plus two checks that I feel are important.

def verify_ipn(data):
	# prepares provided data set to inform PayPal we wish to validate the response
	data["cmd"] = "_notify-validate"
	params = urlencode(data)
 
	# sends the data and request to the PayPal Sandbox
	req = Request("""https://www.sandbox.paypal.com/cgi-bin/webscr""", params)
	req.add_header("Content-type", "application/x-www-form-urlencoded")
	# reads the response back from PayPal
	response = urlopen(req)
	status = response.read()
 
	# If not verified
	if not status == "VERIFIED":
		return False
 
	# if not the correct receiver ID
	if not data["receiver_id"] == "DDBSOMETHING4KE":
		return False
 
	# if not the correct currency
	if not data["mc_currency"] == "USD":
		return False
 
	# otherwise...
	return True

The verification url provided above is for the PayPal sandbox, a developer testing ground. You can signup for access to test accounts, and tools such as IPN Test submissions at PayPal SandBox. When the app is ready for production, simply remove .sandbox from the url.

Just because PayPal says it’s Verified, only means that a payment has a occured, and that the data received is the same on Paypal’s database. Someone could have paid themselves using your IPN URL; for this reason we want to make sure that the receiver_id matches your own. The Secure Merchant ID can be found on the top your Paypal account’s profile page as shown below:

Next is a currency check, because 100 Zimbewe dollars is not equivalent to 100 US Dollars. The variables mc_gross as well as item prices don’t establish the currency used, but rather simply the value amount. So this check is important to protect against incorrect purchase amounts.

Now we have the main Root class of the IPN app.

class Root(Controller):
	# if the app will not be served at the root of the domain, uncomment the next line
	#channel = "/ipn"
 
	# index is invoked on the root path, or the designated channel URI
	def index(self, **data):
		# If there is no txn_id in the received arguments don't proceed
		if not "txn_id" in data:
			return "No Parameters"
 
		# Verify the data received with Paypal
		if not verify_ipn(data):
			return "Unable to Verify"
 
		# Suggested Check : check the item IDs and Prices to make sure they match with records
 
		# If verified, store desired information about the transaction
		reference = data["txn_id"]
		amount = data["mc_gross"]
		email = data["payer_email"]
		name = data["first_name"] + " " + data["last_name"]
		status = data["payment_status"]
 
		# Open a connection to a local SQLite database (use MySQLdb for MySQL, psycopg or PyGreSQL for PostgreSQL)
		conn = connect('db')
		curs = conn.cursor()
		try:
			curs.execute("""INSERT INTO ipn (id, purchased, txn, name, email, price, notes, status) 
			VALUES (NULL, ?, ?, ?, ?, ?, NULL, ?)""", (time(), reference, name, email, amount, status,))
			conn.commit()
		except sqerr, e:
			return "SQL Error: " + e.args[0]
		conn.close()
 
		# Alternatively you can generate license keys, email users login information
		# or setup accounts upon successful payment. The status will always be "Completed" on success.
		# Likewise you can revoke user access, if status is "Canceled", or another payment error.
 
		return "Success"

In most cases you would host each application on their own domain or subdomain, such as ipn.domain.com. But I generally prefer to run all my apps from a single subdomain, such as apps.domain.com. The Controller above is expecting / for the root base, not /ipn/ like I would have. As a result we add the channel line to inform the controller of the new base.

The variable data will be received as a dict type, we’ll first check to make sure data has come in (by seeing if there’s a transaction ID with the data received), then verify the data with Paypal. Once verified, the desired information can be stored in a database, be it SQLite, MySQL, PostgreSQL or Durus.

Three things I would suggest for this portion: use a log or email to keep track of errors, and if you use sell a number of products online, check the item ID and prices to verify their accuracy. Also if your products are subscriptions, or prone to returns, check to see if a transaction ID already exists in the database and update it’s status accordingly as opposed to creating a new record for every instant payment notification, useful if someone cancels or charges back their transaction.

The next method is not very practical by itself but could lead to more useful ideas. This would be placed under the same Root class above.

	def lookup(self, id):
		if not id:
			return "No Transaction Provided"
 
		conn = connect('db')
		curs = conn.cursor()
		try:
			# Pulls a record from the database matching the transaction ID
			curs.execute("""SELECT name FROM ipn WHERE txn = ? LIMIT 1""", (id,))
			row = curs.fetchone()
			ret = row[0]
		except sqerr, e:
			ret = "SQL Error: " + e.args[0]
 
		# The response will either by the name of the buyer, or a SQL error message
		return ret

The above method would allow you to access a URL such as http://domain.com/lookup/transactionid/ or http://domain.com/lookup/?id=transactionid, and in return see the name of the buyer of that transaction.

We still need a way to serve this app to the web so that Paypal can reach it.

# Standard TCP method		
#(Server(("127.0.0.1", 9000)) + Root()).run()
 
# Unix Socket Method - make sure webserver can read and write to the socket file
(Server(("ipn.sock")) + Root()).run()

On my server I tend to prefer unix socket files as a means of connection since they’re easier to recongnize in a webserver configuration (The app name is right in the socket file name as opposed to “was it on port 9005 or 9008?”). However a standard TCP setup is more recognizable, and has wider support with most webservers. It is also important to note that using 127.0.0.1 will only allow connections from other services on the same server. To allow a service from outside the server to connect directly to the application, you will need to use a public IP address or 0.0.0.0

Notes
Using Unix Sockets
As of writing this, Unix Sockets are only supported by the development build of Circuits.web, which can be accessed with mercurial using https://dev.circuits.googlecode.com/hg/. The current 1.2 stable release can be obtained from Circuits – Google Code, or by supplying easy_install with the package ‘circuits’. An alternate web framework of similar syntax is CherryPy.

Error Outputs
Paypal always considers a response code of 200 as confirmation that the notification has been successfully delivered. In your production copy you’ll likely wish to change the output to log the errors, and send back a generic error message to the screen, possibly by creating another method.

View the next page to see the example code in uninterrupted form, as well as information on serving the example app with the Nginx webserver.

Nginx and Django

September 14th, 2009

In a previous guide I showed how to use Passenger (aka mod_rails) to work with Python (WSGI) scripts. While this proved effective for simple wsgi applications, a framework such as Django required a bit more love. This guide assume you already have everything installed from the previous guide, and that Python has already been configured with Django and other modules.

Aside from having the usual setup, you will need the original Django source files. One way of going about this is downloading the latest source from the Django Project. The next couple of steps assume you have nginx installed at the /usr/local/nginx/ location, and already have a src folder under your home folder.

# cd ~/src
# wget http://www.djangoproject.com/download/
# tar zxvf Django-1.1.tar.gz
# cd Django-1.1
# mv ./django /usr/local/nginx/html

I should note, my /html/ folder isn’t publicly accessible because all my sites are subfolders under that location. I use it primarily to store resources for the folders beneath it. You can change where you save the source, just remember where you put it and that the webserver has access to it.

Open up your nginx configuration where you have already setup a passenger app from the previous guide. Should look something like this.

server {
	server_name project.domain.com;
	root /usr/local/nginx/html/project.domain.com/public/;
	passenger_enabled on;
}

The first thing you have to do to set this up for Django (at the very least), is to modify the server block above to something more like this:

server {
	server_name project.domain.com;
 
	location / {
		root /usr/local/nginx/html/project.domain.com/public/;
		passenger_enabled on;
	}
 
	location /media {
		alias /usr/local/nginx/html/django/contrib/admin/media;
	}
}

Now you have either create or move your django application into the site folder. For example, if your site is defined as /usr/local/nginx/html/project.domain.com like above, and your django application is called mytest , you will end up having it located in /usr/local/nginx/html/project.domain.com/mytest. In the end you should have a folder structure similar to this (based off the Django tutorials).

Directory Structure

Now open up the passenger_wsgi.py from the previous guide which may look something like this:

import sys
import os
 
def application(environ, start_response):
	start_response("200 OK", [])
	ret = ["%s: %s\n" % (key, value)
		for key, value in environ.iteritems()]
	return ret

We’re going to turn this into a WSGI initializer for the Django application like so:

import os, sys
 
#automatically finds application's current path
nginx_configuration= os.path.dirname(__file__)
project = os.path.dirname(nginx_configuration)
workspace = os.path.dirname(project)
sys.path.append(workspace) 
 
os.environ['DJANGO_SETTINGS_MODULE'] = 'testapp.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

After saving, should restart the nginx server, then load up the site in a browser. If this is a new Django application you’ll be greeted with a “Welcome to Django” page, otherwise the application you had already created.

Static content can be access by creating additional Aliases in Nginx, or by using django.views.static.serve method in your applications url settings.

Using Python with Nginx via Passenger

September 14th, 2009

Since Nginx 0.6.* I been looking for an effective way to run Python (WSGI) applications thru the Nginx Webserver. A couple of options were available at the time:

Update 1/31/2011: There is now an article showing how to deploy Python (namely with Circuits.web) via Nginx + uWSGI.
Deploying circuits.web with Nginx/uwsgi.

Update 9/29/2010: As of version 0.8.40 Nginx now supports wsgi applications running over the uWSGI protocol natively. Click Here for more details. The 0.8.x branch is now the current stable branch.

Apache + mod_wsgi
One possible solution was to run Apache as a backend server listening on 127.0.0.1:8080 (or a port of your choice), which could also be easily used to serve up other dynamic content such as php. Many have used this setup for a great deal of internal load balancing, letting Nginx serve strictly static content, and Apache to handle the dynamic content.

In the sense of a WSGI Python application, you would have a server block like this in Nginx:

server {
	#server_name is what nginx responds to
	server_name  mydomain.com www.mydomain.com;
 
	location / {
		#setting a Host header is handy if you have apache
		#setup with name-based virtual hosts
		proxy_set_header Host mydomain.com;
 
		#requires nginx to be compiled with --with-http_realip_module
		proxy_set_header X-Real-IP $remote_addr;
 
		#This can also be the IP address of another server
		proxy_pass http://127.0.0.1:8080;
	}
}

With the above method you could go with the easy route of having mod_wsgi (as of this writing, mod_wsgi is available at 2.5 supporting Python 2.6) installed via Apache. However doubling up the number of webservers run, might defeat the purpose of running Nginx for some of us, especially for those less familiar with Apache configuration.

Nginx WSGI Module

There was a port of Apache’s mod_wsgi over to Nginx by Manlio Perillo. However this module had only been tested on Nginx 0.5.34, and patched for 0.6.*.

However even when patched, the module remained quite buggy. Implementing this module went something like this:

In the http { } block

    wsgi_python_optimize 0;
    wsgi_enable_subinterpreters on;

In your server { } block

server {
	listen 80;
	server_name yourdomain.com;
 
        location / {
            wsgi_pass  /var/www/username/domain/application.wsgi;
 
            include wsgi_vars;
 
            wsgi_var  SCRIPT_NAME         /var/www/username/domain/application.wsgi;
            wsgi_var  DOCUMENT_ROOT       /var/www/username/domain;
 
            wsgi_pass_authorization off;
            wsgi_script_reloading on;
            wsgi_use_main_interpreter on;
        }
}

This module is no longer maintained and is does not to build on modern versions of Nginx (0.7/0.8), and was barely compatible with 0.6 by using a patch. The creator of the Apache mod_wsgi, Graham Dumpleton commented on his blog regarding the Nginx implementation:


The nginx version of mod_wsgi borrows some code from my original Apache version, but obviously since the internals of Apache and nginx are very different, the main parts of the code which interface with the web server are unique. Although I condoned use of the source code, I do wish I had insisted from the outset that it not be called mod_wsgi due to the confusion that has at times arisen.

Although development on the nginx version of mod_wsgi appears to no longer be happening, this isn’t stopping people from using it and many are quite happy with it. The question is whether they really understand anything about how nginx works and the shortcomings in how nginx and mod_wsgi work together.

Admittedly the author of the mod_wsgi module for nginx has been up front in pointing out that because nginx is asynchronous, and with WSGI not designed for such a system, that once the WSGI application is entered all other activity by the web server is blocked. The recommendation resulting from this is that static files should not be served from the same web server. Use of multiple nginx worker processes is also suggested as a way of mitigating the problem.

Source: Graham Dumpleton: Blocking requests and nginx version of mod_wsgi

There is however a far more practical solution fit for production use, especially for those using modern versions of Nginx.