Archive for the ‘Primary’ category

Apache to Nginx Migration Tips

April 11th, 2010

For a simple beginning example configuration to start with please view my example configuration

Nginx currently holds shy of 6.5% of the known webserver market, which is just roughly shy of 13 million servers. This little lightweight webserver created by a sole Russian developer has been gaining a great deal of popularity over the last few years and is used by sites such as WordPress, Texts from Last Night and Hulu.

This guide will show you common examples in Nginx to make migration from Apache a bit easier.

HTTP Basic Authentication or htpasswd support

If you have been playing with nginx for a the first time you may have noticed that there is no .htaccess support. However Nginx can still utilize HTTP Basic Authenication with existing htpasswd files using the HTTP Auth Basic Module. The module is compiled by default.

If you still have Apache installed (but perhaps turned off or on a different port) you can still use htpasswd to generate a password file. But in the event that you don’t have or don’t want to install Apache you can use Perl or Ruby to generate at least the encrypted passwords. The .htpasswd files should be in the following format.

username:encrypted-password:comment

To generate a password with Ruby in irb:

"your-password".crypt("salt-hash")

Likewise to generate with perl from the terminal:

perl -le 'print crypt("your-password", "salt-hash")'

The two commands above will utilize a 56-bit DES encryption which can be used in the htpasswd file.

Once you have your password file saved, store it outside of the web-accessible location like you would have when using Apache (likewise you can ensure that web access to hidden files are denied, more on that later).

Lets say you have a subfolder on your server called ‘admin’ and you wish to protect it. Here is an example configuration:

location  /admin  {
  auth_basic            "Admin Access";
  auth_basic_user_file  conf/domain.com/admin/htpasswd;
}

If the above did not work, check your error logs as sometimes it will notify you if the path to the password file could not be found.

Ignoring robots.txt, favicon.ico and hidden files

Sometimes you may wish to omit commonly accessed files from being recorded in the access log as well as block access to hidden files (filenames beginning with a period such as .htpasswd and .htaccess). Even though nginx doesn’t support htaccess, you may still wish to secure files left behind from the migration. You can easily do this by adding a couple location blocks to your domain’s server block.

location = /favicon.ico { access_log off; log_not_found off; }	
location = /robots.txt { access_log off; log_not_found off; }
location ~ /\. { deny  all; access_log off; log_not_found off; }

The above will not record access to the favicon.ico or robots.txt in the root of your site, as well as ignore the file-not-found error if you do not have either file present. Also the last line will deny access to any file beginning with a period anywhere within the root path. You can actually save the above into a file such as /conf/drop.conf, and include it at the bottom of each of your server blocks like so.

include drop.conf;

Extremely Easy WordPress Migration

Most people now days take advantage of WordPress’ permalink feature which normally requires mod_rewrite enabled in the .htaccess file. However as of Nginx 0.7.26 and above this can be taken care of very easily with a single directive.

location / { 
   try_files $uri $uri/ index.php;
}

The above will attempt to try the requested URI first as a file, then as a folder, and if both of those come back negative it will fall back to index.php. WordPress can simply parse the request URI sent by the webserver so there is no need to manually add arguments onto index.php. If you instead need to do something further you can change the fallback from index.php to @fallback like so:

location / {
   try_files $uri $uri/ @wordpress;
}
 
location @wordpress {
     # Additional rewrite rules or such you need here.
}

For Nginx versions older than 0.7.26 you can use the older method shown below

location / { 
        if (-f $request_filename) {
            expires 30d;
            break;
        }
 
         if (!-e $request_filename) {
            rewrite ^(.+)$ /index.php?q=$1 last;
        }
}

The two if blocks replace the common RewriteCond used by wordpress to test if the requested filename is an existing file or folder.

On the next page: Utilizing WP Super Cache, rewrite examples and more.

How to Steal an Android Market App

April 10th, 2010

One of the biggest fear plaguing any freelance application developer is piracy. All their hours and hours of work to bring you the next useful little app that they hope you’ll enjoy. So why shouldn’t they be compensated for their hard work. Sometimes however this fear can hurt a new platform more than it can help. That is why in this article I will show you how easy it is to steal even a protected android market app.

Step One – Rooting your phone

Rooting your android device is similar to jailbreaking your iPod Touch or iPhone. Basically gives you the ability to perform tasks normally reserved only for the ‘root’ user of the device. The method of rooting vary greatly depending on the device. I personally own a T-Mobile MyTouch 3G Limited Edition (fender) which is identical to the new T-Mobile MyTouch 3G v1.2 with the headphone jack on top, the only difference is design, and the size of the miniSDHC card that comes with it. I rooted my phone by following the instructions found in this XDA Developer forum thread. If you have one of the older MyTouches you can easily find how-to guides on the same forum corresponding to the Magic 32A and 32B.

Warning: Like most things in gadget life, performing an action such as above not only has the possibility of voiding your warranty, but can also brick your device, that is to say it’ll be no better than a brick that won’t even turn on. So proceed at caution. It is very important to not only carefully read any instructions (such as provided by the link above) but also to verify that your device actually matches the instruction’s requirements. Most people who do this tend to have no problems, but those who fail to actually follow instructions have a very good chance at flashing the wrong recovery or radio image.

Once you are rooted, you can download the Android Software Developer’s Kit, or find the ConnectBot App in the market place. Sometimes protected applications (like the free Paypal app) may not show up after rooting, usually installing the Market Enabler fixes this.

Step Two – Purchase

For this demonstration to work we need to make a purchase. For this article I’ve decided to purchase Retro Defense by Larva Labs. I’ve always enjoyed the games from Larva Labs.

Ok, so we have purchased the game, now what?

Step Three – Back it up

With your rooted phone you can now either open up ConnectBot, or in your terminal run “adb shell” (from the SDK).

Once you are in the terminal as root (you may have to type su if you see $ instead of # to elevate yourself as the root user), you can then follow the following commands, to make a directory and copy the game to that directory.

# cd /sdcard
# mkdir /backapk
# cd /data/app
# ls (you'll see a screen showing you the file content, find the file you want)
# cp com.larvalabs.retrodefense.apk /sdcard/backapk

You may be prompted by SUuser application to allow Connectbot root access (if you used adb shell instead you won’t see this prompt)




At this point all you have done was make a backup of your app/game onto the SD card. Protected application usually exist in /data/app-private as an apk (leaving a small .zip in the /app location)

Migrating Cpanel to DirectAdmin

March 28th, 2010

One of the most frustrating thing someone can do involving their websites is moving them from one hosting provider to another. It’s increasingly more difficult if your hosting was based on a control panel such as Cpanel, and try to migrate to a different kind of control panel or none at all.

Step One – Package your account

If you have root access to the server via SSH (such as if you have a VPS) you can do this quite simply by performing the following command.

/scripts/pkgacct username

The above command will place the cpmove-username.tar.gz file into the /home directory where you can download it via FTP/SCP.

For those who only have user-level cpanel access, you will need to log into your cpanel account. From there under the file section you will see a Backup Wizard icon.

Cpanel Files Section

When you click on that, you’ll be prompted to 1) Backup. 2) Full Backup 3) Provide email.

When the backup packaging is complete you will receive an email notification, and the generated file will be placed in your home directory. (usually the default root of your FTP client when you log in). You will want to download this .tar.gz file onto your desktop.

Step Two – Preparing to upload

For those of you who are windows users, you will need to download something such as 7-Zip in order to unpack the tar.gz file. Create a folder to place the file in and unpack it. The resulting files will be all the data from your Cpanel account (including your mail in MailDir format). Find the homedir.tar and pull it out of the folder and extract it.

Once extracted you’ll see various folders that was in your home directory. In /tmp you may even find some of your statistics such as webalizer which can be opened right from your desktop. The files you want to concentrare are in /public_html.

If you use any add-on domains, move them from the public_html folder. Once you’ve done that you can then compress the public_html folder content as a .zip file. Do the same for each one of the add-on domains. Don’t delete the uncompressed copies just yet.

PayPal IPN Revised for Python

January 24th, 2010

This article adds onto the previous entry Paypal IPN with PHP, by showing you how to process an Instant Payment Notification from Paypal with Python. For more information about setting up your PayPal account or purchase code for Instant Payment Notifications, refer to the link above.

For any python app, you will need a way to launch it from the terminal. You can use the “shebang” below to make the app self serving. You will of course need to make sure it has executable privilages (chmod +x ./app.py on most unix/linux systems). If the instance of python you are using is not the systemwide instance such as python 2.6 installed to /opt, you will then then change it to use python2.6 instead.

#!/usr/bin/env python

You still launch the app via the interpreter directly, omitting the above shebang if you’d like.

With the exception of circuits.web, all of the modules used are built into Python (as of 2.6 to my knowledge).

from time import time
from urllib import urlencode
from sqlite3 import connect, Error as sqerr
from urllib2 import urlopen, Request
from circuits.web import Controller, Server

I have created a separate method for verification to keep the rest of the code cleaner. This is pretty much the same process PayPal provides developers, plus two checks that I feel are important.

def verify_ipn(data):
	# prepares provided data set to inform PayPal we wish to validate the response
	data["cmd"] = "_notify-validate"
	params = urlencode(data)
 
	# sends the data and request to the PayPal Sandbox
	req = Request("""https://www.sandbox.paypal.com/cgi-bin/webscr""", params)
	req.add_header("Content-type", "application/x-www-form-urlencoded")
	# reads the response back from PayPal
	response = urlopen(req)
	status = response.read()
 
	# If not verified
	if not status == "VERIFIED":
		return False
 
	# if not the correct receiver ID
	if not data["receiver_id"] == "DDBSOMETHING4KE":
		return False
 
	# if not the correct currency
	if not data["mc_currency"] == "USD":
		return False
 
	# otherwise...
	return True

The verification url provided above is for the PayPal sandbox, a developer testing ground. You can signup for access to test accounts, and tools such as IPN Test submissions at PayPal SandBox. When the app is ready for production, simply remove .sandbox from the url.

Just because PayPal says it’s Verified, only means that a payment has a occured, and that the data received is the same on Paypal’s database. Someone could have paid themselves using your IPN URL; for this reason we want to make sure that the receiver_id matches your own. The Secure Merchant ID can be found on the top your Paypal account’s profile page as shown below:

Next is a currency check, because 100 Zimbewe dollars is not equivalent to 100 US Dollars. The variables mc_gross as well as item prices don’t establish the currency used, but rather simply the value amount. So this check is important to protect against incorrect purchase amounts.

Now we have the main Root class of the IPN app.

class Root(Controller):
	# if the app will not be served at the root of the domain, uncomment the next line
	#channel = "/ipn"
 
	# index is invoked on the root path, or the designated channel URI
	def index(self, **data):
		# If there is no txn_id in the received arguments don't proceed
		if not "txn_id" in data:
			return "No Parameters"
 
		# Verify the data received with Paypal
		if not verify_ipn(data):
			return "Unable to Verify"
 
		# Suggested Check : check the item IDs and Prices to make sure they match with records
 
		# If verified, store desired information about the transaction
		reference = data["txn_id"]
		amount = data["mc_gross"]
		email = data["payer_email"]
		name = data["first_name"] + " " + data["last_name"]
		status = data["payment_status"]
 
		# Open a connection to a local SQLite database (use MySQLdb for MySQL, psycopg or PyGreSQL for PostgreSQL)
		conn = connect('db')
		curs = conn.cursor()
		try:
			curs.execute("""INSERT INTO ipn (id, purchased, txn, name, email, price, notes, status) 
			VALUES (NULL, ?, ?, ?, ?, ?, NULL, ?)""", (time(), reference, name, email, amount, status,))
			conn.commit()
		except sqerr, e:
			return "SQL Error: " + e.args[0]
		conn.close()
 
		# Alternatively you can generate license keys, email users login information
		# or setup accounts upon successful payment. The status will always be "Completed" on success.
		# Likewise you can revoke user access, if status is "Canceled", or another payment error.
 
		return "Success"

In most cases you would host each application on their own domain or subdomain, such as ipn.domain.com. But I generally prefer to run all my apps from a single subdomain, such as apps.domain.com. The Controller above is expecting / for the root base, not /ipn/ like I would have. As a result we add the channel line to inform the controller of the new base.

The variable data will be received as a dict type, we’ll first check to make sure data has come in (by seeing if there’s a transaction ID with the data received), then verify the data with Paypal. Once verified, the desired information can be stored in a database, be it SQLite, MySQL, PostgreSQL or Durus.

Three things I would suggest for this portion: use a log or email to keep track of errors, and if you use sell a number of products online, check the item ID and prices to verify their accuracy. Also if your products are subscriptions, or prone to returns, check to see if a transaction ID already exists in the database and update it’s status accordingly as opposed to creating a new record for every instant payment notification, useful if someone cancels or charges back their transaction.

The next method is not very practical by itself but could lead to more useful ideas. This would be placed under the same Root class above.

	def lookup(self, id):
		if not id:
			return "No Transaction Provided"
 
		conn = connect('db')
		curs = conn.cursor()
		try:
			# Pulls a record from the database matching the transaction ID
			curs.execute("""SELECT name FROM ipn WHERE txn = ? LIMIT 1""", (id,))
			row = curs.fetchone()
			ret = row[0]
		except sqerr, e:
			ret = "SQL Error: " + e.args[0]
 
		# The response will either by the name of the buyer, or a SQL error message
		return ret

The above method would allow you to access a URL such as http://domain.com/lookup/transactionid/ or http://domain.com/lookup/?id=transactionid, and in return see the name of the buyer of that transaction.

We still need a way to serve this app to the web so that Paypal can reach it.

# Standard TCP method		
#(Server(("127.0.0.1", 9000)) + Root()).run()
 
# Unix Socket Method - make sure webserver can read and write to the socket file
(Server(("ipn.sock")) + Root()).run()

On my server I tend to prefer unix socket files as a means of connection since they’re easier to recongnize in a webserver configuration (The app name is right in the socket file name as opposed to “was it on port 9005 or 9008?”). However a standard TCP setup is more recognizable, and has wider support with most webservers. It is also important to note that using 127.0.0.1 will only allow connections from other services on the same server. To allow a service from outside the server to connect directly to the application, you will need to use a public IP address or 0.0.0.0

Notes
Using Unix Sockets
As of writing this, Unix Sockets are only supported by the development build of Circuits.web, which can be accessed with mercurial using https://dev.circuits.googlecode.com/hg/. The current 1.2 stable release can be obtained from Circuits – Google Code, or by supplying easy_install with the package ‘circuits’. An alternate web framework of similar syntax is CherryPy.

Error Outputs
Paypal always considers a response code of 200 as confirmation that the notification has been successfully delivered. In your production copy you’ll likely wish to change the output to log the errors, and send back a generic error message to the screen, possibly by creating another method.

View the next page to see the example code in uninterrupted form, as well as information on serving the example app with the Nginx webserver.

Releasing IP Addresses in WHM/Cpanel

December 28th, 2009

If you are hosted with a WHM/Cpanel based server or VPS, you may have had some difficulties trying to get an alternate webserver such as Nginx or Lighttpd installed especially if you wanted to use the default port 80. This article shows you how you can release extra IP addresses to be used by those services without conflicting with Apache.

There are two areas wee need to address, Apache, and then WHM/Cpanel.

Apache

By default Apache listens to every interface coming into the machine by listening to 0.0.0.0:80/443. Normally you could edit the httpd.conf file directly in order to change the listen line, however that may cause problems with Cpanel’s automation.

To acheive this more safely, log into Webhost Manager (typically http://yourdomain.com/whm) and find Service Configuration followed by Apache Configuration. Then click on Reserved IPs Editor.

Here you will need to check the boxes of the IP addresses you do NOT wish for Apache to use. Once you save your selections, a new configuration file will be configured to listen on all the other IPs that were not checked. This will allow other services such as alternate webbrowsers to listen on those IP addresses without conflicting with Apache.

Webhost Manager

We freed some IP addresses from Apache, but now we have to make sure that WHM/Cpanel doesn’t attempt to use those same IP addresses for other services.

Navigate to IP Functions, followed by Show/Edit Reserved IPs. Here you should check the same IPs that were selected in the Apache Reserve list above. Once this is done WHM/Cpanel will avoid using those IP address when setting up new accounts and services.

Configuring Nginx
Assuming Nginx is already configured you will need to make a minor adjustment to the server blocks in order to start it up.

server {
	listen an-ip-address:80;
	server_name yourdomain.com www.yourdomain.com;
	...
}

Normally you would just have the port number identified, but if you tried to start it as just that, it would conflict with Apache if already running (otherwise vice-versa). So we need to make sure to bind each server block to an available IP address.

While I personally prefer to run Nginx standalone on it’s own server such as the one Kbeezie.com is running off of. Some may wish to use the benefits of Nginx for certain projects while still having the ease of use of Cpanel for other sites.

On a side note, if PHP is setup as a Fast-CGI executable in WHM you can share the same PHP instance with Nginx or Lighttpd, otherwise you’ll need to make sure to either compile a separate instance of PHP, or launch the fast-cgi daemon manually for the existing installation (spawn-fcgi, php-fpm, etc).

Handling wildcard subdomains with PHP

November 5th, 2009

Subdomains can be a very handy way to make your urls more friendly looking. They can also be incredibly useful for membership driven websites to allow members to have their own custom subdomain. But how do make manage dynamic subdomains with PHP?

There is two parts to making dynamic subdomains work in this case. The first is updating your DNS and webserver to expect wildcard requests. On the DNS side its a simple matter of adding * as an ‘A’ record, or cname pointing to the primary domain or IP. In most cases if you are using shared hosting, you’ll need to contact your hosting provider to enable wildcard DNS on your account. Otherwise if you are on a dedicated server or virtual private server, this can be most likely handled by yourself, such as editing the DNS zone in WHM.

Once you have enabled wildcard DNS, you’ll need to configure your webserver to expect a *.domain.com request. Otherwise it will ignore all requests other than the base domain and pre-defined subdomains.

In Nginx simply modify the server_name directive:

server_name yourdomain.com *.yourdomain.com;

In Apache you would add a ServerAlias directive to the appropriate virtual host entry.

ServerAlias *.yourdomain.com

On most Cpanel driven servers the http.conf file for Apache can be found in /usr/local/apache/conf.

Once you have configured both the name server and web server for wildcard requests you can now use php to not only detect the provide subdomain, but also generate them. The code below will obtain the subdomain being visited.

$sub = str_replace(".yourdomain.com", "", $_SERVER["HTTP_HOST"]);

The above will simply take the hostname stored in the HTTP_HOST enviroment variable, and attempt to replace the base domain with nothing (if no subdomain was provided all that remains is the base domain).

Once the subdomain is determined, you can filter out requests either with an if/then block or in a switch/case statement. In the event of using memeber names as subdomains you can use a switch statement to make sure that the subdomain doesn’t match the homepage (www / yourdomain.com) or a reserved page name such as register.yourdomain.com and assume anything else must be a member name. With that information you can pull the members information from a file or database.

Using PHP to handle the subdomain is certainly easier for most novice as opposed to modifying an htaccess to do rewrite rules passing the subdomain off as a query (which takes more resources than str_replace) or having to modify the webservers configuration file, especially if you do not have direct access to the webserver’s configuration file or do not have override permission.

If your current hosting provider does not support wildcard subdomain, and you are running a memeber based site, consider upgrading to a Virtual Private Server or use a shared hosting provider such as HostGator that allows you to turn on such capability via a support ticket.