Hey ./manage.py runserver, Y U No Work?

Recently I was working on a Django-based project that integrates with Android and Embedded Linux devices. One day a coworker called me and asked about an error he had encountered. It seemed like he was not able to execute ./manage.py runserver to any port he had previously used. He was careful to use Control + C to stop the server, but for some reason the port was held onto by something. Looking at the output of “ps aux” was not showing any running processes using ./manage.py.

Who’s using that port?

Next instead of looking at processes, I wanted to see what was holding onto that port.  I ran the following command:

netstat -lpn | grep 8000

The output gave me the answer:

tcp    0    0 127.0.0.1:8000        0.0.0.0:*      LISTEN    15902/adb

So ADB was the culprit. A simple kill -9 to the process ID and the issue was fixed (temporarily). The issue keeps occuring, so maybe something in my code may be keeping ADB holding onto that port. I’m using the subprocess module to interact with ADB. I tried adding a call to terminate from the result of my subprocess.Popen call, but that ended very badly and had no effect on the hanging port. I’m sure when I have time to dig deeper (read “After My Looming Deadline”), I’ll try to write some tests to explore this issue more thoroughly.

This is yet another post that’s really a reminder for me, but you’re welcome to it if it helps. I’ve been working on the Android OS for a few months now and at first I was physically touching the phone to reboot it into fastboot mode and then back to the normal boot mode. I grew tired of this manual process and so one day it dawned on me that fastboot also had a reboot command & I could make a bash alias and run all the commands to do this process at once.

m && cd out/target/product/passion/ && adb reboot-bootloader && fastboot flash system system.img && fastboot reboot && cd ../../../../

You’ll hear the phone vibrate when it’s booting back up normally and know it’s time to get back to work…

I’ve been working on a Django-based project since October 2010 that includes some integration with Scrapy. Scrapy is a web scraping framework for Python and due to its design inspiration, Django developers will be quickly familiar with it. I have installed Scrapy no less than 10 times at this point and decided it was time for a script. This post is more of a reminder to me for the next time I install Scrapy, but if it works for you as well – great!.

This script works for Ubuntu 9.10, 10.04 and 10.10.  If you’re on an older release, you’ll need to install some dependencies first and then follow these instructions.

sudo apt-get install curl
curl -s http://archive.scrapy.org/ubuntu/archive.key | sudo apt-key add -
DISTRO=$(lsb_release -cs)
echo deb http://archive.scrapy.org/ubuntu $DISTRO main | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install scrapy-0.12

Have you ever edited your models.py file and shortly after when trying to see the change list for a model in the admin site you received this error?

Caught an exception while rendering: coercing to Unicode: need string or buffer, NoneType found

I did today and it was one of those “bonehead” moments. You know, the one where you’ve been coding for a bit and for some reason make a simple error that leaves you thinking “I’ve done that exact same action a hundred times, why would I get an error this time?”, when you realize you made a really simple mistake. The answer was quick and easy. It was not occuring on the new model form, only on the change list so I opened my models file and looked over the model in question and then realized my typo. I had forgot to add “return” to the __unicode__ method on my model. So if you get this error, make sure that you’re returning something and that its really unicode.

I’ve been working on a project that interacts with Microsoft Office SharePoint Server 2007 (MOSS 2007) for the last few months and one of the tools I’ve been using daily to support my work is IronPython. IronPython has made it easy for me to come up to speed quickly on the MOSS 2007 API but what has made my life even easier is the IronPython Console.

*Note: If you have a browser that supports Silverlight, you can try Python in your browser now.

IronPython ships with a console virtually identical to the stand Python console. The IronPython console provides me with a constant live connection directly to the API. It’s very different and somewhat empowering to transition from the standard workflow in Visual Studio of write code/build/run to something more like a conversation with the code. I think the term conversation is a very good analogy because it’s not just looking at classes and methods, it’s also data and interaction. It’s experiencing everything all at once!

Let’s take a look at what I’m talking about. To run these samples, I had IronPython 2.6.1 installed on a server running Windows Server 2003 and MOSS 2007.

When you first start up the console, you need to reference the Microsoft.SharePoint .dll. You SharePoint types probably know to find this in either the 12 Hive or the GAC.

>>> import clr
>>> clr.AddReference("Microsoft.SharePoint")

We need to import the SPSite class before we use it too.

>>> from Microsoft.SharePoint import SPSite

Now we can create a connection to a site and open its default SPWeb object.

>>> site = SPSite(baseurl)
>>> web = site.OpenWeb()
>>> web

We can also navigate around the subwebs of a given web by using the Webs property. You can run into permissions issues though. Here we try to access the first subweb in the Webs collection.

>>> web.Webs[0]
Traceback (most recent call last):
File "", line 1, in
SystemError: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENI
ED))

To get around this error, we can just use the SystemAccount token attached to each site. I really don’t know the reason behind the design of this API, but you can use one site reference to give you a connection with elevated privileges on another site.

>>> temp = SPSite(baseurl)
>>> site = SPSite(baseurl, temp.SystemAccount.UserToken)
>>> web = base_site.OpenWeb()

Now you can start exploring around and seeing what exits. When you want to see
what members a given object has, you just use the dir() method. Let’s take a look at
the first object in the Lists collection.

>>> dir(base_web.Lists[0])
['AddWorkflowAssociation', 'AlertTemplate', 'AllRolesForCurrentUser', 'AllowCont
entTypes', 'AllowDeletion', 'AllowEveryoneViewItems', 'AllowMultiResponses', 'Al
lowRssFeeds', 'AnonymousPermMask', 'AnonymousPermMask64', 'Audit', 'Author', 'Ba
seTemplate', 'BaseType', 'BreakRoleInheritance', 'CanReceiveEmail', 'CheckPermis
sions', 'CheckedOutFiles', 'ContentTypes', 'ContentTypesEnabled', 'Created', 'Cu
rrentChangeToken', 'DefaultApprovalWorkflowId', 'DefaultContentApprovalWorkflowI
d', 'DefaultItemOpen', 'DefaultView', 'DefaultViewUrl', 'Delete', 'Description',
'Direction', 'DocumentTemplateUrl', 'DoesUserHavePermissions', 'DraftVersionVis
ibility', 'EffectiveBasePermissions', 'EffectiveFolderPermissions', 'EmailAlias'
, 'EmailInsertsFolder', 'EnableAssignToEmail', 'EnableAttachments', 'EnableDeplo
yWithDependentList', 'EnableDeployingList', 'EnableFolderCreation', 'EnableMinor
Versions', 'EnableModeration', 'EnableSchemaCaching', 'EnableSyndication', 'Enab
leVersioning', 'EnsurePropsFresh', 'EnsureRssSettings', 'Equals', 'EventReceiver
s', 'EventSinkAssembly', 'EventSinkClass', 'EventSinkData', 'ExcludeFromTemplate
', 'Fields', 'FirstUniqueAncestor', 'Folders', 'ForceCheckout', 'Forms', 'GetCha
nges', 'GetContentTypeIdByUrl', 'GetDefaultViewForContentType', 'GetDirectChildC
ontentType', 'GetDistinctFieldValues', 'GetHashCode', 'GetItemById', 'GetItemByU
niqueId', 'GetItems', 'GetItemsInFolder', 'GetPropertiesXmlForUncustomizedViews'
, 'GetType', 'GetUncustomizedViewByBaseViewId', 'GetView', 'HasUniqueRoleAssignm
ents', 'Hidden', 'ID', 'ImageUrl', 'IrmEnabled', 'IrmExpire', 'IrmReject', 'IsCa
talog', 'IsContentTypeAllowed', 'ItemCount', 'Items', 'LastItemDeletedDate', 'La
stItemModifiedDate', 'Lists', 'MajorVersionLimit', 'MajorWithMinorVersionsLimit'
, 'MemberwiseClone', 'MobileDefaultViewUrl', 'MultipleDataList', 'NoCrawl', 'OnQ
uickLaunch', 'Ordered', 'ParentWeb', 'ParentWebUrl', 'Permissions', 'PropertiesX
ml', 'ReadSecurity', 'Recycle', 'ReferenceEquals', 'RemoveWorkflowAssociation',
'RenderAsHtml', 'RequestAccessEnabled', 'ResetRoleInheritance', 'RestrictedTempl
ateList', 'ReusableAcl', 'RoleAssignments', 'RootFolder', 'SaveAsTemplate', 'Sch
emaXml', 'SendToLocationName', 'SendToLocationUrl', 'ServerRelativeDocumentTempl
ateUrl', 'ServerTemplateCanCreateFolders', 'ServerTemplateCanReceiveEmail', 'Set
AttributesForPropertiesXML', 'ShowUser', 'TemplateFeatureId', 'Title', 'ToString
', 'Update', 'UpdateWorkflowAssociation', 'Version', 'Views', 'WorkflowAssociati
ons', 'WriteRssFeed', 'WriteSecurity', '__class__', '__delattr__', '__doc__', '_
_format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclas
shook__', 'm_ListAttributesDict', 'm_Lists', 'm_Permissions', 'm_arrListProps',
'm_iRow']

Now this may or may not be readable to you. Alot of times when there’s a ton of text to deal with in the console, I’ll iterate through the methods and print them each on their own line.

>>> for method in dir(base_web.Lists[0]):
...     print method
...
AddWorkflowAssociation
AlertTemplate
AllRolesForCurrentUser
AllowContentTypes
AllowDeletion
AllowEveryoneViewItems
.... [Omitted to save space]
__class__
__delattr__
__doc__
__format__
__getattribute__
__hash__
__init__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__
m_ListAttributesDict
m_Lists
m_Permissions
m_arrListProps
m_iRow

You can inspect classes too. You don’t need an instance. If you inspect the class you’ll just need to import the type first.

>>> for method in dir(SPListCollection):
...     print method
...
Traceback (most recent call last):
NameError: name 'SPListCollection' is not defined
>>> from Microsoft.SharePoint import SPListCollection
>>> for method in dir(SPListCollection):
...        print method
...
Add
CopyTo
... [Omitted to save space]
__add__
__class__
__delattr__
__doc__
__format__
__getattribute__
__getitem__
__hash__
__init__
__iter__
__len__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__

If you’re coming from a C# or VB background, you’re probably familiar with the “Using” block. Python has something similar to that with the “with” statement. The with statement is used with ContextManagers. Many of the .Net framework classes support the ContextManager methods so you can use the with statement. You can see the __enter__ and __exit__ methods below that help make this possible.

>>> dir(SPSite)
['AddWorkItem', 'AllWebs', ...[Omitted for space], '__class__', '__delattr__', '__doc__', '__enter__
', '__exit__', '__format__', '__getattribute__', '__hash__', '__init__', '__new_
_', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__s
tr__', '__subclasshook__']

Now we can refactor our earlier sample to use the with statement. This is something I use more in scripts that reference MOSS 2007, but I thought you may find some uses.

>>> web = None
>>> with SPSite(baseurl) as temp:
... 	with SPSite(baseurl, temp.SystemAccount.UserToken) as site:
...			web = site.OpenWeb()
...

It’s all pretty basic and the console lends itself well to those who like to explore. I really like this method versus digging around the object browser or Reflector all of the time. I like the instant feedback when exploring and the ability to see data and not just structure. I’ve also successfully used IronPython to create custom event handlers and for some administrative tasks for a 2003 to 2007 upgrade. I highly recommend you try it if you find yourself having to do any development using the SharePoint API.

I am finally writing another post after falling off the grid for a couple of months.  I had high hopes of writing a post every day in November, but the holidays combined with a full time job and paying side job keeping me up until 2AM or 3AM left me with little motivation to squeeze out a post.  Yesterday I finally completed the project and so I’ve got some more time on my hands.

I’m not at liberty to announce the project yet, but it is something that will be public and it involves my favorite web framework. Once it launches officially I will provide some more details to how it was created.

So now I have a few things I’ll be working on.  They’re going to be including Django, jQuery, CouchDB, Amazon S3 and Rackspace Cloudfiles.  At work I’ve been busy creating a solution with and ASP.Net MVC front end, A MOSS 2007 backend and some custom workflow type components built with Iron Python.  Hopefully that will lead to some more interesting posts here.

Over the last week I’ve been reading a few articles about CouchDB and playing with some sample code.  Yesterday I read Eric Florenzano’s article on Using CouchDB with Django and it elevated my interest.  CouchDB is a a document-oriented database from Apache.  It can be queried and indexed in a MapReduce function in javascript.  CouchDB also provides an incremental replication feature that finds and resolves conflicts bi-directionally.

Eric’s article provided an example of using CouchDB that didn’t fit what I was looking to do, but provided me with two key pieces of data; an example of using the python library for CouchDB and introducing me to an awesome project CouchDBX.  CouchDBX is basically an InstantCouchDB for Mac OSX.  It was very easy to download and start using CouchDB immediately.

Unfortunately this post has no code sample.  I haven’t slayed CouchDB yet.  Whether I use a Python class that inherits from object or from the couchdb-python library Document object, I end up with a json serialization error “<object> is not JSON serializable”.  It’s annoying, but there’s just something I’m still missing.  I’ll figure out (or some Google searching will).

I’m planning a demo at work.  Since I still work for a Microsoft based shop, most of these alternate technologies are like a traveling freak show from the circus for them.  I do see some places where CouchDB would work well, although security is an issue.  Where I work they want everything protected with CAC’s or certificates of some sort, so I’ll have to eventually look into how to do that with CouchDB.

UPDATE:

I was able to start storing data once I created a proper object.  It seems that a public property of a collection type isn’t json serializable.  I had a property “answers = []” and it was causing the serialization error.  Creating and updating data is working well now.  Time to tackle queries and then replication.

This weekend we drove up to Raleigh for the simple pleasure of giving Apple, Inc. a handful of money. I’ve wanted one for over a year, but the timing or the budget just didn’t line up until now. We purchased an iMac last year as a “Shared Family” computer. Wrong. After 3 days, my wife decided it was hers and the rest of us were never to touch it.  Ever.

First Mac

I ended up getting the 2.53Ghz, with a 320GB HDD and 4GB of RAM.  I also purchased a wired keyboard with the number pad and a magic mouse.  The mouse is pretty good, like magic.  So far it’s been very easy to set up and use.

Now that I have the Mac I hope to take a little time to get to know the platform, try out some Obj-C, PyCocoa, and maybe try my hand at an iPhone app.  I’m very curious to see where this differs from my Linux platform.

If you want the play by play, here’s the Whrrl story to document my experience.

Ok so the annual November is the month for blogging thing is going on and this year I’m going to attempt it.  Like many other developers I don’t keep up with my blog as much as I should.  I don’t feel guilty about letting the reader down….it’s not like I have a huge following anyway.  The blog is more for me for getting out my ideas (somewhat cautiously still) and as a place to find things I need when I’ve forgotten.

So for the month of November, I will give it my best to write something daily.  It may not be long or particularly interesting but that’s not the goal!  Here’s to setting the bar low!

I’ve had the occasion to setup new Ubuntu environments several times over the last couple of months and out of laziness I created a simple script to install easy_install, pip, virtualenv, virtualenvwrapper and fabric.  I’m no bash ninja, but this script makes it quick and easy for me to install the python tools I need to play.  Sharing this with you is only half of my motivation; I’m also putting this here for someday when I forget it…

#!/bin/bash
 
#prerequisites
echo ensuring buid-essential and required dev headers are installed
sudo aptitude install build-essential python2.5-dev python2.6-dev
 
#easy_install
echo installing easy_install
mkdir ~/tmp_sources
cd tmp_sources
wget http://peak.telecommunity.com/dist/ez_setup.py
sudo python ez_setup.py
rm -rf ez_setup.py
 
#pip
echo installing pip
sudo easy_install pip
 
#fabric
echo installing fabric
sudo pip install fabric
 
#virtualenv
echo installing virtualenv
sudo pip install virtualenv
sudo pip install virtualenvwrapper
 
#clean up
rm -rf ~/tmp_sources
echo all done!