Hey ./manage.py runserver, Y U No Work?

Recently I was working on a Django-based project that integrates with Android and Embedded Linux devices. One day a coworker called me and asked about an error he had encountered. It seemed like he was not able to execute ./manage.py runserver to any port he had previously used. He was careful to use Control + C to stop the server, but for some reason the port was held onto by something. Looking at the output of “ps aux” was not showing any running processes using ./manage.py.

Who’s using that port?

Next instead of looking at processes, I wanted to see what was holding onto that port.  I ran the following command:

netstat -lpn | grep 8000

The output gave me the answer:

tcp    0    0 127.0.0.1:8000        0.0.0.0:*      LISTEN    15902/adb

So ADB was the culprit. A simple kill -9 to the process ID and the issue was fixed (temporarily). The issue keeps occuring, so maybe something in my code may be keeping ADB holding onto that port. I’m using the subprocess module to interact with ADB. I tried adding a call to terminate from the result of my subprocess.Popen call, but that ended very badly and had no effect on the hanging port. I’m sure when I have time to dig deeper (read “After My Looming Deadline”), I’ll try to write some tests to explore this issue more thoroughly.

I’ve been working on a Django-based project since October 2010 that includes some integration with Scrapy. Scrapy is a web scraping framework for Python and due to its design inspiration, Django developers will be quickly familiar with it. I have installed Scrapy no less than 10 times at this point and decided it was time for a script. This post is more of a reminder to me for the next time I install Scrapy, but if it works for you as well – great!.

This script works for Ubuntu 9.10, 10.04 and 10.10.  If you’re on an older release, you’ll need to install some dependencies first and then follow these instructions.

sudo apt-get install curl
curl -s http://archive.scrapy.org/ubuntu/archive.key | sudo apt-key add -
DISTRO=$(lsb_release -cs)
echo deb http://archive.scrapy.org/ubuntu $DISTRO main | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install scrapy-0.12

Have you ever edited your models.py file and shortly after when trying to see the change list for a model in the admin site you received this error?

Caught an exception while rendering: coercing to Unicode: need string or buffer, NoneType found

I did today and it was one of those “bonehead” moments. You know, the one where you’ve been coding for a bit and for some reason make a simple error that leaves you thinking “I’ve done that exact same action a hundred times, why would I get an error this time?”, when you realize you made a really simple mistake. The answer was quick and easy. It was not occuring on the new model form, only on the change list so I opened my models file and looked over the model in question and then realized my typo. I had forgot to add “return” to the __unicode__ method on my model. So if you get this error, make sure that you’re returning something and that its really unicode.

I’ve been working on a project that interacts with Microsoft Office SharePoint Server 2007 (MOSS 2007) for the last few months and one of the tools I’ve been using daily to support my work is IronPython. IronPython has made it easy for me to come up to speed quickly on the MOSS 2007 API but what has made my life even easier is the IronPython Console.

*Note: If you have a browser that supports Silverlight, you can try Python in your browser now.

IronPython ships with a console virtually identical to the stand Python console. The IronPython console provides me with a constant live connection directly to the API. It’s very different and somewhat empowering to transition from the standard workflow in Visual Studio of write code/build/run to something more like a conversation with the code. I think the term conversation is a very good analogy because it’s not just looking at classes and methods, it’s also data and interaction. It’s experiencing everything all at once!

Let’s take a look at what I’m talking about. To run these samples, I had IronPython 2.6.1 installed on a server running Windows Server 2003 and MOSS 2007.

When you first start up the console, you need to reference the Microsoft.SharePoint .dll. You SharePoint types probably know to find this in either the 12 Hive or the GAC.

>>> import clr
>>> clr.AddReference("Microsoft.SharePoint")

We need to import the SPSite class before we use it too.

>>> from Microsoft.SharePoint import SPSite

Now we can create a connection to a site and open its default SPWeb object.

>>> site = SPSite(baseurl)
>>> web = site.OpenWeb()
>>> web

We can also navigate around the subwebs of a given web by using the Webs property. You can run into permissions issues though. Here we try to access the first subweb in the Webs collection.

>>> web.Webs[0]
Traceback (most recent call last):
File "", line 1, in
SystemError: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENI
ED))

To get around this error, we can just use the SystemAccount token attached to each site. I really don’t know the reason behind the design of this API, but you can use one site reference to give you a connection with elevated privileges on another site.

>>> temp = SPSite(baseurl)
>>> site = SPSite(baseurl, temp.SystemAccount.UserToken)
>>> web = base_site.OpenWeb()

Now you can start exploring around and seeing what exits. When you want to see
what members a given object has, you just use the dir() method. Let’s take a look at
the first object in the Lists collection.

>>> dir(base_web.Lists[0])
['AddWorkflowAssociation', 'AlertTemplate', 'AllRolesForCurrentUser', 'AllowCont
entTypes', 'AllowDeletion', 'AllowEveryoneViewItems', 'AllowMultiResponses', 'Al
lowRssFeeds', 'AnonymousPermMask', 'AnonymousPermMask64', 'Audit', 'Author', 'Ba
seTemplate', 'BaseType', 'BreakRoleInheritance', 'CanReceiveEmail', 'CheckPermis
sions', 'CheckedOutFiles', 'ContentTypes', 'ContentTypesEnabled', 'Created', 'Cu
rrentChangeToken', 'DefaultApprovalWorkflowId', 'DefaultContentApprovalWorkflowI
d', 'DefaultItemOpen', 'DefaultView', 'DefaultViewUrl', 'Delete', 'Description',
'Direction', 'DocumentTemplateUrl', 'DoesUserHavePermissions', 'DraftVersionVis
ibility', 'EffectiveBasePermissions', 'EffectiveFolderPermissions', 'EmailAlias'
, 'EmailInsertsFolder', 'EnableAssignToEmail', 'EnableAttachments', 'EnableDeplo
yWithDependentList', 'EnableDeployingList', 'EnableFolderCreation', 'EnableMinor
Versions', 'EnableModeration', 'EnableSchemaCaching', 'EnableSyndication', 'Enab
leVersioning', 'EnsurePropsFresh', 'EnsureRssSettings', 'Equals', 'EventReceiver
s', 'EventSinkAssembly', 'EventSinkClass', 'EventSinkData', 'ExcludeFromTemplate
', 'Fields', 'FirstUniqueAncestor', 'Folders', 'ForceCheckout', 'Forms', 'GetCha
nges', 'GetContentTypeIdByUrl', 'GetDefaultViewForContentType', 'GetDirectChildC
ontentType', 'GetDistinctFieldValues', 'GetHashCode', 'GetItemById', 'GetItemByU
niqueId', 'GetItems', 'GetItemsInFolder', 'GetPropertiesXmlForUncustomizedViews'
, 'GetType', 'GetUncustomizedViewByBaseViewId', 'GetView', 'HasUniqueRoleAssignm
ents', 'Hidden', 'ID', 'ImageUrl', 'IrmEnabled', 'IrmExpire', 'IrmReject', 'IsCa
talog', 'IsContentTypeAllowed', 'ItemCount', 'Items', 'LastItemDeletedDate', 'La
stItemModifiedDate', 'Lists', 'MajorVersionLimit', 'MajorWithMinorVersionsLimit'
, 'MemberwiseClone', 'MobileDefaultViewUrl', 'MultipleDataList', 'NoCrawl', 'OnQ
uickLaunch', 'Ordered', 'ParentWeb', 'ParentWebUrl', 'Permissions', 'PropertiesX
ml', 'ReadSecurity', 'Recycle', 'ReferenceEquals', 'RemoveWorkflowAssociation',
'RenderAsHtml', 'RequestAccessEnabled', 'ResetRoleInheritance', 'RestrictedTempl
ateList', 'ReusableAcl', 'RoleAssignments', 'RootFolder', 'SaveAsTemplate', 'Sch
emaXml', 'SendToLocationName', 'SendToLocationUrl', 'ServerRelativeDocumentTempl
ateUrl', 'ServerTemplateCanCreateFolders', 'ServerTemplateCanReceiveEmail', 'Set
AttributesForPropertiesXML', 'ShowUser', 'TemplateFeatureId', 'Title', 'ToString
', 'Update', 'UpdateWorkflowAssociation', 'Version', 'Views', 'WorkflowAssociati
ons', 'WriteRssFeed', 'WriteSecurity', '__class__', '__delattr__', '__doc__', '_
_format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclas
shook__', 'm_ListAttributesDict', 'm_Lists', 'm_Permissions', 'm_arrListProps',
'm_iRow']

Now this may or may not be readable to you. Alot of times when there’s a ton of text to deal with in the console, I’ll iterate through the methods and print them each on their own line.

>>> for method in dir(base_web.Lists[0]):
...     print method
...
AddWorkflowAssociation
AlertTemplate
AllRolesForCurrentUser
AllowContentTypes
AllowDeletion
AllowEveryoneViewItems
.... [Omitted to save space]
__class__
__delattr__
__doc__
__format__
__getattribute__
__hash__
__init__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__
m_ListAttributesDict
m_Lists
m_Permissions
m_arrListProps
m_iRow

You can inspect classes too. You don’t need an instance. If you inspect the class you’ll just need to import the type first.

>>> for method in dir(SPListCollection):
...     print method
...
Traceback (most recent call last):
NameError: name 'SPListCollection' is not defined
>>> from Microsoft.SharePoint import SPListCollection
>>> for method in dir(SPListCollection):
...        print method
...
Add
CopyTo
... [Omitted to save space]
__add__
__class__
__delattr__
__doc__
__format__
__getattribute__
__getitem__
__hash__
__init__
__iter__
__len__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__

If you’re coming from a C# or VB background, you’re probably familiar with the “Using” block. Python has something similar to that with the “with” statement. The with statement is used with ContextManagers. Many of the .Net framework classes support the ContextManager methods so you can use the with statement. You can see the __enter__ and __exit__ methods below that help make this possible.

>>> dir(SPSite)
['AddWorkItem', 'AllWebs', ...[Omitted for space], '__class__', '__delattr__', '__doc__', '__enter__
', '__exit__', '__format__', '__getattribute__', '__hash__', '__init__', '__new_
_', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__s
tr__', '__subclasshook__']

Now we can refactor our earlier sample to use the with statement. This is something I use more in scripts that reference MOSS 2007, but I thought you may find some uses.

>>> web = None
>>> with SPSite(baseurl) as temp:
... 	with SPSite(baseurl, temp.SystemAccount.UserToken) as site:
...			web = site.OpenWeb()
...

It’s all pretty basic and the console lends itself well to those who like to explore. I really like this method versus digging around the object browser or Reflector all of the time. I like the instant feedback when exploring and the ability to see data and not just structure. I’ve also successfully used IronPython to create custom event handlers and for some administrative tasks for a 2003 to 2007 upgrade. I highly recommend you try it if you find yourself having to do any development using the SharePoint API.

I’ve been getting to know IronPython a little lately and one of the first things I wanted to know was how to debug IronPython in an IDE.  Since I use Visual Studio 2008 at work, I thought it would be a good start.  I searched Google and found a text representation of the steps involved over at Harry Pierson’s blog.

I found it pretty simple to do and somewhat useful for longer scripts.  It will be nice in the future to have first class support for IronPython in Visual Studio 2010.

You should have IronPython 2.0 Final and Visual Studio 2008 Professional or Better* installed to complete this demo.

Create a python file to debug:

1
2
3
4
5
6
7
8

#debug.py
class Foo:
def __init__(self, name=None):
    self.name = name
 
if __name__ == "__main__":
    f = Foo("LVS")
    print f.name

From Visual Studio, open the IronPython executable (ipy.exe). If it doesn’t do so for you, add the ipy.exe as an existing project.

From the Solution Explorer, right-click on Properties and set the Command Arguments property.  Use “-D” to tell ipy.exe to create debugging output, “-i” to start an interactive session at the end of debugging and add the full path to the script you made before.

Properties Dialog

Click Ok.

Open the script you created earlier in Visual Studio and place breakpoints as desired.

Adding a Breakpoint

Run the script in debug mode (F5).

If you added the “-i” argument in step 3, after the last breakpoint is released it will launch an interactive session for you.  You will be able to inspect your script at this point, much like the immediate window in Visual Studio.

Interactive Console

* I tried this in the Visual Studio 2008 Team Developer Edition and the Visual Studio Express Edition.  The Express Editions do not work as they don’t allow you to open the ipy.exe as a solution item. Sorry “hobbyists”, we’ll see if the Express sku’s of Visual Studio 2010 will target IronPython / IronRuby.

I’m sitting next to the creator of python and listening to the History of Django talk.  It’s very interesting to see how the Django framework morphed into what we know it today.

Jacob Kaplan-Moss and Guido Van Rossum just had an interesting side discussion about how Django should do things, making it easier to run on Google App Engine.  After seeing Guido’s talk, the latest backend changes to allow a non rdbms database and the side conversation, I think there will be many exciting developments in Google App Engine in the next year.