Hey ./manage.py runserver, Y U No Work?

Recently I was working on a Django-based project that integrates with Android and Embedded Linux devices. One day a coworker called me and asked about an error he had encountered. It seemed like he was not able to execute ./manage.py runserver to any port he had previously used. He was careful to use Control + C to stop the server, but for some reason the port was held onto by something. Looking at the output of “ps aux” was not showing any running processes using ./manage.py.

Who’s using that port?

Next instead of looking at processes, I wanted to see what was holding onto that port.  I ran the following command:

netstat -lpn | grep 8000

The output gave me the answer:

tcp    0    0 127.0.0.1:8000        0.0.0.0:*      LISTEN    15902/adb

So ADB was the culprit. A simple kill -9 to the process ID and the issue was fixed (temporarily). The issue keeps occuring, so maybe something in my code may be keeping ADB holding onto that port. I’m using the subprocess module to interact with ADB. I tried adding a call to terminate from the result of my subprocess.Popen call, but that ended very badly and had no effect on the hanging port. I’m sure when I have time to dig deeper (read “After My Looming Deadline”), I’ll try to write some tests to explore this issue more thoroughly.

I’ve been working on a Django-based project since October 2010 that includes some integration with Scrapy. Scrapy is a web scraping framework for Python and due to its design inspiration, Django developers will be quickly familiar with it. I have installed Scrapy no less than 10 times at this point and decided it was time for a script. This post is more of a reminder to me for the next time I install Scrapy, but if it works for you as well – great!.

This script works for Ubuntu 9.10, 10.04 and 10.10.  If you’re on an older release, you’ll need to install some dependencies first and then follow these instructions.

sudo apt-get install curl
curl -s http://archive.scrapy.org/ubuntu/archive.key | sudo apt-key add -
DISTRO=$(lsb_release -cs)
echo deb http://archive.scrapy.org/ubuntu $DISTRO main | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install scrapy-0.12

Have you ever edited your models.py file and shortly after when trying to see the change list for a model in the admin site you received this error?

Caught an exception while rendering: coercing to Unicode: need string or buffer, NoneType found

I did today and it was one of those “bonehead” moments. You know, the one where you’ve been coding for a bit and for some reason make a simple error that leaves you thinking “I’ve done that exact same action a hundred times, why would I get an error this time?”, when you realize you made a really simple mistake. The answer was quick and easy. It was not occuring on the new model form, only on the change list so I opened my models file and looked over the model in question and then realized my typo. I had forgot to add “return” to the __unicode__ method on my model. So if you get this error, make sure that you’re returning something and that its really unicode.

I’ve been working on a project that interacts with Microsoft Office SharePoint Server 2007 (MOSS 2007) for the last few months and one of the tools I’ve been using daily to support my work is IronPython. IronPython has made it easy for me to come up to speed quickly on the MOSS 2007 API but what has made my life even easier is the IronPython Console.

*Note: If you have a browser that supports Silverlight, you can try Python in your browser now.

IronPython ships with a console virtually identical to the stand Python console. The IronPython console provides me with a constant live connection directly to the API. It’s very different and somewhat empowering to transition from the standard workflow in Visual Studio of write code/build/run to something more like a conversation with the code. I think the term conversation is a very good analogy because it’s not just looking at classes and methods, it’s also data and interaction. It’s experiencing everything all at once!

Let’s take a look at what I’m talking about. To run these samples, I had IronPython 2.6.1 installed on a server running Windows Server 2003 and MOSS 2007.

When you first start up the console, you need to reference the Microsoft.SharePoint .dll. You SharePoint types probably know to find this in either the 12 Hive or the GAC.

>>> import clr
>>> clr.AddReference("Microsoft.SharePoint")

We need to import the SPSite class before we use it too.

>>> from Microsoft.SharePoint import SPSite

Now we can create a connection to a site and open its default SPWeb object.

>>> site = SPSite(baseurl)
>>> web = site.OpenWeb()
>>> web

We can also navigate around the subwebs of a given web by using the Webs property. You can run into permissions issues though. Here we try to access the first subweb in the Webs collection.

>>> web.Webs[0]
Traceback (most recent call last):
File "", line 1, in
SystemError: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENI
ED))

To get around this error, we can just use the SystemAccount token attached to each site. I really don’t know the reason behind the design of this API, but you can use one site reference to give you a connection with elevated privileges on another site.

>>> temp = SPSite(baseurl)
>>> site = SPSite(baseurl, temp.SystemAccount.UserToken)
>>> web = base_site.OpenWeb()

Now you can start exploring around and seeing what exits. When you want to see
what members a given object has, you just use the dir() method. Let’s take a look at
the first object in the Lists collection.

>>> dir(base_web.Lists[0])
['AddWorkflowAssociation', 'AlertTemplate', 'AllRolesForCurrentUser', 'AllowCont
entTypes', 'AllowDeletion', 'AllowEveryoneViewItems', 'AllowMultiResponses', 'Al
lowRssFeeds', 'AnonymousPermMask', 'AnonymousPermMask64', 'Audit', 'Author', 'Ba
seTemplate', 'BaseType', 'BreakRoleInheritance', 'CanReceiveEmail', 'CheckPermis
sions', 'CheckedOutFiles', 'ContentTypes', 'ContentTypesEnabled', 'Created', 'Cu
rrentChangeToken', 'DefaultApprovalWorkflowId', 'DefaultContentApprovalWorkflowI
d', 'DefaultItemOpen', 'DefaultView', 'DefaultViewUrl', 'Delete', 'Description',
'Direction', 'DocumentTemplateUrl', 'DoesUserHavePermissions', 'DraftVersionVis
ibility', 'EffectiveBasePermissions', 'EffectiveFolderPermissions', 'EmailAlias'
, 'EmailInsertsFolder', 'EnableAssignToEmail', 'EnableAttachments', 'EnableDeplo
yWithDependentList', 'EnableDeployingList', 'EnableFolderCreation', 'EnableMinor
Versions', 'EnableModeration', 'EnableSchemaCaching', 'EnableSyndication', 'Enab
leVersioning', 'EnsurePropsFresh', 'EnsureRssSettings', 'Equals', 'EventReceiver
s', 'EventSinkAssembly', 'EventSinkClass', 'EventSinkData', 'ExcludeFromTemplate
', 'Fields', 'FirstUniqueAncestor', 'Folders', 'ForceCheckout', 'Forms', 'GetCha
nges', 'GetContentTypeIdByUrl', 'GetDefaultViewForContentType', 'GetDirectChildC
ontentType', 'GetDistinctFieldValues', 'GetHashCode', 'GetItemById', 'GetItemByU
niqueId', 'GetItems', 'GetItemsInFolder', 'GetPropertiesXmlForUncustomizedViews'
, 'GetType', 'GetUncustomizedViewByBaseViewId', 'GetView', 'HasUniqueRoleAssignm
ents', 'Hidden', 'ID', 'ImageUrl', 'IrmEnabled', 'IrmExpire', 'IrmReject', 'IsCa
talog', 'IsContentTypeAllowed', 'ItemCount', 'Items', 'LastItemDeletedDate', 'La
stItemModifiedDate', 'Lists', 'MajorVersionLimit', 'MajorWithMinorVersionsLimit'
, 'MemberwiseClone', 'MobileDefaultViewUrl', 'MultipleDataList', 'NoCrawl', 'OnQ
uickLaunch', 'Ordered', 'ParentWeb', 'ParentWebUrl', 'Permissions', 'PropertiesX
ml', 'ReadSecurity', 'Recycle', 'ReferenceEquals', 'RemoveWorkflowAssociation',
'RenderAsHtml', 'RequestAccessEnabled', 'ResetRoleInheritance', 'RestrictedTempl
ateList', 'ReusableAcl', 'RoleAssignments', 'RootFolder', 'SaveAsTemplate', 'Sch
emaXml', 'SendToLocationName', 'SendToLocationUrl', 'ServerRelativeDocumentTempl
ateUrl', 'ServerTemplateCanCreateFolders', 'ServerTemplateCanReceiveEmail', 'Set
AttributesForPropertiesXML', 'ShowUser', 'TemplateFeatureId', 'Title', 'ToString
', 'Update', 'UpdateWorkflowAssociation', 'Version', 'Views', 'WorkflowAssociati
ons', 'WriteRssFeed', 'WriteSecurity', '__class__', '__delattr__', '__doc__', '_
_format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclas
shook__', 'm_ListAttributesDict', 'm_Lists', 'm_Permissions', 'm_arrListProps',
'm_iRow']

Now this may or may not be readable to you. Alot of times when there’s a ton of text to deal with in the console, I’ll iterate through the methods and print them each on their own line.

>>> for method in dir(base_web.Lists[0]):
...     print method
...
AddWorkflowAssociation
AlertTemplate
AllRolesForCurrentUser
AllowContentTypes
AllowDeletion
AllowEveryoneViewItems
.... [Omitted to save space]
__class__
__delattr__
__doc__
__format__
__getattribute__
__hash__
__init__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__
m_ListAttributesDict
m_Lists
m_Permissions
m_arrListProps
m_iRow

You can inspect classes too. You don’t need an instance. If you inspect the class you’ll just need to import the type first.

>>> for method in dir(SPListCollection):
...     print method
...
Traceback (most recent call last):
NameError: name 'SPListCollection' is not defined
>>> from Microsoft.SharePoint import SPListCollection
>>> for method in dir(SPListCollection):
...        print method
...
Add
CopyTo
... [Omitted to save space]
__add__
__class__
__delattr__
__doc__
__format__
__getattribute__
__getitem__
__hash__
__init__
__iter__
__len__
__new__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__str__
__subclasshook__

If you’re coming from a C# or VB background, you’re probably familiar with the “Using” block. Python has something similar to that with the “with” statement. The with statement is used with ContextManagers. Many of the .Net framework classes support the ContextManager methods so you can use the with statement. You can see the __enter__ and __exit__ methods below that help make this possible.

>>> dir(SPSite)
['AddWorkItem', 'AllWebs', ...[Omitted for space], '__class__', '__delattr__', '__doc__', '__enter__
', '__exit__', '__format__', '__getattribute__', '__hash__', '__init__', '__new_
_', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__s
tr__', '__subclasshook__']

Now we can refactor our earlier sample to use the with statement. This is something I use more in scripts that reference MOSS 2007, but I thought you may find some uses.

>>> web = None
>>> with SPSite(baseurl) as temp:
... 	with SPSite(baseurl, temp.SystemAccount.UserToken) as site:
...			web = site.OpenWeb()
...

It’s all pretty basic and the console lends itself well to those who like to explore. I really like this method versus digging around the object browser or Reflector all of the time. I like the instant feedback when exploring and the ability to see data and not just structure. I’ve also successfully used IronPython to create custom event handlers and for some administrative tasks for a 2003 to 2007 upgrade. I highly recommend you try it if you find yourself having to do any development using the SharePoint API.

I’ve been getting to know IronPython a little lately and one of the first things I wanted to know was how to debug IronPython in an IDE.  Since I use Visual Studio 2008 at work, I thought it would be a good start.  I searched Google and found a text representation of the steps involved over at Harry Pierson’s blog.

I found it pretty simple to do and somewhat useful for longer scripts.  It will be nice in the future to have first class support for IronPython in Visual Studio 2010.

You should have IronPython 2.0 Final and Visual Studio 2008 Professional or Better* installed to complete this demo.

Create a python file to debug:

1
2
3
4
5
6
7
8

#debug.py
class Foo:
def __init__(self, name=None):
    self.name = name
 
if __name__ == "__main__":
    f = Foo("LVS")
    print f.name

From Visual Studio, open the IronPython executable (ipy.exe). If it doesn’t do so for you, add the ipy.exe as an existing project.

From the Solution Explorer, right-click on Properties and set the Command Arguments property.  Use “-D” to tell ipy.exe to create debugging output, “-i” to start an interactive session at the end of debugging and add the full path to the script you made before.

Properties Dialog

Click Ok.

Open the script you created earlier in Visual Studio and place breakpoints as desired.

Adding a Breakpoint

Run the script in debug mode (F5).

If you added the “-i” argument in step 3, after the last breakpoint is released it will launch an interactive session for you.  You will be able to inspect your script at this point, much like the immediate window in Visual Studio.

Interactive Console

* I tried this in the Visual Studio 2008 Team Developer Edition and the Visual Studio Express Edition.  The Express Editions do not work as they don’t allow you to open the ipy.exe as a solution item. Sorry “hobbyists”, we’ll see if the Express sku’s of Visual Studio 2010 will target IronPython / IronRuby.

This was a great talk where Malcolm Tredinnick spelled out what he’s looking for in Django patches and what to avoid if you don’t want your patch tossed out.  Although his talk was targeting the Django project, he really dispensed good advice for anyone maintaining software regardless of project, language or platform.  The room was completely packed!  All of the chairs and couches we’re filled and people were sitting on the floor and standing at the sides of the room.

You’re doing it wrong if:

• the word “print” is in your patch
• you don’t have a test (fail before, pass afterwards)
• you think “PEP 8″ is an energy drink (PEP 8 is the style guide for python)
∘ make your patch look like django style code

Create patches by running diff from the top of the repository to get all changes.  The svn add command works locally even if you don’t have rights to commit to the repository so files you add will be added to the patch.

“Code style is opinionated and yours doesn’t matter!”

What he’s saying here makes sense in any open source project.  There’s a style defined and you should not deviate from it no matter how much you like your own special coding habits.  You might get away with minute changes, but don’t push it.

“Read the contributing document.”

This seems to be simple and common sense.  Skipping over information someone has left for you is a recipe for confusion.

Some tips given on comments:
• comments should last
• comments should be correct
• comments should explain they “why”, we can already read the how in the code.

Some general tips:
• Fix problems and not symptoms!
• Research is not a four letter word.
• The crowd is smarter than you

And what I believe was his last advice for the talk (at least I stopped taking notes after this) was to contribute and keep contributing.  Even when your patches don’t get in, don’t get discouraged.

It was a good talk and the first time I had heard Malcolm speak.  He was very informative and encouraging.  Overall I liked it and it was one of my favorite talks besides James Bennett and Cal Henderson.

I’m here at the GooglePlex sitting a row away from the stage and enjoying the atmosphere.  It’s a great vibe here.  This does feel different than the Microsoft conferences I have been to.  Some observations so far is that the crowd seems a little younger for the most part and there is a comfortable atmosphere.  Of course there’s free food and drinks, free wifi, a great stage and screen for the presentations and the audio is flawless.  The only complaint I have thus far is the uncomfortable plastic daycare chairs.

Earlier I attended Guido’s App Engine keynote (blog post to follow later), and I’m now attending the High Performance Django talk.  I will try to make a post for each talk I attend.

Check back soon…

Because I can. Because I should. Because cool developers do it. I’m really doing it because I agree with an article from Geoffrey Grossenbach of the Nuby On Rails and PeepCode (actually, the guy does tons of stuff – go check his site out) fame. In it he says,

“I’ll add to this and say that every beginning Rails developer should write their own blog software. It’s a great learning experience and you can try things that aren’t possible with just an app running on localhost. It’s also a great environment for learning without the pressure of a mission-critical app. When you’re working for a client and deploying an important application, you’ll have made all the beginner mistakes on your own time (hopefully).”

It’s somewhat of a personal pride thing, as I want readers that come to my site to know that I took the time to build my own blog engine. It was a great introduction to Django and provided me a means to applying the knowledge I have gained from the tutorial and other reading.

So far I’ve done three 45 minute sessions during lunch and I have the Articles, Categories, Links, and Announcements working.  I’ve been keeping track of the time and once I’m done I’ll provide the stats on time and effort along with the source.