Archive for the ‘pypy’ Category

Translating pypy-c with 2.5 support

Thursday, August 28th, 2008

Last week I did some fixes to PyPy 2.5-features branch because the translator was crashing. That’s because some of the code I wrote in the last four months were not RPython.

The changes were mostly to help the annotator (with assertions). Now you can have a pypy-c with full 2.5 support.

There are some bugs so stdlib may not be fully working. I’m working on that.


svn co http://codespeak.net/svn/pypy/branch/2.5-features/ pypy-2.5-features
cd pypy-2.5-features/pypy/translator/goal/
python translate.py targetpypystandalone --allworkingmodules
# wait........
./pypy-c

Please, read PyPy’s documentation on translating and if you find any problem while translating let me know. Note that this docs are for the trunk so some of the options may not be available in 2.5-features branch.

I wrote this post last week and it was in my drafts. Actually I fixed some other bugs (one related to PEP 352) and it seems that my changes broke the translator again. Talking in #pypy Armin said that this is probably a bug in the annotator. I don’t know much about the translation process or about the annotator… I guess it’s time to start looking at it :-)

PyPy 2.5-features – Starting to port the standard lib

Wednesday, August 6th, 2008

A little late I know.

This week I finished to fix all the failing tests and to implement PEP 328 (Absolute/Relative import).

Now it’s the final step for supporting Python2.5 on PyPy, to port the standard library.

I’m starting with it today and I really hope I can finish it all (including tests and everything else) until the end of next week.

Well, one more time, if anyone want to help me, test your python programs (specially the ones that use python2.5 specific features, like with statement, conditional expressions and new generator stuff) in PyPy-2.5-features (svn co http://codespeak.net/svn/pypy/2.5-features pypy-2.5-features).

How to test? Simple:


$ cd pypy-2.5-features
$ ./bin/py.py your_python_program.py

Be careful and patient, it may take a little bit to things start happening :-)

Of course you can translate PyPy using the translator if you want. Take a look at PyPy’s website for more information.

PyPy 2.5-features – Yet another status update

Friday, July 18th, 2008

Here comes another GSoC status update :-)

Some finished tasks:

  • throw() method on generators
  • close() method…
  • faking 2.5 behavior for IMPORT_NAME and IMPORT_FROM opcodes
  • changing the default value for magic attribute of PyCode objects
  • changing the magic number that goes in .pyc files compiled by PyPy
  • fixing tests and more tests…

From those changes the only one I would like to comment is the change of both magic numbers.

First, they have different values and meanings in PyPy. The magic number for .pyc files (defined in pypy/modules/__builtin__/importing.py) is the number that identifies the bytecode “version”. It’s used to know if the interpreter should use the .pyc file or should recompile the .py. PyPy’s value is different from CPython’s one. As we are changing some opcodes (as IMPORT_NAME, mentioned before) this number had to be changed. The old value in PyPy was 1024 (or 1024 + 2 or 1024 + 4 or 1024 + 2 + 4, depending on some command line options), the new value is 1034 (or ….). We are just using the same policy CPython uses to change the value, add 10 to the old value.

Now the PyCode magic attribute. This value is the CPython magic number (the one explained above)., the old default value was the value from CPython 2.4 (62021). Some checks against this value are made through the code to decide if the bytecode should be interpreted one way or another. One example is the IMPORT_NAME opcode. In Python 2.4 IMPORT_NAME did not have the level parameter, this parameter is new in 2.5 because of the absolute import feature. So if the bytecode represented by a PyCode object is 2.4, when we visit a import statement we should not try to pop the level value from the stack (because it’s not there), but if it is 2.5 we should! So we check the magic value. The problem is, the default value was 2.4 but we changed the opcodes to behave like 2.5, so those checks were not working. The solution was to change the magic default value to 63231 (2.5c2 value I think), so now our bytecode interpreter is (almost?) compatible with 2.5 bytecode. And our compiler is generating 2.5 compatible bytecode as well.

PyPy 2.5-features – PyParser, Grammar and PEPs

Monday, July 7th, 2008

After two weeks without any writing, here comes another status update of my GSoC (and I promise I’ll do it more frequently).

Today I’m working on PEP 342 – “Coroutines via Enhanced Generators”. I’ve finished to implement the .send() method and all the stuff it should do “behind the scenes” (e.g. push the value into the frame stack).

Last week some other PEPs and small changes were supported as well.

PEPs 308 (conditional expressions) and 343 (with statement) were already supported, I just needed to write tests and remove a SyntaxWarning for conditional expressions. PEP 352 (BaseException and raise “string”) is now supported as well.

Some changes were made to Grammar2.5 because pyparser could not read it the right way. Those changes were already present to Grammar2.4. The main problem is that when the parser finds a rule that has alternatives (like, ‘is’ | ‘is’ ‘not’ ) and the first alternative matches (imagine it trying to match ‘is not’, it will match ‘is’) it doesn’t look for the other alternatives and then fails (because there is no rule that matches a ‘not’ after an ‘is’). The solution in that case is to change the order (which doesn’t make much sense, before looking the parser’s code), so the rule should be ‘is’ ‘not’ | ‘is’.

Next steps: check PyCode magic number and finish PEP 342 :-)

ImportWarning, UnicodeWarning, -W and parser

Wednesday, June 18th, 2008

After almost three weeks (more?) I’m back :-)

Well, this week I’ve fixed two small 2.5 features in PyPy.

The first one was UnicodeWarning. From Python’s docs:

…UnicodeWarning is triggered when you attempt to compare a Unicode string and an 8-bit string that can’t be converted to Unicode using the default ASCII encoding…

In PyPy to trigger a Warning when in Interpreter-level you can use the method space.warn, this method, internally, will use the warnings module to trigger the warning properly.

The second was ImportWarning. Again from Python’s docs:

ImportWarning warning is triggered when an import would have picked up a directory as a package but no __init__.py was found

After searching for a while I’ve found the pypy/lib/__buitins__/importing.py file that holds, obviously, pypy importing stuff. I’ve looked to CPython’s implementation to see when I should trigger the ImportWarn and load_part() seemed to be the right place.

The ImportWarning should be silently ignored unless you provide the -Wd command line option but PyPy does not accept the -W option, so I’ve implemented it. It is simple, you just need to append the argument (‘d’ in this case) to the sys.warnoptions list.

While working on this issues I’ve found a bug in the parser when using the 2.5 Grammar. It is raising a SyntaxError when you call a function passing a keyword argument (like f(x=3)). Because of this (and other small bugs) I’ve decided that I will work on the pyparser to make it compatible with the 2.5 Grammar now.

PyPy 2.5 features – news

Tuesday, May 13th, 2008

Changes! Now I’m free to work “full-time” on my GSoC project, last Friday was my last day on my old job (Ikwa) :-)

Today I’ve read lots of docs and code, specially the pyparser code. I’ll try to maintain a list of features I’m currently working on here.

Last week (ok, Friday before last Friday) we went to Werneck’s cave to code. One thing I’ve made there was implement the defaultdict object in the collections module. This module is written in C originally, so I needed to port it.

The code is relatively simple and I’ve just “copied” the C implementation (of course it’s clearer and more beautiful in Python :P ). Werneck also helped me with the pyparser but we couldn’t find a solution for supporting the full Python 2.5 Grammar .

Today I’ve worked on a small Python 2.5 change, the empty base classes list syntax.

Until Python 2.5 the list of base classes could not be empty. So you couldn’t define a class like:

class A():
    pass

The right way for doing this was

class A:
    pass

Now, in Python 2.5, both ways are legal.

Last week I’ve committed a slight changed Grammar for Python 2.5 syntax support to PyPy’s 2.5-features branch (the only change I’ve made was removing the support for the new import syntax, as it crashes the parser, I need to work on that).

With part of the 2.5 Grammar already supported, to support this new class definition syntax the only change needed was in pyparser (more specifically to the AST builder).

While the build_classdef method expected 4 atoms (< class keyword >, < white space >, < class name > and < comma >) or 7 (the same 4 plus < ( >, < base classes > and < ) > before the < comma >) in Python 2.4, now in Python 2.5 it can receive 6 atoms too (< class keyword >, < white space >, < class name >, < ( >, < ) > and < comma >). The change was really simple, but it was a good exercise for me because I could apply what I’m learning in my compilers classes at school (while reading the sources) :-)

Some code (pypy/interpreter/pyparser/astbuilder.py:634):

def build_classdef(builder, nb):
    ...
    if l == 4: # class NAME:
        basenames = []
        body = atoms[3]
    elif l == 6: # class NAME():  # 2.5
        basenames = []
        body = atoms[5]
    else:
        assert l == 7
        basenames = []
        body = atoms[6]
        base = atoms[3]
    ...

I don’t think I need to explain this code, it’s very simple (of course, it’s Python! :-) ).

I’ve made some tests and it seemed to work great, but I’ve decided to compare the bytecode generated by PyPy and by CPython (both 2.4 and 2.5). While in CPython 2.4 the statement “class A: pass” produces the following bytecode:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 BUILD_TUPLE              0
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

CPython 2.5 produces for both statements (the old syntax and the new one) something slight different:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 LOAD_CONST               3 (())
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

The only difference between 2.4 and 2.5 is the instruction BUILD_TUPLE in 2.4 against the LOAD_CONST in 2.5. I'm not a bytecode expert, but it seems that this is an optimization, of course I may be completely wrong :P

Well, as I expected the bytecode produced by PyPy is identical the one produced by CPython 2.4, so I think I have more things to change to complete this task. ;)

But not tonight, after 14 hours of PyPy code reading and some debugging (plus Linear Programming and Computer Theory classes) I think I should sleep.

:-)

PyPy 2.5 features: __hash__ behavior

Thursday, April 24th, 2008

One small change in Python 2.5 is the __hash__ method behavior. In earlier versions the __hash__ method was supposed to return a normal integer always, but now due to changes in the id() builtin function the __hash__ method behavior was changed and it can return a long or an int.

This was easy to change in PyPy, the hash() builtin function implementation is part of the ObjectSpace as almost all the builtin functions and types.

It’s in pypy/objspace/descroperation.py:

def hash(space, w_obj):
        w_hash = space.lookup(w_obj, '__hash__')
        if w_hash is None:
            if space.lookup(w_obj, '__eq__') is not None or \
               space.lookup(w_obj, '__cmp__') is not None:
                raise OperationError(space.w_TypeError,
                                     space.wrap("unhashable type"))
            return default_identity_hash(space, w_obj)
        w_result = space.get_and_call_function(w_hash, w_obj)
        if space.is_true(space.isinstance(w_result, space.w_int)):
            return w_result
        else:
            raise OperationError(space.w_TypeError,
                     space.wrap("__hash__() should return an int or long"))

One don’t even need to know PyPy well to understand that the lines

if space.is_true(space.isinstance(w_result, space.w_int)):
    return w_result
else:
    raise OperationError(space.w_TypeError,
             space.wrap("__hash__() should return an int or long"))

are responsible for raising a TypeError when the returning value from __hash__ is not an integer.

So just changing the condition

space.is_true(space.isinstance(w_result, space.w_int)):

to

(space.is_true(space.isinstance(w_result, space.w_int)) or
 space.is_true(space.isinstance(w_result, space.w_long))):

will solve this, I hope!

But that’s not all the code needed, I promised Carl to write tests for everything, but as I like to use TDD (test-driven development) it was already done.

Oh, and should I mention that I love PyPy’s documentation? Thanks guys!

Supporting Python 2.5 features in PyPy – GSoC 2008

Tuesday, April 22nd, 2008

As you probably know my GSoC proposal to work on PyPy interpreter, supporting the Python 2.5 features, has been accepted. This post is a short explanation of my Google Summer of Code project.

PyPy

The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn’t it?

PyPy is divided into the python interpreter and the translator framework. The interpreter is written in RPython (a subset of Python) and the main goal is to provide an easy-to-hack (for python programmers ;) ) Python interpreter.

PyPy’s interpreter currently supports only a few features introduced in Python 2.5, like __index__ (for slicing) and some part of the ‘with’ statement. Also, the 2.5 changes to the language and to the standard library have not been ported yet (although some have been, like the ctypes).

Goal

This proposal idea is to bring Python 2.5 features and changes to PyPy interpreter, this includes porting the modules and writing code for the features it self.

Here are some of the features I’ll work on:

  • PEP 308: Conditional Expressions[*]
  • 328: Absolute and Relative Imports
  • PEP 342: New Generator Features
  • PEP 343: The with statement[*]
  • PEP 352: Exceptions as New-Style Classes

[*] part of this features are already supported by PyPy

Benefits

With support to 2.5 features and changes more the PyPy interpreter will be up-to-date so people who wants to use this features can be happy using PyPy. This probably means more people using PyPy (i hope!) :-)

Why this project?

The first reason I’ve chosen the PyPy project is that I really like their ideas and goals. I was looking for collaborating to it in some way for a long time, but it took me some time to read the docs and to really understand the project.

As I like PyPy ideas and I’m starting to plan my graduation project/bachelor’s thesis, my work on PyPy interpreter will be a big part of it. Maybe I can explain my ideas in another post :-)

I would like to thank Google (and the SoC folks), Python Software Foundation, PyPy and specially to Carl Friedrich Bolz* who will be my GSoC mentor and Leonardo Santagada who helped me to understand PyPy and to write my application.

* morepypy is a blog where PyPy developers talk to the community, it’s not Carl’s personal blog.

Thanks to my friends from Python Brasil community too.

Approved in GSoC

Monday, April 21st, 2008

As mentioned before I’ve applied to GSoC to work on PyPy for the Python Software Foundation.

Today I’m very happy to say that my project got approved and I will work on PyPy interpreter the next three months. It means that a lot of PyPy posts are coming, and the first one will be a introduction to the project, what is PyPy, the interpreter and the translation framework :)

Stay tuned if you are interested in PyPy! Congratulations to everybody that have applied to GSoC this year =)