The changes were mostly to help the annotator (with assertions). Now you can have a pypy-c with full 2.5 support.

There are some bugs so stdlib may not be fully working. I’m working on that.


svn co http://codespeak.net/svn/pypy/branch/2.5-features/ pypy-2.5-features
cd pypy-2.5-features/pypy/translator/goal/
python translate.py targetpypystandalone --allworkingmodules
# wait........
./pypy-c


Please, read PyPy’s documentation on translating and if you find any problem while translating let me know. Note that this docs are for the trunk so some of the options may not be available in 2.5-features branch.

I wrote this post last week and it was in my drafts. Actually I fixed some other bugs (one related to PEP 352) and it seems that my changes broke the translator again. Talking in #pypy Armin said that this is probably a bug in the annotator. I don’t know much about the translation process or about the annotator… I guess it’s time to start looking at it

This week I finished to fix all the failing tests and to implement PEP 328 (Absolute/Relative import).

Now it’s the final step for supporting Python2.5 on PyPy, to port the standard library.

I’m starting with it today and I really hope I can finish it all (including tests and everything else) until the end of next week.

Well, one more time, if anyone want to help me, test your python programs (specially the ones that use python2.5 specific features, like with statement, conditional expressions and new generator stuff) in PyPy-2.5-features (svn co http://codespeak.net/svn/pypy/2.5-features pypy-2.5-features).

How to test? Simple:


$ cd pypy-2.5-features
$ ./bin/py.py your_python_program.py


Be careful and patient, it may take a little bit to things start happening

Of course you can translate PyPy using the translator if you want. Take a look at PyPy’s website for more information.

Some finished tasks:

• throw() method on generators
• close() method…
• faking 2.5 behavior for IMPORT_NAME and IMPORT_FROM opcodes
• changing the default value for magic attribute of PyCode objects
• changing the magic number that goes in .pyc files compiled by PyPy
• fixing tests and more tests…

From those changes the only one I would like to comment is the change of both magic numbers.

First, they have different values and meanings in PyPy. The magic number for .pyc files (defined in pypy/modules/__builtin__/importing.py) is the number that identifies the bytecode “version”. It’s used to know if the interpreter should use the .pyc file or should recompile the .py. PyPy’s value is different from CPython’s one. As we are changing some opcodes (as IMPORT_NAME, mentioned before) this number had to be changed. The old value in PyPy was 1024 (or 1024 + 2 or 1024 + 4 or 1024 + 2 + 4, depending on some command line options), the new value is 1034 (or ….). We are just using the same policy CPython uses to change the value, add 10 to the old value.

Now the PyCode magic attribute. This value is the CPython magic number (the one explained above)., the old default value was the value from CPython 2.4 (62021). Some checks against this value are made through the code to decide if the bytecode should be interpreted one way or another. One example is the IMPORT_NAME opcode. In Python 2.4 IMPORT_NAME did not have the level parameter, this parameter is new in 2.5 because of the absolute import feature. So if the bytecode represented by a PyCode object is 2.4, when we visit a import statement we should not try to pop the level value from the stack (because it’s not there), but if it is 2.5 we should! So we check the magic value. The problem is, the default value was 2.4 but we changed the opcodes to behave like 2.5, so those checks were not working. The solution was to change the magic default value to 63231 (2.5c2 value I think), so now our bytecode interpreter is (almost?) compatible with 2.5 bytecode. And our compiler is generating 2.5 compatible bytecode as well.

Today I’m working on PEP 342 – “Coroutines via Enhanced Generators”. I’ve finished to implement the .send() method and all the stuff it should do “behind the scenes” (e.g. push the value into the frame stack).

Last week some other PEPs and small changes were supported as well.

PEPs 308 (conditional expressions) and 343 (with statement) were already supported, I just needed to write tests and remove a SyntaxWarning for conditional expressions. PEP 352 (BaseException and raise “string”) is now supported as well.

Some changes were made to Grammar2.5 because pyparser could not read it the right way. Those changes were already present to Grammar2.4. The main problem is that when the parser finds a rule that has alternatives (like, ‘is’ | ‘is’ ‘not’ ) and the first alternative matches (imagine it trying to match ‘is not’, it will match ‘is’) it doesn’t look for the other alternatives and then fails (because there is no rule that matches a ‘not’ after an ‘is’). The solution in that case is to change the order (which doesn’t make much sense, before looking the parser’s code), so the rule should be ‘is’ ‘not’ | ‘is’.

Next steps: check PyCode magic number and finish PEP 342

Well, this week I’ve fixed two small 2.5 features in PyPy.

The first one was UnicodeWarning. From Python’s docs:

…UnicodeWarning is triggered when you attempt to compare a Unicode string and an 8-bit string that can’t be converted to Unicode using the default ASCII encoding…

In PyPy to trigger a Warning when in Interpreter-level you can use the method space.warn, this method, internally, will use the warnings module to trigger the warning properly.

The second was ImportWarning. Again from Python’s docs:

ImportWarning warning is triggered when an import would have picked up a directory as a package but no __init__.py was found

After searching for a while I’ve found the pypy/lib/__buitins__/importing.py file that holds, obviously, pypy importing stuff. I’ve looked to CPython’s implementation to see when I should trigger the ImportWarn and load_part() seemed to be the right place.

The ImportWarning should be silently ignored unless you provide the -Wd command line option but PyPy does not accept the -W option, so I’ve implemented it. It is simple, you just need to append the argument (‘d’ in this case) to the sys.warnoptions list.

While working on this issues I’ve found a bug in the parser when using the 2.5 Grammar. It is raising a SyntaxError when you call a function passing a keyword argument (like f(x=3)). Because of this (and other small bugs) I’ve decided that I will work on the pyparser to make it compatible with the 2.5 Grammar now.

Today I’ve read lots of docs and code, specially the pyparser code. I’ll try to maintain a list of features I’m currently working on here.

Last week (ok, Friday before last Friday) we went to Werneck’s cave to code. One thing I’ve made there was implement the defaultdict object in the collections module. This module is written in C originally, so I needed to port it.

The code is relatively simple and I’ve just “copied” the C implementation (of course it’s clearer and more beautiful in Python ). Werneck also helped me with the pyparser but we couldn’t find a solution for supporting the full Python 2.5 Grammar .

Today I’ve worked on a small Python 2.5 change, the empty base classes list syntax.

Until Python 2.5 the list of base classes could not be empty. So you couldn’t define a class like:

class A():
    pass

The right way for doing this was

class A:
    pass

Now, in Python 2.5, both ways are legal.

Last week I’ve committed a slight changed Grammar for Python 2.5 syntax support to PyPy’s 2.5-features branch (the only change I’ve made was removing the support for the new import syntax, as it crashes the parser, I need to work on that).

With part of the 2.5 Grammar already supported, to support this new class definition syntax the only change needed was in pyparser (more specifically to the AST builder).

While the build_classdef method expected 4 atoms (< class keyword >, < white space >, < class name > and < comma >) or 7 (the same 4 plus < ( >, < base classes > and < ) > before the < comma >) in Python 2.4, now in Python 2.5 it can receive 6 atoms too (< class keyword >, < white space >, < class name >, < ( >, < ) > and < comma >). The change was really simple, but it was a good exercise for me because I could apply what I’m learning in my compilers classes at school (while reading the sources)

Some code (pypy/interpreter/pyparser/astbuilder.py:634):

def build_classdef(builder, nb):
    ...
    if l == 4: # class NAME:
        basenames = []
        body = atoms[3]
    elif l == 6: # class NAME():  # 2.5
        basenames = []
        body = atoms[5]
    else:
        assert l == 7
        basenames = []
        body = atoms[6]
        base = atoms[3]
    ...

I don’t think I need to explain this code, it’s very simple (of course, it’s Python! ).

I’ve made some tests and it seemed to work great, but I’ve decided to compare the bytecode generated by PyPy and by CPython (both 2.4 and 2.5). While in CPython 2.4 the statement “class A: pass” produces the following bytecode:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 BUILD_TUPLE              0
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 LOAD_CONST               3 (())
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE