Archive for the ‘free software’ Category

Translating pypy-c with 2.5 support

Thursday, August 28th, 2008

Last week I did some fixes to PyPy 2.5-features branch because the translator was crashing. That’s because some of the code I wrote in the last four months were not RPython.

The changes were mostly to help the annotator (with assertions). Now you can have a pypy-c with full 2.5 support.

There are some bugs so stdlib may not be fully working. I’m working on that.


svn co http://codespeak.net/svn/pypy/branch/2.5-features/ pypy-2.5-features
cd pypy-2.5-features/pypy/translator/goal/
python translate.py targetpypystandalone --allworkingmodules
# wait........
./pypy-c

Please, read PyPy’s documentation on translating and if you find any problem while translating let me know. Note that this docs are for the trunk so some of the options may not be available in 2.5-features branch.

I wrote this post last week and it was in my drafts. Actually I fixed some other bugs (one related to PEP 352) and it seems that my changes broke the translator again. Talking in #pypy Armin said that this is probably a bug in the annotator. I don’t know much about the translation process or about the annotator… I guess it’s time to start looking at it :-)

back to gentoo linux

Thursday, August 28th, 2008

After months postponing I’ve installed Gentoo in my laptop. I really like portage and probably everything I know about configuring linux using /etc and a bunch of commands I know because of gentoo.

The first time I installed Gentoo it I had a copy of the Gentoo Handbook and that was it. No internet connection to help. It was a great experience and I’ve learned a lot about how linux works internally. Before Gentoo I was using Fedora Core 2.

After those days I decided that I should became a programmer and not a sysadmin, so I did it, and I decided to install Ubuntu… now after two years using Ubuntu I’m back to Gentoo.Things are different of course. The installation process is easier (with a graphical interface LiveCD!). But, it’s nice to decide what I want to install and have full (or almost full?) control of my machine again! :D

Well.. let me go back to work, portage just finished to compile… :-P

PyConBrasil 2008

Saturday, August 9th, 2008

This year’s PyConBrasil will be held at Universidade Veiga de Almeida, Rio de Janeiro – RJ. I’m definitely going! Last year’s PyConBrasil was great.

So, if you are in Brazil or planning come near September 18th, 19th and 20th, don’t miss the chance to meet the awesome guys from Python Brasil community :-)

PyPy 2.5-features – Starting to port the standard lib

Wednesday, August 6th, 2008

A little late I know.

This week I finished to fix all the failing tests and to implement PEP 328 (Absolute/Relative import).

Now it’s the final step for supporting Python2.5 on PyPy, to port the standard library.

I’m starting with it today and I really hope I can finish it all (including tests and everything else) until the end of next week.

Well, one more time, if anyone want to help me, test your python programs (specially the ones that use python2.5 specific features, like with statement, conditional expressions and new generator stuff) in PyPy-2.5-features (svn co http://codespeak.net/svn/pypy/2.5-features pypy-2.5-features).

How to test? Simple:


$ cd pypy-2.5-features
$ ./bin/py.py your_python_program.py

Be careful and patient, it may take a little bit to things start happening :-)

Of course you can translate PyPy using the translator if you want. Take a look at PyPy’s website for more information.

PyPy 2.5-features – Yet another status update

Friday, July 18th, 2008

Here comes another GSoC status update :-)

Some finished tasks:

  • throw() method on generators
  • close() method…
  • faking 2.5 behavior for IMPORT_NAME and IMPORT_FROM opcodes
  • changing the default value for magic attribute of PyCode objects
  • changing the magic number that goes in .pyc files compiled by PyPy
  • fixing tests and more tests…

From those changes the only one I would like to comment is the change of both magic numbers.

First, they have different values and meanings in PyPy. The magic number for .pyc files (defined in pypy/modules/__builtin__/importing.py) is the number that identifies the bytecode “version”. It’s used to know if the interpreter should use the .pyc file or should recompile the .py. PyPy’s value is different from CPython’s one. As we are changing some opcodes (as IMPORT_NAME, mentioned before) this number had to be changed. The old value in PyPy was 1024 (or 1024 + 2 or 1024 + 4 or 1024 + 2 + 4, depending on some command line options), the new value is 1034 (or ….). We are just using the same policy CPython uses to change the value, add 10 to the old value.

Now the PyCode magic attribute. This value is the CPython magic number (the one explained above)., the old default value was the value from CPython 2.4 (62021). Some checks against this value are made through the code to decide if the bytecode should be interpreted one way or another. One example is the IMPORT_NAME opcode. In Python 2.4 IMPORT_NAME did not have the level parameter, this parameter is new in 2.5 because of the absolute import feature. So if the bytecode represented by a PyCode object is 2.4, when we visit a import statement we should not try to pop the level value from the stack (because it’s not there), but if it is 2.5 we should! So we check the magic value. The problem is, the default value was 2.4 but we changed the opcodes to behave like 2.5, so those checks were not working. The solution was to change the magic default value to 63231 (2.5c2 value I think), so now our bytecode interpreter is (almost?) compatible with 2.5 bytecode. And our compiler is generating 2.5 compatible bytecode as well.

PyPy 2.5-features – PyParser, Grammar and PEPs

Monday, July 7th, 2008

After two weeks without any writing, here comes another status update of my GSoC (and I promise I’ll do it more frequently).

Today I’m working on PEP 342 – “Coroutines via Enhanced Generators”. I’ve finished to implement the .send() method and all the stuff it should do “behind the scenes” (e.g. push the value into the frame stack).

Last week some other PEPs and small changes were supported as well.

PEPs 308 (conditional expressions) and 343 (with statement) were already supported, I just needed to write tests and remove a SyntaxWarning for conditional expressions. PEP 352 (BaseException and raise “string”) is now supported as well.

Some changes were made to Grammar2.5 because pyparser could not read it the right way. Those changes were already present to Grammar2.4. The main problem is that when the parser finds a rule that has alternatives (like, ‘is’ | ‘is’ ‘not’ ) and the first alternative matches (imagine it trying to match ‘is not’, it will match ‘is’) it doesn’t look for the other alternatives and then fails (because there is no rule that matches a ‘not’ after an ‘is’). The solution in that case is to change the order (which doesn’t make much sense, before looking the parser’s code), so the rule should be ‘is’ ‘not’ | ‘is’.

Next steps: check PyCode magic number and finish PEP 342 :-)

Global Python Sprint Weekend in São Paulo 2

Thursday, June 19th, 2008

This weekend (Jun 21st and 22nd) will happen (again) the Global Python Sprint Weekend and one more time our Python User Group (GruPy-SP) will meet to work together.

Our meeting will be held at Google’s São Paulo Office. Thanks Rodolpho and Google :-)

If you are in São Paulo you can take a look at our wiki (portuguese only) to see how you can join us. If you are not in São Paulo but in Brazil visit the PythonBugDay page to see if anyone in your town is organizing a meet too ;-)

ImportWarning, UnicodeWarning, -W and parser

Wednesday, June 18th, 2008

After almost three weeks (more?) I’m back :-)

Well, this week I’ve fixed two small 2.5 features in PyPy.

The first one was UnicodeWarning. From Python’s docs:

…UnicodeWarning is triggered when you attempt to compare a Unicode string and an 8-bit string that can’t be converted to Unicode using the default ASCII encoding…

In PyPy to trigger a Warning when in Interpreter-level you can use the method space.warn, this method, internally, will use the warnings module to trigger the warning properly.

The second was ImportWarning. Again from Python’s docs:

ImportWarning warning is triggered when an import would have picked up a directory as a package but no __init__.py was found

After searching for a while I’ve found the pypy/lib/__buitins__/importing.py file that holds, obviously, pypy importing stuff. I’ve looked to CPython’s implementation to see when I should trigger the ImportWarn and load_part() seemed to be the right place.

The ImportWarning should be silently ignored unless you provide the -Wd command line option but PyPy does not accept the -W option, so I’ve implemented it. It is simple, you just need to append the argument (‘d’ in this case) to the sys.warnoptions list.

While working on this issues I’ve found a bug in the parser when using the 2.5 Grammar. It is raising a SyntaxError when you call a function passing a keyword argument (like f(x=3)). Because of this (and other small bugs) I’ve decided that I will work on the pyparser to make it compatible with the 2.5 Grammar now.

PyPy 2.5 features – news

Tuesday, May 13th, 2008

Changes! Now I’m free to work “full-time” on my GSoC project, last Friday was my last day on my old job (Ikwa) :-)

Today I’ve read lots of docs and code, specially the pyparser code. I’ll try to maintain a list of features I’m currently working on here.

Last week (ok, Friday before last Friday) we went to Werneck’s cave to code. One thing I’ve made there was implement the defaultdict object in the collections module. This module is written in C originally, so I needed to port it.

The code is relatively simple and I’ve just “copied” the C implementation (of course it’s clearer and more beautiful in Python :P ). Werneck also helped me with the pyparser but we couldn’t find a solution for supporting the full Python 2.5 Grammar .

Today I’ve worked on a small Python 2.5 change, the empty base classes list syntax.

Until Python 2.5 the list of base classes could not be empty. So you couldn’t define a class like:

class A():
    pass

The right way for doing this was

class A:
    pass

Now, in Python 2.5, both ways are legal.

Last week I’ve committed a slight changed Grammar for Python 2.5 syntax support to PyPy’s 2.5-features branch (the only change I’ve made was removing the support for the new import syntax, as it crashes the parser, I need to work on that).

With part of the 2.5 Grammar already supported, to support this new class definition syntax the only change needed was in pyparser (more specifically to the AST builder).

While the build_classdef method expected 4 atoms (< class keyword >, < white space >, < class name > and < comma >) or 7 (the same 4 plus < ( >, < base classes > and < ) > before the < comma >) in Python 2.4, now in Python 2.5 it can receive 6 atoms too (< class keyword >, < white space >, < class name >, < ( >, < ) > and < comma >). The change was really simple, but it was a good exercise for me because I could apply what I’m learning in my compilers classes at school (while reading the sources) :-)

Some code (pypy/interpreter/pyparser/astbuilder.py:634):

def build_classdef(builder, nb):
    ...
    if l == 4: # class NAME:
        basenames = []
        body = atoms[3]
    elif l == 6: # class NAME():  # 2.5
        basenames = []
        body = atoms[5]
    else:
        assert l == 7
        basenames = []
        body = atoms[6]
        base = atoms[3]
    ...

I don’t think I need to explain this code, it’s very simple (of course, it’s Python! :-) ).

I’ve made some tests and it seemed to work great, but I’ve decided to compare the bytecode generated by PyPy and by CPython (both 2.4 and 2.5). While in CPython 2.4 the statement “class A: pass” produces the following bytecode:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 BUILD_TUPLE              0
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

CPython 2.5 produces for both statements (the old syntax and the new one) something slight different:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 LOAD_CONST               3 (())
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

The only difference between 2.4 and 2.5 is the instruction BUILD_TUPLE in 2.4 against the LOAD_CONST in 2.5. I'm not a bytecode expert, but it seems that this is an optimization, of course I may be completely wrong :P

Well, as I expected the bytecode produced by PyPy is identical the one produced by CPython 2.4, so I think I have more things to change to complete this task. ;)

But not tonight, after 14 hours of PyPy code reading and some debugging (plus Linear Programming and Computer Theory classes) I think I should sleep.

:-)

Global Python Sprint Weekend in São Paulo

Thursday, May 8th, 2008

This weekend (May 10th and 11th) will happen the Global Python Sprint Weekend. Like the last PythonBugDay our Python User Group (GruPy-SP) will meet to work together.

This time our meeting will be held at Universidade de São Paulo IME (Instituto de Matemática e Estatística) thanks to RBP and the CCSL (Free Software Competence Centre) =)

Other Python User Groups from Brazil are also meeting to work together, check the wikipage for the PythonBugDay to know if there will be a meeting near you :-) (or you can work from your home sweet home).