Archive for the ‘computer science’ Category

Translating pypy-c with 2.5 support

Thursday, August 28th, 2008

Last week I did some fixes to PyPy 2.5-features branch because the translator was crashing. That’s because some of the code I wrote in the last four months were not RPython.

The changes were mostly to help the annotator (with assertions). Now you can have a pypy-c with full 2.5 support.

There are some bugs so stdlib may not be fully working. I’m working on that.


svn co http://codespeak.net/svn/pypy/branch/2.5-features/ pypy-2.5-features
cd pypy-2.5-features/pypy/translator/goal/
python translate.py targetpypystandalone --allworkingmodules
# wait........
./pypy-c

Please, read PyPy’s documentation on translating and if you find any problem while translating let me know. Note that this docs are for the trunk so some of the options may not be available in 2.5-features branch.

I wrote this post last week and it was in my drafts. Actually I fixed some other bugs (one related to PEP 352) and it seems that my changes broke the translator again. Talking in #pypy Armin said that this is probably a bug in the annotator. I don’t know much about the translation process or about the annotator… I guess it’s time to start looking at it :-)

PyConBrasil 2008

Saturday, August 9th, 2008

This year’s PyConBrasil will be held at Universidade Veiga de Almeida, Rio de Janeiro – RJ. I’m definitely going! Last year’s PyConBrasil was great.

So, if you are in Brazil or planning come near September 18th, 19th and 20th, don’t miss the chance to meet the awesome guys from Python Brasil community :-)

PyPy 2.5-features – Starting to port the standard lib

Wednesday, August 6th, 2008

A little late I know.

This week I finished to fix all the failing tests and to implement PEP 328 (Absolute/Relative import).

Now it’s the final step for supporting Python2.5 on PyPy, to port the standard library.

I’m starting with it today and I really hope I can finish it all (including tests and everything else) until the end of next week.

Well, one more time, if anyone want to help me, test your python programs (specially the ones that use python2.5 specific features, like with statement, conditional expressions and new generator stuff) in PyPy-2.5-features (svn co http://codespeak.net/svn/pypy/2.5-features pypy-2.5-features).

How to test? Simple:


$ cd pypy-2.5-features
$ ./bin/py.py your_python_program.py

Be careful and patient, it may take a little bit to things start happening :-)

Of course you can translate PyPy using the translator if you want. Take a look at PyPy’s website for more information.

Google Code Jam – Round 1

Wednesday, July 30th, 2008

Just to update my blog, let’s get ready to GCJ Round 2 :-)

Saturday 4pm UTC (it’s 1pm here in Brazil, right?!)!

Coming Soon: GSoC Status Update (I promise!!!) :P

PyPy 2.5-features – Yet another status update

Friday, July 18th, 2008

Here comes another GSoC status update :-)

Some finished tasks:

  • throw() method on generators
  • close() method…
  • faking 2.5 behavior for IMPORT_NAME and IMPORT_FROM opcodes
  • changing the default value for magic attribute of PyCode objects
  • changing the magic number that goes in .pyc files compiled by PyPy
  • fixing tests and more tests…

From those changes the only one I would like to comment is the change of both magic numbers.

First, they have different values and meanings in PyPy. The magic number for .pyc files (defined in pypy/modules/__builtin__/importing.py) is the number that identifies the bytecode “version”. It’s used to know if the interpreter should use the .pyc file or should recompile the .py. PyPy’s value is different from CPython’s one. As we are changing some opcodes (as IMPORT_NAME, mentioned before) this number had to be changed. The old value in PyPy was 1024 (or 1024 + 2 or 1024 + 4 or 1024 + 2 + 4, depending on some command line options), the new value is 1034 (or ….). We are just using the same policy CPython uses to change the value, add 10 to the old value.

Now the PyCode magic attribute. This value is the CPython magic number (the one explained above)., the old default value was the value from CPython 2.4 (62021). Some checks against this value are made through the code to decide if the bytecode should be interpreted one way or another. One example is the IMPORT_NAME opcode. In Python 2.4 IMPORT_NAME did not have the level parameter, this parameter is new in 2.5 because of the absolute import feature. So if the bytecode represented by a PyCode object is 2.4, when we visit a import statement we should not try to pop the level value from the stack (because it’s not there), but if it is 2.5 we should! So we check the magic value. The problem is, the default value was 2.4 but we changed the opcodes to behave like 2.5, so those checks were not working. The solution was to change the magic default value to 63231 (2.5c2 value I think), so now our bytecode interpreter is (almost?) compatible with 2.5 bytecode. And our compiler is generating 2.5 compatible bytecode as well.

Google Code Jam: Qualification Round

Thursday, July 17th, 2008

Most of you probably know that yesterday started the Qualification Round for the Google Code Jam competition. In 24 hours there were 3 problems to be solved using you preferred method (or language).

To go to the Round 1 the coder just needed to achieve 25 points. For each problem there were two kinds of input, a Small and a Large one, the correct answer for the small input counts 5 points and the correct answer for the large input counts 20 points, so the only way to go to Round 1 was solving at least one problem for both inputs. I did it for problems A and B. Of course I’ve solved them using Python :-P

Problems.

The first problem was to minimize the number of switches a central engine should make between multiple search engines. You receive a list of search engine names and a list of query, each query is the name of a search engine. The search engine shouldn’t search for itself or the universe will “implode” (or something like that :P ). Well, the basic way to solve this is to always switch to the last next engine in the queries list.

The second problem was to optimize the number of trains that should be in station A and in station B at the start of the day. This given the time that each train leave A and reaches B (and vice-versa). My solution here was to create a list of coming trains (actually of time that the train would be prepared to leave) for each station and then popping out them when a train that should the station leaves after one came (so you reuse the train).

Well, that’s it. Now let’s get ready for Round 1 :-)

ImportWarning, UnicodeWarning, -W and parser

Wednesday, June 18th, 2008

After almost three weeks (more?) I’m back :-)

Well, this week I’ve fixed two small 2.5 features in PyPy.

The first one was UnicodeWarning. From Python’s docs:

…UnicodeWarning is triggered when you attempt to compare a Unicode string and an 8-bit string that can’t be converted to Unicode using the default ASCII encoding…

In PyPy to trigger a Warning when in Interpreter-level you can use the method space.warn, this method, internally, will use the warnings module to trigger the warning properly.

The second was ImportWarning. Again from Python’s docs:

ImportWarning warning is triggered when an import would have picked up a directory as a package but no __init__.py was found

After searching for a while I’ve found the pypy/lib/__buitins__/importing.py file that holds, obviously, pypy importing stuff. I’ve looked to CPython’s implementation to see when I should trigger the ImportWarn and load_part() seemed to be the right place.

The ImportWarning should be silently ignored unless you provide the -Wd command line option but PyPy does not accept the -W option, so I’ve implemented it. It is simple, you just need to append the argument (’d’ in this case) to the sys.warnoptions list.

While working on this issues I’ve found a bug in the parser when using the 2.5 Grammar. It is raising a SyntaxError when you call a function passing a keyword argument (like f(x=3)). Because of this (and other small bugs) I’ve decided that I will work on the pyparser to make it compatible with the 2.5 Grammar now.

PyPy 2.5 features – news

Tuesday, May 13th, 2008

Changes! Now I’m free to work “full-time” on my GSoC project, last Friday was my last day on my old job (Ikwa) :-)

Today I’ve read lots of docs and code, specially the pyparser code. I’ll try to maintain a list of features I’m currently working on here.

Last week (ok, Friday before last Friday) we went to Werneck’s cave to code. One thing I’ve made there was implement the defaultdict object in the collections module. This module is written in C originally, so I needed to port it.

The code is relatively simple and I’ve just “copied” the C implementation (of course it’s clearer and more beautiful in Python :P ). Werneck also helped me with the pyparser but we couldn’t find a solution for supporting the full Python 2.5 Grammar .

Today I’ve worked on a small Python 2.5 change, the empty base classes list syntax.

Until Python 2.5 the list of base classes could not be empty. So you couldn’t define a class like:

class A():
    pass

The right way for doing this was

class A:
    pass

Now, in Python 2.5, both ways are legal.

Last week I’ve committed a slight changed Grammar for Python 2.5 syntax support to PyPy’s 2.5-features branch (the only change I’ve made was removing the support for the new import syntax, as it crashes the parser, I need to work on that).

With part of the 2.5 Grammar already supported, to support this new class definition syntax the only change needed was in pyparser (more specifically to the AST builder).

While the build_classdef method expected 4 atoms (< class keyword >, < white space >, < class name > and < comma >) or 7 (the same 4 plus < ( >, < base classes > and < ) > before the < comma >) in Python 2.4, now in Python 2.5 it can receive 6 atoms too (< class keyword >, < white space >, < class name >, < ( >, < ) > and < comma >). The change was really simple, but it was a good exercise for me because I could apply what I’m learning in my compilers classes at school (while reading the sources) :-)

Some code (pypy/interpreter/pyparser/astbuilder.py:634):

def build_classdef(builder, nb):
    ...
    if l == 4: # class NAME:
        basenames = []
        body = atoms[3]
    elif l == 6: # class NAME():  # 2.5
        basenames = []
        body = atoms[5]
    else:
        assert l == 7
        basenames = []
        body = atoms[6]
        base = atoms[3]
    ...

I don’t think I need to explain this code, it’s very simple (of course, it’s Python! :-) ).

I’ve made some tests and it seemed to work great, but I’ve decided to compare the bytecode generated by PyPy and by CPython (both 2.4 and 2.5). While in CPython 2.4 the statement “class A: pass” produces the following bytecode:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 BUILD_TUPLE              0
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

CPython 2.5 produces for both statements (the old syntax and the new one) something slight different:

>>> from dis import dis; c = compile("class A: pass", "/dev/null", "exec"); dis(c)
  1           0 LOAD_CONST               0 ('A')
              3 LOAD_CONST               3 (())
              6 LOAD_CONST               1 ()
              9 MAKE_FUNCTION            0
             12 CALL_FUNCTION            0
             15 BUILD_CLASS
             16 STORE_NAME               0 (A)
             19 LOAD_CONST               2 (None)
             22 RETURN_VALUE

The only difference between 2.4 and 2.5 is the instruction BUILD_TUPLE in 2.4 against the LOAD_CONST in 2.5. I'm not a bytecode expert, but it seems that this is an optimization, of course I may be completely wrong :P

Well, as I expected the bytecode produced by PyPy is identical the one produced by CPython 2.4, so I think I have more things to change to complete this task. ;)

But not tonight, after 14 hours of PyPy code reading and some debugging (plus Linear Programming and Computer Theory classes) I think I should sleep.

:-)

FLISOL 2008 – Python Talk

Saturday, May 3rd, 2008

Last Saturday I gave a Python talk at FLISOL 2008 in Sào Paulo. The public was students who were learning programming (C and Visual Basic), so it was a very basic talk. The event was great and so was the talk, the only problem was that I thought that the public would be programmers with some experience, but not really. Anyway, the FATEC students seemed to like Python and some of then came to talk to me after the event to know more about the language and to ask where they could find more information!

I’ll upload my presentation somewhere.

I’ve helped to install GNU/Linux (Ubuntu) on a FATEC lab, people there were very nice!

Well, I felt like my “mission” was accomplished :-)

FLISOL 2008 in São Paulo – Python Talk

Friday, April 25th, 2008

I’ll give a talk about Python in FLISOL 2008 São Paulo. It’s scheduled to start at 15h00. It’s a basic talk for people who code in other languages and don’t know much about Python.

Take a look here for more information about FLISOL.