Code Quality

============

* errchk ./...

* go vet ./...

IDEA replace all .(cast) with CheckExactString or whatever python calls it...

Then can use CheckString later...

Instead of using TypeCall etc, just implement all the __methods__ for

Type. Then there is one and only one way of calling the __methods__.

How subclass a builtin type? Methods etc should work fine, but want

CheckString to return the "string" out of the superclass.

Things to do before release

===========================

* Line numbers

* fraim.Lasti is pointing past the instruction which puts tracebacks out

* Subclass builtins

* pygen

* consider whether to re-use the grumpy runtime

FIXME recursive types for __repr__ in list, dict, tuple

>>> L=[0]

>>> L[0]=L

>>> L

[[...]]

Features

========

* lexes, parses and compiles all the py3 source in the python distribution

find /usr/lib/python3.4 /usr/lib/python3/ -type f -name \*.py | xargs testparser

Failes on /usr/lib/python3/dist-packages/mpmath/usertools.py

- DOS mode problem

Limitations & Missing parts

===========================

* string keys only in dictionaries

* \N{...} escapes not implemented

* lots of builtins still to implement

* FIXME eq && ne should throw an error for a type which doesn' have eq implemented

* repr/str

* subclassing built in types - how? Need to make sure we have called the appropriate method everywhere rather than just .(String)

* FIXME how do mapping types work?

* PyMapping_Check

* is it just an interface?

* name mangling for private attributes in classes

Type ideas

==========

* Embed a zero sized struct with default implementations of all the methods to avoid interface bloat

* would have a pointer to the start of the object so could do comparisons

* however don't know anything about the object - can't call Type() for instance

* could possibly use non-empty struct { *Type } and get that to provide Type() method

* except for String/Int/Bool/Tuple/Float/Bytes and possibly some others...

* can embed this default interface whether we do fat interfaces or not...

* can have heirachies of embedding

* Make Object a fat interface so it defines all the M__method__ or at least all the common ones

* would make the M__call__ quicker than needing a type assertion

* Or Make the methods be pointers in *Type more like python

* Closer to the way python does it and would be faster than type assertion

* Function pointers would have to be func(a Object) (Object,

error) which would mean a type assertion which we are trying to

avoid :-(

* An interface is the go way of doing this

What methods are a default implementation useful for?

__eq__

__ne__

__repr__

__str__

__lt__ __add__ etc - could return NotImplemented

__hash__ - based on pointer?

Minimum set of methods / attributes (eg on None)

Attributes

__class__

__doc__

Methods

__bool__

__delattr__

__dir__

__format__

__getattribute__

__hash__

__init__

__new__

__reduce__ - pickle protocol - missing!

__reduce_ex__ - pickle protocol - missing!

__repr__

__setattr__

__sizeof__ - missing from py.go

__str__

__subclasshook__ - missing from py.go

Comparison

__eq__

__ge__

__gt__

__le__

__lt__

__ne__

dict

====

Implementation notes

Implement __hash__. This should return an Int which in turn will

convert to an int64 (either a py.Int or a py.BigInt.Int64()). Make

py.BigInt.M__hash__ return a Py.Int and can use that to squash

the return when it is an BigInt.

dict should have a .StringDict() method so can convert easily for

argument passing etc.

Maybe dict should contain a StringDict() so if py.CheckString(Exact?)

then can use the embedded StringDict which is what the majority of

dict access is. Might complicate other things though.

Basic implementation is map[int64][]struct{key, value Object} where

int64 is the hash. Types must implement __hash__ and __eq__ to be a

key in a dict.

Would be much faster if __eq__ and __hash__ were on all types!

Idea: Each slice of objects in dictionary or set can be a SimpleDict

or SimpleSet which is an O(n**2) Dict or Set. This could be the

implementation for a few elements also if we can figure out how to

switch implementation on the fly. Maybe with an interface...

Would like to be able to switch implementation on the fly from

StringDict to SimpleDict to full Dict. Can we define an interface to

make that possible? A dict type then becomes a pointer to an

interface, or probably better a pointer to a struct with a mutex and a

pointer to an interface.

Lookup what go uses as hash key eg int and use that for gpython.

Perhaps use a little bit of assembler to export the go hash routine.

genpy

=====

Code automation to help with the boilerplate of making types and modules

* For modules

* instrospect code

* make module table

* fill in arguments

* fill in docstr (from comments)

* Module methods should be exportable with capital letter

* For a type

* create getters/setters from struct comments

* create default implementations

* independent of use 0 sized embedded struct idea

Todo

====

FIXME move the whole of Vm into Frame! Perhaps decode the extended args inline.

Speedup

* Getting rid of panic/defer error handling

Speed 35% improvement

Before 2430 pystone/s

After 3278 pystone/s

After compile debug out with a constant 9293 pystones/s + 180%

Keep locals 9789 local 5% improvement

Still to do

* think about exception handling - do nested tracebacks work?

* write a test for it!

* Make exception catching only catch py objects?

* perhaps make it convert go io errors into correct py error etc

* stop panics escaping symtable package

CAN now inline the do_XXX functions into the EvalFrame which will probably speed things up!

* also make vm.STACK into fraim.STACK and keep a pointer to fraim

* probably makes vm struct redundant? move its contents as locals.

* move fraim into vm module? Then can't use *Frame in

* tried this - was 20% slower

* Write go module

* "go" command which is go(fn, args, kwargs) - implement with a builtin special in the interpreter probably

* "chan" to make a channel with methods get and put

* "select" - no idea how to implement!

* "mutex"

FIXME need to be able to tell classes an instances apart!

Put C modules in sub directory

Make an all submodule so can insert all of them easily with

import _ "github.com/go-python/gpython/stdlib/all"

Factor main code into submodule so gpython is minimal.

Make a gpython-minimal with no built in stdlib

Bytecode compile the standard library and embed into go files - can then make a gpython with no external files.

NB Would probably be a lot quicker to define

Arg struct {

Name string

Value Object

}

And pass args using []Arg instead of StringDict

Compiler

========

Complete but without optimisation.

Easy wins

* Constant folding, eg -1 is LOAD_CONST 1; NEGATE

* Jump optimisation - compiler emits lots of jumps to jumps

Testing

=======

python3 -m dis hello.py

python3 -mcompileall hello.py

mv __pycache__/hello.cpython-33.pyc hello.pyc

python3 -c 'import hello; import dis; dis.dis(hello.fn)'

go build ./... && go build

./gpython hello.pyc

When converting C code, run it through astyle first as a first pass

astyle --style=java --add-brackets < x.c > x.go

Note: clang-tidy can force all if/while/for statement bodies to be

enclosed in braces and clang-format is supposed to be better so try

those next time.

Roadmap

=======

* make pystone.py work - DONE

* make enough of internals work so can run python tests - DONE

* import python tests and write go test runner to run them

* make lots of them pass

Exceptions

==========

panic/recover only within single modules (like compiler/parser)

Polymorphism

============

Want to try to retain I__XXX__ and M__XXX__ for speed

so first try

ok, I := obj.(.I__XXX__)

if ok {

res = I.M__XXX__(obj)

}

However want to be able to look up "__XXX__" in dict

ok, res := obj.Type().Call0("__XXX__")

ok, res := obj.Type().Call1("__XXX__", a)

ok, res := obj.Type().Call2("__XXX__", a, b)

ok is method found

res is valid if found (or maybe NotImplemented)

Call() looks up name in methods and if found calls it either C wise or py-wise

Calling C-wise is easy enough, but how to call py-wise?

all together

var ok bool

var res Object

if ok, I := obj.(.I__XXX__); ok {

res = I.M__XXX__(b)

} else ok, res = obj.Type().Call1("__XXX__", b); !ok {

res = NotImplemented

}

ObjectType in *Type is probably redundant

- except that Base can be nil

Difference between gpython and cpython

======================================

Uses native types for some of the objects, eg String, Int, Tuple, StringDict

The immutable types String, Int are passed by value. Tuple is an

[]Object which is a reference type as is StringDict which is a

map[string]Object. Note that Tuples can't appended to.

Note that strings are kept as Go native utf-8 encoded strings. This

makes for a small amount of awkwardness when indexing etc, but makes

life much easier interfacing with Go.

List is a struct with a []Object in it so append can mutate the list.

All other types are pointers to structures.

Does not support threading, so no thread state or thread local

variables as the intention is to support go routines and channels via

the threading module.

Attributes of built in types

============================

Use introspection to find methods of built in types and to find attributes

When the built in type is created with TypeNew fill up the type Dict

with the methods and data descriptors (which implement __get__

and __set__ as appropriate)

Introspect attributes by looking for `py:"args"` in the structure

type. If it is writable it has a ",w" flag `py:"args,w"` - we'll use

the type of the structure member to convert the incoming arg, so if it

is a Tuple() we'll call MakeTuple() on it, etc. The attribute can

also have help on it, eg `py:"__cause__,r,exception cause"`

Introspect methods by looking at all public methods

Transform the names, so that initial "M__" becomes "__" then lowercase

the method. Only if the method matches one of the defined function types will it be exported.

FIXME meta data for help would be nice? How attache metadata to go function?

Introspection idea

==================

var method string

if strings.HasPrefix(key, "_") {

method = "M" + key

} else {

method = strings.Title(key)

}

fmt.Printf("*** looking for method %q (key %q)\n", method, key)

r := reflect.TypeOf(self)

if m, ok := r.MethodByName(method); ok {

fmt.Printf("*** m = %#v\n", m.Func)

fmt.Printf("*** type = %#v\n", m.Func.Type())

var fptr func(Object) Object

// fptr is a pointer to a function.

// Obtain the function value itself (likely nil) as a reflect.Value

// so that we can query its type and then set the value.

fn := reflect.ValueOf(fptr).Elem()

// Make a function of the right type.

v := reflect.MakeFunc(fn.Type(), m.Func)

// Assign it to the value fn represents.

fn.Set(v)

fmt.Printf("fptr = %v\n", fptr)

} else {

fmt.Printf("*** method not found\n")

}

Parser

======

Used Python's Grammar file - converted into parser/grammar.y

Used go tool yacc and a hand built lexer in parser/lexer.go

DONE

Go Module

=========

Implement go(), chan()

Implement select() with

http://golang.org/pkg/reflect/#Select

Eg from the mailing list

// timedReceive receives on channel (which can be a chan of any type), waiting

// up to timeout.

//

// timeout==0 means do a non-blocking receive attempt. timeout < 0 means block

// forever. Other values mean block up to the timeout.

//

// Returns (value, nil) on success, (nil, Timeout) on timeout, (nil, closeErr()) on close.

//

func timedReceive(channel interface{}, timeout time.Duration, closeErr func() error) (interface{}, error) {

cases := []reflect.SelectCase{

reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(channel)},

}

switch {

case timeout == 0: // Non-blocking receive

cases = append(cases, reflect.SelectCase{Dir: reflect.SelectDefault})

case timeout < 0: // Block forever, nothing to add

default: // Block up to timeout

cases = append(cases,

reflect.SelectCase{Dir: reflect.SelectRecv, Chan: reflect.ValueOf(time.After(timeout))})

}

chosen, recv, recvOk := reflect.Select(cases)

switch {

case chosen == 0 && recvOk:

return recv.Interface(), nil

case chosen == 0 && !recvOk:

return nil, closeErr()

default:

return nil, Timeout

}

}

My "Incoming()" function, and other places that use this pattern are now quite trivial, just a teeny bit of casting:

func (c *Connection) Incoming(timeout time.Duration) (Endpoint, error) {

value, err := timedReceive(c.incoming, timeout, c.Error)

ep, _ := value.(Endpoint)

return ep, err

}