Packet Forth dev log. For more info see http://zwizwa.be/packetforth Entry: forth bootstrapping Date: Mon Jan 30 03:23:26 CET 2006 been thinking about gradual forth bootstrapping. like: -- LOW LEVEL -- * integer core * float words (E) * array/grid processing (E) * polymorphy (L) * scheme like pairs & garbage collection * true reflection (infinite reflective towers?) -- HIGH LEVEL -- where (E) stands for early binding, or compile time binding, and (L) stands for late binding, or run time binding. there is one practical level which is too hard to implement for a unix-based portable forth, which is the assembler level. but it should be possible to have reflection (meta compilation) on the C level, wich might also facilitate C library interaction. so the compiler, written in forth, is a forth -> C compiler. reloading is done by dumping out C code, running gcc and reloading the kernel. this almost requires an in-house editor. Entry: PF versus SC Date: Mon Jan 30 03:39:40 CET 2006 the supercollider idea: (A) very fast & simple audio server (B) very convenient & flexible control language both are tied using a messaging system (OSC) since the B events are 1 to 3 orders of magnitude less frequent, it is possible to trade some cycles for convenience. one of them is clearly garbage collection, which enables lisp-like freedom, and all kinds of neat language tricks that are not really possible without the ability to account for cyclic graphs. ( in PF there are some cyclic graphs, though they are burried deeply. one less burried example is 'defer'. so PF needs pointers that can be null, or can reference a non-existing or recucled object. ) the problem i encounter is: if you enable GC, there is no fast reuse of image packets, so this will be slow (cache misses) and memory inefficient (huge memory usage). the solution is to decouple both: the language should not be able to manipulate images directly, only image manipulators. (in SC: you cannot process audio buffers, only control synths, which process audio buffers). this is a real choice that has consequences everywhere: * PF is really packet central, not processor central. a bit like Pd. So it is really not possible to have m/s GC in the core, only ref count based resource management. * SC is processor central. true OO information hiding, which enables a clear separation of memory allocation strategies for server and client. so.. another view. SC client is a compiler for SC server. SC server is the VM. this brings me to the following possibility, to split PF into two parts. * a true forth server with grid support (i.e. an efficent 'map' word / code gen) * a Joy like RPN language (implemented in scheme for example) as a client. BUT BUT BUT! the cut does not have to be too explicit, does it? in light of the previous entry on bootstrapping, it should be possible to write an reflective engine on top of the core forth that bootstraps itself into a scheme-like eval machine... (note: this idea is worked out in chicken & egg) Entry: C code cleanup Date: Tue Feb 21 17:19:47 EST 2006 looking at the C code, there's a lot of stuff that went terribly wrong. half of it is type checking and low level list manipulation. very very wrong. how can this be cleaned up? is it possible to remove lowlevel list manipulation? some common practices: * packets: stack->int->header->struct * scalars: check type and copy to c local variable * packets: check if packet + inspect struct to see if its workable so it's time for some guidelines about writing plugins: * write type checking macros. don't do this manually. * store all temporary data on the data stack. this way multiple exits are easy to implement * don't access lists directly, use a set of macros * use pf interface only for glue to existing C modules final word: need to fill pf/macros.h with good words and ACTUALLY USE THEM in an example plugin. Entry: tail calls Date: Thu Feb 23 13:22:05 EST 2006 Trying to turn all calls into tail calls when possible, however, i'm running into a problem with : x >r r> ;; this gets translated to >r do-pass r> if the last XT in a link modifies the return stack, it won't work, since do-pass pops RS before executing the xt. can we make it pop RS *after* calling? Entry: lists, stacks & queues Date: Sun Feb 26 20:39:34 EST 2006 The list atom in packet forth has a 'last' pointer. Why do i need that? To make queues. Now, it is possible to make queues using just 2 stacks (singly linked lists). I thought of that before, but never saw it was the key to clean up a lot in packet forth, mostly getting rid of pf_list_t, pf_atom_t, and have a pf_pair_t, which is basicly like lisp pairs with the restriction that it can only be used to build trees. Then there's another thing i'd like to get rid of: the end-of-link pointer. For the latter case, there's only: xt>ct xt>doc xt>name and linkhead, only used in debugging for the rest, comma assumes code fields (primitives) and data fields have a certain order. not too hard to change probably. So, in order to clean up lists and atoms, which basicly means * removing __last (replace by method) * removing __elements (replace by method) * unifying list and atom We have to get rid of places where these are used as lvalues. The main point of annoyance is the 'forth_sentinel' in pf_atom_link comma_to_link print_ip and PF_XT_TO_LNK in the words above What about temporarily setting XT=LINK? it's supposed to be an abstract data type anyway. WONT WORK! This breaks "does>" which promotes an arbitrary atom pointer to an xt, which is really what we want anyway.. so we'll have to go around it in another way, directly eliminating the sentinel structure. Entry: lists, stacks & queues II Date: Mon Feb 27 10:55:20 EST 2006 This won't be easy because - 'here' really needs an address. the whole compilation architecture needs to be rewritten if here is to disappear - i tried to break the circularity of the sentinel by using only the circularity of XT using an explicit list containing meta data and an xt. this is hard to do because it requires a distinction between link and meta. so, for now i just leave it like it is. it's messy, but at least it works.. Entry: stacks (recovered from stacks.txt) Date: Thu Mar 2 12:02:04 EST 2006 This file dealed with: (A) more stacks other than data/return/dict (B) memory management for circular refs These are largely solved. (A) We don't have more than these 3 stacks, since I/O stacks are (were) there to provide 'execution context' which can be easily put on the data stack, for fast changing context, or on the return stack, for slower changing context. For (B). Everything which uses circular refs is not managed. All data structures are strictly trees: meaning all data is owned by one owner at a time. All the rest is solved using 'atom pointers' or variables. These pointers do not 'own' the data they point to, so if the underlying data is destroyed, pointers become stale. Another remark. Some objects are not managed using reference counts, i.e. they are not packets. This should be fixed: * display lists Entry: multimethods Date: Thu Mar 2 15:35:40 EST 2006 when i started out on pf, i didn't want to make an object-oriented language. this is wrong. i DID want to make an OO language, but i didn't want methods to be 'owned' by classes. i just wanted simple data objects + multimethods. (i'm reading 'The Interpretation of Object-Oriented Programming Languages' 2nd Edition by Iain Craig. which i should have started way earlier.) But, i don't want inheritance, and i dont want to create new classes to abstract solutions to problems. my objects are the basic data types, and the simple list/stack/tree/queue containers that group them. PF is mainly about arithmetic (DSP) and data-flow. My types are simple: image, matrix. My I/O is fixed, and written in C (X11, xvideo, ...) But, i do need to solve one problem properly: multimethod dispatch. Currently it sort of works, but it's slow, and a big hack. Entry: immediate words Date: Thu Mar 2 15:50:40 EST 2006 something to think about.. pf is a hybrid between forth and something like joy (snarfed in brood/cat). one example: (dup) (1) ifte if dup else 1 then there is no real need for many immediate words in pf, because blocks can be implemented using sublists, and there's the { } local word context. i know i can forget about making pf clean. it's a practical thing, and though i like to dwell over theory, i'm a practical person mostly.. there's a lot of different constructs available. there's more than one way to do it! see brood/doc/ramblings.txt for more of this [EDIT: this is actually quite an important example. because of the raw nature of forth, there is indeed more than one way to do it. using highlevel functions in things as simple as if..then constructs adds a lot of possibility for factoring. : ifte >r >r if rswap then r> rdrop execute ; this enables things like { 123 } { 456 } ifte, where the XTs generated by the angle brackets can be reused in other places. i use this construct a lot in brood, since it's basic there. and grew quite fond of it. it's cleaner than if-else-then.] Entry: new parser Date: Tue Mar 28 13:27:30 CEST 2006 Things that need to change: * pre-emptable * extensible The easiest way to have this is to write it in forth. Then the problem is to bootstrap it. To cut the feedback loop (bootstrapping is like a Y combinator :) is to have a generator which translates forth code to C code that does not need the parser. This is actually the quest for more introspection / reflection. So we need a way to feed pre-parsed code. (Fixed). Can't think right now, but generating preparsed code seems to be pretty simple. Got stuck in the parser again. I wonder if there's an easier way to describe it using grammar. Do i really need specific parsers? The only thing necessary is a forth parser, and an s-expression parser. Yes. There's several things that don't really work: parsing from string, readline line-based stuff, ... So... What about worker threads? Since it's written in C anyway, there's no real reason to not attach a thread to a stream... In that case, the whole pre-emption story is moved to the OS. Entry: input stream blues Date: Fri Mar 31 13:00:27 CEST 2006 What was i thinking when writing the input stream code?? This is really really really badly written. Instead of having an input stream data item, i should just redefine 'accept', and have this word map to a name to determine the name of an input stream. I should make a list about places where 'current input / current output' are necessary, and if they can be replaced by a deferred word. Entry: temp code Date: Fri Mar 31 15:57:37 CEST 2006 Getting confused about temporary code again. What i need is to be able to store temporary code on the return stack, to be able to use proper dynamic scope tricks without using global variables. This can be done by * compiling an anonymous function to the dictionary * obtaining its xt * popping the function from the dictionary * storing it on the return stack Using list-interpret is the easiest, but also requires list-interpret to work. Since i want to use this to implement list-interpret, it might be better to go another route. The latter is a lazy hack anyway. The idea is: wrap a variable in an xt. For example, mimick this scheme code without using permanent storage: (define add-element (lst a) (map (lambda (x) (+ x a)) lst)) Just the lambda (accumulator) gives: : make-accum # ( a -- xt code ) mark ` xxx link enter, postpone literal postpone + postpone ; latestxt mark> ; : make-accum mark ` xxx link enter, postpone literal postpone + postpone leave latestxt mark> ; This seems to +- work for streams. Now i just need lists and string accepters. Entry: copying code Date: Sat Apr 1 08:28:37 CEST 2006 I think i remember now. Copying code doesn't work, because it's littered with pointers to atoms. So it's necessary to be really careful with task swapping and temp functions. The error was in .stack : atoms are copied before they are printed. Entry: linear lisp - memory management Date: Sat Apr 1 11:06:44 CEST 2006 I think i sort of get it now. Linear lisp: no circular references. Every PAIR has a single owner. Objects which do not contain pointers (terminals) can be managed by reference counts. This is packet forth. This brings me to see the very nice thing about functional programming and indefinite extent and debuggin: it is possible to save EVERYTHING in the state it was at a certain point, if you want to have a look at things. This is something i really miss in pf. Entry: abort stack Date: Sun Apr 2 09:12:51 CEST 2006 Ran into a problem with the abort stack using coroutines. Apparently this doesn't work: load-coroutine generator x :: (1 2 3) { . yield } consume ; ' x start x x x x x x .n Because consume messes with the abort stack, which is also used to perform file loading.. The core of the problem is the following: the abort stack is used to isolate environments, i.e. when loading files, or executing code on the console. In this setting, it is illegal for a subprogram to look beyond its containing environment. Abort stack = stack of environments + exit words = GLOBAL The problem here is that the generator above uses consume to generate a new environment with stuff on the abort stack: it uses a global resource to perform a local state saving/restoring feature (consume). This is OK, as long as the environments are CONTAINED. Making it a generator violates that: control is returnned to caller with context left on the abort stack. How to solve this? The most straightforward way is to make map & consume not use the abort stack, so they can be used in coroutines. Entry: core changes epilogue Date: Sun Apr 2 14:20:10 CEST 2006 Lots of things changed last couple of days. * eliminated need for new (pre-emptible) parser by running parser in a separate thread. (plugins/reader) * the above required a different kind of 'accept' semantics (not just a current input stream, but a current input accepter word), so i went on eliminating the input/output stacks from the VM and replaced it with a single deferred 'accept' word. * i tried to use temp code on the return stack to implement the current accept semantics: like lisp closures without GC. i tried this before, but didn't learn: it's too hard to get the memory management working: too many things can go wrong. * so i split the closures into an explicit code/data pair: 'accept' contains the code word, while the top of the input stack contains the data part of the stream, which in most cases is just a stream packet. * this allowed a single interpreter for all streams, a good thing. error message reporting is done in the interpreter's abort handler, using the 'stream-info' and 'error-location!' words. (stream-info is hacked to not produce errors: can't do that inside an error handler). * abort stack semantics are clarified: it is for global linear control flow only. for local control flow only the return stack can be used (so this control can be frozen by swapping return stacks.) for this to work 'map' and 'for-each' (former 'consume') had to be changed to leave the abort stack alone. * code copying has been clarified: see pf_stackatom_dup in libpf/list.c : this means 'dup' and '@' have slightly changed semantics: they will return a copy only if objects are copyable, and a ghost otherwize. technically it should now be safe to do temp-code tricks, but i prefer to keep it out of the core functionality. Entry: polywords Date: Sun Apr 9 13:52:06 CEST 2006 Found myself a new problem: polywords.. They are not re-entrant, and i appear to be in need for that to implement the recursive list>string word. [later that day..] So, in the end i didn't need re-entrant polywords for list>string, but it's better to have them this way. I went ahead and cleaned up the stream C code. This was a bit of a mess. It seems to work now: all binary input/output is unified in streams. Non-binary output can be unified on the XT level, with an output stack containing 3 atoms: data, xt.write and xt.print in the same spirit as the input stack. Things to do: * Printing strings with null characters (as hex?) * Output stack * Stack printing: special case: write strings but print other packets. Entry: non-pure packet copy/create semantics Date: Mon Apr 10 13:38:34 CEST 2006 Need to sort out how to solve packet creation/copy for packets that can only be created using a factory method (like streams). Entry: quines in pf Date: Sat Apr 15 00:08:03 CEST 2006 I swear i started an entry already.. Can't find it. Anyways. Trying to find a way to make quines in pf. Instead of an incremental brute force approach, let's think about it first. What is a quine? A fixed point of some function. Which function depends on the rules of the game. Usually this is compile followed by run. In pf, compile followed by run is generally equivalent to interpret. Wether the game is trivial depends on what we see as compile and run, and what we see as input or output. For example, we could look for fixed points of the function 'interpret'. This prodices a lot of trivial quines: all non-symbols, and symbols evaluating to themselves. This is considered cheating. There has to be at least some operation of unquoting/interpretation of the source code. The next thing to look at is fixed points of 'interpret-list'. There are no trivial quines here, since it involves unquoting. So that's the first place to look for interesting quines. In 'interpret-list' the containing list structure is removed, so there has to be at least one operator which generates a new list from non-list atoms. This could be 'pack'. This operation needs an integer argument, so ' pack' should be part of the deal. Maybe it's easier to write a reproducing program first. I'll use the shortcuts : i interpret-list ; : p pack ; Playing with (1 p) i and dup gives ((1 p)) i -> (1 p) dup i -> ((1 p)) So we see ((1 p)) is a fixed point of { i dup i } and (1 p) is a fixed point of { dup i i } What about this: you need a quoted payload that when executed will duplicate the top of stack and pack it: dup 2 pack Found this one ((dup 2 p) dup 2 p) -> ((dup 2 p) (dup 2 p)) Which evenually lead to the first quine: ((dup 2 pack split +) dup 2 pack split +) Which reads as: * copy quoted code dup * combine in one package 2 pack * unqote one half, recombine split + Analyzing that one gives a shorter one ((dup unsplit) dup unsplit) Which reads as: * copy list of code * combine by inserting one list as a data argument into the other This is very similar to the Y combinator, where you pass a function to itself as a parameter to implement recursion. However, in that case the reproduction eventually stops. In CAT/PITH this quine is ((dup cons) dup cons). Both elements of the first pair in this list point to (dup cons). The car one is quoted because it is not a symbol, and the cdr one is interpreted because it's cars are symbols. http://www.drunkmenworkhere.org/170 Entry: libquicktime Date: Mon Jul 3 14:31:42 EDT 2006 First, i had "quicktime/*" included instead of "lqt/*". Apperently, somewhere along the road this broke things. I updated my install to 0.9.9, which behaves differently in other ways, namely the differences between BC_YUV420P video range yuv 16-240 BC_YUVJ420P full range yuv 0-255 The first one is expected by video codecs, while the second one is from photo jpeg. Confusing stuff. Basicly, this is about "black level" and "white level", which i don't remember exactly what purpose this serves, other than that it is part of video standards. I changed all references from BC_YUV420P to BC_YUVj420P for now, which seems to fix the problems i ran into. Version 9.7 fails to compile with this feature. Probably 9.8 works, (following the release notes) but i didn't test this. Now i got this very strange problem. When linking pf with -lquicktime, it finds the symbol quicktime_match_32 when the plugin libquicktime/lqt_mjpeg.so is loaded. However, if pf is not linked with -lquicktime, i get the error message: Trying to load module /usr/local/lib/libquicktime/lqt_mjpeg.so...failed, /usr/local/lib/libquicktime/lqt_mjpeg.so: undefined symbol: quicktime_match_32 That last thing is only fixable by linking the main program with -lquicktime Entry: containers and arrays Date: Fri Jul 7 13:09:52 EDT 2006 My idea is falling apart. If i see how it's being used by other people, element access is something important apparently. Pushes my face to things that are wrong in pf.. Somehow, not having element access makes things really bloated, since i have to write code for every possible way you'd like to access groups of elements, leading to combinatorial explosion. The idea is still valid, since in most cases i can find a way to keep the data sizes static, and perform parallel operations. The combinatorial explosion can be handled by automaticly generating iterators. Back to square one.. This is the whole idea behind brood, in the end.. But i feel the need to sob over it some more. See brood/doc/ramblings.txt for the whole deal. Entry: swinging Date: Fri Jul 21 00:01:53 BST 2006 thinking about switching to Factor, or at least investigating it well enough to see if it would be worthwhile. in any case, pf has to be severely simplified. most important things are * threads need to go, especially fine grained locks on data * unified type system * unified exception system what about this. i already have atoms as basic structure. atoms consist of 3 things: * type * data * next i don't think i need generic pairs. changing that would be too much. but, the type can be changed to something else. actually, 'type' should be a function, or a relation 'member of' i.e. an indicator function. it's hard to kill your baby.. rushing between wanting to change pf, simplify it. then seeing that this is actually a whole lot of work which can better be done by rewriting the tower of abstraction bottom up, instead of top down simplification and optimization. Entry: threads Date: Mon Jul 31 16:25:53 CEST 2006 command queues have to go, since i want to remove the locks from the list/packet memory allocation. currently it's used in: * osc: can be removed : select based mainloop + assume output doesnt block * readline + reader : both just input text in a separate thread. maybe combine with osc? the snigulps have to be changed too. probably means the end of udp, glade and xfree so the roadmap looks like: 1. create select based pf loop + find out what to plug in 2. either make the parser re-entrant, or have it use a serialized intermediate format maybe a fast binary serial data format could be an interesting approach. this would solve the console related problems above, plus gives fast inter-pf communication. it could also be a good opportunity to unify the type system. the binary format is 32bit host data format. endianness needs to be solved elsewhere. * int float tag, value * symbol tag, words, characters * string/raw tag, words, characters, words, data * list tag, elements, ... all other cannot be serialized. for lists: a whole list is sent at once (list=message). to do incremental sends, split a list in records (atoms). knowing the number of things to read in advance easier than trying to figure this out from the data stream. the parser is then just transforms a text stream into a binary stream. binary stream decoder -> internal data structures is very straightforward, as is the serialization. Entry: lists and atoms Date: Mon Jul 31 16:40:18 CEST 2006 so what is the difference between a list and a chain of atoms? the former has a finite length, and a last element, but the latter does not. so an 'atom' is more like a 'record' or a 'message'. Entry: foamy things Date: Wed Aug 9 10:10:56 CEST 2006 two problems to fix today: * colour space conversion: i420 -> rgb * disable nested functions to work around apple arrogance i thought i already fixed the colourspace conversion problem.. apparently not. currently there's the function 'image:crotate' which is the general colour transform. (problem was in matrix inverses.. somehow it gave nan results.. after rewrite to use some macros.h stuff, it seemed to work.) some more remarks. digging in the matrix code, fixing some things. noticed this: often it happens i want to clone something i only have a matrix_t reference to. the thing what's wrong is that there's no connection from the lowlevel data (subheader) to the management layer (packet). so what's there: * packet (integer) * raw packet (pf_packet_t) * real data (subheader) something that seems to fix things is the PUSH_CLONE matrix. Entry: foam days Date: Wed Aug 9 12:36:51 CEST 2006 1. walks and talks 2. show and tell, install fest, possible directions 3. install fest, bug fixes Entry: removing double precision? Date: Wed Aug 9 13:09:37 CEST 2006 what about keeping double precision matrices inside PF? the only place where this is important is for ill conditioned 'cancellation' problems which probably only occur as intermediate states in some algorithms, and not as source data or end product. currently it's only the LU decomp that requires double precision, which is only used in the matrix:inverse word atm. in the future the SVD might show up somewhere, and some levinson / shur like algorithms. Entry: binary data pipes Date: Wed Aug 9 18:56:54 CEST 2006 so this should actually replace the procqueue stuff. each end of a pipe has two interfaces: read/write style raw data and pf data atoms. so it's a collision of: - parser pre-emption - fast out-of-process pf comminication - binary data save/load - pf core synchronization (select) - streams argh.. brain. Entry: pf rewrite Date: Wed Aug 9 23:11:58 CEST 2006 cycle 44 or somting. trying to summ the features that are necessary, but need to be implemented in a simpler way - pre-emptive parser (separate process?) / lisp compatible? - proper pairs and lisp dot syntax? - lock free operation in core - operate without libc malloc? / malloc once? - proper type/object system - multimethod dispatch (how to encode types and use that?) - well thought out serialization: message based comm + persistent state - are floats/ints/symbols different from packets? - what about type symbols vs type lists? - write a parser in forth (how to bootstrap this?) - bytecode should not be lists but arrays - any reason why the stack should be a list? - need call threading for proper tail recursion (push bit) - proper exceptions (continuations, dynamic-wind) - don't ever use ints again to represent objects! - don't ever use 2 type descriptions again! so. forth + prototypes + multimethods some things (like the tail recursion, and arrays instead of lists for bytecode storage) really can't be fixed. the only thing that can be saved are the plugins, but most of them will need to be adapted. type tags. what about using some 'optimally' distributed hash for each and every type encountered, stored in a central database. since i basicly want a binary protocol, how would this be designed? 32bit word, followed by size and raw data. how about that thing, but per application, serving just as compiled in type dispatch. Entry: objects and dispatching Date: Sat Aug 12 15:15:29 CEST 2006 "lala.mov" mov open where * 'mov' is a parsing word with interpret and compile semantics, changing the binding of 'open' * 'mov' is an object which changes the binding of 'open' using multimethod dispatch. the parsing version is ugly, but should later be replacable by the clean version. the parsing version is also faster since it performs method lookup at compile time. Entry: waking up again Date: Fri Aug 25 02:20:56 CEST 2006 time to end the summer lazyness, and fix things! maybe time for yet another list: * binary protocol: stream of chunks * parser translates text -> binary pf stream * main select loop * main pf body single thread / process (see also brood doc/ramblings.txt) Entry: the expression problem Date: Mon Aug 28 17:35:08 CEST 2006 something i ran into when reading lut: http://lambda-the-ultimate.org/node/1641 http://lambda-the-ultimate.org/node/1453 http://www.ipl.t.u-tokyo.ac.jp/~tops/done_seminar06.html "In object-oriented languages, it is easy to extend data by defining new classes, but it is difficult to add new functions. In functional languages, the situation is reversed: adding new functions poses no problems, but extending data (adding new data constructors) requires modifying existing code. The problem of supporting both directions of extensibility is known as the expression problem." Looks like what i'm trying to accomplish is very much related to this. Some bones to chew on before diving into the theory: are binary operations enough? Unary operations are really just object methods. For what i need them, can i assume i can construct multi-ary operations from binary ones? Entry: binary protocol Date: Wed Aug 30 02:30:37 CEST 2006 - a message is a single atom (can be composite) - an infinite stream is as a stream of messages - character streams (program text) are dealth with out-of-core - message protocol needs to be real-time (bounded time), which means predictable memory (length precedes message) Entry: lock-free Date: Wed Aug 30 03:39:46 CEST 2006 forgot to jump on that train! i think most of the machinery is already present in libpf/thread.c i lost the train of thought.. why go single threaded? to fix some of the problems with memory allocation, which is currently a bit of a mess. probably this lock-free approach is worthwhile to look at first. ok, the train was: to use poll/select in the core, i need to make the parser pre-emptive. the way i wanted to do that, is to make the core single threaded to make it cleaner, and have the parser be a thread that converts io to synchronous binary stream. so, a little survey on the occurence of locks in pf - packet reference count - atomic stack for memory allocation of list/atom structs - symbol hash - packet pool lock these are for true blocking operations, so need to be preserved - thread -> command_queue Entry: packet pool Date: Wed Aug 30 04:19:42 CEST 2006 the packet pool maps integers -> pointers. the original reason for this is to be able to check wether packet id's are valid, without having to use hash tables for this dereference them. in pf this is no longer the case. packet id's are nowhere accessible, so there's no reason for them not to be pointers. it's a pure PDP relic, and it should be possible to remove it, and replace it with a stack to keep track of all the packets for later ad-hoc 'GC' algos. i probably can't change the fact that a packet is represented by an int, since it's all over the code, so it might be wize to just leave this in, or byte the sour apple and go to packet_t and leave int behind. i probably need to fork to fix this one. thinking about it very briefly sends shivers down my spine. the assumption that a packet is an int is really everywhere. one for the list.. i really need a new core. Entry: yada yada redesign Date: Wed Aug 30 19:19:57 CEST 2006 seems very difficult to have a stable opinion about how to proceed with pf. one thing is clear, the core needs to go and moved to a simpler forth base, with memory allocation and multimethod dispatch clearly identified as separate modules. but what with the plugins? in all, the core is not too hard to rewrite, and can probably be simplified 10x, since it's so convoluted at the moment due to legacy PDP stuff, but the plugins.. it's going to be a long way to port all functionality to the new poke core once it works. so, i need to get over it :) i can't change PF's design. it's convoluted. but i can try to keep it going, keeping my eyes wide shut. Entry: postponed Date: Fri Sep 1 02:47:46 CEST 2006 looks like pf dev can be postponed for now. the dispatcher (select wrapper main loop) has to wait till next time. need to tackle cat first, norway is nearing. Entry: packaging Date: Fri Sep 1 20:11:25 CEST 2006 but but but. i need to find out how to make debian packages, so i'm going to do this for pf first. what is needed is: - tarball snapshot from darcs archive at goto10 (bin/snapshot) - debian source package - debian dest packege - zwizwa apt mirror - darcs patch howto so, to create a snapshot from the darcs archive do: bin/snapshot or, if you want to add a date stamp in the name, do bin/snapshot -d Entry: debian/ubuntu Date: Tue Sep 5 16:59:50 CEST 2006 what's the difference between xlibsmesa-gl (my debian) and libgl1-mesa (thomas' ubuntu) ? looks like the latter is testing, which is what ubuntu uses.. Entry: polling & listen Date: Wed Sep 20 19:14:43 CEST 2006 seems like the easiest way to do the polling is to mimick select. in ((stream xt) (stream xt)) out: list with active streams seems to work. now it's time to rewrite the polling loop using this construct. first i need a dictionary, or change the polling function so it takes a payload like above, currently it's just a list of streams that gets split into ready / notready. but, there's another problem: socket listeners for tcp and unix sockets behave different than pipes. they are actually pipe constructors: sombody connects, and it splits of a pipe fd. so the question is: how is this going to be handled in pf? it's probably a good idea to keep the 'listen' and 'poll' mechanisms separate. Entry: server/client Date: Sat Sep 30 07:00:09 CEST 2006 fixed some things for server/client yesterday. looks ok. giving up on automatic binary packet saving over text pipes. Entry: interrupts & abort Date: Sun Oct 8 22:09:07 CEST 2006 added interrupt checking, mainly for ctrl-C, possibly some timer/task stuff later. though this made me think about why i am using abort? nothing but trouble.. Entry: writing packets Date: Sat Oct 28 09:38:00 EDT 2006 a smarter parser? it's terribly convenient to be able to write out raw packets. it might introduce endianness problems etc, but currently there's no way to quick hack saving a CA for example. so it needs to go in. what's needed to support this is an escape code for the parser. previously this was just symbolic PF code which would enable a file to be 'load'ed instead of 'read'. that's not enough. it's more flexible, yes, but really not safe and it doesn't work for packets in lists. if you want something like that, it's still easily coded in PF. so, what's going to be the escape character? something that's not used yet. plenty of room since PF takes only a subset of ASCII atm. NUL is a good candidate, since the stream will contain binary data anyway, NUL won't be so special. another possibility is EOF. interference? need to check later. something which i should have done a long while ago is to make stream parsing a vm method, and not a stream method. this mainly allows for better error handling. things to fix: * emacs? * interrupted system call error message * check read/write for chunks length etc.. * check if fread can return short * check all blocking points Entry: pipes and deadlock Date: Sat Oct 28 18:28:11 EDT 2006 problem with multiprocessing: buffers need to be big enough to avoid deadlock if 2 processes are trying to send to each other at the same time. ways to solve this: 1) thread decouple reads 2) thread decouple writes 3) software flow control 4) write while waiting is this an occasion to get rid of all threads? 3) merely avoids the problem. 1) and 2) use threads and can be quite complicated. it seems number 4) is the easiest to get right: handle writing at wait points, which means, select and read. this does require some form of context saving for each postponed write, so i do wonder if it's not easier to just use threads, since i use libc ops everywhere: can't pre-empt them without writing to a memory buffer first. there is no need for a solution as long as the connectivity graph is acyclic, which happens in quite a lot of applications that come to mind, so maybe i just need a special 'write-atom-buffered' word which buffers exactly one atom? this will still give proper control flow (it will block on the 2nd atom if the first one is never read) so, i cannot give up atom based control flow, meaning: - buffer size is counted in atoms and can to be specified, default = 1 - implementation: one thread per stream, or writer thread? when i use the stdio streams, i need threads. the only way to work around this is to print to a string buffer, and have some write struct deal with the data later. so it's either A) one thread per output stream B) serialize to flat memory + use select based global write state machine. i already have a mechanism to write to strings. this could be used to go for option B. so it's going to be the 'serialize' word, together with write-buffer. ... writing this code, i'm thinking it might be a lot simpler to have only one select word, as the central point of time keeping and I/O scheduling. ... ok. select based stuff seems to work.. now the problem is, read might still block!! suppose A and B nodes are sending each other. if they start sending at the same time, they might both be tricked into a stdlib read, which will prevent further sending because that read can block. so the moral of the story is: - select-input only garantees there is data - this will block (for a long time) depends on wether the sender is blocked. what i need is to explicitly use nonblocking I/O and perform a pf_poll_output() before a retry call after EAGAIN is encountered. Entry: stream actions Date: Sun Oct 29 09:10:48 EST 2006 basicly the choice is - modify select to take a ((stream action) (stream action) ...) list - add a simple way to do this mapping (association lists) i think association lists are more general, so i should go for that. however, the first one is more efficient, since it doesn't involve any searching (binding). Entry: console fix again Date: Sun Oct 29 19:10:40 EST 2006 moving from raw data pipe -> command queue and string communication. seems to work now. Entry: select fixups Date: Mon Oct 30 08:53:48 EST 2006 code got a bit messy.. things to do fix: - emacs (reader plugin) - sleep time delay -> no output poll during sleep now = wrong - make all fd's non-blocking by default and iron out bugs from there - identify all points that need AGAIN handling + provide better polling routine Entry: write and stdlib streams Date: Mon Oct 30 13:05:43 EST 2006 maybe i should completely get rid of all stlib streams since mixing both fd I/O and stream I/O gets very confusing. i can't guarantee much if things pass through different gates.. so: - stream_read/write have the same semantics as system read/write - get rid of all stream references - printf prints to string first - get rid of putbackchar (ungetc) something about putbackchar.. the parser really needs this, since ( " and # are word terminators.. this means i need a real buffer. what about making file streams a 'subclass' of string streams? -> bad idea: it's not the same thing.. buffering.. how to do this -> check code.. seems to work. still need to fix the packet read/write code -> ok so the final word is: - input is buffered for small reads ( < 512 ) - output is only buffered for 'write-atom-buffered' - all output is tied to VM fixes: * sometimes an extra xxx sleep is necessary to send out a packet?? -> done * check stream.h for cruft -> done * slurp on program output needs to read until EOF -> done looks like i'm done. nope. still to do: - emacs (reader plugin) - sleep time delay -> no output poll during sleep now = wrong fix sleep time first. it's not too hard. -> done for the emacs thing, i'm going to open another entry. Entry: parser and pre-emption Date: Tue Oct 31 12:51:12 EST 2006 if i'm to take this threadless design seriously, i need to get rid of blocking parser. maybe i should take advantage of the pf_vm_idle() function that's used everywhere? what about cooperative threads in C? is there a safe way to switch tasks to different stacks? what about converting all the functions to forth words as a first step to have both an extensible parser and a pre-emptable one? that would be rather intrusive and probably have annoying bootstrap problems too.. i have one that works atm, and is relatively fast. making it pre-emptable would be a nice exercise to do the same with other code written in C. so, conclusion: i need a way to switch the C stack. Entry: shut up and code Date: Fri Nov 3 10:11:52 EST 2006 i just finished a pre-emptable version of the parser. didn't find any way to easily do the C stack switching, so i just made a single function with a return stack as pf list. looks even simpler than it was before. FIXME: raw packet reading still uses blocking (iterated poll + sleep) I/0. this might be incorporated into the pre-emptive parser later. allright. so now it's really possible to get rid of all thread code. guilty parties are: - console : readline/emacs can be unified now - osc : i guess udp 'streams' are easily added let's try console first. maybe i should re-define a console? so it can easily be attached later? maybe i should write the console in lisp/python whatever? the thing which is important here is flow control: the console needs to know when the prevous input was successful. maybe i should distinguish two modes of operation: - bidirectional: console sends message + waits for reply - unidirectional: console sends raw text - in-band-hack: unidirectional + escape code like 'emacs: ' the second one is trivial, and already present. the first one needs flow control, so it's best to avoid ad-hoc protocols and just use the pf text protocol: means, the other end is pf, and the communication is one atom at a time. actually, the first one is pretty trivial also: open a pipe, start sending/receiving atoms. so what am i whining about? both readline and emacs do not fit this: they provide raw text, and require in-band data. maybe i should try the emacs thing first and model readline after that, using a special purpose hack: make pf talk back in their own language. ok.. emacs.pf works without reader.c now. sweet. probably best to go after osc next: an opportunity to factor it into udp 'streams' and a plain osc parser which operates on streams/strings. still to fix: - hex digit parsing -> can't save binary buffers quoted as strings now (done) - osc-receiver -> no blocking receive atm so the command console is the only thing that uses threads atm. all the rest is thread free.. it would be nice to take advantage of the occasion to make the readline console behave a bit better. alright.. Entry: readline console Date: Fri Nov 3 16:03:37 EST 2006 the black sheep. it sucks. can we do anything about it? the question is, do we do line based, or multiple line io? in the latter case, command completion won't work inside a list. what about using 2 pipes? one for query, and one for continuous I/O? alright.. what about this: - readline in a thread, like before - communication over 2 pipes: zero terminated strings the wordlists comes as a space separated zero terminated string i need to get this over with so i can start doing some interesting stuff again.. forgive me, what where you doing? trying to purge pf of threads. and maybe rewrite malloc. kidding. of course, i know what the real solution is going to look like: each pf session opens a unix socket to which multiple consoles can connect. for interactive (readline and emacs) use, 2 connections are needed: one for pre-emptable user input, and one for command completion and maybe other queries: both channels are essentially independent. haha. i thought i was done :) there's the interpreter state, which is global, not stream bound. this is easily solved though: allow only one 'human' console, which owns the compiler, and several isolated message based command pipes. or use only one-shot interpretation (find a single sybol) instead of list-interpret. Entry: thread changes Date: Sat Nov 4 17:43:56 EST 2006 ok, so the central reason is of course: i can't make the VM multithreaded using posix threads. my core data structures are not intended for that use. there is essentially no way back, because i use raw access everywhere. using threads anyway complicates matters since it requires C code to export functionality that deals with the locking/atomicity problems outside of PF. switching the entire design to non-blocking IO makes it possible to use cooperative PF threads to do essentially the same thing, using more forth code instead of tedious special purpose C. that's one. two is, that if i ban pthreads completely, i gain access to more platforms, and a possibility to run it standalone, or under simple operating systems: select,read,write and malloc are not so incredibly hard to implement. i don't really need pthreads, so the whole system becomes simpler to use and easier to interface with other components. - i only use it because it's there and because some libraries like readline do not provide non-blocking IO. threads can be convenient, but atomicity problems in my current implementation make it very awkward to used. - the blocking parser is gone, and i use buffered output: these were the main reasons to use threads. - other blocking operations can now be easily coded in forth without awkward atomicity issues: i can then essentially get the same convenience of using OS threads. in the thread-less version, all memory allocation and reuse (atoms, packets, ...) can be cleaned up. this is a biggy, but should make code a lot simpler. Entry: we have a winner Date: Mon Nov 6 19:47:42 EST 2006 this qualifies as the worst PF core bug since ages. i had this: : >abort current-abort @ abort-stack push current-abort ! ; : abort> current-abort @ abort-stack pop current-abort ! ; but it should be this: : >abort current-abort @> abort-stack push current-abort ! ; : abort> current-abort @> abort-stack pop current-abort ! ; the thing which failed was a for-each in a return stack that was put '>abort', meaning, the pointer got copied, but the original list to which it pointed got erased. i don't think i can do much about that: pointers are poison.. i tend to forget they can really cause problems, since i never used them directly, only in lowlevel words, where they still cause problems. Entry: console protocol Date: Tue Nov 7 12:15:10 EST 2006 so.. a console. some synchronization is necessary, if only for displaying the readline prompt. i was thinking about ascii only, single line protocol. it should be something that's readable by any program, or scanf. ok, i'm tired of this shit: - only ascii 32 - 127 - control char = newline : send line, receive line using 'sanitize' and read-line this can be enforced. all binary data needs to be quoted: the daemon's console code can be instructed to process the output resulting from the interpretation of a command line using arbitrary commands. the result of this is that the basic protocols for communicating with PF are: - strict line based console message channel using only ascii : all non-ascii characters are set to spaces. this is mainly for apps that don't speak PF. if binary data is needed, the pf quoted strings are easily implemented. - text/binary native PF atom read/write channel to connect multiple PF instances and load/save native PF data. - raw packet streams: connect to anything that produces chunks of raw data in a known format, i.e. raw video pipes. got the readline part working too now. TODO: - fork from pf (done) - it's possible to crash pf with ": xxx 1 2 3" - fix backtrace (+- done) - unix pipes: remove file on close (done) - pf_console build (done) - de-threadification (+- done : leaving relics behind) - sometimes 'p' doesnt print in new console.. (fixed) - hotkey mode? Entry: de-threading Date: Sat Nov 11 12:01:01 EST 2006 so, i have a dilemma. i waited for a couple of days to think, but i'm still not sure. i went to a lot of trouble to make pf thread-free, but do i want to change things that are unfrendly to threads in general? let's see.. now, in the code there are several locks. should i take them out? the packet allocation code could be cleaned up a whole lot, and unified with stream stuff and general purpose object ordiented things.. everything should become simpler when the locks are removed. i'm convinced i don't need threads, so there's really no reason to keep the locked code around. alright, i'm tagging the code now, and will put a forked archive on ataraxia. ok, here we go. the first thing i run into is the reuse mechanism: i should make lists and atoms use the same reuse mechanism as other packets : tied to symbols. so, what is data in pf? it is either owned by a VM, or it is owned by a re-use list tied to a symbol. So the root of the data tree is the symbol hash together with a list of all VMs. Symbols are 'global' in the C sense. They are properties of a process, not a VM. Atoms and lists are owned by the 'symbol' and 'list' symbols. summary: - all locks removed - toplevel data tree = symbol hash + VMs - symbols never get freed - other things never get freed unless it's asked for explicitly - fast allocator removed, use reuse stack - list/atom reuse mechanism unified with packet reuse mechanism (symbol.c) - unify packet id with it's pointer in memory pf_packet_header() -> identity() - a packet is something with a reference count - join all global variables in one struct for cache opti - atom/list reuse and packet reuse are not the same due to refcounts - strings are just wrappers for malloc : they are not really deterministic check if this is a problems probably best to do make packet ID == packet pointer at this moment, since it would clean up a lot of cruft. going to make a safety commit though. probably i need to keep two toplevel free lists for lists and atoms, to not interfere with them being used to store packets. too lazy and hungry atm to figure out if it's possible to do without. maybe i should save all global variables that are accessed a lot close to each other for cache opti? got it to compile, however there's something wrong with the packet management: fixed, refcount problem. retired strings have their buffers stripped. what this means is: strings are really just a wrapper around malloc, meaning, they are not really reusing their data. so if you want predictable performance, you cannot use strings. there's really no way around this, since hreusing the buffers could lead to huge memory inefficiencies. looks like it's working now.. PF is officially tread-free :) Entry: more things to clean up Date: Sat Nov 11 18:03:41 EST 2006 - SEGV and other signal handling using setjmp. - print/write semantics about print/write semantics of strings. - 'write' followed by 'read' is the identity for serializable atoms - 'print' is human readable description of an object - 'save' dumps in raw/binary/... format print for strings at the moment is neither of those, it's dequoting, or a form of interpretation. so i need a new word. maybe have a look at how it's called in other languages... in scheme it's 'display', 'write' and 'print'. probably i need 'save' just for buffers. so, 'save' basicly means: dump in raw/binary/whatever form. currently, only implemented for strings, but might be implemented for other things too. FIXME: save-string etc.. maybe call it save-atom? doesn't work. "abc" print -> abc so not 'save'... sucks somethings not right here.. let's try again - 'write' is a machine and +- human-readable format for serialization - 'print' is dequote: go from internal representation to raw result - 'report' is a human readable summary, or 'write' if that makes sense the core of the problem is of course that i want "abc" p -> abc ((1 2 3>) >matrix p -> 1.000 2.000 3.000 means i mix print/report semantics as described above its quite a job to clean this up probably.. so in this sense, printing a list doesnt make sense.. argh.. maybe i should reserve the '.' word for this mixed semantics? ok, that's how it's done now.. 'p' is just an alias for '.' with : . string? if print else report then 32 emit ;; Entry: glx stuff Date: Sat Nov 11 23:38:52 EST 2006 closing stuff down shows bugs in glx stuff. but maybe i should fix this first: ERROR: line 114, Function intelInitDriver, File intel_screen.c libGL error: InitDriver failed libGL error: reverting to (slow) indirect rendering that's fixed.. now i need to poke around in the X11 plugin a bit. SDL seems to work fine, but x11.pfo won't load.. probably pthread stuff. yep. ldd libGL.so: linux-gate.so.1 => (0xffffe000) libX11.so.6 => /usr/lib/libX11.so.6 (0xb7e1a000) libXext.so.6 => /usr/lib/libXext.so.6 (0xb7e0c000) libXxf86vm.so.1 => /usr/lib/libXxf86vm.so.1 (0xb7e07000) libm.so.6 => /lib/tls/libm.so.6 (0xb7de1000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7dcf000) libdl.so.2 => /lib/tls/libdl.so.2 (0xb7dcb000) libdrm.so.2 => /usr/lib/libdrm.so.2 (0xb7dc3000) libc.so.6 => /lib/tls/libc.so.6 (0xb7c8b000) libXau.so.6 => /usr/lib/libXau.so.6 (0xb7c88000) libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0xb7c83000) /lib/ld-linux.so.2 (0x80000000) why is this? possible to do without? ok, got it: > "/home/tom/build/libpf/plugins/x11/x11.pfo" loadplugin :1: e_inval: can't load plugin: /home/tom/build/libpf/plugins/x11/x11.pfo: /home/tom/build/libpf/plugins/x11/x11.pfo: undefined symbol: glXGetVideoSyncSGI took it out.. is there a way to put that back in? so no reason to panic. Entry: loading plugins Date: Sun Nov 12 11:26:11 EST 2006 - resolve primitive names using dladdr ? - a library is a packet: store it in dict (done) - SEGV handling -> not necessary: bug is bug - better error messages the old a_extension things are still in there.. used in some code. Entry: video Date: Sun Nov 12 18:18:06 EST 2006 maybe more pressing: fix raw video / pf atom format / libquicktime / mencoder ... so they behave similarly. Entry: but first: 'binders' Date: Mon Nov 13 13:54:11 EST 2006 let's refresh the concept: a binder is a parsing word which interprets the next word in a different context. this is really just a namespace. so i should add a parsing word: : create-binder mark interpret-list mark>binder ; Entry: weird bug in console interpreter Date: Mon Nov 13 19:10:01 EST 2006 any error that's throw from within the console interpreter gives something like this: > e_undef throw :1: e_type: invalid argument type > .bt :1: e_type: invalid argument type error: # xt: # DS: <2> # # RS: <2> -># -># that's on first run. on subsequent runs it's in , so the error is probably stale. ok. got it. stale error message: now run clear before i'm interested in an error message. maybe i'm getting carried away, but it might actually be possible to fix - exception handling (throw) - backtrace handling - error reporting exceptions work fine, but the error reporting and backtrace mechanism are a pain. what's needed is something that goes like: mark do stuff if something went wrong, get the backtrace and error message let's separate 2 cases * low level error occurs, and we want to know the entire backtrace up to the user or file command that eventualle led to the error. * low level error occurs, we want to really recover from it. most of the time: throw/catch are used to restore global state = keep dynamic binding consistent. what i'm interested in is a deep stack trace where all the states are saved. i'm talking out of my ass. this is way too complicated. maybe i need an explicit 'dynamic-wind'. now, dynamic wind is really 'abort' with shielding of the return stack. the functionality provided is exactly that: provide a consistent exit point, and make sure the return stack cannot be messed up. now, backtraces don't make much sense outside these contexts, since an RS is swapped, anything can happen. so i guess my semantics of 'clear-error' is not really so bad: recover will clean up the stacks, but won't actually delete any data. clear-error will. Entry: clean up readme : the stuff that nobody reads Date: Sun Nov 19 12:57:01 EST 2006 What is this? First of all, it's actually an application, but it's packaged as a library to ease integration in other systems. Currently there are 2 operational interfaces: as a unix command and as a puredata object. Second of all, it's actually a language. The application core is written in C, and can be used in programs that have a C language binding. Most of the core functions are available as primitives in a language based on Forth. These primitives can be used to build things related to media processing. Where does it come from? This project is a second incarnation of a media processing library called pf, which is an extension for the computer music program Pure Data. Interfaces. Since one of the goals of pf is to serve as a 'media glue language' there are a lot of ways to connect to other programs or objects. The pf object for Pure Data connects Packet Forth with original pf, pidp and 0.12.x pf protocol compatible objects. Most of this works, or at least should work. You can help by filing a bug report to tom@zwizwa.be or the pure data mailing list if you find a broken or missing feature in the pf version of pf. The unix command acts as an interactive interpreter to the forth dialect. When connected to a terminal it lauches a readline enabled command line console. The line editor runs in a separate thread, and is polled by the worker thread, which runs the user definable 'schedule' and 'sync' functions. If the input is not a terminal the readline part is skipped and the stream is parsed in blocking mode, just like would be done when the argument is a file. This way you can send forth commands to a pf instance, i.e. from perl or an external gui. Packet forth supports streams. These can be files, sockets, fifos or the input/output of inferior processes. Load and save of packet data in the native format is supported for all streams. yuv4mpeg is supported, so you can use it in conjuction with programs that support this (veejay, mjpegtools, mplayer and some smaller utilities.) Pf can be mapped to guile. This effectively enables all the powers of pf combined with scheme syntax, garbage collection and all kinds of sillyness. Plugins Most library dependencies are wrapped in plugins to decouple them from the core. See the file PLUGINS for more information. Why the fork? I don't really have an explanation for this. But i enjoyed the year of turbulence which has brought it to what it is now. And remember.. there is no fork. Why the forth? Quirky as it is, it stimulates the brain in different, and even orthogonal pleasure dimensions. It is the ultimate in one word hacking, and man, have you ever got lost in a 3 line program? Why not an existing scripting language? Because those that i considered (python and scheme) were too heavyweight and did not fit some requirements. The most important reason is of course lack of control about internal workings. Forking an existing language is more complicated than writing a forth from scratch. Other languages will eventually be supported using direct api mappings. I didn't even remotely think of considering perl as a core language, but it does work very nice together with perl just doing 'open PF, "|pf";', or in conjunction with sockets. This would be the way to connect an external gui to pf. GC Packet forth does not have a garbage collector, for the simple reason that it is not necessary. Packets are managed using reference counts, and because they cannot refer to other packets themselves, this is sufficient. Add to that the fact that a real-time garbage collector is not an easy thing to write, i've opted to leave it out entirely. Packets use reference counts and copy on write. All other atoms and containers (lists) are copied. Abstract objects (i.e. make-xv) need explicit freeing. Most objects are for io and need explicit resource cleanup anyway. Note that memory allocated for packets does not get freed. Instead all packets use a FIFO reuse stack per unique type symbol to ensure the memory is 'as hot as possible'. For normal operation this is not a problem, but if you start creating a whole lot of packets of different types (i.e. loading different image sizes) you need to manually clear the type cache when you are done using a type. This is not 'intended use' of pf and actually considered a good thing, because it keeps things simple and predictable. Pf does not solve all your problems :) Portability & Requirements I set out to make libpf as portable as possible, but unfortunately i had to drop this constraint to keep it small and simple. At the moment, GNU libc is required, and gcc-3.x So compiling on a GNU or similar system other than GNU/Linux should work with relatively minor effort (OSX/fink is one of the intended targets), as is MSW/cygwin. Porting to a non-GNU system might be quite an ifdef ride though. Security I did my best to eliminate the most obvious buffer overflows, but i can't guarante anything. Remember that almost all data files are actually executable, (which can be worked around), and pf can execute arbitrary programs and directly modify memory and disk when running as root. Sending forth code over the network is a very powerful mechanism, but incredibly insecure. If you intend to use pf as a server or as part of a server exposed to the internet or another non-trusted network, be careful. It is very promiscuous. Entry: scheduler Date: Tue Nov 21 13:01:25 EST 2006 it really shouldn't be so hard to make output tasks = anything that's waiting for a file descriptor = OS I/O. what form would this take? currently i have things stored as 'stream + stuff list'. the necessary information is this: * stream * input/output/error * task (DS RS) there's one problem atm: blocking reads/writes can not be handled in C code, unless i use some kind of setjmp method to jump back to the interpreter. best of all is to avoid the C stack entirely and do the scheduling incrementally. so, the scheduler should be * round-robin, in case it's not tied to an OS stream * use select in case it is. currently pf_vm_idle() is used in - PF_PRIMITIVE(pf_word_read_thing) (called by ..read_atom) - pf_vm_stream_write() - pf_vm_stream_read() blocking I/O is used when writing directly to a stream. the blocking loop itself is handled in C: pf_vm_idle() can eventually be called from pretty deep in the C stack, so I don't want to do any stuff there that can make it blow up. what i want is 'blocking I/O on the forth level' which means: do a non-blocking OS read/write, and save the task in a list whenever it's not done. during C level blocking I/O, only buffered packet write can be used. hmm.. that's clumsy. what i need is to exactly know when the scheduler pf_vm_idle() is invoked. this should not happen during printing.. another question: what can be done when blocking I/O is used in 'single thread mode'. just like the parser, the printer should be pre-emptable too. the easiest way to do that is to always print to string first, then to a pre-emptable single write. so, i just wrote this in print.c: there are 2 regimes here.. * printing to string will never block.. use this in conjunction with non-blocking write to have non-blocking print. * printing directly to a non-string stream can block. the idle time is filled up with low level tasks (see select.c) the thing is, i don't want to give up the latter part. or at least, i don't want to really thing about how to do that. maybe it's better to go for it anyway, write all the 'single threaded' blocking print code explicitly as locally looped code. basicly, i need to rewrite printf everywhere again.. pff.. again a large chunk of work. but it should be better when it's done: printing can be done in well-deligned chunks, and non-blocking IO handling then only works with simple write semantics, instead of nested printfs. OK. first part is done. now the blocking points are nicely separated out, but i still need to write them. RESTART CURRENT that's what i need to do proper task switching: need to restart the current execution token. this means vm->current to data stack and call scheduler. seems to work out for writing (without testing) added explicit schedule support to pf_vm_t/forth.c need to do the same for reading of course. adding chunk read support to the parser. OK that seems to work now, on to writing. seems everything's in place now.. check select.c again ok, now the poller C code could be replaced by forth code. the scheduler could be a true select scheduler + a bag of things to poll once ever X ms. e_yield reused e_yield to signal rescheduling event to the interpreter loop, instead of a pf_vm_schedule() function, to make sure the current function will exit. there are only a couple of blocking points atm: read/write and sleep. Entry: ready for scheduler Date: Wed Nov 22 20:37:40 EST 2006 ok. it was a bit less trivial than i thought, but i guess most of it is in place now. time to rewrite the buffer output code to forth code, so i can do proper spawining/killing/... of tasks. the current buffer writing tasks could be simply a single XT that's restarted, basicly doing a 'output-buffered' operation, followed by a kill-task. simple enough. then i just need a mechanism to spawn (add it to the list) and to wake up on I/O. tasks can wait for OS fd's, or for sync variables, which can be just empty (undef) to indicate a waiting condition, so a task structure is somthing like: thread = (channel DS RS) active lowlevel I/O tasks can be identified by select, while higlevel forth I/O could be identified by a single variable. Entry: abort Date: Mon Nov 27 14:14:50 EST 2006 need to re-think abort. basicly, it's a 'global' thing. there are 2 kinds of error handlers: - task-local: saved on the return stack - global: handled by the scheduler before i can make tasks work, i need to figure out how to do local error handling. the only place that has abort atm is interpretation. the reason for this is simple: swapping/killing the return stack during interpretation is problematic. the real problem here is global resources. and this opens up a can of worms: dynamicly bound variables should be task-local. the two biggest problems are 'abort' and 'accept'. i need to have support for parsing words in the interpreter. changed 'accept' to 'read' which is a bit cleaner. what about this: implement dynamic barriers properly using * rewrite exceptions so they map RS -> (RS) * add task switching words that respect this structure this gets rid of the e-marker stuff. so.. again... why do i want this rewinding business? to make exceptions a bit easier to handle. and cleaner to implement. the thing is, it would be nice to know exactly how many data atoms are consume when a try block is entered: there is no way to separate the data stack like the return stack can be separated. that makes it hard to use.. tough luck. so, an exception handler consists of 3 nodes: try: wrap return stack + save data stack size register error handler cleanup: OK -> clean up error handler + clean up stack shielding handle: restore wrapped stacks the real problem i'm trying to solve is this: - when an error occurs: * save current context -> backtrace * execute error handler this is really the following: - a try block spawns a task - when the task returns an error maybe i should do the spawning first. this has an explicit 'n' argument for the number of data atoms to detach. : detach ( xt n -- task ) ; if this mechanism is used for exceptions: spawning a task while waiting for it's completion - backtrace is easily implemented: it's a task's state. i'm random walking again... conclusion: winding a return stack doesn't really help much implementation-wize. i think keeping things as they are: abort and 'current' I/O are task bound, is ok for now. so, let's do this one at a time: Entry: implementing copy-stream-buffered Date: Mon Nov 27 17:27:18 EST 2006 problem 1: blocking io now uses a restarted XT. that's not compatible with tasks. i need to re-implement blocking I/O from the start. not to confuse with buffered I/O. so todo: - make blocking I/O go through the scheduler - implement buffered I/O in the scheduler solution: implement write-atom-buffered as a task. seems to work now. todo: * add a proper 3 way select to the scheduler * implement sleep using events: synchronize to some i/o Entry: wait Date: Tue Nov 28 18:21:40 EST 2006 in order to complete the scheduler, a task needs to specify what it's actually waiting for. there are 4 cases STREAM QUEUE INPUT if iq OUTPUT of oq where a FILE is an OS event, and a QUEUE is an internal task communication mechanism (that can be added later). these best behave in a similar manner, so inter-task communication and inter-process communication behaves the same. (an internal queue is just a listvar: pointer to a list/queue) blocking events are handled by placing the task + the stream/listvar in a wait list. i currently have 2 points: of -> calls schedule word explicitly (which returns e_yield) if -> primitive returns e_yield going to have a look at graham's "on lisp". it has a chapter about this stuff, which i skimmed over when i read it. ok.. no reading, just thinking. it seems to work now. still need to test a bit. Entry: abort Date: Wed Nov 29 09:47:44 EST 2006 abort is for console only. if a program does not have an attached console, abort really doesn't make any sense. so, what i need to do is to replace the abort handler in the stream loading code with an exception handler, and move the abort handler to the console code. error during console command interpretation -> caught by exception error during any other task -> kill tasks ok, streams now use exceptions. that's better. let's see for abort. ok, that seems to work with a remaining bug for the error reporting. Entry: task local dynamic variables Date: Wed Nov 29 10:17:44 EST 2006 now the tough nut: dynamic variables and tasks. one way to do that is to store a dictionary in the return stack. Entry: error reporting and backtrace handling in task context Date: Wed Nov 29 10:29:35 EST 2006 now, what should a backtrace be? probably a restartable task. unfortunately, that's not always possible.. but, maybe it's a good idea to try to make it always possible. vm->current needs to be valid at all times, so restarting is possible. the primitive doesn't really need to be saved. the xt is more important. so what about making backtrace capture explicit? in other words: if you want debugging, you need to start a word in a different task. that would take all the backtrace stuff out of the core. the next problem is 'clear-error' and the way errors are registered. ok.. currently i think it's sort of ok. the ad-hoc error message stuff is moved to a separate word, and i have a restartable backtrace using tasks, with all internal backtrace code removed (throw will just drop stuff). Entry: interrupt Date: Wed Nov 29 14:20:19 EST 2006 there are still problems with interrupt. if it occurs when the scheduler is running, the only thing i can think of is to completely reset the interpreter by calling warm. however, this discards all I/O and messes up consoles. maybe i should just restart a console then? when do you need interrupt? if something gets stuck in a loop. otherwize it doesn't really do much. running into some weird bugs.. need to clean up again. what about taking out 'tick' and writing it as tasks? ok.. the thing is: the signal handler sets an interrupt flag, which is then translated to an e_interrupt in the inner interpreter. individual functions may not return e_interrupt, since this will mess up the masking. e_interrupt is only for interrupting forth code, but it should be possible to mask it. i need to start over with the whole interrupt thing.. it's broken. the problem is: the console receives the interrupt, but the daemon receives it too, since they have the same controlling tty. i think i'm just going to give up on that for a sec: just don't make infinite loops. Entry: task local variables Date: Thu Nov 30 11:16:31 EST 2006 necessary for things like current input/output. really important when multiple consoles are attached. i need to think about this.. it's easy if a certain variable is 'globally thread-local', but not if there's a possibility to shadow definitions. as a matter of fact, this could be used to implement objects too, since it's just local namespaces. what about this: :dyn ...... ; # create dynamic word dyn # interpret dynamic word ` interpret-dynamic the definitions are simply dictionary links that are searched on the return stack. the trade-off seems to be - dynamic lookup + fast task switching - static lookup (compiled) + slow task switching (safe/restore dynamic context) seems the first one is best, since i mostly need it for the interpreter 'current' io. this is a good opportunity to unify dictionary lookup and generic assoc list lookup. IMPORTANT: so, what's needed here is to look into some binding strategies: * dynamic * static at task compile time * other? -> virtual member tables? Entry: redesign Date: Thu Nov 30 17:33:59 EST 2006 on august 9 i had a severe case of coder depression, thinking bad things about pf. see where we are now with last month's spurt. done / no longer apply / not so bad - pre-emptive parser (separate process?) / lisp compatible? - lock free operation in core - well thought out serialization: message based comm + persistent state - proper type/object system (better now) - are floats/ints/symbols different from packets? (yes) - write a parser in forth (how to bootstrap this?) - don't ever use ints again to represent objects! - don't ever use 2 type descriptions again! - proper pairs and lisp dot syntax? (no -> requires GC) still a problem : can't fix this - operate without libc malloc? / malloc once? - multimethod dispatch (how to encode types and use that?) - what about type symbols vs type lists? - bytecode should not be lists but arrays - any reason why the stack should be a list? - need call threading for proper tail recursion (push bit) - proper exceptions (continuations, dynamic-wind) so.. dynamic wind i think i sort of fixed. the problem shifted a bit thanks to tasks: these need task local dynamic variables. i think blocking task switches is a bit overkill. Entry: on threading models Date: Thu Nov 30 20:16:56 EST 2006 there are only 3 of any importance (I) indirect threading (D) direct threading (S) subroutine threading i'm getting quite fond of the last one, because it has genuine tail calls: the compiler needs to decide between primitive / high level word at compile time, so can immediately decide on compiling a tail call to a highlevel word when required. for indirect threading, this can be messed up. alright. that's confusing. what i mean is: i can't have simple (automatic) tail calls in pf. when i would use a 3 way interpreter based on the already present type tags: - execute primitive - call highlevel word - jmp highlevel word which would eliminate codefields. i need to think about this some more.. can't really explain it well yet. one of the reasons for having threaded code as lists of addresses is to save space: there is no overhead for encoding meta-data inside the instruction stream, and the interpreter is very simple. however, in PF, there is already a lot of meta data in the instruction stream, so why not use it? Entry: things to mess up next Date: Mon Dec 4 10:42:22 EST 2006 * task local variables * change threading model -> automatic tail calls * internal task wait conditions * get rid of sentinel (too difficult since can't look back in list..) Entry: threading model Date: Mon Dec 4 10:55:51 EST 2006 why do i want to change the threading model to something like subroutine threading? -> return addresses and XT = same -> make normal lists executable -> easier tail calls: literal/execute and jump/call is determined by atom type: 'execute' bit. -> the general case: can only look back at the last atom: compilation using standard forth methods is problematic using the current code threading model and requires a lot of hacks (like the sentinel). other explanations: -> there already is a type system: this can easily be modified to provide 'executable' bits. -> good opportunity to clean up the interpreter guts. so, how would this work? -> ordinary atoms are transferred to the data stack -> primitives are executed if they contain an 'execute' bit -> atom pointers are called if they contain an 'execute' and 'call' bit, and are jumped to if they have the call bit zero. progress log * changed all types to LIT(type) which reserves two executable bits. * control flow words do not change, since they take an address argument * 'compile,' maps XT -> executable code let's see. i would like to do this gently: install all words to have both threading models, then switch from one to the other by redefining a couple of words. this is an interesting exercise: how to write a forth largely independent from the threading model, but keep it fast? in the pf case it might not be possible because not being able to look back in the code thread complicates things: changing the last atom in place is no problem, but changing jump targets is. maybe this is the real reason why i want to change the threading model. at this point, i cannot see the big picture, so i need to go one error at a time, breaking things and work from there. commit now. what i need is: atom -> CT XT -> CT CT -> compiled code (execute) now, in the new model, an XT is an atom pointer with CALL+EXECUTE bits enabled. interpreter: primitives that read from the instruction stream (vm->ip) or the word body (vm->current) could maybe be incorporated as opcodes? bottom line here again is to not implement similar mechanisms with different tools: there is already a type system, so all 'routing' should use this instead of a second approach (here: having things look ahead). half baked, hard to express. Entry: compiler rewrite attempt failed Date: Mon Dec 4 14:04:04 EST 2006 broke too much at once.. need to start over. what about just compressing some stuff.. there's too much code dup. just cleaning out some things now.. reminds me i need to fix dodoes, currently it's a bit awkward, using wide codefields. - fix dodoes - fix dictionary lookup (assoc list) hmm.. about create does> i don't really need a whole buffer as argument, a single variable for context is more than enough. so what about just ditching the does> and replacing it by something as : xxx { } undef bind ; where 'doer' takes an XT and binds it into a word, yep.. 'does>' needs to change since it's one of those forced things.. Entry: bad day Date: Tue Dec 5 10:59:36 EST 2006 yeah... reasons are probably external, but i had a bit of a bad day yesterday doing the stuff described above. reflecting on that a bit, i can see the following problems: PF does not use linear memory for threaded code. this gives problems when you have to look back/forward. nothing that can't be solved, and i think writing the interpreter such that there's a single atom per functional unit should fix it (this means, encoding instructions in the type descriptor bits). disentangling this however is quite a job, since forth.c is quite a mess. i must say that after a couple of weeks of fixing deep problems, i'm getting overconfident. but, i'm starting to believe again in PF being a solid base for other experimental work. i managed not to break any important things in the stream/scheduler rewrite, which is a good thing. gives me hope to iron out some more conceptual inconsistencies. Entry: backward edit Date: Thu Dec 7 12:49:37 EST 2006 i tried editing this file.. doesnt really work. let's try again. Entry: check travel notes Date: Wed Dec 13 03:23:15 CET 2006 the basic ideas where to use the type bits in atoms also for code interpretation: (1) use a 'return' bit embedded in the atom. the bit indicates that after executing the atom, the code following this atom will be ignored, and the interpreter returns to the previous context. (2) highlevel functions are just atom pointers: arbitrary lists can contain code. atom pointers can occur in 2 forms: quoted (as data) they are just loaded on the data stack or executable. (3) lowlevel functions are just void pointers. they can also have an executable bit. the first one really simplifies tail recursion. the other two will eliminate probably all words that look ahead in the stream. (is this really true? jumps: literal followed by conditional execution.) Entry: OSX build problems Date: Wed Dec 13 17:39:22 CET 2006 some things to figure out: * is /sw standard? how to do nonstandard * how to compile stuff from source linking it to /sw libs Entry: thinking about compilers again Date: Sun Jan 28 18:20:42 GMT 2007 maybe i do need to change the core machine language and the interpreter to prepare for proper linear lisp and to have PF ready to be a BROOD target. it might be wize to converge on the 3-instruction machine: PRIM/HIGHLEVEL/LIT + return bit. it's quite elegant. (HIGHLEVEL is tree structure, PRIM/LIT are leaf nodes) Entry: emacs mode Date: Tue Jan 30 18:50:25 GMT 2007 fixed most of it. still need to figure out some way to 1. find proper "standard" key bindings 2. figure out this meta/escape key for X11/terminal Entry: incremental improvements Date: Tue Jan 30 21:09:25 GMT 2007 instead of rewriting the core interpreter bottom up, it might be a good idea to keep the core as it is.. i went through a couple of cuts that would have benefitted from a bit more thought.. i'm not sure wether incremental rework actually takes more time.. maybe bug wize there's a gain. however, it does require a bit more planning.. so. towards a rewritten interpreter, following the lit/prim/word approach with return stack bit. -> eliminate most instruction lookahead: replace with a single conditional exec. -> eliminate most return stack tricks (not necessary?) -> separate out dictionary and error handling (error = packet?) that's probably enough already. woah. can of worms :) Entry: error handling Date: Tue Jan 30 21:19:32 GMT 2007 maybe the most important one is error handling.. that's really broken. the real problem is that everywhere in the code, a simple return value is used for error handling. what about turning these things into objects? there are 2 ways to generate an error: 1. return e_error 2. THROW(e_blabla, "message") this should be not so difficult, as long as nothing gets lost. a place where things could get lost is in using EXEC and ignoring the error. so. i just changed all EXEC occurances in libpf/libpf to either a TRY or a TAIL case. all of those should respect conservation of error. Entry: lambda the ultimate Date: Mon Feb 19 22:48:45 GMT 2007 ``singly-linked lists of cyclic doubly-linked lists with back-pointers to head nodes'' That's PF! :)) http://lambda-the-ultimate.org/node/2074 Entry: ideas Date: Thu Feb 22 01:29:07 GMT 2007 - errors need to be abstract objects (see above) - interpreter needs tail calls - interpreter needs minimal nb of quoting instructions Entry: what is PF? Date: Tue Feb 27 11:23:41 GMT 2007 another answer of the recurring question. forth + linear lists/queues + generic functions + simple type system to get to a fresh implementation of the core idea, i need to rewrite the whole shebang. however, it might be possible to separate out the stream processing and all the plugins, generating only the core from within BROOD. a brief look at the previous iteration tells me that it's still the same. most other comments are about implementation. an interesting remark is code in a linear language. currently i take the stance that compiled code cannot be duplicated, since compiled code contains pointers OUTSIDE of the normal linear list structure. basicly, it has circular references. distilling: circular references are ONLY allowed in compiled code, and compiled code is abstract. Entry: debian packages Date: Sat Apr 28 01:52:04 CEST 2007 to generate debian source packages, it's necessary to build a tarball first, since it needs a real version. the tarball's version is 'darcs'. also, the darcs tree is too raw. so: make snapshot tar zxf packetforth-0.13.20070428.tar.gz cd packetforth-0.13.20070428 make deb-src Entry: time for pf again Date: Sun Apr 22 14:30:10 CEST 2007 need to fix some things, but first i need to figure out how tasks work. most of the code is in scripts/scheduler.pf a task has independent return and data stacks. the udp stuff apparently returned just a single list. question: does it make sense to have udp connections respond to connection refused errors? (from ICMP port unreachable). ok, something for the list of rules: the interface in pf for reading messages is either: * a task which handles one at a time, and blocks when done * a function that returns the list of pending messages blocking the entire app is depricated, but i think should still be possible in terms of the other 2. faking blocking i/o: just yield if something's not ready. Entry: pf is not scheme Date: Sun Apr 22 17:32:44 CEST 2007 and i sort of regret that.. lexical scoping is important, and cons cells are important too. however, because i DONT use late binding, some other tricks are possible: global variables don't really cause name clashes when they are used 'close together'. i.e. variable accumulator : accumulate something () accumulator ! get-list { accumulator push } for-each accumulator @> ; it's safe to just re-use the name accumulator later on, which will shadow the already defined use, but that's ok, since the name served its purpose. tricks like these of course only work when the code will never be re-entered, so only cooperative multitasking is possible. of course, the real solution is to use lexical scoping, or even just dynamic scoping.. Entry: data structures Date: Mon Apr 23 22:47:17 CEST 2007 pf uses single ended deques. maybe that's a mistake, and maybe that's not even the right name.. the idea is that i only need stacks and queues, and both are implemented using the same structure. however, i've been working with lisp more lately, and regret i didn't use cons cells. but does that make sense? i'm working on a 'linear lisp' VM. a VM for a pf-like language with linear data structures (no shared storage). in PF, i don't really do that properly: i use pointers a lot. some things are important to me: * i need O(1) queues and stacks * preferrably using a single data structure however, in core PF i often find i need to go back too.. so i probably need doubly linked lists. funny that i started out not using them because they are 'inefficient'. maybe that should be the next core change? if i can't have cons cells, move to double lists. maybe linus is right? :) the second core change should be a garbage collector. i think i understand the problem better now: i know how to mostly avoid garbage collection using stacks and 'unobservables', but it's too convenient to miss out on temporary code, and i think the artificial compile mode / interpret mode distinction is too archaic. interestingly, trying to make accesses abstract separates the reading from the writing. i probably need to rewrite stack.c and list.c to use only a couple of muting primitives. that's where the core of the problem is. so, i fixed up list.c there's another one thats worse: stack.c i'm wondering if this really pays off. maybe it's easier to try to generate the code instead of refactoring it. there are so many special cases, i don't know where to begin.. and if it will ever end.. this is crazy if i can't automate it. it's probably better to automate it, and regenerate completely. that's going to be more interesting anyway, what i just did is slave labour. one question though: how come i keep falling for this trap? thinking i can refactor PF, take out the assignments and hardcoded bits.. it's a bit hard for me to accept i wrote a rigid piece of code. and even more that i knew i was doing this: for efficiency and simplicity. in fact, it is quite modular etc, but the data structures are pretty much fixed, as a result the core is not factored enough, and that's a thorn. disentangling this is probably too much work, but stays a challenge, since there's so much knowledge encoded already.. i know that the only real solution is to write a linear subset inside of a scheme language, possibly as a distributed system: linear core on appliances, with scheme on the cluster. Entry: problem with tasks Date: Fri May 4 12:39:26 CEST 2007 i don't have task-local variables or proper dynamic wind. a problem seems to be to print in daemon mode: print should not spawn an output task there! the real solution is: * proper dynamic-wind: any exit from a protected block should restore the dynamic state of a task. in this case, it is ok for a console task to block: it just needs to know where to send its input. so, how to do this? starting a task is: - saving all variables in a temp location - storing it's variables stopping is the same, but reversed, so i need an operation that swaps a list of variables, and store the saved list in a global variable. a note about swapping tasks: why not just concentrating the swapping on the return stack only, since i have other things to swap.. ? Entry: default I/O Date: Tue May 8 13:23:32 CEST 2007 now that task local I/O stacks are solved, it's time to fix 'read'. currently i don't really remember how it works, so that's a bad thing.. 'read' is a core word, and currently implemented as a deferred word. however, deferred words are global things, not task-local. currently 'read' refers to read-current-stream. the latter is currently also implemented as a deferred word, which is what needs to change: it needs to take the top of the input stack (a variable that can be task local) as definition. seems to work. Entry: debug output Date: Wed May 9 11:58:29 CEST 2007 the thing that's necessary: every word should store it's position, so errors can have source code information. one way to solve the problem is to make a hash of words -> source locations. another way is to just store it in the definitions, maybe after the docstring. Entry: errors Date: Wed May 9 15:43:11 CEST 2007 maybe it's better to just make errors to be the union of: = | ( "message" ...) and get rid of the 'errno' behaviour, because it is way to compilicated to use in a multi-task environment.. it should be possible to have 'or-throw' and 'throw' support this. let's give it a try. -- i cut out a whole lot now.. e_idle is broken. i don't understand why this is there any more, so i'm going to take it out, including the reset vector. -- ok, looks like i pulled it off. quite a cut, i took out a lot. the pf binary is just a wrapper now, and the command line options are disabled. next thing is the error printing for tasks. Entry: things that might change next Date: Wed May 9 23:34:43 CEST 2007 - get rid of XT cache - slim down forth.c : write more in forth - try to get to tail recursion & tagged words LIT/CALL/PASS, to eliminate multi-atom instructions Entry: tagged (colored) code Date: Wed May 9 23:38:20 CEST 2007 i started this already: there are 2 bits reserved in the type field of an cell. maybe i should think a bit about the 'essence' of an atom? the sane way: a cell can contain an object, the object is tagged. in PF the cell is tagged, but that's just how you look at it: objects are always in cells, so wether the bits belong to the object or to the cell, doesn't matter. it's just a poor choice a while ago.. so: * an OBJECT in C is a pf_word_type_t and a pf_word_t * a cell contains an object (owns it). there are no copies. * packets are ref-counted. this should be seen as an optimisation: they are also copied. Entry: indexing Date: Sat May 12 18:48:59 CEST 2007 in a linear language, data is owned by only one object. to my big surprise, this works really well. it is natural to 'move' and 'dup' data in forth, instead of implicitly storing a reference as in lisp. except for indexing. when you have a set of objects, and want to augment that set with an index (a dictionary), you're stuck. it doesn't work. there are no shortcuts. you need to include the index in the data structure.; this seems kind of strange. first i need to clarify something. of course, i am cheating. it's possible to use varaibles in PF, so it's not really linear: varaibles are just pointers, and there can be more than one reference. so why can't i use a data structure of pointers? because a variable is guaranteed to be static. generic pointers to list atoms are not. delete the list and the pointer becomes stale. before i go writing a new core of pf, i need to read a bit more about these things. i dont think i fully understand yet. Entry: raw video support Date: Tue May 15 11:32:25 CEST 2007 so, raw video.. the following problems: - colourspace there is no proper colourspace conversion in PF. there are enough libraries that handle this, so i'm not goint to be bothered by it. the rules are: stick to one colourspace. use RGB for opengl work, and some YUV format (i.e. i420) for video work. - meta data raw data is raw data. i need a standard way to tell PF how to open a raw file. the easiest way to do this is to save a 'read' word in a .pf file with the same name as the data file. Entry: synchronization variables Date: Tue May 15 13:43:17 CEST 2007 instead of blocking on streams, it should be possible to block on variables or queues. variables are probably not general enough, so let's block on queues. read: queue is empty -> yield write: just push works. was pretty easy... Entry: main sync Date: Tue May 15 14:23:09 CEST 2007 this is something fairly important. something external needs to give sync. fixed rtc. Entry: copy method Date: Tue May 15 19:23:15 CEST 2007 maybe it's a nice moment to take out the packet copy method, since it's not really used anywhere.. at least i think so. oops. can't take it out. 'reserve' uses it. so i'm going to change the default to non-copyable, unless there is a copy method, and add a copy method to the packets that need one. Entry: the big change: packet from int to abstract Date: Tue May 15 21:17:33 CEST 2007 since casting a pointer to an int doesnt work for 64bit, i need to change it. this is also the time to take out invalid packets. a packet is always valid, except in constructors, which need to use proper error handling. went pretty smooth actually. Entry: pure data Date: Wed May 16 21:40:12 CEST 2007 re-re-fixing the puredata bindings. this time i hope i get it right. using task abstractions: each instance is a task. this way i can do interesting sequencing in pd. also, i switched to a pure message passing coroutine approach. no re-entrance. all this is nice practice for tasks in purrr. Entry: getting rid of subheader Date: Thu May 17 15:11:00 CEST 2007 it's probably a lot easier to just have the packet == header == subheader, with the subheader containing a header_t instance. i really don't know where this comes from.. todays fix? hmm.. too unimportant and probably too much work. Entry: unify variables and deferred words Date: Fri May 18 12:14:48 CEST 2007 i really need local executable words instead of just variables.. how to do so? context switching might just exchange the first body element?? it's fixed now as: : local-variable dup @> 2 list dynamic-parameters push ; : local-xt xt>body local-variable ;; which works ok.. i changed some C code to make it more robust. don't use it for non-deferred words.. maybe i need to simplify deferred words a bit, unify them with "closures" and add an extra slot to primitives? hmm.. deep change. not for now. Entry: non-linear memory Date: Fri May 18 14:14:52 CEST 2007 so: lists + packets are managed -> cannot contain circular structures. pointers are not managed => code is not managed => dictionary is not managed assume: dictionary is persistent: violations: pointers to list structures (data not in dict) can be void -> only used in isolated lowlevel words LIN/NLIN separation requires that pointers, which can only point to nonlinear data, are accessible from the root, so they can be GCd by the nonlinear part. Entry: puredata Date: Sun May 20 11:30:15 CEST 2007 i think i sort of managed to get the bottom ready: - pd serves a set of control (swaptask, return, error) and pd/setup methods (send, make-outlet, ...) - pd asks pf to do context switching (pd-enter pd-leave) result is most intelligence can be written in pf, without pd object depending too much on C interface. also: all pd data is shielded from pf, because pf has no control about extent of objects. Entry: pure data objects Date: Tue May 29 12:22:10 CEST 2007 to make objects, i can use the shallow binding from swaptask, and a way to add variables. i.e. 'property', which does both global var alloc, and task local spec. i think i'm almost there now: 1. mainloop swaps in the current object before responding to a message 2. object creation/deletion respect this protocol 3. the current task at object create/deletion is always the main loop what about creating a new task every time a message is sent? hmm.. that defeats the whole idea of having local state.. it's probably better to make a swaptask that takes an extra argument: the number of data atoms to exchange together with the task.. ok, seems to work a bit. now, to simplify: empty list means return. this makes it easier to just send a list of forth code to an object. dsp part is working too. TODO: - sync stuff: take out all delay loops: make sure scheduler runs once/dsp tick. - create a 'method' word to implement shallow binding. sync stuff seems to work. now for the 'method' things. Entry: shallow binding Date: Wed May 30 15:47:10 CEST 2007 i don't have much choice for method binding, given the architecture. shallow binding using the 'swaptask' dynamic variables, and deferred words. a method is thus: - a deferred word - a local-xt what i need is a convenient syntax. something like this is probably best: { split interpret pd-done } method pd-float ok. i think i got it sorted out now. Entry: linear lisp Date: Tue Jun 5 00:08:51 CEST 2007 http://home.pipeline.com/~hbaker1/LinearLisp.html PF is almost the same: lists have RC = 1 (not counting pointers, i use as non-linear cheating without GC) and atoms can be packets, which are ref-counted. so, basicly. there are only 2 different types: cons cells which always have reference cout = 1, and atoms which are not (necessarily) managed. in the paper above, FREE, COPY and EQUAL are first constructed as deconstruct/construct operations, and later made more efficient by implementing them as primitives, but preserving the linearity. i don't get the "hash consing" part though.. Entry: notes.org Date: Fri Aug 24 12:03:07 CEST 2007 from notes.org: * packets montevideo done ** pf debs ** terminal output bug fixed ** task local variables ** window -> view coordinate transforms for events ** better error reporting ** blob generation, polar coordinates with uniform angle ** text rendering ** display lists as objects ** quicktime output ** splines, nurbs, interpolation, subdivision ? ** pf object for pd Entry: from old zwizwa site Date: Thu Oct 4 00:33:19 CEST 2007 the following posts are from the old zwizwa wiki. Entry: Is PF really forth? Date: Thu Oct 4 00:34:01 CEST 2007 So, what is forth, really? Forth is more like a design pattern, than a real language. Hence we say 'a Forth' instead of just 'Forth'. I think the easiest way to describe Forth is by means of how it works. Basically, a forth is an interpreter. Interpreter means here, very practically, a translater of some language to machine code. Deep down, a Forth consists of very small chuncks of machine code. In PacketForth, thise chunks are actually C code, so the lowest layer is abstracted away from the programmer. In BadNop, the lowest layer is C for the host, and machine code for the target. A basic Forth contains 2 interpreters. This means there are at least 3 language levels, or 3 representation levels for code. M. machine code X. execution tokens A. ascii source code PacketForth contains 4 levels, and 3 interpreters to move between the levels. M. compiled C code X. execution tokens S. symbols A. ascii source code The presence of the S. level enables to implement lisp-style symbolic macros along side forth-stile compile time executing macros (create does>). The power of a language could be somehow measured by the amount of reflection possible. In PacketForth, the 3 interpreters enable you to move to a lower level, while it is possible to move to a higher level by generating either X S or A code. http://en.wikipedia.org/wiki/Reflection_%28computer_science%29 Entry: make art presentation Date: Thu Oct 4 00:35:11 CEST 2007 Packet Forth presentation for the http://makeart.goto10.org festival in Poitiers, France. So, I get to start from a clean slate, hey :) It's strange, I've been thinking so much about language lately that I've forgotten how to communicate a simple idea. Problem is, I don't know you. I don't know your background. And I've gotten tired of explaining what PacketForth really is. Probably because I don't really understand it myself. So instead, I give you the story of me, and how I make art. The Making of an Artist ----------------------- First, the moral of the story. I've been using Pd http://www-crca.ucsd.edu/~msp/software.html for quite a while now, summer of 2000 to be precise, and I like it a lot. I think it's the first piece of software that made me go WOW after I went through a 3 year long sysadmin-wannabe period, getting to know Linux, GNU, Debian GNU/Linux and collecting 386 boxes and networking cards, and realizing there were no 'applications' for Linux. Especially not for making music. Only weird things that never worked, that I never understood. Before the summer of 2000 I'd been toying with hardware synths and effects. There it was: virtual boxes and virtual patch cords, much like the things I knew from real life, costing absolutely nothing, running on Linux, making sweet noises I could only dream of the years before this discovery. I was hooked! I was corrupted! So there were applications for Linux after all. Then it started to dawn on me. There are actually tons and tons of applications for Linux, they just look different. Actually, they concentrate on doing something right instead of just looking good. And a great deal of them are weird and look like the works of a madman. The reason for this is that most of these tools are made by the people who use them, and not dictated by the ideas of a mass market analysis department. These people take pride in understanding a problem and solving it, in such a way that they put their name on it and show every little detail (source code) to the world. During that period I got to know Perl http://www.perl.org/, a weird programming language originally written by a madman. Now, that's not an end user application, is it? The idea I try to bring across is that Perl IS a user interface, it's like a living, spoken language. It evolves, defies absolute rules and adapts to arising needs. Perl is dirty like real life is, but allows you to get things done quickly, like real life demands. Am I making art http://makeart.goto10.org yet? No. It's still 2001, and I'm still building Pd patches alone in my room. Nobody cares, unless I produce something with a familiar http://mala.wha.la/cgi-bin/prj?ioidi 4/4. I wanted to build a synth of my own, before I got to know Pd. But now that I had Pd, what was there to do? Luckily I found some things missing in Pd, so I started to translate some of the ideas to http://zwizwa.be/darcs/creb/modules/xfm.c C code. Alright, you say, C code. That's not what I'm here for! Right you are, indeed. As computer languages go, C is about the worst one still in wide use. To be fair, C is not all bad, but it's great to scare people off. This is exactly why some people think that programming is only for obscure geniusses. Bollocks! It used to be like that 20 years ago, sure. But today, there is a broader palette. After writing http://zwizwa.be/pdp PDP, around november 2002, I started to realize I could do more interesting things if I would combine ideas from Perl with things i saw happening in Pd. If you look at Pd, it has a lot of the qualities of a programming language: lots of tiny objects that do one thing well, which you can connect together to build something bigger: a patch. Now a patch is really just a program. So why not make this all explicit? Now we're januari 2006. Now I went completely crazy and wrote myself a new language. Of course, it has been done before, but mine is a little bit different. But more important: it's mine. My way of talking to my machine. Interaction ----------- It's strange when people start calling you an artist, when you see yourself as 'anything but'. I still have to convince some friends of mine I don't believe in all that crap, really. That I'm still an engineer. That there's still some SCIENCE in what I'm doing, and that art is about experience, not about concepts or reflections (shudder!). So, instead of fighting this label people attach to my forehead, I started to redefine for myself what art is. Decided I really don't care what anyone else thinks it is (making me a true artist, for that matter :), so I define art as interaction. Not interaction with audience per se, but interaction with my machine. I am designing a user interface. Like a trumpet, only I'm not using brass, but code. If you look around you, graphic user interfaces are everywhere. The biggest laugh I had recently was when I was programming my dad's DVD recorder. Why do these interfaces suck? Because of two things: computers (any electronic device today is a computer) are getting too powerful to use them to do just simple tasks, and they are not like human beings. It is incredibly difficult to build a system that interacts in a proper way with a human being, for the simple reason that people don't think digital. Now, there's two things you can do about that. Either you wait till some http://www.singinst.org/ bright soul finds a way to make computers think more like humans, or you start to think more like a computer. When the first thing happens, I think I got myself a new hobby, but the second thing is what I want to talk about now. Being incredibly stubborn, I always want my gizmo to do something different. Something the maker of the gizmo did not think of. This void is filled by 'good applications'. They leave the last part of assembly to the user. Instead of getting your meal served on a plate, you take 3 packages of 'almost-ready' food, and add them together so you at least have the idea of doing something original. I think the basic idea here is POWER. We like computers because we like power, because we like slaves. We want the computer to LISTEN and DO what we SAY. Maybe ART is when you combine POWER with ORIGINAL? How to talk to a computer ------------------------- (A) Basicly, what we would want is: HUMAN: "blabla .. bla, now do it." COMPUTER: "owkay, boss!" (B) The way it works in the current world is: COMPUTER: "these are the options. choose one." HUMAN: "eerrrr.. damn. how do i?.. aahh.. click" COMPUTER: "error! error! beep beep beep!" Now, there's a strange pattern. Even if in (B) the options are reasonably simple, people still need to go through the 'beep beep beep' phase before they actually learn what the options are. Reading a set of options is one of the most boring things you can do. It makes you feel dumb, not in control. Reading a manual, if you don't know what you're looking for, is madness. It puts the programmer (or anyone who wrote the manual) in control of you, cuz you need to LISTEN! Not in command, but subordinate! That's no fun, at least, for most of us. On the other hand, making an error, UNDERSTANDING what you did wrong, and then finding out what the right thing is, is a rather fun process. Especially if you get over the fear of being thought of as dumb. Luckily, a machine doesn't really care. So what if we make the collection of options incredibly huge, and let the human make errors until he finally figures out something that works? Or even better. Give a couple of examples he can change around to get the idea behind things? This looks a lot like the ordinary language we speak day in day out. First time you use a word, it's usually wrong. A shame and blush later, you know what it means, and probably never use it in the wrong way again. So, to get to the point. My idea of an ideal computer program is one that I never really fully understand. One with an infinite amount of options. A thing that i can interact with, gradually learning how to do things by making mistakes and experimenting. If I completely understand something, it's time to move on, because boredom awaits. This is what PacketForth is for me: my current level of comfortable chaos. It is a language, but it has things that work well for video or other graphics. Better than other languages. It is a toolbox full of small tools that can be strung together to make something new. It might not be the thing for you, it might not be the thing for many people, but it is an attempt to make interaction with a machine possible in ways that are not possible otherwize. The Message ----------- The real message is hard to bring across. If only I could download what I know into other peoples minds so I could go on with experimenting -- making art if you want. Unfortunately, it don't work that way. But I think the real message is that we are only at the tip of the iceberg of seeing what computers can do for art and aestetic experience. And the most interesting things I've seen thusfar all come from madmen. It's an interesting time we live in, but it's quite chaotic! People are maturing in the way they interact with present day technology. New ideas are being born. Personally, I find the world of free software incredibly interesting, and I am hopelessly addicted to the amount of control you can gain over computers AND aestetic experience if you learn tricks from people motivated mostly by curiosity. Entry: linux graphics Date: Thu Oct 4 00:47:15 CEST 2007 PacketForth uses mainly two extensions: xvideo and opengl. By default these are accessed through SDL. xvideo (video port) - hardware accellerated scaling / colourspace conversion / video overlay. - can be a physical video port (nvidia?) opengl (drawing and geometry engine) - non-accellerated uses Mesa (open source opengl clone) - accellerated uses dri - agpgart agp driver kernel module: (gart = memory mapper) - drm direct rendering manager: 2 kernel modules: drm + mga/i810/... - mesa-dri system lib: opengl level driver (dri client) http://dri.freedesktop.org/wiki/DRI http://utah-glx.sourceforge.net/gart/README http://dri.freedesktop.org/wiki/DRM http://www.mesa3d.org Entry: next refactoring run Date: Mon Oct 29 22:47:15 CET 2007 were i to change PF for the better, what needs to change? rewriting is not an option: too much knowledge encoded, and things depending on it. same goes for PDP. * it needs a new inner interpreter. i doubt i can make this change without messing up a lot, but i do need: - proper tail calls - a single quoting (literal) mechanism * get rid of most preprocessor macros, or write macros in a fixed style so they can be parsed: i'd like to move the code base into s-expressions and try some automatic refactoring. PF is large enough to make this process non-trivial, but there are quite a lot of points where some automatic refactoring would be a good idea: i.e. to make atom and list structure accesses abstract. simply replacing every atom->t by type(atom) is already quite a challenge. maybe i should use some specific C refactoring tools first to clean it up some. gulp @ verbosity. i just looked at libpf/forth.c and it's not going to be easy! Entry: the future of PF Date: Sat Jul 5 15:23:17 CEST 2008 Been quite busy with Staapl (Brood) last half year. It looks like I'm finally starting to understand the problem I set out to solve with PF: build a system for prototyping + deployment in the domain of sound + video processing. This problem was a bit more complicated than I anticipated, but I'm getting to a point where it is possible to see the trees through the forest. I'm oscillating between planning complete abandonment of PF, and trying to make the best of it. It's wade through the accumulation of dead ends (see previous post), but maybe PF can be saved as a stand-alone tool. What it does right: * Memory management. Stacks + added GC. * System interface. * User interface: polymorphy through multiple dispatch. What it does wrong: * Interpreter and type system implementation and semantics. * Specialization: Partial evaluation, code generation, code transformation through deforestation. This can probably be combined with a static type system to write an interface to PF that compiles straight code: getting rid of dynamic dispatch. * Create a malloc-free core: once initialization code has run, all non-deterministic garbage collection should be disabled, which means no more malloc/free or automatic GC other than the refcount based packet management. What direction it should go: * OpenGL: the dead end. Last couple of years I've focussed more on OpenGL, but this has turned out to be a mistake. The real beef is in image processing. For OpenGL, PF is merely a scripting language: a place where Scheme, or at least Scheme based metaprogramming, is probably a better alternative. As far as I can see, Dave Griffiths has solved this problem with Fluxus. * Image processing + linear algebra. PF fits really nice as a metaprogramming target in the 'algebra of programs' approach I'm trying to develop (Inspired by Backus' FP). I wonder if I can find a way to keep the system interface, and replace all the internal parts one by one. The easiest thing to do would be to run a metaprogrammed PF2 inside PF, reusing only system interface. Then one by one PF's processing primitives could be replaced by generated PF2 code, using a 'codelet cache' or something based on LLVM. Entry: Sobel edge + Hough Date: Sat Jul 5 16:01:26 CEST 2008 Moving prototyping for Snowcrash to C phase, means the construction of a Hough transform algorithm. Let's build a wrapper around OpenCV first. Skeleton in place, but probably it's not necessary.. Entry: Corrected Hough transform Date: Fri Jul 11 15:24:33 CEST 2008 Giving up on generating code in the staapl/ip framework: it's not ready yet: not general enough. I need to think about comprehensions + shift operators, and how to combine them with general programming constructs. So, let's go on in PF. The problem with Hough is that there are a huge number of parameters. Is there a way to somehow compute ranges from the accumulator dimensions? Probably best to keep the parameters in C, and use a C based edit/compile/run loop. Entry: glare Date: Fri Jul 11 17:51:10 CEST 2008 Instead of thresholding (throwing away information) an interesting operation is to spread out the energy to preserve the intensity integral. Entry: speeding up Hough Date: Sat Jul 12 10:34:46 CEST 2008 bitgrids: Sobel + Hough: Time to speed up the kernel routines: use bitgrids instead of images to represent the image to reduce memory access. updating sin/cos: this gave a significant speedup, about 2x/3x Entry: cleaning up PF type checking Date: Sun Jul 13 12:39:22 CEST 2008 I need to solve the type checking in a library. Just been writing some PF wrapper words and it's completely insane the amount of red tape.. Basicly, convert every argument to a pointer which points to the struct that represents the data. (don't handle int and float as values -> this gives a proper common interface). Hmm.. since i'm not going to save any lives anyway, maybe clean up PF a bit to see if i can save PF for the next protyping run. * subheader includes the header (they point to the same) * the first pointer in the subheader points to the concrete data Let's make a list of all the data types in the mean time: pf_matrix_t pf_image_t (+ bitmap) pf_bitgrid_t module_t pf_stream_t pf_string_t qt_t x11_t gslmatrix_t texture_t bignum_t To port, remove the following constants in packet.h // #define PF_HEADER_SIZE 4*64 ///< size of the packet header, including type specific subheader // #define PF_SUBHEADER_OFFSET 4*16 ///< header part used by pf_header_t // #define PF_SUBHEADER_SIZE (PF_HEADER_SIZE - PF_SUBHEADER_OFFSET) All structures will have the appropriate malloced size, and are subclasses of the generic packet header which includes type and memory info. This requires a change in packet_alloc, now takes 2 arguments: headersize and datasize. This seems to be enough to drive all changes. Note that not making 'super' abstract is going to bite me in the ass some day. I guess it's easy enough to change though.. OK: i got this to run. Now, should the code be refactored to not use the header/subheader stubs any more? What code is interesting? Stream and string. They're central, and they're packets. Seems to work. The rest will be a bit more effort. The matrix code is really ugly.. Entry: no data member access Date: Sat Jul 19 02:04:41 CEST 2008 This would be a nice excuse to try out the Haskell C parser library: write some code to modify the PF source to replace all direct structure accesses by accessor inline functions. http://www.sivity.net/projects/language.c I recently made an error here by using access like image->super.type instead of SUPER(image)->type. The idea being that at least the link between packets and super should be abstract (can either be inline or a pointer). OTOH, i am using direct casts also in the change i made yesterday, so maybe i'm making a mess of it again.. Entry: refactoring Date: Tue Jul 22 00:18:33 CEST 2008 The thing which gets in the way for proper parsing the code for refactoring is preprocessor macros. Those are too nasty to do anything with. Maybe a good way to spend those not-so-awake hours on moving as much macros as possible to inline functions? The first one i did: INT in macros.h actually doesn't work because of the early exit. Maybe i should use a proper exception mechanism? Maybe the macros are really a symptom? Entry: setjump Date: Fri Sep 19 19:48:34 CEST 2008 Maybe the easiest way to solve the macro.h problem is indeed to replace the return value checking with setjump everywhere. That way, all type checking can be done in functions instead of macros. FIXED: didn't properly check before. it seems to work now: the 'throw' methods are functions intead of macros based on 'return'. Entry: 2 arguments Date: Fri Sep 19 20:11:57 CEST 2008 It's probably also easier to remove the stack argument for primitives. If I recall this was for 'performance reasons' but I doubt it does any good at all.. There's a problem with stack.c, which mixes 2 meanings: operations on a stack object, and operations for Forth which should be on the VM. This needs to be disentangled. The reason for stack/vm split is the same as pure Coma and the ``control'' extension in Staapl. OK... i think i got all occurences now.. was a nice trip through years of different tricks to get at the stack arguments ;) I really need a better PF<->C interface, based on the following ideas: * data is always owned by the stack * only remove arguments when done (so errors leave data intact) got a problem.. in scheduler.pf the lists waiting-write and waiting-read return undef the 2nd time the scheduler executes.. FIXED: the problem was a local shadowing of the s parameter used with CHECKN and ARG0... Entry: name space management Date: Sat Sep 20 10:56:31 CEST 2008 split the view of pf_vm_t into private and public, with private only accessible from libpf/forth.c, and public only access to the data stack. Entry: type checking Date: Sat Sep 20 11:58:03 CEST 2008 time to factor out the type checking / dispatch a bit. i like the INT(arg) accessors, so let's stick to them. get rid of the CHECK1,CHECK2, ... macros removed 3 and 4.. looks like 1 and 2 are all over the place and used in functions that do lowlevel manipulation. Entry: X11 bug Date: Sat Sep 20 15:27:56 CEST 2008 something broke in the type checking: e_type when using the default xv blitter. EDIT: on the matrox card there's an error too, but not on the S3. Entry: PLT Scheme Date: Sun Sep 21 11:22:16 CEST 2008 Initialization is working. The script/plt.pf code will handle the PF side of things, basically just a receive-eval-send loop. The problem is with data types and memory management. Essentially, because of linearity, all PF data objects should be unobservable from scheme, with only the operations and data constructors (literals) accessible. With PUSH FIND and RESUME we're done for Scheme -> PF control. Data access can be done on an as-needed basis, copying data straight from packets. NEXT: figure out how build a scheduler that cooperates with Scheme's. I got it running using 2 PF threads: one to run the 'tick' method and one to poll the C thread (c-yield / pf_vm_resume() ) Now, because PLT Scheme is a non-realtime language, and due to linearity we want to hide the PF memory from PLT Scheme (share-nothing parallellism), I think the best approach is to run PF in a separate OS thread. O horror yes. But that's for later. Actually, pipe comm + thread decoupling could be implemented separately.. Entry: different compiler Date: Sun Sep 21 15:40:36 CEST 2008 It might be interesting to build a Staapl-like compiler on top of the VM code, bypassing the built-in outer interpreter. Entry: pf-alloc Date: Sun Sep 21 17:13:52 CEST 2008 just added a print statement in pf_alloc and pf_dealloc to see if it runs statically. it does for demo/ca.pf except for the window and console, which probably use strings. Entry: next fix Date: Mon Sep 22 00:45:29 CEST 2008 after setjmp/longjmp and the function prototype changed, the next one would be to make atoms, lists and packet structs abstract: x.member -> type_member(x) or to remove the list_pop operations? in the kernel they are used sometimes to transfer things.. better to use pop_atom there, but hey, there's a lot. so i put them back in. for the plugins however, it might be better to remove them. Entry: type error when creating xv blitter Date: Fri Oct 3 13:26:24 CEST 2008 The error happens here: > "xv.pf" load "" xv-blitter xv > : blit xv blit ; # Something wrong with the parsing word 'xv' This works fine: > "" x11:make-xv > constant XV > XV p #<0809b1a0:x11/:0.0> > > > ` bitmap/i420/320/240 new > XV x11:blit > So it looks like the error is in create-binder. Looks like the elements popped from the allot stack get accessed somewhere still.. > mark : boo 123 ; mark> > p ((boo # (# "" 0) # # 123 # #)) > ok, but this: > mark : boo 123 ; mark> constant BOO > BOO p (()) > is not ok. It's bad.. The type field of the listvar contains garbage at the point static PF_PRIMITIVE(pf_forthword_dict_find) is invoked. Ok.. kick me. Double deref... Entry: Ocaml FrontC Date: Fri Oct 3 15:19:58 CEST 2008 http://pauillac.inria.fr/cdk/newdoc/htmlman/cdk_182.html Might be interesting to use this instead of the Haskell C parsing library. I can't find anything like this in Scheme. Entry: c.plt Date: Tue Mar 24 20:10:01 CET 2009 Looks like time has come to try out C parsing: there is now Dave Herman's c.plt for PLT Scheme. Plan: - forth.c refactoring: structure deref -> (inline) functions - plugins: cleanup so they can be used in PLT Scheme The idea is that either PF needs code refactoring so it can grow as a C program, or it will be disassembled and replaced with a meta Scheme layer generating the VM. The basic idea behind PF is ok, it just needs to be simplified. Doing this manually however would be tedious and take too long. This could be a driver for code analysis tools. Entry: eliminating macros Date: Wed Mar 25 09:49:04 CET 2009 I'm using macros mostly for using implicit context. What should be done then is to make the macro -> function layer thin, and map directly to a fully bound function. The place to start is pf/macros.h Let's get the regression tests online again first though. Entry: tests Date: Wed Mar 25 09:52:24 CET 2009 - run all executions in gdb Hmm.. This is problematic due to a GDB bug: Cannot find new threads: generic error So i fixed the gdb attach a bit (see bin/pf-gdb and libpf/debug.c) Entry: bugs Date: Wed Mar 25 10:55:03 CET 2009 - v4l packet doesn't have desc field -> memset erased super init Entry: cpp output Date: Wed Mar 25 10:59:57 CET 2009 - c.plt needs plain C input, so we need to expand source files. - __extension__ not recognized - __attribute__ not recognized I'm going to use the following trick: add an option to disable parsing of header files. Entry: cplt report 1 Date: Wed Mar 25 11:35:50 CET 2009 #ifdef CPLT #define __const #define __extension__ #define __restrict #define __attribute__(...) #endif typedef int (*__compar_fn_t) ( void *, void *); # When I changed it to this: typedef int (*__compar_fn_t) ( void *a, void *b); # It produced: car: expects argument of type ; given () === context === /home/tom/.plt-scheme/planet/300/4.1.4.3/cache/dherman/c.plt/2/1/private/syntactic-context.ss:168:0: looked-ahead? /home/tom/.plt-scheme/planet/300/4.1.4.3/cache/dherman/c.plt/2/1/private/syntactic-context.ss:197:0: pop-parser-declarator! /usr/local/plt-4.1.4.3/collects/parser-tools/yacc.ss:321:16: parsing-loop Entry: PF ideas Date: Thu Mar 26 16:29:07 CET 2009 Some of today's intuitive flashes.. * pf object tree in filesystem Instead of managing data objects internally, why not export them as a filesystem? This would open some doors for serialization. Since the whole idea of PF is to run in two regimes: startup (compile code / build datastuctures) and run, this might not be such a bad approach. The only question is: is this expensive? Once disk objects are memory-mapped, do they behave as files or as memory? And is a ramfs filesystem really fast enough? * functions from types: all functions = type convertors One of the early ideas in PF is to make all data-conversion automatic. Thinking about this some more: a lot of programs these days translate data from one form into another, and don't really do much more (they don't "act"). Can this be extrapolated to a point where types themselves define programs, and functions between them can be derived automatically? I ran into some google video presentation about code that does this in a more mathematical setting, but can it be done for simple applications? (I.e. given a pair of images (which provides order) a binary operator (-) and the "map" function represented as a type, the combined operation can be derived.) Entry: cleanup Date: Thu Jul 9 16:29:50 CEST 2009 The C analysis stuff isn't going anywhere. It's much more difficult than I expected to get c.plt working - it's got bugs that I can't fix myself yet, and Dave Herman seems to be busy with other things. So.. What's next? Is there anything I can do that would make cleanup of the VM a bit more managable without too much manual labour? Entry: removing hash.c Date: Thu Jul 9 16:59:27 CEST 2009 It's not used, and it relies on 32bit ints so let's just take it out. { unsigned int knuth_magic = 2654435761U; /* see http://www.concentric.net/~Ttwang/tech/inthash.htm */ unsigned int k = ((unsigned int)key) >> HASH_SHIFT; k *= knuth_magic; k &= (1 << x->logsize) - 1; return k; } Entry: deps on current debian unstable Date: Thu Jul 9 17:09:28 CEST 2009 apt-get install \ libreadline5-dev libquicktime-dev libgsl0-dev Entry: 64bit woes Date: Thu Jul 9 17:21:42 CEST 2009 I know what to do: port to 64bit clean code. Now that I'm in 64bit land myself this should be straightforward. In 1995, a number of major UNIX vendors agreed to standardize on the LP64 data model for a number of reasons[1]... So, in short, long = pointer. Problem: longjump takes int argument, and you can't encode a pointer in this.. How to fix? Wait. This generalizes to all functions that return int: they can no longer encode pointers! So this needs some fixing.. (osc.c) Ok, was straightforward: there was already context, so just added a member. After changing w_int in pf_word from int -> long the interpreter seems to run. Let's try the rest. Looks like the opengl code is working. Video code however seems broken. Could be just the CA too since I don't have any video atm.. [1] http://www.unix.org/whitepapers/64bit.html Entry: boo! Date: Thu Jul 9 18:39:27 CEST 2009 tom@zni:~/packetforth/demo$ $PF tv.pf boo! e_type: No global abort handler. This error is fatal. I seem to remember it from somewhere.. Ok, it's in the setup of the v4l plugin. Something was going on here where I didn't know where the exit came from. Entry: fixing load-v4l Date: Fri Jul 17 09:32:28 CEST 2009 the problem occurs here: ` i420 "/dev/video" grabber problem was a missing assignment .type = PF_V4L in the constructor Entry: XV doesn't work on 64bit (it does) Date: Fri Jul 17 09:54:25 CEST 2009 Hmm.. It doesn't seem to be XV. Let's write a noise generator test for this. Indeed. Entry: Scaf problems on 64bit Date: Fri Jul 17 10:23:57 CEST 2009 Makes more sense the problem is with the bitneukerij.. Ok, there was a cast to (int) which should have been word_t = long. Now it works. Entry: PF is weird Date: Fri Jul 17 10:26:50 CEST 2009 Looking at the code again, It does look like I learned a lot with Staapl.. Overall PF is usable, but full of these little wrong things all the way to the bottom. I'm not sure if it will be possible to clean it up to my liking. Some are quite naive approaches in retrospect. I.e. I'm using 'eval' a lot, which indicates I really missed a proper macro system. But most of them are the dirty kind: writing too lowlevel (splatted) code. Instead of writing code that is factored as a sequence of consistent steps, a lot of C code "unpacks" into inconsistent temporary representation.. I'm not sure if I'm explaining this well though.. It's mostly about memory management. Entry: PF / Staapl / PLT Scheme Date: Fri Jul 17 14:54:52 CEST 2009 So.. Is there hope? I constantly oscillate between times where I think I should just rip out the guts and start over with a new VM, and times where I think things are not so bad and the current VM can be grown into something more mature. Unfortunately, the latter happens mostly after not looking at the code for a while.. The overall structure, which is what I remember after leaving the code alone, is quite allright, it's just the leaves on the ground that are rotten... What needs fixing? 1. packet types The type system and the way polymorphism is handled is currently implemented in a very bizarre way: there is the type ID and the symbolic representation. I would like to unify this into one. 2. macros There are too many C preprocessor macros that make source code analysis difficult. The bulk of these should be replaced with inline functions, and macros that are in some sense "typed" or have a clear semantics such that a C parser with possibly some extensions (i.e. Dave Herman's c.plt) can be used to parse it. 3. duplication in the kernel code A lot of the code can be factored and macro-generated. However, I don't dare to approach this without automated tools 4. abstracting plugins to a simpler API This can be done manually: most of the plugins can be separated into some decoupled C code that is useful in other projects, and a C<->PF bridge. Entry: Packet Types + Macros Date: Fri Jul 17 15:04:31 CEST 2009 Let's see if this can be fixed. What about replacing pf_header.type with a pointer to an object that can produce a pf_symbol_t type description? Actually, this can be gotten from pf_header.theclass So, let's simply remove .type and .desc and provide accessors for them. Ok, this is a deep cut.. Might also need to do some of the macro refactoring while I'm at it. I see last time I already replaced the "return" based exception handling by a "longjmp" based one, so most of the type checkers can actually be abstracted as functions. Maybe it is possible to limit all macros to "implicit context" macros that reference the vm struct? These kinds are ok: - simple renames: these can be left as-is, because they don't increase the complexity of the code. - "context" macros that implicitly bind an operation to a lexical parameter (i.e. the vm). these are reasonably predictable and when expanded don't raise complexity either. - typecasting Macros that perform composition that can also be performed by inline functions should be eliminated. (statement sequencing, expression nesting, struct dereferencing). The PF_STACK_CHECK macros raise errors so they only make sense in a vm context. however, this is I believe abused in a lot of places to do "local" stack checks. Ok, it's simpler to use packet_t.type as the name for the class: this keeps most code that performs such checks intact, replacing the PF_XXX by class instances. Entry: something wrong with e_type? Date: Sat Jul 18 03:34:05 CEST 2009 this used to be "return e_type;" but now goes through pf_vm_abort() no doesn't seem to be it.. symptom: bunch of values like these on the stack: # something doesn't get dropped properly, probably due to error escape.. Entry: interpret-stream Date: Sat Jul 18 10:21:41 CEST 2009 the error seems to be in "interpret-stream" which leaves the stream instance on the top of the stack. "stream>input" seems to be the culprit no, it works: something doesn't clean up.. but what is it? probably some exception that's not reported.. i find myself trapped now in the non-debuggable part of the code: the exception mechanism is not transparent. also, the fact that this read code is bootstrapped doesn't help.. maybe time to clean it up a bit? having both nonlocal exits and primitives that call other primitives and inspect the error return value causes problems.. most of the code seems to work though, except for this weird bug.. Entry: getting rid of the return value Date: Sat Jul 18 11:57:01 CEST 2009 Currently the combination longjmp and return for error signalling is too complicated. It's probably best to get rid of the return value all together and use guarded execution in setjmp wherever it's necessary to call primitives from C. Basic idea: if a primitive has an error, it returns to the interpreter loop and abandons its C stack. Ok, this looks like it's tedious but quite straightforward. One thing though: write (serialize) should be able to fail. ok i got it to build.. now let's spend some time in the debugger.. Entry: shutting it up Date: Sat Jul 18 15:31:21 CEST 2009 i'd like to boot it in a way that doesn't have an inner polling loop.. currently it seems to be cought up in some kind of (error raising/handling) loop.. ok: bug in the inner loop: didn't start throw properly on error. seems to run: all tests pass except for the previously mentioned bug of # instances lying around.. Entry: trace Date: Sat Jul 18 16:44:09 CEST 2009 > 1 >trace : asdf ; 0 >trace rdrop rdrop jmp jmp lit >r <8a0aa0> emarker>r read read-top@ input-stack head <8844a0> >r <89a8d0> r r> <89a8d0> follow <89a8d0> <89a8d0> @ <89f350> <89a8d0> execute <89a8d0> @read-atom <89a8d0> @ <89a8d0> # FETCH stream read-atom lit do-pass read-thing read-loop read-task dup : undef : : scalar:= : : jz 0 : leave : >r : drop drop # DROP stream r> leave : leave : leave : rdrop : rdrop : jmp : lit : >r <8a0ca0> : emarker>r : interpret-atom : interpret : : fancy-: boot-: do-pass : read read-top@ input-stack head <8844a0> >r <89a8d0> r r> <89a8d0> follow <89a8d0> <89a8d0> @ <89f350> <89a8d0> execute <89a8d0> @read-atom <89a8d0> @ <89a8d0> # FETCH read-atom lit do-pass read-thing read-loop read-task dup asdf undef asdf asdf scalar:= asdf asdf jz 0 asdf leave asdf >r asdf drop drop # DROP r> leave asdf leave asdf leave asdf boot-colon asdf leave record-location input-location input-stack top <8844a0> head <8844a0> @ <89a8d0> # FETCH -> but here it doesn't disappear. leave type # returns wrong symbol! lit variable do-route stream variable lit variable do-route list variable drop variable lit lit "" leave -1 "" swap -1 "" latestxt "" -1 xt>doc "" -1 linkhead "" -1 follow <8a7800> "" -1 follow <889690> "" -1 leave <889c30> "" -1 >r <889c30> "" -1 r "" -1 queue <889c30> "" -1 r> -1 queue <889c30> -1 leave leave rdrop rdrop jmp jmp lit >r <8a0aa0> emarker>r read read-top@ input-stack head <8844a0> >r <89a8d0> r r> <89a8d0> follow <89a8d0> <89a8d0> @ <89f350> <89a8d0> execute <89a8d0> @read-atom <89a8d0> @ <89a8d0> read-atom lit do-pass read-thing read-loop read-task dup ; undef ; ; scalar:= ; ; jz 0 ; leave ; >r ; drop drop r> leave ; leave ; leave ; rdrop ; rdrop ; jmp ; lit ; >r <8a0ca0> ; emarker>r ; interpret-atom ; interpret ; execute ; ; rdrop rdrop jmp jmp lit >r <8a0aa0> emarker>r read read-top@ input-stack head <8844a0> >r <89a8d0> r r> <89a8d0> follow <89a8d0> <89a8d0> @ <89f350> <89a8d0> execute <89a8d0> @read-atom <89a8d0> @ <89a8d0> read-atom lit do-pass read-thing read-loop read-task dup 0 undef 0 0 scalar:= 0 0 jz 0 0 leave 0 >r 0 drop drop r> leave 0 leave 0 leave 0 rdrop 0 rdrop 0 jmp 0 lit 0 >r <8a0ca0> 0 emarker>r 0 interpret-atom 0 interpret 0 rdrop 0 rdrop 0 jmp 0 jmp 0 lit 0 >r <8a0aa0> 0 emarker>r 0 read 0 read-top@ 0 input-stack 0 head <8844a0> 0 >r <89a8d0> 0 r 0 r> <89a8d0> 0 follow <89a8d0> <89a8d0> 0 @ <89f350> <89a8d0> 0 execute <89a8d0> 0 @read-atom <89a8d0> 0 @ <89a8d0> 0 read-atom 0 lit 0 do-pass 0 read-thing 0 read-loop 0 read-task 0 dup >trace 0 undef >trace >trace 0 scalar:= >trace >trace 0 jz 0 >trace 0 leave >trace 0 >r >trace 0 drop 0 drop 0 r> 0 leave >trace 0 leave >trace 0 leave >trace 0 rdrop >trace 0 rdrop >trace 0 jmp >trace 0 lit >trace 0 >r <8a0ca0> >trace 0 emarker>r >trace 0 interpret-atom >trace 0 interpret >trace 0 >trace 0 > bug was this: (type.c) - case a_packet: ts = pf_packet_type(pf_word_packet(a->w)); + case a_packet: ts = pf_packet_type(pf_word_packet(a->w)); break; Entry: rearranging throw and abort Date: Sat Jul 18 17:24:59 CEST 2009 Quite a marathon. The internal interfaces are getting more brittle, which helps a lot for manual refactoring like this. Let's look at all the demo files to see if they work. Looks like there are quite some bugs left though. These seem to be all C-level bugs, so let's first install some proper gdb startup routines.. The idea is that the global abort will run a debug trap. Problem: it's only known whether there are any exception handlers after the longjump. Todo: move this to before. ok works. The advantage of the current form is that it is relatively easy to JIT into a subroutine threaded Forth. Entry: next Date: Sat Jul 18 21:33:11 CEST 2009 lots of small bugs that prevent anything to work properly.. blob.pf works but: * cubes.pf doesn't work * v4l / xv problem? Entry: always using gdb Date: Sun Jul 19 19:49:44 CEST 2009 Let's switch to a default way of starting pf: - from modules in ~/packetforth/build - inside gdb #!/bin/bash . /home/tom/packetforth/build/vars CMDS=`tempfile gdb-pf-` cat <$CMDS r $* EOF cat $CMDS gdb -x $CMDS $PF rm $CMDS Entry: starting the console Date: Sun Jul 19 19:59:20 CEST 2009 pf starting pf-console the way it does now is problematic in emacs, and also doesn't work in gdb. it does work on console, so let's keep it as the default let's focus on starting pf in single-process mode with a proper abort handler. this doesn't seem to work well: :: 123 p ; >abort done. now call "safe-stdin" to have "stdin" with a default error handler that aborts to "quit", which starts a repl on standard input. Entry: gdb --args Date: Wed Jul 22 16:19:15 CEST 2009 gdb [options] --args executable-file [inferior-arguments ...] Entry: Primitives are tasks Date: Mon Aug 3 11:31:35 CEST 2009 I've been very confused about a couple of things lately, but it seems to be coming together. It is related to the similarity between programmin with tasks+channels and functions+datastructures. The core idea is that a C primitive useable in a scripting language should really be a task / thread / coroutine / ... Basicly, when you tie C into a scripting language, you want _both_ memory management and control management to be decoupled from the C side. C (or anything a bit less arcane with similar semantics) really isn't so bad when you don't use dynamic memory. I see two versions atm: - `pure' C tasks: they allocate all local memory on the C stack and never use malloc() / free() or any other external resources that are context-based. - `impure' C tasks: area allowed to use external resources but cannot be terminated. Depending on where the stack is placed, there are two different behaviours. - stack is the ordinary C-stack, which is saved and restored. this is the least optimal method, but allows for transparent task cloning (no pointer translation necessary). - stack is allocated on the heap: each task is a unique object and cannot be duplicated or observed. with restrictions on pointers (i.e. only relative to current stack context) this can be fixed. Keep in mind that the goal is to make something efficient. Therefore it is imperative that the primitives themselves don't introduce any intrinsic inefficiencies. However, when one starts with full control over virtual memory (to make stacks grow) or relative addressing (to be able to re-locate) this mechanism in itself could be quite powerful, and the actual implementation might not be so important. Entry: New interpreter Date: Mon Aug 17 10:01:30 CEST 2009 I started working on a new PF interpreter in [1]. The key changes are 1. proper tail calls 2. a safe memory model The new memory model is a combination of a graph memory with a simple asynchronouse garbage collector (GC) for code and variable box storage, and a clearly separated tree memory with linear lists (2 stacks + free lists) and a synchronous reference counting (RC) based leaf object (= packets) manager. This architecture is based on the assumption that for applications PF is intended for (DSP scripting), compilation and variable allocation happen only at program load time, while the ``infinite loop at the end'' is usually linear, and benefits from the synchronous finalization property of an RC-based memory manager, which leads to smaller heap sizes and better data locality: ideal for DSP applications. Instead of separating these in a staged solution where a compiler executable generates an executable with proper linear or static memory management, the new PF engine has both of them in the same interpreter image. The compiler uses GC, while the compiled programs do not. In other words: when your script loads you can use all kinds of macro-fu to generate the DSP code. Then you can essentially turn off the asynchronous GC and run the resulting program. This cycle can be repeated without leaving the application. The new PF core is implemented in a dynamically typed expression language EX, which is essentially Scheme running on a C stack. The EX language is also used to define SC, a proper Scheme interpreter that could be used as a metaprogramming system for PF code. The main goals at this moment are simplicity of implementation, reflection (very useful for debugging), and small code size. Current PF primitive C functionality (i.e. data processing, video IO and OpenGL) is not ported yet. This will be done as needed. [1] http://zwizwa.be/darcs/libprim Entry: Category Theory and Data Converters Date: Sun Sep 27 09:28:01 CEST 2009 One of the ideas behind packetforth is to be able to perform data conversion easily, and possibly automatically. Getting this organized requires some framework to capture the structure and approximations. Concepts from CT might help here to provide a design.