Software Architecture / Engineering Notes about the practical side of building software and getting it to where it needs to be. This is a counterpart to more technical issues which are handled in the comsci notes[1]. [1] entry://../compsci/ Entry: Coding The Architecture Date: Mon Mar 1 16:57:49 CET 2010 I'm reading some on here[1]. First impression: close gap between architects and implementors. What is architecture about, really? The role of a software architect: DEFINITION: * Management of non-functional requirements * Architecture definition * Technology selection * Architecture evaluation * Architecture collaboration (?) DELIVERY: * Ownership of the bigger picture * Leadership * Coaching and mentoring * Quality assurance * Design, development and testing [1] http://codingthearchitecture.com Entry: Architecture document Date: Mon Mar 1 17:08:15 CET 2010 Summary of [1]. 1. An outline description of the software architecture, including major software components and their interactions. 2. A common understanding of the drivers (requirements, constraints and principles) that influence the architecture. 3. A description of the hardware and software platforms on which the system is built and deployed. 4. Explicit justification of how the architecture satisfies the drivers. [1] http://www.codingthearchitecture.com/pages/book/software-architecture-document-guidelines.html Entry: What did I learn this time? Date: Sun Mar 21 10:06:39 CET 2010 The project I'm about to finish was something new to me. In some sense it was quite frustrating as it wasn't much about design and programming, but more about communication and politics. All my work up till now has been higly technical: understand the problem deeply, write a solution, debug, iterate. This time the technical challenges where minor, apart from absorbing knowledge about a new system and its quirks (Android) and a new programming language (Java). Most work went into: - Very difficult path from requirements to specifications. Things seemed obvious at first, but workarounds for secondary effects out of our control greatly complicated the design and implementation. New use-case requirements kept popping up. Lack of adequate tests to see what is really needed. - No real feedback from particular use case to general requirements. This seems to be largely a consequence of hierarchical way of planning and making decisions common in large companies. This project was special as it was a small one, not important enough for upper layers to bother. - Documentation. Getting the point across is difficult when using concepts from functional programming in an OO-oriented shop. Writing good design documentation is hard. Keeping it up to date is harder. Make sure that documentation and implementation are linked! Turn every question ever asked into a FAQ item. - Struggle with tools. The android development tools are not very good. They are slow, buggy and error-prone. The large write-compile-test cycle makes working with this system quite frustrating. - Meetings drain energy. My efficient work modes are problem solving (long stretches of deep concentration) and process management (keeping and eye on and streering a lot of not too complicated semi-automatic processes). Dealing with people and disagreements is not something that fits into this plan. It is as if _everyting_ in my mind needs to be swapped to secondary storage to make room for the social interaction rules. Entry: Robust filesystem Date: Sun Apr 24 21:59:35 EDT 2011 I'm burning my fingers on a current project. Some things that went wrong: - No specification, evolutionary what-can-we-get-away-with-cheaply design. Considering the application that was actually not such a bad approach: functional requirements where very simple and straightforward. Eventally there was one ill-specified requirement that caused a bit of complexity. - Complete underestimation of non-functional requirements: robustness. - Difficult refactoring: merging two subsystems that where "almost the same" caused many headaches when they where actually placed on top of the same abstraction. - Splitting another part of the code into separate modules proved difficult due to insufficient understanding of the coupling involved. - Premature optimization. Not for speed, but for memory usage, in this case disk buffers. This lead to an implementation that was very hard to change. - Inadiquate test suite: the stateful nature of the system, and the nature of the errors that should be recovered (lots of invalid intermediate state) makes it very hard to test. If I can name one major point, it is state. The system as it is at this moment has too many degrees of freedom. This makes many things very difficult: * Testing: almost impossible to cover all corner cases. * Change: the higly sequential nature of operation makes it very difficult to separate responsabilities over multiple objects, or perform simple, incremental changes. * Ownership: at least one bad factorization is due to unclear ownership of data structures. * Temporary storage management: A non-functional requirement is to use a minimal memory footprint. If state is the problem, the solution I imagine is almost immediately to switch to a transaction-based approach where pre and post conditions can be expressed clearly (even if only in the test suite). In a fully transaction-based approach, there is a very simple, even _stupid simple_ way of handling errors: whenever it fails, retry. In the current implementation that approach doesn't work completely: physical errors are system state changes: the system is changed from a consistent to an inconsistent state. The difficulty is in recovering from that. Entry: Robustness : external mutators Date: Sun Apr 24 22:31:05 EDT 2011 Follow-up of last post. How to make a data storage system robust? - In a reliable system (no externally introduced faults), inconsistent state has to be a consequence of bad design, as there are tried and true principles to build such systems correctly. The main idea is to clearly define what a state change is, and allow it to occur or not, but never show to produce any intermediate state. This is captured in more detail in the ACID[1] rules used in database design: * Atomicity: a transaction succeeds or fails. No intermediate (inconsistent) state is ever visible. * Consistency: each transaction maintains consistency rules. * Isolation: concurrent interaction should not interfere. * Durability: a completed transaction persists. All these are reasonably obvious, especially if you stick to the simpler approach of serialization as isolation principle: a state change either succeeds or fails. - In a system with transient errors, recovery is possible through transaction abort+retry if inconsistencies are discovered soon enough. Here "soon enough" means before an inconsistent read leads to a write that would not occur. Let's call these "read faults" or "wire faults". In practice, such errors can be caught using redundancy. Checksumming, error detecting and correcting codes, ... As an optimziation, nested transactions can be retried locally. - In a system with "external state mutation", consistency maintenance requires extra effort, i.e. actual "repair". This is hard, as it requires "intelligence", i.e. knowledge that is not inside the system and its consistency rules. It seems that the best approach is to design the system in such a way that such repair operations are kept to a minimum. If the external mutations are known, they could possibly be caught by not allowing them to cause actual inconsistencies. [1] http://en.wikipedia.org/wiki/Database#The_ACID_rules Entry: Abstraction isn't always the solution Date: Mon May 9 09:07:50 EDT 2011 It's hard to quantify this impression. Some heterogenous examples: * http://zwizwa.be/darcs/staapl/staapl/pk2/libusb.ss Originally, Jakub wrote this code with a naming scheme that's close to the C header file. I changed it to have ``pretty'' dash-separated scheme-ish names. I later changed it back so that all entities that are direct wrappers of C functions and structures are verbatim copies of those names ( and struct:). Then only the functionality that is built on top of this hase the pretty scheme names. This has the advantage that libusb documentation remains valid. In essence, the original (lower level) API remains visible. In a 2nd layer this can then be abstracted out if necessary. Morale: Exposing the low-level names seems dirty at first (that's why I originally changed Jakub's naming) but seems to make a lot of sense from a documentation pov. If you include the existence of libusb as a C library, and the fact that many people that would use it from scheme would also use it from C, then the overall complexity seems to be decreasing if the horribility of the the C API is leaked into the Scheme code. ( Where it can be abstracted over later if necessary, but only app-specific, not wrapper-specific. ) * Exposing data structure internals in interfaces. C-specific: When you expose a structure in a header file, you forever break binary upgradability of the representation (apart from adding members to the end of the struct). For core data structures, exposing the internals is almost always a bad idea. For mutable structures which need to respect data invariants accessors that explicitly maintain invariants are absolutely necessary. In some cases though it seems not such a big deal to just use a wide open struct and be done with it, instead of a pletora of constructor and accessor methods that do not add much value. However, this seems to work only for "constant" structures: those that never change shape after they have been constructed. Such a structure behaves as a "configuration file". ( In C, one additional reasons to expose internals of a struct is to allow temporary structured data alloced as local variables, avoiding malloc() issues. ) Note that when the use of such a public configuration structure is limited to one or a handful of API methiods, upgrade is not such an issue. Simply remove one method together with its configuration structure, and replace it with another method with another structure. Here the binary linking issues are the same for structures and functions. ( The general idea seems to be that when structures are read-only, the difference with functions starts to look more blurry. This is especially apparent in a pure functional language like Haskell [1]. ) [1] entry://../compsci/20110509-093238 Entry: On doing things differently Date: Mon Oct 24 19:08:58 EDT 2011 If you violate a rule or invariant in a sound model, or design pattern, then you should know *why* you violate it. Meaning, the violation should not be out of ignorance of how to do it properly, but out of knowing why you have to do it differently. Often it is indeed easier to start from scratch and build something simpler, with less moving parts. Though this should never be out of ignorance, because usually in good designs some possibly hard to understand complexity might actually be essential in ways you don't get yet. Entry: Writing stateful code Date: Fri Nov 25 14:14:34 EST 2011 I've recently had the pleasure (!) to write some file system code. The main problems I run into are 1. robustness (error detection and recovery) and 2. simply getting it to work correctly in the first place. I talk about robustness somewhere else[1]. This post is about how to handle stateful code. File systems generally have a lot of state and a lot of invariants covering relationships between state elements. Often a lot of this state is cache or index data: some *duplication* of state that is available from other state, but is too expensive to compute. A practical way to solve this is to use an approximation of [2]. In logic, structure is expressed as predicates. To use this in programming, make sure these predicates can be evaluated by reusing some of the code that computes the caches: 1. Make sure the code that computes the cached/index data is readily available as subroutines. 2. Add assert checks that verify stored caches against computed data after major state updates occur. Make these optional, so they can be run only during testing/debugging or under specially constructed conditions. [1] entry://20111125-141642 [2] http://en.wikipedia.org/wiki/Hoare_logic Entry: Robustness Date: Fri Nov 25 14:16:42 EST 2011 Main issues are: - Transient vs. permanent errors. Transient errors are errors that do not violate system state, but are a consequence of communication errors. The criterion is that they can often be solved by detecting them (using data redundancy) followed by simply retrying the operation. Permanent errors are those that permanently violate invariants. The issue here is to identify which invariants are broken, and how to fix them by throwing away part of the information causing minimal damage. - Local or global recovery. It is hard to implement robust code that is also modular. There seems to always be a tension between abstraction (solving problems locally) and need for broad-ranging information (solving problems globally). Entry: Darcs Emacs Date: Mon Nov 28 15:10:33 EST 2011 After getting used to git a bit more (magit), I really miss decent support of darcs in emacs. I currently have these installed: vc-darcs.el[1] xdarcs.el[2] [1] http://www.loveshack.ukfsn.org/emacs/vc-darcs.el [2] http://chumsley.org/download/xdarcs.el Entry: State machines are hard Date: Sat Dec 17 10:32:46 EST 2011 I'm on an embedded project with a lot (lot lot) of state that is very hard to test properly. We're basically like "oh yes, this is a case that needs to be tested also." but only after we see it failing in more expensive black bock testsint, past the first line of developer tests. How can this problem be reduced in a more systematic way? I sort of saw it coming, but did not have a way to counter.. Entry: Testing a stateful monster Date: Tue Dec 20 14:10:35 EST 2011 I'm done with this approach.. Same project as last post, more state problems. The thing is also that there really aren't enough seconds in all of time to write test cases for every possible initial state. How to turn a stateful program into property based testing in a systematic way? Entry: Branching in Darcs / Git Date: Sat Dec 31 07:41:58 EST 2011 Been using git for work for a while. I actually like it better than darcs. Maybe it's the "chronological" model that meshes better with how I think about code. I don't think I ever used the "clean, commuting patch" model of darcs. I'm just not that clean! Or it's more about the kind of early-stage development that has large, cross-cutting changes instead of small feature additions or bug fixes.. So I'm thinking of moving my stuff to git, at least the current Haskell work, or figure out how to branch and inspect branches in darcs. Entry: Clean it up when it becomes a library Date: Mon Jan 28 15:31:01 CET 2013 In practice it is often benefitial to not clean up "top level" code, i.e. the configuration layer of an application, the thing that glues things together. Only clean up when a part becomes "library", when there is some other code depending on it, and it would help to make the interface clean. Entry: A Great Old-Timey Game-Programming Hack Date: Mon Dec 16 10:17:11 EST 2013 > The challenge wasn't overwhelming complexity, as it is today. The challenge was cramming your ideas into machines so slow, so limited that most ideas didn't fit. https://news.ycombinator.com/item?id=6913467 Entry: Debugging - thinking Date: Wed Sep 23 18:33:14 EDT 2015 You can add convenience and tooling, but you're never going to eliminate the need for deep thought when debugging. This is science! You need to design an experiment to gain information about the system like narrowing down a bug. Entry: Stack ripping Date: Sat Oct 3 13:14:36 EDT 2015 https://thesynchronousblog.wordpress.com/2008/08/28/the-problem-with-events/ Another bad consequence of this code separation is that the state kept in callbacks are independent of each other as all local variables residing in their stacks are lost on returning. To keep state, their stacks must be manually reconstructed in the heap and passed as extra parameter from callback to callback. This phenomenon is known as stack ripping. Entry: It's called hardware because it makes everything hard Date: Tue Oct 6 01:30:08 EDT 2015 Rump kernels: drivers are "real-world, bug-compatible". http://www.fixup.fi/misc/usenix-login-2015/login_oct15_02_kantee.pdf Entry: Joe Armstrong on OSC and music Date: Sat Feb 13 22:45:59 EST 2016 http://joearms.github.io/2016/01/26/The-Unintentional-Side-Effects-of-a-Bad-Concurrency-Model.html http://joearms.github.io/2016/01/27/Controlling-Live-Music.html http://joearms.github.io/2016/01/28/A-Badass-Way-To-Connect-Programs-Together.html http://joearms.github.io/2016/01/29/Controlling-Sound-with-OSC-Messages.html Entry: Static Erlang Date: Tue Feb 23 17:54:18 EST 2016 How to get better timing constraints? Erlangs model of proceses and messages works very well. On a (bare-bones) microcontroller, state machines can take the role of true proceses. Often it is possible to put constraints on the depth of a call stack. However, dealing with message queues in memory-constrained systems is harder. What happens when the queue is full? Easy: a task should block. So how to mix this with strict timing constraints? Entry: Feature change, a 3-step process Date: Wed Mar 9 12:05:29 EST 2016 To implement a feature change that requires a structural change (current structure A, desired structure B), it is usually best to factor the approach in 3 steps: 1: Abstract the implementation such that it can represent both structure A and structure B based on a configuration data structure. 2: Change the configuration data structure to switch to structure B. 3: Optional: remove (evaluate) the configurability if it is (absolutely) certain that structure A is no longer needed. The abstraction step 1 is most difficult, but it can use exisisting regression tests. This way one avoids the "rewrite pitfall", which usually fails due to missed requirements that were implicitly covered by the original implementation. Entry: Why are code examples so important? Date: Wed Mar 16 14:59:02 EDT 2016 Because they document protocol: sequencing and composition, a piece of information that is usually absent from API documentation. Entry: unikernels Date: Sun May 15 19:17:13 EDT 2016 http://queue.acm.org/detail.cfm?id=2566628 Entry: Accidental Complexity - Fine Grained Redundancy Date: Mon Sep 5 11:11:51 EDT 2016 Things I keep underestimating: - The amount of arbitrary details caused by interface-mismatches: modules are similar, but not quite compatible. - Getting the details right: moving from abstract design (which is usually correct) to filling in a clean implementation. - Difficulty of debugging embedded systems: no safety nets. The boil down to not being very precise. Redundancy helps, but is not always easy to obtain in fine-grained form. Entry: Engineering problems Date: Sun Nov 27 08:15:05 EST 2016 A list of actual problems I face recently, as complexity rises - establishing canonical interfaces - i.e. identifying interface duplication and removing it through unification. - leaky abstractions when attempting to change interfaces to support unification - not being able to remember the previously established canonical interfaces and abstractions, and so making bad choices - things break when making the slightest changes - too many distractions, tangents, interactions and things to consider Entry: protocols Date: Wed Dec 7 18:26:46 EST 2016 Looking back at the last couple of years of professional embedded software work, what is very clear, is that it is mostly about implementing protocol endpoints. I would like to find a way to make this less problematic. Maybe by implementing more protocols? A fundamental issue in my understanding is that it is easy. It is not. The reason is mostly about statefulness. Protocols are one thing, mostly about ser/deser. However, dealing with references to existing state is something completely different. I saw a talk by Joe Armstrong mentioning this. It might be part of the UBF project. Maybe UBF(b) in https://github.com/ubf/ubf also http://scarl.sewanee.edu/CS326/Lectures/ubf.pdf Entry: The real problem with software engineering Date: Fri Dec 9 23:01:23 EST 2016 1. You're not smart enough to build everything up front, so you work in several iterations and changes. 2. It might take some time before you realize all the side effects of a seemingly simple change. Entry: genius consultant Date: Fri Dec 23 23:43:51 EST 2016 https://yow.eventer.com/yow-2014-1222/stop-treading-water-learning-to-learn-by-edward-kmett-1750 The consultant = co-genius idea is funny. Takeaway: - Go deeper every time you revisit a topic - Use "Feynman Algorithm" on problems _and_ solutions Entry: what is a good protocol? Date: Wed Dec 28 01:12:51 EST 2016 the problem with communication is often not bandwidth, but delay. in that sence, a good protocol is one that doesn't involve a lot of ping-pong, e.g. is not very stateful, or doesn't have a lot of indirection that needs to be fed back (explicit state passing). in general it seems there is a trade-off between size of messages (+) and amount of indirection (-). Entry: subdivide Date: Thu Dec 29 12:12:46 EST 2016 - divide problems into isolated subproblems - ensure the composition is sound (e.g. no abstraction leaks) - if there are unavoidable leaks, make sure they are known and accounted for Entry: programmer psychology Date: Fri Dec 30 15:20:28 EST 2016 Emotional cues that the approach is not right: - overwhelm - boredom Eliminating the causes of these will often eliminate despair and so create a boost in energy. The two emptions indicate depletion of resources, and seem to be a safety mechanism of the body to back off and rest, repair. The resources are being depleted as a consequence of "too much": Overwhelm is a consequence of loss of mental capacity after trying to contain too much complexity, and boredom is a consequence of depleted willpower, being forced to take a path that is "not right". These seem to correspond to defeat due to too much actual, as of yet unconquered complexity, and defeat by too much accidental, badly handled complexity. overwhelm --------- How to interpret mood queues? What I've seen is that a lot of times when fatigue and overwhelm enter the picture, the problem is that there is no clear next step. It is as if I uncounsciously know I'm deluding myself when assuming I understand the problem. The next step should be the right size, meaning that: - the goal should be clear and verifiable - it should be perceived as doable with current energy level The last one is very important: if the problem is so big that it involves a risk of getting very confused in the process, it is not understood yet, and the next problem is actually to split the chosen next problem into a couple of steps that implement an incremental change. The danger for this is to invent incremental changes that are actually side steps. The first one is also important: if it is hard to test, the problem is actually to write the test first. boredom ------- In theory, it should not be possible to be bored while programming, as there is always a way to automate a problem that is simple and repetitive. So why does it persist? It seems that boredom in context of programming always has an element of accidental complexity: of being confronted with things that "should have been handled better in the past". This judgement reduces motivation, and thus requires willpower which will eventually deplete. In a high-energy state, it is usually possible to trade boredom for overwhelm, e.g. keep eliminating boring work until the elimination process iteself becomes overwhelming. conclusion ---------- There seems to be only one problem: overwhelm. The way to deal with overwhelm is to accept it -- do not fight it as it does mean STOP. Then increase allotted time or reduce the problem set. After acceptance, there is calm resolve which can bring about a next iteration of subdivision and possible backtracking. Entry: video tearing, the poster child of bad abstractions Date: Tue Jan 3 18:35:45 EST 2017 It's been a thorn in my side ever since I learned about the problem doing VGA graphics on a 386. The solution is simple (use double buffering) but for some reason it persists into 2017 simply because of the towering abstractions and their inability to abstract this properly! The mess we're in :) Entry: Creating context Date: Mon Jan 16 14:46:37 CET 2017 As architectures get more complex, what I find is difficult is to create context in which to express a solution. Often, it is not hard to express mechanism once all components are brought together, but more and more, the bringing together of components becomes the main problem. Communication is the real problem in software architecture. Entry: Kinds of errors Date: Tue Jan 17 16:55:13 CET 2017 In order of difficulty to resolve, most problematic first. - architectural Architectural problems are those that occur when a secondiary or sometimes (when it's really bad) a primary requirement is not reflected properly in the architecture. Structural limitations turn seemingly simple problems into huge issues that require big changes. - algorithmic Or logic errors. Often, in data-structure intensive code, not everything can be encoded explicitly in a way that correctnes is easy to verify through code inspection. I.e. the problem is encoded in data structures that are subjet to _implicit_ invariants. To expose the latter as explicit code, it helps to make an effort to encode more in the data structures - e.g. make it impossible to represent illegal states, and to further make other assumptions explicit as functions that evaluate properties, to be used in a (property-based) test or emulation suite. - protocol / type errors / representation bugs Usually minor "brain farts" that lead to obviously incorrect behavior, but can linger due to lack of type or test coverage. Testing and/or static analysis can usually find these. Entry: Eliminate accidental complexity Date: Sat Feb 4 12:03:40 EST 2017 ( wording is very fuzzy - a little under the weather today ... ) Easier said then done... I still find myself constantly battling ridiculous problems, most of them caused by solutions that are not thought out well, mostly because of reliance on other accidental complexity. How to change this? A typical pattern: 1. I'm tired of this shit. Let's abstract it. (time passes) 2. How did that work again? Why did I do it like that? The problem seems to be that nothing is straightforward. Straightforward things are easy to remember. Abstractions however often create more trouble: yet another thing to remember, beause they leak, the underlying structure needs to be in the mind as well. It is as if the relation between the underlying structure and the abstraction remains in working memory as a background echo while developing the abstraction, but when that cache disappears, it seems that the abstraction no longer makes sense. It is as if the original structure needs to be restored to memory to understand the abstraction itself. That is really not a good abstraction! Let's find a word for it: re-abstracting through understanding an abstraction's implementation / relation between low and high. Entry: Conflicts Date: Thu Feb 9 16:31:07 EST 2017 Design is about finding trade-offs between conflicting ideas. Be a centrist! auto-generate protocols (e.g. use closures) vs. explicitly use protocols as decoupling points and contracts late binding vs early binding static types vs dynamic types Entry: Implementing protocols Date: Mon Feb 13 11:16:24 EST 2017 The main problems: - get a good grip on state changes - identify transactions In reality, there is usually a level of interpretation required: some transactions might require mutual exclusion across a state transition sequence made up of primitive commands. Entry: The kind of unknown Date: Thu Feb 16 10:42:18 EST 2017 When the "kind of unknown" is unknown, the only remedy is to get to know it first. Too much of software engineering is based on assuming that this kind is known, i.e. there is a pre-applicable pattern! If this is the case, you're doing something really really boring. Entry: Centralize code Date: Fri Feb 17 14:09:47 EST 2017 Reason: during development, it makes sense to change protocols. This means the entire system would need to be upgraded. In that case it is best that at boot, code is downloaded from a central repository. Having version diverge between nodes is very annoying. What is more annoying is to have firmware upgrade break because of protocol changes. Entry: Distributed computing Date: Mon Feb 20 21:45:52 EST 2017 Suddenly, everything is distributed. At least everything I deal with on a day-to-day basis. There are no more "simple computers" and it seems that those are never coming back. So I guess it's time to embrace connectedness. A consequence of this is that all new code written is communication code. A lot of code I write feels as if it is just routing information. Not doing anything useful other than being a network hop. Entry: Structure is what is important Date: Wed Feb 22 09:04:53 EST 2017 And somehow, it is what I keep forgetting. When I look at a problem, I seem to only account for time necessary to implement some stripped down version of it, and most of the time Entry: Exponential sensitivity Date: Sun Mar 5 08:25:40 EST 2017 It seems that for most programming work, there is a very clear "leap into chaos" once complexity rises. Complexity is combinatorial. One one hand, it is easy to generate: add one element to a design that is not decoupled properly, and the number of combinations multiplies by this one element. On the other hand, the brain somehow needs to keep track of the enumeration of cases in case of a non-decoupled property. Once this gets too large, it's no longer possible to track. This is a very strong sentiment that I reach all of the time now. The only mitigation is to squash the interaction: to ensure that either the design element is removed, or that it is implemented in such a way that it is orthogonal to other components, or at least has very limited interaction as to not multiply the number of cases (e.g. adding a case is not a problem). Entry: When to use OO? Date: Sat Mar 25 14:40:13 EDT 2017 OO is needed when a concept in your program has identity, existence over time, state. All those are maybe rooted in how we thing about "object". Anything that doesn't change over time, is not an object. Often it is simpler to think in terms of data (a transition of an object is actually two different data items). In my work, I encounter objects as hardware: physical things that have state that you cannot represent as explicit data. Programs that deal with these things are centered around COMMUNICATION. Almost everything else is data PROCESSING, which is better expressed as functional programs. An architecture is often a combination of these two. These two views are dual (actions and the things acted upon). Which side to view it as is most of the time dictated by the physical reality. Also, these can be layered: - functions can have (hidden, short-lived) objects in their implementation - objects can have functions implementing their state transitions Entry: The Debugging Mindset Date: Tue Apr 4 22:20:36 EDT 2017 http://queue.acm.org/detail.cfm?id=3068754 A good writeup. Some takeways: - Debugging method, summarized: (1) Develop a general theory of the problem. (2) Ask questions leading to a hypothesis. (3) Form a hypothesis. (4) Gather and test data against the hypothesis. (5) Repeat. - Perseverance is key. I would add to that: think bigger, wider. Question your assumptions, make them more explicit. It can take a long time to find a bug if you keep looking in the wrong places! - Fixed vs. growth mindset. Over time, I've moved from fixed to growth more consciously. Being at the edge of chaos is necessary to learn. And learning is the gratifying thing, not "being smart". Entry: Shell programming Date: Sun Apr 9 18:05:58 EDT 2017 Not sure where to put this... Bash sucks, but still I find myself writing a lot of small shell scripts. Why? Programs are the primitives, which is a big plus. There are no two distinct things (language functions and program executions) -- it is all one: bash functions look the same as program invocations. Entry: NASA coding Date: Tue Apr 11 22:03:59 EDT 2017 1. pseudocode specs 2. two isolated teams: coders and testers 3. document changes 4. fix mistake AND cause of mistake, blame the process https://www.fastcompany.com/28121/they-write-right-stuff My remarks: it is possible to evolve specs (one-off tests, property checks, model verifications) and code if there is not enough information to produce specs. Entry: Writing embedded software Date: Tue Apr 11 23:33:49 EDT 2017 So, let's embrace the inevitable: you are going to write state machines. In fact, it will be the vast majority of all the code you will write. And the perceived lack of tools will make your toes curl. You are going to want to write a code generator. Or a macro package. Or some other kind of verifier or mocker to be able to make sense of it. You are going to wish you could turn state transition diagrams into code. You are going to have to put your functional programming ideals on the shelf. Or not. You are going to try not to let go of those. You will write a macro package, embedded in a functional language, and think you've solved the real problem. But there will still be state machines. And they will hurt you. Entry: Register configuration language Date: Fri Apr 21 10:25:39 EDT 2017 A configuration language should be nothing more than a series of assignments to bitfields, possibly parameterized by offsets. The main problem is that C bitfields can't be trusted, and that this needs to be handled by read-modify-write of machine words, or on Cortex M3, the bit-band access. Entry: State and restarts, compilation caches Date: Sat May 6 11:33:24 EDT 2017 Redesign system from ground up using restartable agents, relying only on easily migratable core state and shared "constant" data == actually compilation cache. Entry: Question-oriented programming Date: Wed May 10 12:19:09 EDT 2017 Trying something new - first in staapl. Keep a list of explicit questions that are in the way of progress. When unclear what to do, answer one of those questions or make it irrelevant through a course change. Entry: User interfaces Date: Mon Jun 5 09:34:47 EDT 2017 Several points of convergence: - hatd webpages - internal monitoring pages - synth ui I need to learn about how to implement user interfaces in a way that does not have the "infinite loop" problem so often encountered in oo-style gui notification propagation. One way to get rid of that is to create a model that is based on some constraint program, and have the presentation derive from that, either through full redraw (e.g. animation), or by incremental updates in a tree-diff approach like react. Entry: GUI: Event propagation graphs Date: Tue Jun 6 10:45:19 EDT 2017 Some ideas: - Eventually, the computer this runs on needs to execute a sequence of commands, propagating data in a way that is not circular. - The problem seems simple if we didn't have to worry about this circularity: - Push: a value changes, and it is expressed manually which other values have to change in response. If this leads to cycles, the system will break the cycle by distinguishing old vs. new. - Pull: every value in the system is a function of another value. This is the "reactive" paradigm? - Constraint: there are a number of constraints between values, and the system would resolve these when one of them is updated. TODO: - Read Conal's push/pull functional programming. The push and pull ideas are central to the idea and I might have reinvented parts of his idea in the data elements implementation. - I believe what I really want is a compiler from a constraint-based specification of a GUI to an static eager imperative event handler. Read more about constraint programming. Entry: View splitting Date: Wed Jun 7 14:46:23 EDT 2017 While driving, this struck me: - the model is functional (= directed, acyclic) - the user input is unambiguous ("one hand") - the user presentation is where "choice" pops up The main insight is that the choice is where the user uses its "one hand". Then extended, multi-hand is possible as long as the controls are not linked through constraints. Basically, when the user "chooses" one side of the view, the other side is no longer an input, but becomes a presentation, functionally dependent on the active input. The topology is something like this: - DAG (engine) - constraint network (presentation+input) - DAG (presentation) It is the combined presentation and input that causes the problems: this is almost always multi-faceted. Change one "knob", and others move along. At the point the user picks one of the linked controls, the network is directionalized and the problem reduces to FRP. So, the two main problems are to implement: - functional reactive programs - directionalization of constraint networks EDIT: The thing is, if the whole model is recomputed, these cyclic things can disappear. The view is always a deterministic rendering of the model. It is not necessary to directionalize anything. Maybe that is exactly what the react approach solves? Recompute everything, then optimize redraws through diffing. This as opposed to directionalizing constraints. It just bypasses all the hairy stuff. In an animated gui, there is never a need for any directionalization if the gui gets updated as a whole on each frame. With the ubiquity of GPUs this should be assumed. Then, it might be possible to do the tree-diffing when no GPU is present? EDIT: Two approaches: - model recompute + tree diffing to compute imperative updates - constraint directionalization Entry: Gui: two approaches Date: Thu Jun 8 09:36:29 EDT 2017 recap: - model recompute + tree diffing to compute imperative updates (react style) - constraint directionalization It is remarkable that by glossing over the problem (react style), a whole class of problems is avoided. It is a reformulation of the problem that is quite remarkable. My main question now is: does it make sense to invest in the directionalized graph approach, or is the slight inefficiency of the full model update + tree diff worth it in all cases? Let's name them: - tree-diff - orient Entry: Orient vs. tree-diff Date: Thu Jun 8 12:40:20 EDT 2017 Emotionally, I am inclined to use orient, because it seems more direct, more efficient, and more widely applicable. It seems good to have the imperative update available as a backup, to leave "holes" in a UI that get updated by a different mechanism. To figure out how to do this, I need a real world example that exhibits the problem: two knobs with different scales: log an lin. I also need a platform, and that probably should be web with imperative dom updates. Entry: Flux, tree-diff Date: Sun Jun 18 23:59:21 EDT 2017 If I understand properly, the point of flux is to re-render everything after a user event. There is a single direction of data flow which terminates in the update of the user's display. The main benefit: it allows the view implementation to be functionally dependent on the view specification. No fussing around with state that we didn't know where it came from. React just re-renders in an efficient way. But what if we don't really need to do that. What if our specification -> implementation function is simple enough? Entry: privileged-ports-are-causing-climate-change Date: Fri Jul 7 15:59:56 EDT 2017 "nobody ever thought to adapt its networking layer to accommodate the needs of network service multi-tenancy." http://adamierymenko.com/privileged-ports-are-causing-climate-change/ But to be fair, boxes in boxes is because of a social problem: things are broken and badly integrated.. Entry: embracing events and distributed caches Date: Fri Jul 14 13:55:45 EDT 2017 Coming from an FP world, I've been trying to avoid "distributed state" for a while. In reality, it is not possible to get rid of it altogether, so best to embrace it. Locally, it's possible to perform state updates in a more orchestrated way to avoid untractable event propagation patterns. E.g. this is what ReactJS does: change the granularity of the message/object structure. Basically, this comes up a lot: - Thing goes live, queries other things for current state and registers a notifier. - Notifications come in and local cache/representation is updated, possibly notifying other things it has registered as listeners. The source of this problem is that it is much too expensive to transfer the entire state from one (physical) spot to another, if only a subset is needed on the receiver or only a subset has changed on the sender side. Incremental updates are a very important behavioral pattern that looks as if it is an optimization, but is so extremely inefficient when not performed, it presents itself more often as an explicit feature or requirement. Entry: Caches Date: Fri Jul 14 14:10:05 EDT 2017 Maybe obvious, but getting into this as being a central problem to a lot of architecture problems. Cache trades management of local state against resource use, i.e. communication overhead, and computation time. Essentially computation time is also communication delay, so really the idea is: "remember shit so you don't have to bother anyone to get your job done". Caches rely on reliable change notifications, and (optionally) on reliable initialization procedures. At the level of data structures, this is about representing differences -- the update operators -- in a way that is efficient. Entry: Many small functions hurt readability Date: Wed Jul 19 22:53:24 EDT 2017 https://lobste.rs/s/8e4wkl/code_health_comment_not_comment What this needs is tooling: Optionally inline the function with the click of a button. Inlining is a wole lot easier to do than automatic factoring. Factoring _adds_ information: the blessing of the writer that this is really a detail that is not relevant in the current context. Entry: Run .NET languages on linux? Date: Thu Jul 20 16:03:34 EDT 2017 Basically, I want to develop a .NET library. How to do this in a language that is sane? Maybe not really.. Entry: Which language to use? Date: Mon Jul 24 18:08:51 EDT 2017 Erlang seems to be really good for system and network glue. Simple language, and not too strict to make hacks easy. Entry: Server or Procedure? Date: Mon Jul 24 18:11:00 EDT 2017 One of the main distictions to make is "does it have local state or not". If it has state, it is an object/server, otherwise, it is a function/procedure. Entry: How to manage a software project Date: Mon Jul 24 19:09:52 EDT 2017 - get settled so you can take a hit - take hits Entry: I no longer think binary logs are a good idea Date: Thu Jul 27 23:51:41 EDT 2017 Especially when they get large, and the format changes, they are hard to work with. Next time: parsable logs, one entry at a time. Entry: One version Date: Wed Aug 16 10:17:53 EDT 2017 Have been trying to get my head around developing multiple applications that depend on the same library. The simplest way to do this is to keep only one version of each, and at commit points: - commit library to main repo - pull library into application - commit application with new version (either submodule, or other way to pull git hash) Entry: Events and chunks Date: Mon Aug 28 15:02:38 EDT 2017 Doing distributed embedded work, it seems a lot of time is wasted just in chunking/unchunking data. How to solve this permanently? The main issue is to construct nested data types without a memory manager. Or to incrementally build an object into a queue. Look into what John's project does and see if it makes sense to work on this. Otherwise, it seems simpler to find a way to work with chunked data on a uC, essentially "virtual memory". Entry: Modularize through projecting on aspects Date: Tue Sep 12 09:35:51 EDT 2017 So what is an "aspect". It is a projection of the program onto some "concern". In a functional program, this seems to be easier to do. E.g. in Erlang, it is mostly a projection onto sequences, which can actually be implemented as processes that listen to events generated during runtime. The method is to always be projecting onto some concern after, and having this guide the refactorings. In a functional program, this actually seems to lead to better modularity. Entry: Do less Date: Wed Oct 18 19:37:25 CEST 2017 It's really the only way forward. Simplify. Create leverage. Converting from net to star seems to be a big part of this. Reduce structural complexity from N^2 -> N+1. Where the +1 is the simplifying representation. Entry: Another reason to use strong types Date: Mon Oct 23 13:28:02 CEST 2017 Practical real-world data littered with real-world edge cases can get _really_ boring. Manipulating complext structures based on what they actually contain vs. "they contain what I need" creates significant cognitive load that I just can't bear in cases where program structure is all I can think about. Entry: Working with sequences? State machines are your friend Date: Thu Oct 26 10:18:48 CEST 2017 There are so many ways to combine sequences -- but the base representation which works also on the lowest level -- is a state machine. Entry: When it's not crystal clear Date: Fri Oct 27 11:45:36 CEST 2017 Don't fall for it. When it's not crystal clear, you DO NOT get to estimate time correctly! These problems are hard: - Multiprocessing, Synchronization - Testing non-pure code - Testing things that have physical state (non-emulatable, non-mockable). Entry: This is how you write imperative code Date: Sat Oct 28 17:55:30 CEST 2017 Ingredients: - pure functions - abstract data types (needed by functions) - sequential FFI 1. Create a pure representation of state updates, meaning a data structure that needs to be interpreted. == "update script". 2. Create an interpreter to apply these pure update scripts against a stateful object. Preferrably, using transaction semantics for updates that can fail. 3. Put all the logic to compute the update in pure functions Entry: Now, what are the actual problems? Date: Fri Nov 3 08:52:41 EDT 2017 1. Pure functions + abstracted recursion patters and separated logic and effects are great tools, but they require types when implemented by an aging brain. 2. Build and integration systems are a real problem. 3. Linux is more complicated than you think How to solve this? The problem is in tools. Maybe the problem is really the build system used to integrate all the heterogenous components. It doesn't seem possible to get rid of the heterogeneity. Is there a build system that can be made to work reliably? Entry: Types Date: Fri Nov 3 10:48:36 EDT 2017 So more and more, I am pointed in the direction that types really are the solution to managing complexity. Basically, complexity is unavoidable. Types make it possible to make more complex structures with less congnitive load, including more complex reusable abstractions that reduce application complexity by shifting complexity. Entry: Microcontroller Development Date: Sun Nov 5 08:16:19 EST 2017 Previous approach: put a GDB stub on the target. The advantage here is that it works without any other infrastructure, when exposed as a serial port. However, the stub protocol is messy, and is really better implemented elsewhere, with the "main tethering line" kept clean. So, either find a way to embed a different packet protocol in the GDB protocol (e.g. "monitor"), or move the stub code off the uC. Entry: Systems programming needs a real language Date: Sun Nov 5 09:35:40 EST 2017 What I'm trying to do in Haskell is not going to cut it. Something needs to catch the need for turing completeness at the lowest level. Once the problem domain is separated from the hardware, it might be possible to use reduced expressivity languages. Entry: While implementing, make tiny arbitrary choices explicit Date: Thu Nov 9 13:38:35 EST 2017 Basically, document the decision points. These are very useful for understanding why something is there. Also, often I find that decisions tend to be more "emotional" when they are not made explicit, using more general aesthetic rules instead of a decision process informed by the actual problem. Entry: "static" OS Date: Mon Nov 13 09:11:50 EST 2017 What I see mostly is the mixing of state machines. I spend a lot of time transferring bytes from one buffer to the other, where the "chunking" is the cause of context switching. It seems this cannot really be avoided. However, the core problem in the implementation is where the loops will be: on the read end or on the write end? It is largely arbitrary for the meaning of the program, but it makes a lot of difference for the efficiency, because context switches are not free. How to make this core problem explicit? The purpose of an operating system is to connect I/O between multiple tasks that have individual run state -- either an explicit finite state machine, or a (finite) stack. Elements: - tasks are exposed as state machines that produce inidividual bytes through a tick() function. - efficiency comes from unrolling the state machine tick() functions. user should only specify buffer sizes and all the rest should be done automatically. - source transformation converts any kind of iterator specification (fold, source, sink, lazy list) into state machines. So the compile time loop inlining is an essential element. The key thing to understand here is that the USER INTERFACE is to write state machines in terms of nested loops. In practice, most state machens follow that approach. When you actually need a raw dispatcher and explicit state switcher, it's probably worth it to implement that in terms of the task abstraction, instead of the other way around. A source transformation should be able to do this. Basically, straight-through code with i/o is Erlang, and this is a really neat model to work with, as it allows effects and purity to be separated. See also next one. Entry: Streams of functions : where does the loop go? Date: Mon Nov 13 09:54:20 EST 2017 What I see a lot is the need to turn loops "inside out". I.e. instead of folding a function over a list of elements, what happens often is to fold an element over a list of functions. In C: parameterize a sequence of destination pointers. The question is always: where does the loop go? The answer should be: this is an optimization that should be arbitrary to change. We do want the programmer to specify where this goes to be able to make efficiency tradeoffs, but we do not want to burden inner functions with this. Entry: C interface for generic source/sink Date: Mon Nov 13 09:58:51 EST 2017 Since I'm not quite there yet writing a DSL for this, how to get the benefit of abstraction on the C level? The base abstraction should be state machines in the form of state + read() / write() functions. I think sm.h already has an abstraction for this. Work with that. EDIT: This is just Unix pipe read/write. Entry: Compile time pipes Date: Mon Nov 13 10:01:32 EST 2017 A good way to look at the idea is to see it as compile time pipes: tasks have byte-level interfaces, and can read/write one byte at a time. The task of the DSL / compiler writer is to make this efficient. Anything else can be built on top of that. Is this CSP channels? But once byte channels are there, one probably also wants object channels. Those need a central store. This might be a problem. Is it possible to keep it "linear", e.g. always copy objects? Byte channels might be too simplistic, but they do compile well down to any architecture. Let's give it a try. These are byte channels. Buffers are not needed: when one byte is written, the reader can continue to execute. Buffering is then an implementation detail and it should be possible to visualize a program in a graph to see where buffering would be beneficial. When buffers are added, yielding is necessary, to indicate "done for now", time to switch tasks. The elements then become: - Write programs in terms of byte pipes, or word pipes, or whatever unit makes sense for the program. Often though, microcontrollers need to do byte-oriented protocols, so bytes are probably a good default. - Observe that execution speed comes from buffering writes and reads, which removes task switch overhead for the producer and consumer loops. - Make the entire connectivity graph concrete, and annotate it with static buffer sizes. Entry: In C: buffers are actually just state machines Date: Mon Nov 13 10:15:16 EST 2017 When you find yourself writing to a buffer, what you are actually doing is writing "into the reader" of a state machine. Conversely, reading from a buffer is really just reading from the writer of another state machine. The big lesson is that it really doesn't matter when code runs, only that it runs in time to produce the right data. Entry: Data is decoupled control Date: Mon Nov 13 10:28:29 EST 2017 The purpose of data is to be consumed by code. I've long a go read a nice articulation about this regarding algebraic data types and pattern matching, boiling down to the idea that constructing a data structure is the same as invoking all the code that matches on it. In essence, it is function call into the future. On the lower level: a write to a data structure is a delayed jump to the code that reads it, and can often be folded. Not sure what to do with this generalization, other than the concrete idea "a buffer write is a write to the state machine that reads it". Entry: Assumption: write sink-parameterized code? Date: Mon Nov 13 10:39:16 EST 2017 The assumption is that while sinks and sources are completely dual, it often make sense to keep the producer concrete, but make the sink abstract. I.e. sink-parameterized generators. Entry: Why are abstract sinks such a good abstraction? Date: Mon Nov 13 10:53:07 EST 2017 Why is this such a good abstraction compared to left fold? The problem with left fold for embedded software is the reliance on allocation to produce the results. While calling a function to produce a result is abstract, this is not so easy to do in C in an efficient manner. This would only work well if there were some kind of linear allocator. It seems more sense to avoid it all together, and have the iterator do less: just push elements into some other (stateful) object. Essentially, it is cheaper to have "multitasking" in C than it is to have fully abstract function evaluation! Let's call sinks/sources what they are usually called: channels. So, doing this at compile time needs a transformation between folds and channel programming. It would be interesting to do this first in a language that has accessible tasks such as Erlang. Entry: Lazy Generalization Date: Mon Nov 13 12:28:02 EST 2017 Generalization often takes up a lot of resources, both programmer and run-time resources. If there are no multiple use cases, write it concrete unless it would otherwise be hard to read or write. Some say to generalize at 3 similar use cases. I most often generalize at 2. Entry: Parameterized one-size-fits-all : the weaver antipattern Date: Mon Nov 13 12:29:36 EST 2017 I often feel the need to write a single complex parameterized routine that can implement several similar but not quite the same behaviors. This smells. Code like that should really be written as a composition of some generic patterns. Entry: Eventual consistency / single writer Date: Wed Nov 22 10:47:03 EST 2017 Might be useful to understand the principle, so computer admin can be done in this way. My intuition so far for the problem I'm facing: There is a central list of edits. Each node needs to play back the edits when it comes back online after "missing time" somehow -- can be benign: laptop powering down, or just no internet connection. This is simple because there is only a single writer. Implement it first for /etc/net and ~tom/pool It could be implemented as a git or darcs repo, or even a nix channel. Something that will fetch current head from master and instantiate it. Make it push/pull: - sync after lost time - subscribe to notifications A downside to this is that bad edits that bring down a node will spread, so maybe keep it manual for now? Entry: Do not do automatic updates Date: Wed Nov 22 10:54:12 EST 2017 Basically, you want to be there when something breaks after an update, in the mindset of dealing with the a possible problem. Entry: Unit tests are about design Date: Sat Nov 25 00:43:09 EST 2017 They are much more about illustrating what a unit is for, by placing it in a simplified context, than to "test" it. Entry: Spec functions Date: Sat Nov 25 01:09:28 EST 2017 Next time that I'm being tempted to implement a complication in the name of efficiency, write the simple but inefficient thing first, and use it in a property-based test to "prove" the complication is correct. Entry: tock, mini os in rust Date: Tue Nov 28 19:39:37 EST 2017 https://www.tockos.org/assets/papers/tock-sosp2017.pdf Thursday there is a meeting about this in Detroit area. https://www.meetup.com/rust-detroit/events/244855856 Entry: replace dependency trees by a flat top level builder Date: Tue Dec 5 20:28:49 EST 2017 This allows dependency resolution to be performed in one spot. Entry: Cached builds Date: Sat Dec 9 13:16:59 EST 2017 Do this right, once and for all. The question is: what are we building? The answer is: a cache directory. EDIT: The core of the problem is this: Projects should _specify_ their dependencies (inputs), but should not _manage_ them. Management needs to be done in a top level project, which is essentially the build/CI system. Reasons: - rebuilds and caches - acyclic dependency graphs vs. plain trees Entry: Need to go back to imperative programming Date: Tue Dec 12 20:53:23 EST 2017 Fold are nice to work with, but really, sometimes it just makes a whole lot more sense to just make a loop that writes into a damn buffer. Entry: Shake Date: Sat Dec 30 01:27:25 EST 2017 http://shakebuild.com/ http://ndmitchell.com/downloads/paper-shake_before_building-10_sep_2012.pdf Basic contribution: dynamic dependency graph. Entry: Why are things so arbitrarily incomposible? Date: Sun Dec 31 17:03:46 EST 2017 Obviously, value comes from composition. Why is it in practice still so hard to put things together? Why are there so many decisions made that limit reuse, or depend on unportable context? Entry: Starting over Date: Wed Jan 3 21:36:39 EST 2018 Re-scared by this intel thing and all the recent security bugs. Things are getting too complicated. So, what about trying to start over again? Re-embedding all the things I use computers for? Use a decent language, a decent platform? Something that is simple such that it can work on simple machines (towards RISC-V). Imoprtant: 1. the language, needs to be changable 2. the base hardware abstraction What about going to OCaml this time? Simpler, and solves the real problem: types for the aging brain. I'd also expect it to be easier to cross-compile. https://stackoverflow.com/questions/12885960/how-to-build-an-ocaml-cross-compiler Maybe instead of Haskell+Erlang, try Ocaml+Erlang. The problem I have with Haskell is that it is a time sink of type magic. EDIT: But what should such a system do? What do I use computers for? - Reading and writing email, documents, code - Keeping an archive From that perspective there is really no hope. Libraries need a substrate (OS), and substrates need powerful hardware. Entry: if state info can get lost, save it! Date: Thu Jan 4 02:00:05 EST 2018 Mostly about managing bad interfaces. Sometimes a state cannot be queried, only "influenced". Keep a model around. Make the model authoratative to the rest of the application. Basically the same as UI design VM. Entry: State machines should be the default Date: Thu Jan 4 22:48:07 EST 2018 Because it is a much simpler way to deal with events. Often, a design is easier to express as many simple machines somehow connected. I.e. "more physical". Entry: embedded.fm 175 (RTOS) and 179 (hierarchical state machines) Date: Fri Jan 5 18:15:08 EST 2018 RTOS one: basic stuff. Meat & potatoes, but not something I particularly gravitate towards. http://embedded.fm/episodes/175 Next up was hierarchical state machines, which I believe is close to my idea of composable state machines and the sm.h "task like" macros. http://embedded.fm/episodes/179 Some basic ideas: - effects generate new events (missing in my system) - the above requires queues.. how to pick queue size? or is there "pressure" on queue full? So really, the core question is about queues: the memory management, and how priority is handled. What to search for? "actor framework in rust". https://github.com/actix/actix https://actix.github.io/actix/actix/ Entry: Distributed applications and "IoT" Date: Sun Jan 14 22:10:15 EST 2018 Something I read in an online trade magazine about IoT not getting past the stage of "smart sensing". Maybe a little bit of preprocessing or filtering, but it makes sense that most applications do not do much with the data apart from sending it on for central processing. What I'm starting to feel is that the basic problem is the communication network. Ethernet might not be the right thing, as it practically requires a Linux host, and for WiFi it becomes even more strict. It's time to figure out how to handle this better. Make a go-to solution for a multidrop RS485 bus. A good architecture seems to be to have a Linux host be master, so it can also bridge to Ethernet. Entry: Erlang vs. Haskell Date: Mon Jan 15 21:36:11 EST 2018 The thing is, with Erlang, I feel I do not have enough leverage. Something is holding me back, and I'm starting to believe more and more it is the types. Writing Haskell code is more intuitive in the sense that more focus can go into the "dreamy design part" as opposed to keeping the wheels on the track. Now, why is it that every time I want to actually start doing things, I feel I am "stuck in the mud"? Tooling, and "impedance match" to existing protocols. Entry: Privilege separation, untrusted code Date: Sun Jan 21 07:55:44 EST 2018 I still have not solved: - editing root-owned scripts - running root code in Erlang After reading: https://cr.yp.to/qmail/qmailsec-20071101.pdf The key is to eliminate trusted code as much as possible. However he also argues that privilege separation doesn't solve any real problems. What is the subtlety there? In 2.5, look for: "Minimizing privilege is not the same as minimizing the amount of trusted code, does not have the same benefits as minimizing the amount of trusted code, and does not move us any closer to a secure computer system." "The defining feature of untrusted code is that it cannot violate the user’s security requirements. Turning a “DNS helper” into untrusted code is necessarily more invasive than merely imposing constraints upon the operating-system re- sources accessed by the program. The “DNS helper” handles data from many sources, and each source must be prevented from modifying other sources’ data." Entry: Trusted code, an exercise Date: Sun Jan 21 11:26:44 EST 2018 I will need to run some applications that are considered critical. An example is a thermostat. The separation primitives for these will be host-based: I no longer trust same-host privilige separation after the spectre madness, and it seems that accepting this now and dealing with it at a design level is the only sane solution. This means: - Sensor, controller, actuator all run on dedicated hosts (network or usb connected). - Communication is packet switched with broadcast option - Communication is encrypted + authenticated (E+A) It is assumed the medium is not trusted, but the end-point hardware is (i.e. key storage is considered safe). Physical access compromise is considered to be an independent problem. - Attack surface is untrusted physical network. None of these hosts will "pull" information from non-trusted internet systems. All input comes from trusted nodes, e.g. hardware terminals, and is E+A. - Debug access is through SSH, extending the attack surface to To do this on a small microcontroller requires a bit of effort. I cannot point straight at reusable code. So maybe for the time being, roll my own, then replace it with something better later. Entry: Eliminate compilation Date: Wed Jan 24 18:32:59 EST 2018 Maybe it's time to take this serious. Whenever I save a file that is a source file for a "live" system, I want it to update immediately. In Erlang, this is possible. In C it's harder but still possible to get a compilation queue going. Entry: When you don't see the light Date: Wed Jan 24 19:37:06 EST 2018 Sometimes it is hard to see a simple design up front. In that case, it helps to just build _something_. That makes it clear where the ugly bits are, and cleaning those up often leads to a simpler architecture, or at least an idea of how to build one. Entry: Intermediate state Date: Tue Jan 30 08:56:02 EST 2018 One important concept that I do not see handled explicitly in many code tools: when I'm writing something, the code is always in some intermediate half-working state that I do not want to commit. Once I get it running, I commit. Tools should have a way to embrace this intermediate state and create a 1-1 correspondence between code and interaction system. Entry: Sychronization Date: Fri Feb 9 09:32:33 CET 2018 After doing some Erlang, it is quite clear how many system-level functionality has race conditions due to bad "sequentialization synchronization" that happens automatically in an Erlang service architecture. But it's also clear that a simple queueing mechanism fixes most of these problems. E.g. using task-spooler. Entry: Setting up Shake Date: Sun Feb 11 16:08:19 CET 2018 root@zni:/etc/net# stack install shake Run from outside a project, using implicit global project config Using latest snapshot resolver: lts-10.5 Writing implicit global project config file to: /root/.stack/global-project/stack.yaml Note: You can change the snapshot via the resolver field there. Downloaded lts-10.5 build plan. AesonException "Error in $.packages.cassava.constraints.flags: failed to parse field packages: failed to parse field constraints: failed to parse field flags: Invalid flag name: \"bytestring--lt-0_10_4\"" FIX: stack upgrade Entry: Re-use -- we're stuck with this Date: Sun Feb 11 20:57:47 CET 2018 About starting from scratch: not possible. The existing code base is just too powerful to set aside, and many real world problems would be hopelessly complicated to tackle without some kind of protocol library or driver. The entire system is too entangled, and complexity mostly comes from stringing together pieces with different protocols, creating a bigger and bigger mess. The only way out of this is to go back to basics, back to hardware: create devices that do something physical, and create them from scratch. Entry: Too hard to generalize Date: Mon Feb 12 10:55:24 CET 2018 I like the quip by Sussman that programming these days requires "basic science", i.e. trial and error. How did it get to this point? An example: I have a script that performs several ssh connections in a row, which is slow over a high-latency link. The solution here is to reuse the connection using a control master. I.e. a cache. However, this behaves as a global variable, and requires extra machinery to turn it into some RAII mechanism. If this were implemented in a programming language, it would be easy to express. Instead it is part of the OS quilt. So many things that should be simple, are not. Mostly because they are implemented in a concrete way. EDIT: Fixed the control master thingy.. Took about an hour. EDIT: The problem lies in distributed, connected systems that are constantly upgraded. If you just build a dedicated widget (even with "unix-level" tools) and don't change it, it seems to work just fine. Entry: fix 10.1.2.0/24 routing rules Date: Mon Feb 12 12:00:41 CET 2018 Forwarding is allowed. Incoming is not. 10.1.2.0/24 is only for forwarding traffic between "public" networks. Uncommenting these: root@zni:/etc/net/fw# grep -re soekris * | grep INPUT garage.sh:#iptables -A INPUT -i soekris -j ACCEPT pi2.sh:#iptables -A INPUT -i soekris -j ACCEPT pi.sh:#iptables -A INPUT -i soekris -j ACCEPT pt.sh:$IPTABLES -A INPUT -i soekris -j ACCEPT xm.sh:#iptables -A INPUT -i soekris -j ACCEPT Entry: For boring software.. Date: Mon Feb 12 20:25:01 CET 2018 ..it should be possible to use it and modify it without really thinking very hard. This is easy to test: work on it when tired. "slacker code". Because that is the only way the dreamy side of the brain gets activated. It is currently underused. What this means really, is to make it easy to follow the bread crumbs. Easy to debug, or easy to modify. Entry: Backups Date: Mon Feb 12 20:38:35 CET 2018 Don't perform bi-directional syncs. Keep a borg master, then rsync or btrfs-send/receive clones. Entry: Disappointment Date: Mon Feb 12 22:41:07 CET 2018 I wonder how much of this comes from the fact that I've written a lot of sloppy code, and am getting tired of having to redo things over and over again. Espectially when it comes to script level code. Entry: equational reasoning Date: Fri Feb 23 20:36:48 CET 2018 There is this thing about functional programming an equational reasoning that allows some processes to be delegated to unconscious pattern matching. I find this especially prominent when refactoring. Essentially, filling in the gaps in a proof. E.g. doing Agda. Entry: Instead of Shake, use Bazel? Date: Wed Feb 28 16:32:46 CET 2018 https://www.tweag.io/posts/2018-02-28-bazel-haskell.html https://news.ycombinator.com/item?id=16482842 Entry: Am I just whining about types? Date: Sun Mar 11 21:09:12 EDT 2018 Code is really all glue. The code that you would think of as code that actually does something is a very small percentage of all code I write. So no, I'm not whining about types. Glue code better be typed. Because it is utterly boring in its details. It shouldn't be there in the first place if interfaces were well-adapted. Entry: The difference between low level and high level code Date: Thu Mar 15 16:53:43 EDT 2018 Low level code: - write once, until it works. - specs are 100% clear - cyclomatic complexity is low High level code: - change often, change functionality - specs are never 100% clear - cyclomatic complexity is often high Entry: Build systems Date: Sat Mar 17 10:09:47 EDT 2018 It is really hard to properly set up a build system that allows small changes of dependencies without doing full rebuilds. Both nixos and rebar do full rebuilds. It seems almost impossible to do this, unless a monorepo is used. The boundary between multi-repo and monorepo is so arbitrary. EDIT: Here's the thing. - Make a single incremental build system for the project, including all dependencies that are regularly rebuilt (e.g. single makefile). - Create a second level build system to build the dependencies that are constant (e.g. nixos). Monorepo vs git submodules is quire irrelevant: the differences can be smoothed over with scripts. What is important is that the repo + submodules uses a single incremental build system. Entry: Build systems Date: Sun Mar 18 01:18:29 EDT 2018 I simply cannot get out of the swamp! Each projects has its own rough edges, where there is no simple connection between editing code and running the modified version. This is simply unacceptable. Can it be solved? For Erlang, yes. For other things, not so easily. Will usually need a restart. So use erlang as the skeleton? To keep something going. Entry: Incremental build and deployment Date: Sun Mar 18 11:33:37 EDT 2018 This is such a key problem. Let's define it better. It all boils down to managing "diffs". I have a good way to do this in Erlang, already used for presentation model stuff. An important component of this is to be fault-tolerant to approximations of rebuilds. Think buildroot: it is not an incremental build system, so benefits from resetting periodically, e.g. nightly. The approach to take is to build an overlay build system. Something that only computes differences, and can be "reset" in case it messes up, is integrated with editing and can perform fast reload updates. It should be initialized by the standard "rebuild" build system. Entry: Building large projects Date: Sun Mar 18 12:46:24 EDT 2018 What is the actual problem? - Buggy partial rebuilds. - Dynamic types make it hard to catch build system errors. Some errors only appear after a full rebuild + full test. These problems are not going away. A good solution needs to take them into account. What is known to work well? - Automated full rebuilds + deployment + test. - Package level partial builds (e.g nixos. only build updated code). - Not updating the build host's build context that's unnaccounted for Entry: Live coding: focus on the code buffer that sits between push and pull Date: Sun Mar 25 21:46:42 EDT 2018 PUSH = source code "save" event, which "blesses" the code as correct, and triggers a compilation + checks. PULL = something that either executes the code directly, or explicitly fetches it to execute somewhere else. In between sits the code buffer. The main design constraint: The code buffer needs to serve pull until we power up the edit system and perform a new push. Entry: The F6 approach -- breadcrumbs Date: Mon Apr 2 05:43:38 EDT 2018 This is mostly to find better ways to deal with the aging brain. What works? - Leave a Makefile with clearly labeled "start here" targets - In addition to that, provide a "lens", i.e. an expect test. A document that contains code to generate examples. Entry: Normal forms Date: Sat Apr 7 11:01:41 EDT 2018 In distributed systems, the "multipath" problem can be really tough to get under control. It is therefore imperative to have a normal form. Multipath here refers to commutation of functions, as happens with e.g. - file storage + code execution: - run code locally, access files remotely - run code remotely, with local file access - software development of shared libraries: libraries need hosting, so there should be a single, canonical host to perform development, preferribly one that has "everything" that is ever used together. i.e. the "main system". - develop on main system, then push to subsystems - develop on subsystems, then push to other subsystems basically, when things get big, develop on the monolith. make sure you have a "main system" that contains everything you would need to test. Entry: Code locations are so important Date: Tue Apr 10 18:06:35 EDT 2018 I.e. "where does the code go?" Creating a new location is often seen as a defeat: why does this need another module? Can't we incorporate it into something else? The overhead is also sometimes quite substantial. Entry: Erlang + C Date: Thu Apr 12 09:38:42 EDT 2018 Some sweet spot: 1. Run code in external processes. 2. Use {packet,4} 3. Encode commands as an array of uint32_t with inlined binary 4. Log errors as a stream of arrays of uint32_t 5. Encode larger data structures as mmap() files 6. Use native integer endianness inside messages In general, make it such that the C code doesn't need to do any memory allocation, and that it can find its input and output memory in the form of simple arrays. Let it log errors to the output stream terminated by an empty packet. Do all the preparation of the protocol messages and data structures in Erlang. It is surprising how much this can do, letting both languages do what they are good at: - C: loops over memory reads and writes - Erlang: nested data structure manipulation, protocols, providing a general framework to implement tests. I.e. all "bookkeeping". Entry: Need for object-orientation Date: Sat Apr 14 11:52:32 EDT 2018 When data grows large. I.e. there will be a "database", that has a stateful interface with which to interact, instead of owning the data as actual. But then again, what's the difference if this is abstracted in a stable, non-mutable view? Entry: Closures vs. events Date: Wed Apr 18 20:18:24 EDT 2018 I started out the web programming using full closures for actions, but I've been gradually moving towards a viewmodel approach with explicit event processing, viewmodel update and viewmodel diffs to commands. Entry: Learning a new architecture Date: Thu Apr 19 11:42:57 EDT 2018 Why is it so difficult to get started? Entry: GUIs Date: Fri Apr 20 22:46:22 EDT 2018 So what's the real point of all this interest in user interfaces? I want to make custom editors, in the broadest sense. The most important one is build systems which relate some form of textual input to a running piece of code, interacting with the world. Entry: Snapshots are nice Date: Sun May 20 12:23:05 EDT 2018 BTRFS snapshots can be used to create checkpoints (target) and freezing (source), to be used for transactions. Entry: The problem is composition, not primitives Date: Sat May 26 13:50:14 EDT 2018 So why is it so counter-intuitive? Connectivity is usually glanced over as "just hook these things up", with full focus going into how to implement primitive operations or even higher level logic. In practice however, a significant amount of time can be spent in "glue". Why is this not taken serious? EDIT: This is great about Haskell: the library is nothing but composition abstractions. Entry: In defence of design patterns Date: Thu Jun 14 21:03:01 EDT 2018 You can't have simple implementations when they are shared broadly: everyone needs something just a little different, and this adds up. That is where design principle reuse makes sense: keep implementations minimal, but reuse ideas. Maybe this is also in defence of metaprogramming: complexity at the metalevel is usually not a big deal. Entry: 4 hours Date: Wed Aug 22 18:05:30 EDT 2018 4 hours each day of "big picture + details". Then the rest is either only a very specific detail without context, or very vague terms, abstracted through a powerful tool. Entry: Back to Erlang Date: Fri Sep 7 23:17:10 EDT 2018 Took a couple of months playing with Haskell, which was great. It works well for "language-oriented" programming, for code that is algorithmic in nature, and heavy on the data structures. I do want to come back to Erlang though. Which works better for more ad-hoc things. Where the idea is more that of "control" and "events". Entry: build systems solve social problems not technical ones Date: Sat Sep 8 15:30:04 EDT 2018 The problem is to have people agree! Entry: Making incremental builds faster Date: Sat Sep 8 22:24:49 EDT 2018 There has to be a way to do this for other sytems... EDIT (9/10) This problem keeps coming back. And really I don't see how something so simple can get so complex and out-of-hand. What about this: construct an explicit dependency graph. Now, invert it, such that it becomes clear what needs to be recompiled on edit. Then at commit points (save), recompile and build. This idea keeps coming back: - Keep legacy build systems for "offline" builds. E.g. a nightly rebuild, or the final delivery. This is the "reference build". It will necessarily be hacky. - Construct a (partial) incremental build system that is validated against the main build. For validation, one possibility is to use the incremental build system to implement the "edit-compile-run" cycle, and have the main build system in the background run in parallel to check that the incremental system is working properly. In the last couple of weeks I've run into the Haskell Data.Graph package. Maybe have a look at creating an overlay. Also, there is the Haskell build system "shake". This might work as well. EDIT: What about just creating a good Makefile? I.e. do this using a tool. This way, the "offline" builds will benefit as well. Entry: Complex code: 1-1 or not? Date: Sat Sep 15 08:16:11 EDT 2018 There is a tension that arises in building complex systems: - You want the code to be in 1-1 correspondence with the state of the machine, to make it more predictable. - You also want to be able to make quick changes to try things out Not every behavior change can be quickly made parameterizable, so you'll end up waiting for your build system when sticking to the 1-1 correspondence, or you'll end up subverting it if you stick to the second. Bottom line is still that build systems are the real problem when working on cross-cutting integration issues. Entry: glue noise floor, implicit properties Date: Sat Sep 15 12:35:02 EDT 2018 The other real problem is the lack of incremental and coherent program construction. There are too many duplicated interfaces. So how to fix this? The problem here is greed: we want powerful, useful systems, and therefore we put up with "glue", tying together things from different parts. Now, wrt these microctontrollers: often, they just need to do something very specific tied to one of the peripherals, so register access is important. I can mill over this, but there doesn't seem to be any shortcut: things are just complicated, and in all practical situations, the least complicated solution is going to have a combination of "glue noise floor" and "implicit properties", i.e. the difference between what the machine is instructed to do, and what it eventually accomplishes at the macro level. Entry: Presentation models and the unit machine. Date: Wed Sep 19 15:50:01 EDT 2018 There is a concept from mechanics called the "unit machine". It is a collection of parts that interact in a way that requires very tight coordination. Basically, the entire machine is one coordinate space where there are strong constraints between the coordinates. Think of an interference engine, where pistons take turn to occupy space. When the timing belt breaks, this constraint is lost and pistons can crash into each other. Basically, when composing sub-machines into larger machines, a decision needs to be made about modularity: do the machines interlock, or are they largely independent? A same trade-off appears in gui (panel) design. Generally, constraints can be solved by using a presentation model: - Some arbitrary state gets updated by events - A new presentation model gets generated for each update - The diff of the presentation model is used to generate change commands Now suppose I want to put two gui widgets on one panel (web page). The choice is then to interlock them, i.e. create a single presentation model, or to keep them fully independent. Here "independent" means that there are no circular control dependencies. Bascially, a change in one control cannot change multiple controls/views. EDIT: I think the underlying idea is valid. As usual, it is not exposed very well here. The basic architecture revolves around two kinds of hubs: - A global status hub - Some smaller, localized hubs, e.g. PMs for guis Signal flow input -> controller controller -> distributed models distributed models -> status_hub status_hub -> controller controller -> display drawing this, here are some possible signal flow paths: input -> controller -> display input -> controller -> model -> status -> controller -> display model -> status -> controller -> display modelA -> status -> modelB Note that it is still possible to have model -> status -> model loops, so maybe it is a good idea to formalize the event dependency path? Entry: Loopless? Date: Wed Sep 19 17:57:39 EDT 2018 Previous model works as long as models don't loop. The GUI is already loopless. How to solve? It would be trivial with an explicit dependency graph. Is that actually possible? Maybe this is not such a big problem in general. Models can usually be kept simple and structured and naturally are hierarchical. The cycles I've seen are mostly in UIs, where two controls set the same variable, but when one is used it updates the other. EDIT: This needs to ferment a bit. Entry: Composition Date: Wed Sep 19 19:22:55 EDT 2018 What I miss is composition. There are two ways that seem natural: - Composing widgets simply as juxtaposition on the page, as cells inside other layout. This should be trivial if separate processes are used. - Composing presentation models. This seems like it is more work, but it is very structural. I bet if it is parameterized the right way, it could be done better. Entry: Presentation models Date: Sun Sep 23 23:21:49 EDT 2018 So, when do these make sense? Let's illustrate according to the task, which is to set one of a number of words to a particular color using imperative GUI commands. There are two approaches to this: A. Redo: - reset everything to no color (loop) - set the element to the desired color B. Update: - reset the previous element's color (needs bookkeeping!) - set the new element's color The former is conceptually simpler, as it does not require keeping track of the previous state. This can be done by sending the appropriate commands to the GUI. However, it is possible that the loop in A is inefficient. If it is not efficient, we're done. If it is inefficient (e.g. it would require a reflow in a Browser), an alternative approach is to model the GUI, perform the operations on the model, and then use the difference between the previous model and the new one to generate update commands. This is still conceptually simple (the generation of the update commands can be automated), but has the advantage of being efficient at the GUI target end: only minimal edits are applied. Essentially, a presentation model ensures constraints are satisfied implicitly, by re-rendering instead of trying to invent transactions that preserve constraints. Entry: models are necessary Date: Sat Sep 29 23:09:36 CEST 2018 There are two apparent approaches to model-based design: - Explict: generate code directly from model - Implicit: use model to validate implementation (through proof or statistics) Entry: MVC / MVP Date: Mon Oct 1 02:50:46 CEST 2018 https://www.youtube.com/watch?v=o_TH-Y78tt4 I don't like the arrogance of this guy, the wandering side stories, and some of his points I really don't agree with, but.. At around 30min, MVC is mentioned to originally be very granular: i.e. a button. Also, MVP, presented :) But near the end, point gets good: defer decisions, use plugin model. Maybe time to disambiguate all my current understanding: MVC : view talks to model MVP : presenter sits between model and "dumb" view MVVM : MVP with a viewmodel https://www.youtube.com/watch?v=JV63czrUpbI - get rid of fat controller - make "middle man" dumb (a bus, router) - MVP: MVC with thin controller EDIT: So to be honest, it still doesn't really make a lot of sense to me. I can build architectures that have elements of these, and then identify some of the problem spots. But these all look like rough guideines and "squint patterns" to me. Here are some graphs: https://stackoverflow.com/questions/2056/what-are-mvp-and-mvc-and-what-is-the-difference Entry: documentation: 4 types Date: Sat Nov 10 07:59:41 EST 2018 https://www.divio.com/blog/documentation/ There is a secret that needs to be understood in order to write good software documentation: there isn’t one thing called documentation, there are four. They are: tutorials, how-to guides, explanation and technical reference. They represent four different purposes or functions, and require four different approaches to their creation. Understanding the implications of this will help improve most software documentation - often immensely. Entry: Design is places Date: Tue Nov 13 12:19:09 EST 2018 Sometimes a feature request is easy to grant if only the place to put it would exist. Design is to ensure those places are there, or are easy to remodel. Entry: Build systems: push and pull Date: Wed Nov 14 08:41:25 EST 2018 Don't use both. make is pull, so stick to that! Entry: redo it simpler? Date: Mon Nov 19 07:20:59 EST 2018 For many applications, it really does make sense to strip out all of the cruft and make configuration simpler, aimed to support only the variability in the application. However, it is very important to keep in mind that the configuration mechanism itself needs to be evolved, because you are going to make it too simple at first. Essentially, riding the wave of feature creep is unavoidable. Periodic resets might be necessary to avoid historical buildup. Entry: build systems: from push to pull Date: Tue Nov 20 08:53:21 EST 2018 From Conal's paper "Push-pull functional reactive programming" http://conal.net/papers/push-pull-frp/ I don't remember the contents, but sticking just to the push/pull nomenclature, it is clear that when mixing imperative and functional, there is going to be a place in the code where push and pull meet at a buffer. The same happens in build systems: - push: create products - pull: download products This is such a strong pattern that it is (almost?) guaranteed to be explicit in code somewhere. Maybe it is an implementation flaw? What you want is everything to be demand-driven, but in practice you also want to have a "push" phase, even if it is only to check constraints on the code. This phase is development time, or compile time. The "pull" phase is run time: name reference. So bottom line: the functional view is pull-only (binding). The imperative implementation of that is always push (set) and pull (get). Or put differently: binding is done using statements. Entry: how to design when there are no real specs? Date: Tue Nov 20 11:26:43 EST 2018 - make it work using purely local reasoning - take a step back, and clean up the big picture Entry: build systems, and what is "push" anyway? Date: Tue Nov 20 12:23:36 EST 2018 How to represent the idea that some dependencies get lost or are simply not encoded? Typically, this is at the library level, or any time there is some kind of recursive build. The solution is to take an _imperative_ viewpoint. While building in theory is a _functional_ endeavor, in practice it is not and there is simply no easy way around it. So if there is no functional implementation, then at least an idempotent implementation (a "push" implementation) is workable. To go deeper: there is _always_ a push part to a build system, so even just walking the dependency graph can be considered that. The difference between a fast graph walk, and actually executing a fast idempotent build is quite moot. Entry: backpressure TCP video streaming Date: Wed Nov 21 11:51:46 EST 2018 - when frame arrives, send ack - at sending side, use acks to: - estimate delay - if delay grows, reduce framerate Entry: redo Date: Fri Nov 23 09:04:33 EST 2018 https://redo.readthedocs.io/en/latest/ There's an experimental variant of Buildroot that uses redo in order to clean up its dependency logic. https://github.com/apenwarr/buildroot/tree/redo Entry: Build systems Date: Fri Nov 23 14:23:37 EST 2018 Because deps get lost, this is always an approximative hack. EDIT: This is a real problem, almost impossible to solve right. The origin is completely made up out of accidental complexity: people building things that do not cooperate properly, needing more glue and workarounds. Entry: A rebuild should export a dependency list Date: Mon Nov 26 16:03:05 EST 2018 This is the insight, and probably what "redo" is about. The reason why build systems do not compose, is that the dependency list gets lost. There is a need to re-trigger the black box when the coars-grain dependencies change. So, rebuild builds: - product - dependency list - rebuild script With this identified, it is also possible to overapproximate it. Entry: trusting the incremental build Date: Tue Nov 27 11:39:30 EST 2018 1. you are going to use the incremental build. rebuilding requires too much time in any practical system. 2. you will need to be able to trust it, because some of your debugging experiments are going to rely on measuring the effect of small changes. getting the incremental build correct is just another debugging problem, but how to actually specify it, or test its correctness? basically, every time an edit is made and a command is sent to propagate the current code state, it should be verified which files actually changed using time stamp and content hash. Entry: arch and algo hats Date: Tue Nov 27 12:09:44 EST 2018 It seems that making contexts switch between architecture and algorithm is difficult. Entry: Proper hierarchical make Date: Thu Nov 29 12:15:23 EST 2018 The thing is to always abstract a sub-build with explicit dependencies. A simple idea, but for some reason very hard to get into my thick head.. So when creating a generator in a different language, always abstract it as a triplet: (input files, generator code to run, output files) It is probably possible to look at all the files that are touched by some strace trick. EDIT: the input,output list can be specified as a Makefile, similar to gcc producing .d files. EDIT: added this to asm_tools as an example. EDIT: changed it to multi in, multi out. otherwise there is no way to abstract "coarse" build processes. Entry: Naming pull and push side Date: Sat Dec 1 12:01:11 EST 2018 ( This is going to be obscure missing context. No time to explain.. ) What is this pattern? - binary is called "run_codegen" - source is called "makefile.hs" Inbetween sits "build". Why is the outside (the binary) different from the inside (the data structure passed to "build"). Basically, malke "build" explicit. makefile.hs could be a separate module exporting only a data structure. Entry: Automatically wrapping a black-box build Date: Sun Dec 2 10:51:31 EST 2018 Suppose that: - the build alrady has a pretty fast null build - it is easy to list the targets This way it would be possible to make a time stamp. One way to solve it is to always tar up the results. That gives a single file to depend on. I'm missing one crucial point: how to take a phony target (i.e. an idempotent script) and turn it back into a time-stamped target? Generalizes to hash -> timestamp. Again: abstracting a build step requires: - list of input files - list of output files (could be packed to one archive) - build script The condition to run the script in a Makefile is done based on time stamps. Is there a way to: - make that decision more abstract - translate the abstract decision to a time stamp to plug into make There seems to be a fundamental issue to plug this back into Make: once something is a phony target, things are lost... Translating an arbitrary boolean to a timestamp doesn't seem to be possible. It only works for recursive make, which is imperative anyway. Entry: Makefile and guard deps Date: Sun Dec 2 11:55:17 EST 2018 1. Using stamps, initial builds will not have dependency issues. 2. Guard deps are only intended for incremental builds, so they need to be "correct enough" wrt. the changes made by the developer. The bottom line is that: - Single Makefile is preferred, but not always possible. Try at least to make these cover as much ground as possible. - When recursive make is needed, try at least to get the guard dependencies right, and use time stamps. If there is even a single phony target, the entire build is now infected and becomes "idempotent impreative" instead of "dataflow". Entry: Abstracting guis. Date: Sat Dec 8 16:07:45 EST 2018 Working on the Erlang web gui toolkit I come to the conclusion that it is ok to stick to the lower levels, i.e. render HTML directly, and use ad-hoc rendering functions to try to recover some reuse. When it comes down to rendering the initial page and the update, often a lot of reuse is possible. Do the same there: ad-hoc re-use based on functions, locally defined. Trying to tuck this all away into objects makes it really difficult to see what is going on. The web is a mess, and it seems best not to layer too much crap on top of it because debugability is the most important property! Summary: ws.erl and ws_widgets.erl solve the following problems: - data representation - communication - web app startup Entry: Distributed system and unidirectional messages (events). Date: Thu Dec 13 15:44:49 EST 2018 Writing an application as a connection of nodes might seem a little overkill, but there is one really important remark to be made: If the nodes are structured in such a way that communication between them is straigtforward, it is easy to move functionality from one node to another. While nodes might be abstractly "the same", they are often distinctly different in how they are implemented, with very different properties in what they can do wrt non-functional requirements such as response time and throughput. E.g. one node could be a C or Rust process, the other an Erlang node, the other an FPGA or microcontroller on a USB bus. An important part here is to allow for UNIDIRECTIONAL communication. Because that is the thing that easily alows moving stuff around, and keep a data-flow structure in the network. I.e. DONT USE RPC IF IT IS NOT NECESSARY. Entry: Sequential programming is essential Date: Sat Dec 15 13:17:08 EST 2018 Sometimes it is really necessary to work sequentially. E.g. some design constraints really make it very hard to use a "functional" approach, and you will have to use "incremental updates". Processes then allow transactions. Entry: The show-stopping problem: tools Date: Wed Dec 19 10:44:13 EST 2018 I think the programming language issue is solved. I have my basic set: C, Erlang, Rust and Haskell. What remains is to remove the cognitive load of building distributed systems and working with large code bases. What does this mean in practice? - Better code navigation - Better build and deployment Entry: Build, deployment and staging Date: Wed Dec 19 10:46:50 EST 2018 Conflicting requirements: - all code the same vs. isolate part of the system to run possibly broken code without taking down everything else. uniformity vs. variation. - instant deployment: edit a line of code, see the effect instantaneously. ultimately this is all about dependencies, while the tools "forget" dependencies at abstraction boundaries. so this is modularity vs. debuggability. The focus is still on the loss of dependencies. Is there a way to automatically recover this? E.g. take a composite build tool, and turn it into an incremental one? It would be possible if the "leaf nodes" could be identified. E.g. C compiler or linker invocation. This seems like a formidable task. Entry: How to factor code Date: Fri Dec 21 14:51:44 EST 2018 Effort should go into splitting these two concerns: - making code that creates a local context predictable, boring - exposing the functionality we're interested in as a series of step in a convenient context Entry: make and variables Date: Fri Dec 21 17:25:33 EST 2018 I'll be damned... 6.10 Variables from the Environment Variables in make can come from the environment in which make is run. Every environment variable that make sees when it starts up is transformed into a make variable with the same name and value. https://www.gnu.org/software/make/manual/html_node/Environment.html Entry: propagation of changes Date: Mon Dec 31 02:09:45 CET 2018 I'm sure now that the problem is lack of types, or even lack of just binding information. The only way to get over this is to somehow declare protocols, and that can be done by finding a way of declaring types over channels. Entry: Distributed systems Date: Mon Dec 31 14:44:28 CET 2018 This is mostly about fixing some issues with Erlang. While Erlang is great to express distributed designs, it is not great to evolve them. I especially miss "type error propagation" I use a lot while refactoring. How to fix this? - typed protocols - better understand supervisor trees, and structure programs accordingly. The latter is just learning an existing mechanism. The former is something special. (see erlang.txt) Entry: notify + batch Date: Tue Jan 1 14:52:35 CET 2019 Separate notification listener + process. Also make pollable. Entry: discoverability Date: Wed Jan 2 21:46:58 CET 2019 This is what I call "bread crumbs". It gets more important for things that are large and arbitrary. Entry: Is source and binary separation just bad? Date: Mon Jan 7 22:37:28 CET 2019 No. Functions are "not there". But practically, code "save" should be a full upload/recompile. It is only build systems that are inefficient for incremental changes. Entry: mutation: only for testing? Date: Fri Jan 11 14:40:30 CET 2019 What about this: Production code does not allow arbitrary mutation of internal objects, but test code does. This way it is often not necessary to provide APIs to set up mock objects. Just instantiate in the usual way and modify? Entry: Finding makefile bugs Date: Fri Jan 11 23:23:49 CET 2019 If manual entry of dependencies is necessary, how to debug this? One way is reproducible builds: - if binary didn't change, the file that changed (likely) doesn't influence. - if build was not triggered, a verification build would not change the binary. So what about this: Run continuous verification rebuilds in the background which compare incremental binary outputs with new full rebuilds. Problem here is that there is a lot of overhead, so likely this will be culled, leading to new bugs. There is no good solution other than doing this formally: make it impossible to have bugs by generating the dependencies. Entry: Brooks' silver bullet Date: Wed Jan 16 15:53:48 EST 2019 https://www.youtube.com/watch?v=HWYrrw7Zf1k 5 levels of strong productivity gains. - machine code - assembly - fortran - libraries - time sharing (interactive: better turnaround) Remark: Language advancements these days, e.g. static type systems are about higher abstractions and larger code bases and avoiding bugs. Entry: CSV Date: Sun Jan 20 11:13:58 EST 2019 Is actually quite universal, as it can express relations, not just functions. Collections can be expressed as relations as well, using a shared key and an index key. Entry: Inverting the build system Date: Tue Jan 22 07:14:21 EST 2019 Added a "push" interface to Erlang buid. Works well. Very fast. Can this be turned into an ordinary "pull" interface? Both are different views of the same thing. The real problem is to express the build as a functional program. I wonder if this can be piggy backed on Haskell's lazy eval. Entry: State Date: Thu Jan 31 09:26:48 EST 2019 Regarding user interfaces, it is overall simpler to have the user think about state and how the state is influenced. The 'functional' approach is more about not creating extra state when not necessary, i.e. it is an implementation detail. It really only works for batch processes. Entry: To type or not to type Date: Fri Feb 1 08:37:55 EST 2019 - I spend A LOT of time trying to get types right by doing incremental programming in Erlang. Without incremental programming, this is just completely impossible to do. - Changes to Rust code do not have this problem. Even changes to C code are less of an issue. I think that it's quite clear: Erlang needs a static type system, or a gradual type system. Entry: Large systems and consistency Date: Fri Feb 1 12:23:49 EST 2019 It is very difficult to build large software systems that handle "human affairs", becaue they need to be consitent. Typically, it is necessary to have a human handle exceptional cases that are not encoded well in the system. Such exceptional cases happen when two large systems are connected together, creating inconsistent corner cases. Entry: Dynamic typing is the default Date: Sat Feb 2 08:40:53 EST 2019 Why? Protocols are just data. There are no guarantees about who is talking on the other end. This is an important insight. Hardware modules are best separated by wires. It is the physical constraint that determines the structure of the software. Static types only work inside the walled garden. Entry: Inverting dependencies Date: Sat Feb 2 13:28:52 EST 2019 Pushing erlang updates is simple: the file is a compilation unit. For other updates, there needs to be a way to invert the dependencies: given a file change, what changes? It's time to start making dependencies more explicit. Entry: The problem with build systems Date: Sat Feb 2 13:31:17 EST 2019 Summary: they do not compose wrt. incremental behavior. This is often a very deep problem, requiring either a full rewrite, or a modification to allow the export of dependencies for larger scale integration. Entry: Functional Objects Date: Mon Feb 4 07:24:47 EST 2019 Basically: keep state transitions functions pure, such that they can be tested in isolation. Then create objects (services) by composing state transition functions. Entry: Modularity and Limitation Date: Mon Feb 4 10:10:23 EST 2019 Creating modularity is creating limitation. This is also why refactoring is necessary: sometimes the current architecture cannot express a particular new feature set, while "bolting it on" without regard for modularity would not be too hard to do. Entry: Live updates Date: Mon Feb 4 11:52:57 EST 2019 Making some more progress with the live updates. Erlang is great for this. Current approach: emacs after-save-hook -> .push_file: filesystem hierarchy determines handler -> live system performs the update Entry: Libraries vs. Frameworks Date: Mon Feb 4 11:54:44 EST 2019 Simple: - libraries compose - frameworks do not Entry: State vs. Cache Date: Tue Feb 5 14:51:00 EST 2019 What is the equivalence of 'stateless' for entities that can have state that is only comprised of caches? Distribution almost always needs cache. Protocols are almost always stateful during a transaction, but stateless outside of those transactions. Entry: Imperative vs. Functional Date: Wed Feb 6 07:52:45 EST 2019 I think I got over my infatuation with functional programming. Imperative and Functional are two views of the same problem. The important insight is that as long as effects in an imperative program have limited scope, it can usually be abstracted (contained) as a pure function. So the lesson to learn is to limit the scope of effects. Entry: Erlang and Rust Date: Wed Feb 6 08:54:51 EST 2019 A good team. It's a bridge between two worlds. Each world has its own benefits: Erlang - reloads for incremental development - supervision - multithreading + mailboxes - distribution - default protocol "baseline" data language - simple pure functional language Rust - decent static type system - multithreading + channels - C-like performance (CPU and memory usage) - many good libraries It seems that the bridge between these needs to be made somewhere. Doing this at port or TCP level speaking ETF seems simplest. Entry: Protocols: organizational or technical? Date: Wed Feb 6 09:05:48 EST 2019 Protocols are always there to decouple things, but to what end? There is a big difference between organizatinal decoupling, e.g. separating teams of programmers, and technical decoupleing: providing modularity for other reasons such as code organization or physical limitations. The key distinction is whether the protocl is published or not. Typically, technical protocols could be private, allowing them to be changed without expensive version control. I.e. if it is possible to update both ends at once, the protocol is effectively abstracted away. Entry: Local reasoning can't solve everything Date: Thu Feb 7 12:26:37 EST 2019 What helps? The top-level idea that some dependency path should be direct. I.e. an application should behave as if it is a big ball of mud. An event shoul have immediate consequence. All indirection should be unevaluated bindings. I.e. it should be possible for a compiler to substantially reduce the actual code causalities. Abstractions should be zero-cost or at least cheap. A final product is a consequence of a large amount of instantiations and connections. Program design is the mapping of the big ball of mud implementation onto an abstracted, modular implementation, by applying good judgement in where to create interfaces. Entry: crossing language barriers Date: Thu Feb 7 17:22:15 EST 2019 One big argument against using multiple languages is that it is an impediment to refactoring, as it requires language change when code is moved around. One particular nasty case is shell scripting. Shell scripts work well for simple sequencing, but do not work well at all when data abstractions are needed. Entry: distributed systems Date: Fri Feb 8 11:38:32 EST 2019 As long as the system does not need to be robust against network partitions, use a hub-and-spokes model. This is one of the most important overarching design decisions. I currently do not have any code that needs this robustness apart from a disconnected laptop. Hub-and-spokes still allows some partitioning of the leaf nodes, but it creates a central point of failure in the hub. This is usually OK for embedded systems. Entry: So what with Erlang? Date: Fri Feb 8 14:46:28 EST 2019 It's nice for scaffolding. Really nice. But I have trouble with the untyped stuff. There are two avenues: - Move to Rust - Join the Erlang gradual typing approach Entry: Good tools Date: Sat Feb 9 12:31:43 EST 2019 What sucks about using good tools is that it becomes quite apparent that the hard thing has always been to understand the problem. Fred was right. Entry: implement derivatives directly Date: Tue Feb 12 09:22:44 EST 2019 Basically, never write code as a "full build". Write it as an icrement only, then use the increments to create the full build. The reason: it is too hard to automatically generate derivates, but it is trivial to integrate. Entry: What is a top_level project? Date: Wed Feb 13 10:51:17 EST 2019 First, it needs a definition of "system". This can mean anything, but should be all-encompassing. E.g. all dependencies are bound. Defining properties of a top_level project are: - source: root of the source tree of the entire system. - tools: can be linked with a reference to build tools - live: ability to link 1-1 to a live deployment of the build products, preferably with incremental deployment on source change. Entry: Big systems Date: Thu Feb 14 12:59:59 EST 2019 When you want a lot of different things, at some point it becomes practically impossible to keep them all consistent. Entry: redo, push_change, Date: Thu Feb 14 22:07:56 EST 2019 really turns things upside-down, back to simple imperative programming. imperative isn't bad, as long as it is: - idempotent - possibly skipped on no-change i.e. as long as it's inside some kind of dependency framework. then the whole thing is still "pure" or "reactive". Entry: What is a service? Date: Fri Feb 15 16:22:33 EST 2019 Something that handles events and has state. So it maybe is more about data structures and data hiding. Entry: 1-1 continuous deployment Date: Sat Feb 16 08:24:25 EST 2019 An interesting consequence is that configuration files disappear! There is no longer a need to separately edit config files if you can just hardcode things such as board device IDS, IP addresses, ... Entry: State Date: Sat Feb 16 12:41:57 EST 2019 If a services is just state + state update handlers, why cannot everything be expressed as a database (or filesystem) + transaction functions? It probably cam but there is a problem: A running process is a cache of its instantiation in the sense that it is linked to other instantiations. So, while it would be possible to have all references be in a data structure, where all references are symbolic, it is usually not practically feasible. However, thinking about state in this way makes it possible to rebuild the cache at any time. Some roadmaps: - instrument obj:call to make changes to back-end storage - a global namespace is needed for all processes - alternatively, a simple way to snapshot is needed, mapping non-serializable processes and ports to instantiation functions. EDIT: It's actually quite nice to map this to Erlang: always use raw process references, but monitor them so restart = rebind. This does need a way to save the state in unbound form. Entry: frameworks vs. libraries Date: Sun Feb 17 16:07:11 EST 2019 Frameworks can create a lot of leverage (integration), but make a lot of assumptions, and by definition cannot compose, as they "manage the world". So exo is a framework meant to perform integration of a lot of different projects under development. Entry: Updates Date: Wed Feb 20 09:03:35 EST 2019 Event driven systems and smart updates are a central problem to everything I do. Get a better handle on it. There is a constant tension between two viewpoints: - push: change a value and propagate upward - pull: recompute the entire data structure Both can be combined with caching. Usually a system is implemented as a combination of these. If a push is not implemented at a certain level (lack of reified inverted dependencies), it is often possible to go up one level of hierarchy and perform a pull. Maybe it is a good idea to create this as a data structure in Haskell, and then try to relate several things to it. Redo redo? Entry: Is a process always just a cache? Date: Thu Feb 21 00:06:31 EST 2019 A temporary bringing together of information that comes from other places? Entry: redo Date: Fri Feb 22 08:33:29 EST 2019 I think redo is smack in the middle of the push/pull story. I started implementing it for uc_tools/gdb to see where it goes. Entry: glue vs. algorithmic Date: Mon Feb 25 00:01:46 EST 2019 Untyped, fast deploy, test-driven for glue code where the main difficulty is API discovery through trial and error. For anything that is better defined, or has more internal logic, is more algorithmic or data-structure heavy, use static types. It is really the former where there might be complex types, but they do not vary a whole lot. Once it works, there are few special cases. If something fails, add a special case. Entry: Application vs. library Date: Tue Feb 26 08:40:54 EST 2019 The app part is where things start to become intertwined. It seems this is necessarily so. The app part is where the "human" comes in, and the human is messy. EDIT: This is really nothing more than the holy trinity: - operating system (the 'framework' : resources and tasks) - libraries (generic sub-tasks) - application (user interface, links human elements to system elements) Entry: What is a task again? Date: Tue Feb 26 11:44:47 EST 2019 It is a cache for a particular piece of state. This state is usually snapshotted (the germ line). Entry: GUIs and corner cases Date: Sun Mar 3 10:53:23 EST 2019 It seems that the interface between neat structures and what a human wants to see and feel is quite big. I run into so much bad foresight while writing gui code. Entry: the blessed tools Date: Sat Mar 16 16:24:57 EDT 2019 Lists of tools that come from difficult choices. I.e. there are alternatives, but these turn out to be optimal relevant to my application space and constraints. - Linux os1: kerne - buildroot os2: minimal embedded (readonly) root filesystem - debian os2: "large" system - Erlang os3: network-level orchestration, protocols, top-level glue - Rust core data processing routines (data-structure heavy) - C low-level interfacing, data processing (trade off with Rust) - Haskell compilers and code generators (data structure heavy) - nix build tool & library dependency management - redo fast incremental rebuilds - sqlite3 data store - git source control - 2-level - Emacs code user interface Entry: static vs dynamic Date: Sun Mar 17 10:33:15 EDT 2019 So I am partial to Erlang. It is mostly these things: - network transparency and protocols - fast reload while keeping everything up - I've solved web and sql interfaces This makes it very suitable to act as top level, for ad-hoc integration and orchestration. When a task gets more complicated and it becomes loop or data structure heavy, delegate to Rust or Haskell, based on whether memory usage is an issue. Entry: Phase boundaries Date: Wed Mar 20 12:56:43 EDT 2019 You need both dynamic and static typing, because in any project there is going to be a "phase boundary", where there are no guarantees about what data you are going to receive, so there is non-totality as a property of the real world. How you deal with that is up to you. Sometimes it makes sense to keep the non-totality around, and only have a small core with (near) perfect totality. ( Totality as in total functions: no chance on any run time exceptions. ) Erlang's stance is: you ain't going to get the protocols right in all details, covering all corner cases, so just make sure you can limit state loss by making restarts very granular. Entry: Printf debugging Date: Sat Mar 23 12:27:17 EDT 2019 So here's a thing: if incremental code uploads are possible, there is no need for very detailed logging. Take a production system, take a build system that has the exact version of this build system (+ some newer version of debug code), and perform updates on the fly without changing the flash. Entry: Moments of clarity Date: Wed Apr 3 09:28:40 EDT 2019 So it is hard to accept, but it is clear that it's getting worse. My brain has momemts of clarity, where I see all global relations, interspersed with much larger perods of dullness where I can think only locally. It should be the goal to use those moments of clarity to install structure, such that in those momemts of dullness, enough work can be done through local thinking. Summarized, maybe it is fair to say that creating the structure, the design, is the thing that requires the most mental effort? The reason is that it is a very intense synthesis job: a lot of context gets reduced to something simpler. Design is compression. Compression is compute-intensive because it is a "global" operation. Entry: Messy data Date: Wed Apr 3 09:51:51 EDT 2019 The only trick there is, is to modularize. I've just not worked with problems that are hard to modularize. Basically, this means, I've not worked a lot with messy data. The key insight is that sometimes, flat is better. If it is not possible to structurize something, use a flat representation, and solve the ad-hoc part in a large collection of "queries". Another example of this is graphics and UI rendering: a lot of context information needs to be synthesized in ad hoc way. It makes more sense to pass the whole thing to every element (an environment), and make local decisions on what information is necessary (queries). The real lesson here is that as long as there is structure, it makes sense to "push it through" as far as possible. If there is none, don't bother imposing any. Use ad-hoc projections instead. How to call this? Object hierarchy vs. bulk store. Entry: Why is math good for modeling design? Date: Thu Apr 4 10:37:29 EDT 2019 Math is really about concepts that have good properties. Discovering these things is always indirect. We really don't have a way to go from properties to concepts other than 1. stumbling into them by accident 2. recognizing known patterns and applying them Since it is hard to make that shit up, it makes a lot of sense to study these patterns, to get to know them, so you can recognize them when they show themselves. Architecture is about organization. It really doesn't matter how things are organized, as long as the organization principle has some good properties. It is those properties that implement abstraction, information hiding. Why are OO patterns no good? Because they are severely limited by the hidden context associated to the idea of objects. Information hiding is not necessarily a bad thing, but it needs to be done by "actually forgetting" things, i.e. a non-invertable function (a projection), not by assuming some hidden connection. So why is math important for design? Mathematical concepts are "clean". They are constructed in a hig-leverage way. Entry: About state Date: Thu Apr 4 11:22:29 EDT 2019 Some rules of thumb: - If it can be done without state, do it without state first, because that allows more properties to be imposed. - If that's not possible, try to keep the state short-lived, such that at a higer,coarser level there is still a stateless interface. This is often necessary to optimize inner loops. - If that's not possible, try to ensure that state is really just a cache. I.e. it can be thrown away and rebuilt without loosing information. This is often necessary to optimize distributed systems where both ends need a copy of the information. - What's left are true 'objects', which are a combination of two forms: - A database: data on persistent store, possibly indexed and cached to make the implementation efficient. - Models of physical objects that have physical state variables Note that herein is contained the idea that "entanglement to environemnt" is the real problem with object-oriented design. As long as all state is local, the distinction between OO and functional effectively disappears. The core idea is that: "OO is non-local" Entry: Fine granularity and scaffolding context Date: Sat Apr 6 09:25:03 EDT 2019 In practice, it is very hard to isolate subsystems. However it really pays off to do so, because it removes the problem of "reloading context". As I get older, I find that reloading context is becoming more of a problem. Either because I'm just doing more complex things, or because I've become slower in (re-)absorbing arbitrary details. My mind seems to have learned to forget about arbitrariness: it is only kept in short-term storage. Design principles _do_ seem to survive a memory storage cycle. What is the conclusion? It is very important to structure the code and maybe more the build system in a way that changes to small components become easy. Summarized: - components should be small - changing a component's code should have immediate effect - context necessary to look at the component in isolation should either be small (e.g. functions), or at least built transparently without too much indirection. Entry: The cost of mental work Date: Sat Apr 6 09:46:08 EDT 2019 There seem to be two components: 1. Loading the problem into working memory 2. Waiting for the solution to fall out The latter seems to happen automatically once the problem is loaded. Before a problem can be loaded, the gates need to be openend. The mind needs to be tricked into finding it interestion. If not, it will refuse to absorb the information, maybe as a protection mechanism? So really there is a thing that happens first: 0. Trick the mind into making the problem relevant Entry: Complex trees Date: Sun Apr 7 09:07:45 EDT 2019 Why is it that as good as every useful computer program has so many special cases? This seems like an indication that the substrate is not a good match for the problem. Basically, we have this computer thing, and we apply it to everything by amassing special cases. Is that really a good way to go? Entry: Why encoding in types can be useful Date: Sat Apr 20 10:59:58 EDT 2019 A lot of "structurizing" is needed to make complex types work at all. This then solves most of the implementation issues along the way. It acts as a guidance. Turning things into a logic is maybe a more difficult problem, but it forces you to take a route that will shed a whole lot more light on the problem. Instead of spending your time getting arbitrary details right, spend it by finding the intrinsic structure, or how to translate a problem description to make it more structured. Entry: Work on the full stack Date: Sat Apr 20 13:55:09 EDT 2019 Don't try to finish the modules before finishing the whole: there should be an interation step, because your interfaces are going to make the wrong assumptions. It's more important to have a working full stack thing full of holes, than just a single module that is perfect against initial spec. Entry: FP vs. OO Date: Mon Apr 22 09:27:13 EDT 2019 OO-design works for things that are fairly arbitrary, i.e. a consequence of working with institutional and technical constraints (good-enough APIs, hardware limitations). However it is very hard to escape this kind of unstructured world once you are inside of it. FP-design works for anything that has decomposability. Functional design is about high-leverage composable abstractions. These are expensive to create and expensive to understand, but if your problem fits the mold, the leverage they create is substantial and the resulting design has a level of simplicity that can seem magical. The basic idea is that OO isn't very "logical". OO is about how people think about "things". FP is about how people thing about relations between things. Entry: DSP and Macro languages Date: Mon Apr 22 09:54:40 EDT 2019 Why both? The latter solves the composition problem of the first. Math only works for "leaf nodes". In general, systems are complex and at implementation level there isn't really any mathemantics left that can handle it. Entry: Idempotency vs. functional reactive Date: Tue Apr 23 08:37:07 EDT 2019 Writing some Erlang actor code, I'm starting to find it palatable to use a pattern of "dumb" event propagation, together with some local optimizations that avoid "full redraw" when nothing changes. Entry: Lifted closures, restarts, caches Date: Wed Apr 24 09:12:11 EDT 2019 If a process can be restarted from a specification, it should be treated as a cache. Caches are essentially "not there" at the conceptual level. They are an implementation detail. The lessen is to create as many processes as possible that behave as caches, i.e. they have no essential internal state. Is there only one kind of "actual" process? The one that holds intermediate user state? I.e. the stuff between two transactions? Entry: Infrastructure vs. functionality Date: Wed Apr 24 11:16:13 EDT 2019 Things have improved for me since I've been splitting up these two components. Infrastructure is the "glue" that connects the things that actually do something. There is quite a bit of work involved in bringing up infrastructure, and more so if it needs to be simple and consistent, while working with existing real-world constraints (i.e. somewhat broken base systems). If adding a bit of functionality is not trivial, the problem is infrastructure and needs to be fixed from that perspective. Early this year I had a month to work on this, and I've increased granularity of reloads/restarts. This has an incredible effect on freeing mental space while adding functionality. This is all done in the context of "the application" or "the framework". There is only one of those, and it is where composition stops and some global assumptions are made. Therefore it needs to be wide. It contains "all I would want to do with computers while being able to modify its behavior". Entry: Lambda-lifting in C code Date: Sun Apr 28 08:44:11 EDT 2019 For each labmda, a context struct is needed. To do this systematically, use a consistent naming scheme, e.g. _fun for the function and _ctx for the captured variables. Entry: Lock free queue Date: Mon Apr 29 08:52:53 EDT 2019 In the single writer / single reader case, where the reader can just poll, is it enough to have a volatile read/write pointer? https://stackoverflow.com/questions/2334987/looking-for-a-lock-free-rt-safe-single-reader-single-writer-structure It looks like this is ok because: - pointers are only updated by a single thread - pointer is updated after data is written - reads are atomic So the reading side will always see a consistent state. Entry: Compute vs. memory Date: Sun May 5 17:17:14 EDT 2019 Compute is cheap, memory (state) is not. It's still meaningful to implement with low-memory use in mind. It probably will always be like that. Entry: Asymmetric protocols Date: Sun May 5 20:30:46 EDT 2019 For internal communication between systems that are very different, it is often useful to use assymetric protocols, where each communication direction uses a different data encoding. I used to think this was ugly. The example that comes to mind is Pure Data, which uses the Pd protocol and TCL in the gui connection. Going this route can reduce a lot of glue code overhead. Printing is easy, parsing is hard, so let the sender generate something that is easy to parse on the other side. Often there already is a "default" parser available. I run into this a lot for high level language to bare-bones C on microcontrollers and real-time core tasks. It seems that often an application merits its own protocol. At the C receiving side, it often makes sense to use something simple: arrays of uint32, or tagged packet C structs. When data gets more complicated, some nested data structure (e.g. ETF) might be useful. I've used nested dictionaries encoded in ETF with integer tags. At the C sending side, generating nested textual data structures is often just as complex as generating binary ones, making the textual protocols often more appropriate. Entry: Editing "dynamic" code Date: Thu May 9 13:13:52 EDT 2019 Editing "dynamic" code, i.e. late-bound code, pretty much requires running it in some kind of test harness, be it a repl, a test suite or a debugger. It is difficult to refactor without a test suite with large coverage. However, there is something satisfying about changing code and having it take effect immediately inside a (large) running system. Something that is hard to do for early-bound systems. Entry: Distributed systems and "mirroring" Date: Fri May 10 08:12:28 EDT 2019 Compared to shared state systems, in message-only systems there is usually some form of state mirroring going on, where one task mirrors some of the state of another task. This always feels like red tape when implementing. The duplication of state is often quite visible in the duplication of code. Is there a way to make that less of an issue? I.e. can both sides share some code and representation that creates actual duplicates of code structures? Basically, the idea is is that the local version of object oriented programming is valuable as a thing in itself. It is an implementation detail however: it is object access without comminication overhead, e.g. just function calls. It might actually be useful to implement both behind the same interface, e.g. render a direct C struct access method and an RPC protocol with marshalling from the same code. This goes back to most state being caches. Entry: The balance Date: Sat May 11 08:14:45 EDT 2019 Good software engineering is a balance between simplicity and automation. I.e. spend complexity (long term maintenance time loss) only on automation (short and long term productivity gain). As a consequence, development systems themselves tend to become a little complex and high maintenance. But that is ok. Entry: Late binding is necessary for code reload Date: Sat May 11 11:02:45 EDT 2019 And code reload is necessary for incremental development. I'm just looking for ways to justify completely settling on Erlang :) I really like this though. It does show that linking/binding is really the core idea. Entry: focus Date: Wed May 15 08:31:09 EDT 2019 - event driven compilation, reactive programming - language abstraction, macros - resource allocation patterns Entry: composition Date: Wed May 15 09:31:40 EDT 2019 Functional programming is about making things composable, more algebraic, creating "high leverage abstractions", creating "product" algorithms, not "sum" algorithms. As for product vs. sum. It is more efficient to split up a problem into two unrelated algorithms that are combined, than to handle two different cases separately. Entry: computation vs. communication Date: Thu May 16 14:09:17 EDT 2019 The essential concept is communication: getting the data to the correct place where it can be used in a computation. The computation itself is often trivial compared to getting the data there! The computer will probabably spend most of its time in performing the computation inside some kind of tight loop, but the programmer will most definitely spend most time in writing communication and linking code. It took me a long time to realize this, and it is still not quite intuitive. Entry: Types are really necessary Date: Fri May 17 16:26:06 EDT 2019 Types make it possible to work with higher order functions as building blocks in a way that would be impossible to understand without being able to look at the types. I.e. implementation really becomes a detail, and might be counterintuitive even, but often the basic idea is quite clear in the types. Pick "impractical" instead or "intractable" instead of "impossible" if you like, but I think the idea stands. This only seems to happen (for me) in compiler work. Entry: Abstraction is auto-lifting Date: Tue May 21 10:25:07 EDT 2019 It seems that "glossing over" basically boils down to discarding wrapping wrapping. I.e. one thing is another thing, except for some wrapping. Obviously the wrapping is there for a reason, since it carries with it the implementation, the information of why this abstraction is a lie. This can be done both on simple wrapping (take one thing and turn it into another one thing), or abstracting a collection of things as a single prototype thing. It all wraps so eerily to parameterized types. Entry: code review Date: Tue May 21 20:02:53 EDT 2019 - look at the conditionals. can they be eliminated? - conditionals should be obvious, or should be commented Entry: code is a detail Date: Wed May 22 15:47:50 EDT 2019 Obvious in retrospect, but a powerful type system really brings that point home. The only thing that matters are the relations between data types, which are visible in a function's signatures. What's inside the function is just the nitty gritty details about how exactly we bridge the high level abstraction (the type sigature) with machine code that changes one into the other. Entry: Haskell for Embedded Software Engineers Date: Thu May 23 09:12:31 EDT 2019 ( I'd like to write a piece aimed at Embedded people. I think I'm scaring them with the Haskell talk. I'm not doing too well explaining why it is useful either. ) Yes. I know C. So why am I so keen on using Haskell to write custom code generators and code analysis tools? There are already many tools available, but they are all expensive and big. There are not really any tools for the small people. So the answer is: there are actually already open source tools: they are called programming languages. Pick a language that is good at manipulating syntax, i.e. a functional language, and in the set of functional language, pick the one with the most promise: Haskell is pure, lazy, has a powerful type system, and has a good ecosystem. Entry: why redo is so great Date: Fri May 24 13:14:39 EDT 2019 I like functional programming, meaning I like to keep dependencies explicit, and I like to build bigger things from smaller things keeping that context-independent way of thinking. But it is just too hard to glue it to existing imperative infrastructure. Pure functional programming allows laziness. The variant of laziness that maps to imperative infrastructure is reactive systems: event propagation + idempotency. So why is redo so great? Because it is reactive imperative programming constructed by building onto old, imperative infrastructure, and augmenting it with a reactive framework that is also compositional. The latter part is REALLY important. It is what distinguishes it from make. Entry: Iterators in C Date: Sat May 25 22:49:20 EDT 2019 Don't use internal iterators (map, fold): they are really awkward: - Require type casts - Require a type for each callback context - Require a separate callback function External iterators just need first/next, and are easily wrapped as an implicit iterator macro. Entry: functional/lazy vs. event-driven/idempotent Date: Thu May 30 15:28:58 EDT 2019 ... Entry: make things uniform = eliminate accidental complexity Date: Fri May 31 13:01:07 EDT 2019 ... Entry: estimates Date: Sun Jun 9 07:52:03 EDT 2019 https://techcrunch.com/2016/04/30/estimate-thrice-develop-once/ "What engineers actually estimate is the size of the technical foundation building block that makes a feature possible, which is often/always shared among other features at that scale. It is very difficult — and doesn’t really make sense — to try and work out exactly how much of the work going into such a building block will be apportioned to individual features." "The sad ultimate truth is that it takes as long as it takes, and sometimes the only way to find out how long that is is to actually do it. MCEs hate this, because they value predictability over almost all else; thus, they nearly always, on some level, treat estimates as commitments. This is the most disastrous mistake yet because it incentivizes developers to lie to them, in the form of padding their estimates — which in turn inevitably slows down progress far more than a few erroneous task estimates might." Entry: Metaprogramming for embedded engineers Date: Mon Jun 10 07:35:40 EDT 2019 So what's this haskell thing about? To cut right down to the chase: In embedded systems, you often want to write some piece of resource constrained code without having to pay abstraction overhead. Doing this manually often leads to very hard to understand code. The obvious solution to that is to start moving abstraction to compile time by generating code from some representation that is easier to understand and modify. - So basically, you're writing a language and a compiler. Best to realize this early and use good tools. - Examples: languages based on macros, or languages based on algebraic datatypes. In general, functional programming languages are good for this. The fact that ML stands for MetaLanguage should be a strong hint. - In the last two decades, final tagless form has emerged as a way to represent typed metaprogramming. What this boils down to is that Haskell or OCaml is used as the macro language. This is an incredibly powerful idea when you contrast it with explicit metaprogramming. (see metaprogramming.md) Entry: ELF Date: Fri Jun 14 14:42:46 CEST 2019 file:///home/tom/Airs%20%E2%80%93%20Ian%20Lance%20Taylor%20%C2%BB%20Linkers%20part%208.html Programs and shared libraries have an alternative view: segments, which describes what goes in memory and where. Entry: Linux code navigation Date: Fri Jul 5 10:12:16 CEST 2019 This has always been an issue. Some ideas: - Use .o files to collect only the corresponding .c files in cscope.files - Based on those .c files, collecte the header files from the .d files Entry: Forth + pattern matching Date: Mon Jul 22 07:47:19 EDT 2019 Forth is great for writing control programs (lots of _output_), but not so great for writing dispatchers (lots of _input_). Is there a way to look differently at random access? Entry: Debugging distributed systems Date: Tue Jul 30 09:29:52 EDT 2019 Debug and also doc = reconstruction of context. For single-core there is usually a stack trace. What is needed for distributed is something similar that can trace the chain of effects. If this is RPC it's essentially a distributed stack trace, but is there an analogy for things that bounce around? E.g. a rube-goldberg stacktrace? Entry: So why is integration hard? Date: Tue Jul 30 12:06:04 EDT 2019 Heterogeneity. There are basically no standards, or just misaligned standards, which causes a lot of duplication of effort. That's really the only problem with computing: it's too anarchic. No central authority to manage all these programmers being smart. EDIT: It should not be economically viable to make crappy software, or software that is not "rebased" on a common substrate. It's reasonable to expect that when a systenm is mostly working, you'll only be faced with places where there are leaky abstractions, rough edges, small inconsistencies in representation. And those really suck, because are often impossible to fix because they require mountains to shift.. EDIT: Bad integration is the worst. It is worth giving up conveniences for at the lower levels. Don't do anything special, ever! ABOLISH SPECIAL CASES! Entry: guidelines Date: Wed Jul 31 17:54:43 EDT 2019 I run into this a lot: trying something a couple of ways and deciding on one particular way because of good reasons. Time passes. Reasons forgotten. Re-invent the same. Is there a way to remember those pitfalls? Entry: spaghetti is the natural state Date: Wed Jul 31 18:51:40 EDT 2019 If a connection is needed, make it. We're never going to be able to understand brains if we can't understand ordinary spaghetti code. Entry: why do things have to be different? Date: Mon Aug 5 23:19:51 EDT 2019 E.g my .2 and .29 services. Why can't everything run on .2 ? Because .29 can be moved to a different physical location (e.g. laptop), giving it different physical properties such as not being connected to the rest of the network all the time. So distribution is necessary because of physicality that cannot be abstracted. Physicality with consequences. This is a step back! Entry: The distributed "commutation" issue : the solution is a common language Date: Tue Aug 6 10:52:20 EDT 2019 Basically, a lot of code is built out of envent chains that start somewhere physical and end somewhere physical, but the intermediate chain is just communicating code that can run anywhere. Is there a good way to look at this problem such that it can be abstracted? I.e. such that that intermediate code can really run anywhere, or even better, such that it can run in the optimal place? I.e. start/end are hard constraints, but the intermediate locations are soft: based on compute and comm, there might be an optimal place. One of the really awkward limitations here is the change of language when moving code from one "place" to another. That might actually be the most important real-world issue: heterogeneity. Solve that. Maybe find a single common language that can run anywhere? Once that is in place, task migration should be straightforward. Compilation is just cache (e.g. migration is expensive only if recompilation is necessary). EDIT: The thing here is to make an actual model for it, and then just describe ways in which the model cannot be implemented. I.e. first describe the ideal, then in a second level, describe the holes. There are a couple of "languages" or "machines". Both are needed, because state is needed to be moved. - Erlang processes - Linux C processes - uC firmware - FPGA gateware Possibly also Rust, but that is currently not really a requirement. Entry: symmetric connections, asymmetric setup Date: Thu Aug 8 12:18:33 EDT 2019 Even for eventually symmetric connections, it seems there is no way out of an a-symmetric connection setup: one of both ends needs to initiate. There needs to be a first message. Obvious when written down like that, but apparently something that needs to be made explicit when building symmetric structures. Also note that it could be made such that both ends can initiate. Still the setup is asymmetric. Entry: idempotent global state Date: Sat Aug 10 11:36:37 EDT 2019 Here's something I run into a lot: - I have some script that sets up some global state on a machine, e.g. a network connection - In order to make it idempotent I kill ALL instances of network setup and replace it with the one I want. This isn't actually idempotent in the normal sense (don't do anything if it is already set), but is idempotent in the observable sense: make sure something is set a certain way. So if a check is added to not change anything if already set up we can agree that they are basically the same. That check is just a cache. The end result is important. Now what I wonder: is this actually good practice? In practical linux day-to-day work it is a lot of labour to make a proper "state manager", but a quick and dirty "kill any other setups and replace with desired one" is maybe acceptable? Entry: Specifying state machines Date: Wed Aug 14 13:32:56 EDT 2019 I should be able to do this one constraint at a time. Entry: Protocol oriented programming Date: Fri Aug 16 08:59:55 EDT 2019 I now know exactly what I mean by this: - printing is easier than parsing - buffers to the rescue: do "macro expansion" at the transmitter end Essentially, protocol units should be easily interpreted scripts. What is interesting is that buffering is used to perform some kind of macro expension at the transmitting end. Often this allows the receiver to be a fairly direct script interpreter that does not need to keep a lot of state and/or resume points, which can often be troublesome. Otoh, with sufficient buffering, the receiver can be "timesliced" at protocol boundaries. I.e. no resume points are needed to create a single message. So there are other ideas: - Avoid task switching mechanisms by working on complete units at both ends: sender dumps into buffer, receiver polls for a single atomic command at a time. - Trade buffer memory for receiver complexity. The latter is an interesting mechanism: macro expansion / inlining at the sending end, or subroutine calls at the receiving end. The latter requires state, the former just requires buffer space. It is interesting that for low-level code, designing things such no intermediate state needs to be saved can make programs a lot simpler. The basic idea is to replace state machines with instruction-fed machines. Entry: Back to boring? How to write state machines. Date: Sat Aug 17 11:01:42 EDT 2019 - decouple via circular buffers. buffers are great in that they have only one parameter: their size. often a buffer creates "protocol wiggle room", e.g. protocol oriented programming: the tokens that go into the buffer can be chosen to make the reading end simpler. - write it in polling style. this still makes it possible to use event or interrupt driven approach by only polling when an event comes in. - buffer + poll allows the use of smart peek. i.e. the buffer does not need to contain parsed tokens. the peek function can do the tokenizing, which is often simpler to do than to perform tokenizing at the sending end. - represent state explicitly, i.e. don't use sm.h tricks when not necessary, i.e. prefer factoring into simpler machines. with explicit state, a poll method looks like: - try to read a token - dispatch on (token, state) Entry: simpler uC code Date: Sun Aug 18 10:59:18 EDT 2019 Something to think about: uCs are cheap, so it makes sense to have more of them and dedicate tasks. This usually allows severe scale-down of code because static memory allocation can be used, and code can be much tighter bound, i.e. macro instantiation allows optimization. Entry: Tokenizers Date: Sun Aug 18 11:01:49 EDT 2019 To keep design simple: create an application as a chain of tokenizers. - processors are polled - prefer processors to be stateless in favor of "expanding" the language that is transported between two processors - input / output buffers are explicit Entry: Embedded Date: Mon Aug 19 19:53:48 EDT 2019 It never stops being hoop jumping, right? Entry: As A Service Date: Sat Aug 24 15:13:52 EDT 2019 So here's a thing: - Most state is cache - If state is actually cache, there is a level of abstraction at which purity is preserved - What about writing all programs that way? This essentially re-invents microservices, but in a way that makes a bit more sense maybe? Actually, a pure function's implementation is a form of state. So really this is about stateful applications. Entry: stacks vs. queues Date: Mon Aug 26 17:50:12 EDT 2019 Why do nested function calls feel clean, but cascaded event propagation feel like Rube Goldberg machines? Beacause there are so many individual players? For nested calls, there is a single CPU. For communicating processes, there is just a lot more going on, and some intermediates could simply not be there. Is there a way to think differently about this? Entry: Are tasks just traces? Date: Tue Aug 27 08:13:59 EDT 2019 This is starting to dawn on me in a very practical sense: often it is easier to "dump" a state machine's behavior into a trace buffer. CSP also looks at things this way I believe.. Entry: event systems Date: Thu Aug 29 12:47:55 EDT 2019 An important part in debugging event systems is to allow for manual creation of events. This makes debugging a running system simpler, i.e. let it run up to a certain state, then manually send an event that would have been sent automatically to nudge it along. Entry: lazy responses Date: Sat Aug 31 11:34:16 EDT 2019 When creating state machines on constrained devices, it is important to note that it is not always necessary to put response messages in a queue. It can sometimes be more efficient to just poll a series of state machines for output, and let them write a message to a buffer whenever the reader requests a next message. Entry: caches, partitioning Date: Sat Aug 31 15:33:20 EDT 2019 Partitioning is actually a very common problem. Think of an OS as the cache of a build system (e.g. nixos). You really don't want to clear that cache if you're not connected to that build system. Is there a way to give semantics to this? A cache that is required to have a consistent local copy at all times? A cache that can't be simply flushed? Entry: zooming in Date: Tue Sep 3 23:02:53 EDT 2019 Soing some more deeply embedded stuff, and it is quite clear that "zooming in" is more difficult. The fancy integrated dev env does help quite a bit. Entry: Why no RTOS? Date: Wed Sep 4 14:17:11 EDT 2019 1. State machines are much more efficient memory-wise 2. A program written as an event handler can be simpler to debug. Entry: Delegation is just subroutines Date: Thu Sep 5 14:13:12 EDT 2019 This is the idea that links distributed systems with "stack based" single-process decomposition. The latter feels clean, but the former just feels messy. Maybe this is really about realizing that the former isn't dirty. That the world is inherently distributed, because of matter needing space, and the speed of light. RPC is really just CPS in disguise, which is "rube-goldbergy". RPC is not a good abstraction. It is about wanting things to be sequential. Maybe the core of the misunderstanding is that "stacks fail together", but units that run code separately can get out of sync, and failure isn't always obvious. Something needs to oversee that the whole thing is still ok, so if one component fails, everything can be put back in order at a coarser hiararchical scale. This is "let it crash". It is about treating queue-based programming the same as stack-based programming: assume the whole thing just keeps working, and handle things getting out of sync by detecting it and delegating restart up the hierarchy (at the point where a "unit" can be distinguised, instead of several cooperating "elements"). Entry: Incremental development of distributed systems Date: Thu Sep 5 14:31:51 EDT 2019 Set up the causal chain first. Make "something" go from input to output, and gradually fill in the gaps. It's important to have the general picture of the data flow and data dependencies implemented, even if it has many many missing features. It's much easier to keep a system running than it is to bring it up, so bring it up when it is still simple. Entry: decentralizing Date: Thu Sep 5 20:33:47 EDT 2019 The title rings true. Didn't read yet... https://hackernoon.com/decentralizing-everything-never-seems-to-work-2bb0461bd168 It's more about the social and political angle, but technically it often makes sense to go hub and spokes. Entry: stream processing machines Date: Mon Sep 9 14:31:32 IST 2019 Think in terms of queues of instructions passed from one machine to the other. Also, think in terms of abortable operations. Basically, design it like in C, but replace buffers with direct "port" connections. Likely buffering is not necessary for dedicated state machines. Only for sequential time-multiplexed machines. Buffers are space-time converters :) Entry: Time and space Date: Fri Sep 13 15:27:55 CEST 2019 To allow for moving an application to FPGA, first design it as a streaming data application, with modules decoupled by buffers. Replace part of the machinery by state machines and possibly direct (non-buffered) streaming ports. To design the machines, there are a lot of possibilities between clock-rate dedicated machines, and some programmable datapath with sequenced operations or a dedicated GP CPU. Entry: There is no clean rebuild (there are only incremental build systems) Date: Wed Oct 9 05:55:25 EDT 2019 What made me see this is how to manage keys in a build system. Ultimately, this is the idea that software is always relative: there is some information in the system that is a proxy of the "real" stuff, e.g. the hardware that interprets the software into physical effects. Keys have the same kind of problem. Keys are necessary to deploy, but the "initial" build will need to generate deployment keys, so technically this is a bootstrapping problem. It is best to acknowledge bootstrapping problems. Or: when creating a clean build system, you always end up at bootstrapping problems that will permanently prevent the "single root" ideal. Or: all software evolution should be incremental, but preferably separated into a germline that is made explicit, and a phenotype construction line (a cache). ( yeah sorry bordering on mania, trying to salvage some ideas before i descend. ) Basic idea: S (source) relates to D (deployed, running software), but the only real relation we have is that dS maps to dD. The initial state D0 is almost always ad-hoc, provided by a human to "just make it work". So there really are only incremental build systems. Entry: Designing Date: Fri Oct 11 05:15:43 EDT 2019 So it's clear: I just cant do this on paper. Unless "paper" is taken to mean an incrementally written program. Something that has a core structure I can trust. Entry: The boundary between messages and subroutines Date: Sat Oct 12 08:19:42 EDT 2019 Is quite a strong one. I've not seen any programming interfaces that can abstract this. It seems a lot of time is wasted creating message based APIs. These two worlds really don't mix. And it is hard to build intuition about how to split functionality between the two sides. The function/subroutine world should always have preference because it is much easier to isolate, but due to real world constraints, some functionality needs to be implemented as communicating processes. Entry: avoid using timeouts Date: Tue Oct 15 07:06:06 EDT 2019 This has always bugged me. It is not possible to properly pick timeouts in all but the simplest cases. Either they are too short -- you forgot that one case that sometimes can take a while, but is perfectly ok -- or they are too long such that effective error-out happens way too slowly causing a large mitigation delay. Is there a way to somehow compose the generation of timeouts such that the numbers get filled in at places that are reasonable knowledgable, and e.g. add up through composition. Or better, replace every timeout with a better abstraction: error out when the other party disappears. This can be done by adding keepalive to a connection, or something like Erlang monitors. So don't be lazy and add monitoring and error feedback. Do not just rely on timeouts as error mechanism unless there is a clear upper bound, e.g. at the lowest levels where CPU time and network delay are not abstracted away. Entry: Correct and simple Date: Tue Oct 15 09:18:53 EDT 2019 The best way to not have to remember a detail is to CORRECTLY and OBVIOUSLY abstract it. Correctly to make sure you never have to look at it again, and obviously because if it is not correct, you need to find out why. Entry: think more asynchronous Date: Tue Oct 15 11:45:57 EDT 2019 E.g instead of having one PC wake up another one to do e.g. a backup, split it up into separate parts: - wake up periodically - when woken up, perform backup - only go to sleep when backup is done Entry: Get rid of the operating system Date: Fri Oct 18 07:15:43 EDT 2019 Now, this is just a rant, obviously. But it has been brought to my attention again that special purpose uC programs are really neat. A bit harder to debug due to the low level, but the overall system complexity is low, so you can actually end up with working code that never needs to be touched again. Of course the same goes for isolated pure software components part of a larger system. So what am I really saying? Simple is good. Doing Linux development often pulls you into a corner where things are no longer simple. So how to solve this? I think I still want to build a Linux+Rust+Erlang system, without shell or the usual programs. The question is then: on what substrate? That integrated atmel chip looks promising. But also just some PCs might be good enough to get things going. Anyway, what this is really about is that I want to find my niche. I've discovered the bare-bones "routed state machine" uC architecture which works really well. So simple is important, but even more important is the ability to work incrementally, and to keep all behavior visible, making it debuggable and allowing for redundancy to be employed (i.e. use executable spec to observe and validate). Entry: State machines Date: Fri Oct 18 08:30:14 EDT 2019 Look at async / .await in Rust. My guess is that this is part of the general trend to move away from emulating multithreading (multiple stacks), and moving into a state machine world but with some syntax sugar. https://rust-lang.github.io/async-book/01_getting_started/04_async_await_primer.html Entry: distributed systems Date: Fri Oct 18 12:37:33 EDT 2019 I can't get rid of the concept of "logging into a node". Is it possible to get rid of that idea? I guess the abstraction of "this machine" is just a hard one to shake. So what is a "machine" in that sense? A CPU of course, but that is just the thing that mixes information. A machine is really its connection to "devices". That's what I need a ssh for. To get to its disk directly, or its USB ports etc.. For applications, most of that can be abstracted: what the CPU does in that case is to abstract the local devices to participate in network activity. In that sense, a machine is a network endpoint. A last hop router. Entry: debuggability Date: Sun Oct 20 07:48:01 EDT 2019 What does that mean really? Most of the time it means the ability to produce useful program traces so they can be compared to the expected behavior. Programming is about creating hierarchy to keep code organized, while debugging is the complete opposite: it is looking at what is actually happening. How do you do both at the same time? This breaks down when you can't trust the lower layers. Often debugging is 1) creating a full program trace to then find out which lower layer has the error, 2) write an extra unit test for the lower layer and fix the corner case. Entry: There are no names Date: Wed Oct 23 11:19:45 EDT 2019 Or more generally, there are no APIs. The only thing that matters is that APIs and names match on both ends. They are not techinically part of a composed system! Hence, all APIs, protocols are arbitrary. It's the information necessary to untangle, or break apart two or more subsystems. Maybe the whole idea of information is the trying to untangle a system? EDIT: Also tied to: do we really need those complex data structures? And the Chuck Moore minimalism. That only works when there are no protocols. If the purpose of the program is to control hardware that interacts with the physical world, but otherwise does no or very little parsing or formatting. Entry: services Date: Mon Oct 28 08:57:03 EDT 2019 Is it a service, or is it a compiler? It seems that the main differences are whether: - the state is in some special form, or on the file system. - the processing code is resident in a process The two are related. Essentially, a service is a cached compiler: in that it operates on an in-memory data store, using in-memory code. These caches ar a real thing btw. Many environments take a long time to load. (Essentially, compile or link themselves into existence.) I wonder if there is a simple way in linux to snapshot an in-memory state in a VM so it can restart very quickly. Entry: What does a leaf node do? Date: Mon Nov 4 11:01:32 EST 2019 It manages a phsyical or otherwise location-bound resource. E.g. - a machine that performs time-critical functions, - interactionss with the physical word through sensing and actuation, - management of cache and configuration data to perform those tasks. So cache and configuration is secondary but often is the bulk of the work. The information needed usually comes from upstream decisionmaking. So the core problem is decentralization: move responsibilitites elswehere and perform a reduction in bandwidth to the central system. Essentially, reduce communication cost by abtracting time scales. LOCAL FEEDBACK LOOPS REDUCE COMMUNICATION COST Corollary: if there is no local feedback loop, the leaf node is just a translator of sensor or actuation data, and needs no intelligence other than implementing communication protocols. ( I mean, these things are kind of obvious and usually implicitly assumed. This is an attempt to get out of the box by making trade-offs explicit. ) Entry: How much complexity is enough? And where does it really come from? Date: Wed Nov 6 18:09:08 EST 2019 What is the sweet spot between negative cost of software reuse, and positive cost of maintenance. If everything would simply "work", likely complexity would be lower, as much of the existing complexity is there just to smooth over inconsistencies. So is it actually possible to build a layered, modular system that does not need all this glue that adds mass? Entry: desinging state machine Date: Sat Nov 9 06:58:41 EST 2019 I have been resisting this for a while, but it is becoming apparent that state machines need to be designed from the perspective of translating events into state updates. When I write it down like that it seems completely obvious, but for some reason I am still inclined to think of them in terms of blocking code. Most machines that I encounter are protocol parsers. So let's write it from that perspective. The trick seems to be to write things down is such a way that the "core loop", i.e. the part that takes individual tokens and assembles a message is as simple as possible. The odd thing is that once such a core machine is written, to then translate it to blocking code form is usually trivial. Entry: implementation Date: Sun Nov 10 14:35:02 EST 2019 Something that is really difficult to avoid in very low-level scenarios is the introduction of parasitic behavior: e.g. if the inputs are not completely according to protocol, it is often not possible to "raise an error". E.g. extra behavior is implemented. The down side of this is that it can enable bad implementations at the other end of the protocol. Entry: late binding doesn't work for large systems Date: Mon Nov 11 12:04:54 EST 2019 A good example here is nix: replace broken api guarantees (late binding) with version-pinned implementations. In theory late binding can work, but in practice there are bugs in specifications of protocols. So either bind early and let the compiler infrastructure check types, or bind late but introduce some kind of test or validation guarantee. Entry: Combine events and polling Date: Tue Nov 12 07:06:40 EST 2019 Basically, write your code as something that can be polled, then call the poll function when an event comes in. This way it is easier to test when the event propagation is not yet implemented. Entry: security and power are always at odds Date: Wed Nov 13 09:58:53 EST 2019 So just separate them. Allow power on the "inside". Entry: Breadcrumbs Date: Wed Nov 13 13:29:42 EST 2019 Mind really wants to create connections, link things up. So how to leave good breadcrumbs? Discoverability? Entry: Two kinds of code Date: Thu Nov 14 07:34:36 EST 2019 ... that arise when evolving systems over time with radically changing requirements: - Top level system glue - Self-contained modules The reason to split them up is that they are fundamentally different. The latter's point is to LIMIT behavior and make it more mathematical, i.e. ensure it has properties, The former then satisfies the fluffly cloud of human interface and social process requirements, which always seem to boil down to SPAGHETTIFICATION: connect everything to everything without any real properties, just a cloud of objects with arbitrary reactive patterns. In most systems it is possible to move some of the non-modular code into modular code once it is understood, but this is a slow process. Once identified, it should be refactored though, because real spaghetti code will eventually hit a wall. Typically this boils down to introducing 1. function and procedure abstraction (write libraries), or 2. object clustering (e.g. Erlang's supervisor networks that manage clusters of objects as units). Procedure abstraction is preferred, as it is the only approach that is reusable. Object clusters are often too specific to a particular task or (hardware) architecture. The need for separate tasks seems to always be driven by a physical property. To maintain spaghetti code, aim for discoverability, rapid (instantaneous?) development cycle, and don't make things too abstract: stick to objects (services) and message passing. Entry: Objects need to be processes Date: Thu Nov 14 07:57:14 EST 2019 Because if done well, it can be cheap. But the main reason is that the need for parallelism will arise at some point, and it is better to ALWAYS think in terms of transactions. Explicit message handling solves that. In this sense, Erlang's supervisor trees are there to restore macro objects: objects that are built from interacting sub-objects. Entry: A developer's "business software" Date: Fri Nov 15 14:04:47 EST 2019 I have the strong advantage that I do not really need to write user interfaces. That is a lot of work. I can focus on modules with some form of terminal or text file input/output. This often leaves them composable as well. Entry: Transactions Date: Sat Nov 16 11:11:37 EST 2019 Is it that all the problems I have with distributed systems are because the old communication mechanisms are not transactional? Entry: Addressing Date: Sat Nov 16 13:07:02 EST 2019 I'm running more into the problem of addressing, naming, binding, as one of the most fundamental ones for human-computer interfacing. Why? It is the core mechanism that changes flat text (or tree) representations into object networks. EDIT: See Nov 20. Solved it. Entry: Why is intuition so often wrong? Date: Sun Nov 17 14:36:42 EST 2019 I've been doing this for a while, but still I can't really predict where things are going in many cases. Why? Is it just beause it is new? That there are so many different combinations that intuition has to be built up for the particular "machine" that is under study? Entry: If you abstract all addresses, all that's left is connected sources and sinks Date: Wed Nov 20 19:20:10 EST 2019 See epid.erl This is a milestone. The key point here is to abstract an aggregating hub as a collection of objects by intercepting "connect" messages. A slight of hand that makes it possible to treat everything as addressable objects globally: - Intersept connect, registering the connection map - When events come in, loop up in the registry and forward So event aggregation at the source side can be an implementation decision this way. Which is awesome! EDIT: This is the reverse of pub-sub. I.e. I want to get rid of (arbitrary) classes and go back to fine-grained connectivity. https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern Entry: Fine granularity sinks Date: Thu Nov 21 14:43:13 EST 2019 I have wanted this thing for so long. This is amazing! The main idea is that it allows shifting encoding of "address" and "value" in such a way that direct connections are possible, routing can be completely abstracted, and name resolution can be solved in isolation. We're just connecting two primitive things that can have arbitrary high-level names, and the connection "just works". It's even possible to regroup things, e.g. either a particular CC value from a specific MIDI channel, or all messages from a midi channel. The address/value distinction is _arbitrary_ Entry: Failsafe, distributed Date: Sat Nov 23 13:45:32 EST 2019 Seems simple enough: design the device such that it shuts down if no control signal is coming in. I.e. a watchdog. Entry: The sink naming thing.. Date: Sat Nov 23 15:47:49 EST 2019 It replaces pattern matching with explicit tags. Entry: What's the ideal platform? Date: Sat Nov 23 16:10:08 EST 2019 Something with a network interface. That will already give you enough slack to do most things, as the network interface is the thing that is expensive. Entry: The epid idea Date: Sun Nov 24 23:04:59 EST 2019 Replaces pattern matching with functions, and thus makes things compositional. I.e. in some sense, the tags are "pre-matched". EDIT: I heard this expressed before: replace pattern matching by folds. That is generic folds, where each constructor is associated to a case. Supposedly that would make things compositional. I think this here is the same idea: patter matching is replaced by constructor lookup. And all we do really is emulate the sum part of an algebraic type. EDIT: It is actually very similar to ws.erl objects. This is truly a unifier! Entry: Here's a lesson for the future Date: Mon Nov 25 13:58:56 EST 2019 Never mix base OS with application. Make it so that the board is network-operational without the application installed. Basically, get the "bootloader" part of the OS to work reliably so it doesn't need to be changed much, and then allow for easy swapping of application. Entry: The epid, symmetry is important Date: Tue Nov 26 08:09:12 EST 2019 It really seems to open things up. Check out imagewriter.erl Some interesting properties: - Both push and pull interfaces can be provided. It really doesn't matter! One can be implemented in terms of the other. - The "process chain commutation" problem is exposed in imagewriter.erl: where should the decompressor run? Entry: Abstraction discovery Date: Tue Nov 26 09:49:53 EST 2019 I'm starting to think this is just a random walk process in the mind, exploring combinations in the background. Given a good representation, many seemingly hard problems become completely obvious because the representation allows proper factoring. Finding that representation is a whole other problem. Entry: Two kinds of code Date: Wed Nov 27 09:39:23 EST 2019 - computation: library code, functional, batch processing - communication: event-driven, device and/or human interaction Entry: Library code Date: Thu Nov 28 10:27:35 EST 2019 Is most code just there to solve problems created by past protocols? Entry: GOTO and traces Date: Fri Nov 29 08:33:59 EST 2019 So event driven programs, state machines, are essentially programming with GOTO. Should I worry? I don't think so. It seems to be the most natural way to do this. The problem becomes documentation: add comments that illustrate some practical protocol traces. It would be nice to be able to program traces directly. This is essentially what "threaded programming" is, but it does not handle forking traces well. Why? I think that making that explicit will shed some insight. EDIT: Note that threads are not linear. It is possible that inside a thread, execution forks at a conditional statement. But what is different from raw state machines is that for a thread, there is usually a join. I.e. the forking of traces is temporary. EDIT: So what is understood by "trace semantics"? Somewhat in the ballpark: https://www.cs.rice.edu/~vardi/papers/lics09full.pdf But what I need is something simpler: a relation between a state machine, and a linear time protocol sequence (test). Entry: Nested quoting in bash Date: Fri Nov 29 08:46:57 EST 2019 This is a problem that I've had for a while, and it pops up especially in combination with ssh. I don't quite understand how to begin tackling it. EDIT: Use explicit quoting. An approximate implementation isn't that hard. See ecat.erl A correct implementation probably needs to re-implement %q for /usr/bin/printf. Entry: epid Date: Fri Nov 29 12:42:09 EST 2019 The big idea there is to get rid of ad-hoc routing. In almost all Erlang code I've written, I end up with some ad-hoc routing mechanism when I'm doing external interfacing. Entry: async/await Date: Sat Nov 30 06:23:30 EST 2019 https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ Highlights the problem with composition. I guess you choose: threads or state machines. It seems that asynch is the solution when threads are too expensive or impossible in a particular context. Entry: Distributed systems: the problem is name resolution Date: Sun Dec 1 12:33:26 EST 2019 Once things are not organize hierarchically, finding peers becomes problematic. So in many practical cases this is naturally solved by splitting the problem into a name structure and a namer resolver. It seems best to solve it at the top level system layer and have only one name resolution mechanism. Entry: Split naming and addressing Date: Sun Dec 1 15:24:34 EST 2019 I keep coming back to it. Seems to be very important. Addresses are representations of routes. Names can be arbitrary, but should be handled centrally. Name resolution is much less frequent than message exchange for processes that have some kind of long term relationship. Entry: Object oriented design.. Date: Tue Dec 3 09:48:37 EST 2019 ..really is about not really having to design anything. At least that is what it feels like to me. It is a buffer that allows accumulation of cruft to a certain threshold where it becomes clear what is needed, and then things can usually be refactored. But maybe this is not really that specific to OO. Entry: composite keys vs. nested maps Date: Mon Dec 9 20:53:23 EST 2019 These are isomorphic, so the only difference between these structures is implementation. But I find that switching to flat map with composite key representation opens doors. It makes code simpler. It allows for restructuring, as the structure itself (the grouping of keys) can be made compositional, i.e. functions can create collections of keys to isolate substructure that might be different from the (arbitrary) nested structure. This seems to work well when the idea of "coordinate" is more appropriate than "path", i.e. individual dimensions are indepenent, not one conditional on the other. Entry: distributed systems Date: Tue Dec 10 07:29:18 EST 2019 Why so interested in distributed systems lately? At the integration level, there is really no way around it. Also, in hardware, there probably isn't either. Entry: explicit stacks Date: Thu Dec 19 16:08:53 CET 2019 1. Write a recursive version first. 2. Replace all non-tail calls with pushes 3. Replace the now "interpreter" function with a pop + again part at the end. Entry: Using a stack as a code sequence Date: Sat Dec 21 15:15:53 CET 2019 Implementing a binary search, I am reminded of the idea that the explicit stack used in the algorithm can also as a code sequence, by loading the code sequence in the form of continuations in reverse order. Basically, this reduces the complexity of a sequencing structure. Instead of sequence + stack, it is now just stack + an initializer for the stack (which is non-blocking and thus has no abstraction cost). EDIT: Why can't a stack just consist of function calls? Basically, event handlers, instead of jump points. Yeah the hybrid instruction vs. resume point idea. It's somehow dual: event handlers have inputs but no outputs, only side effects (message send + state update). Entry: Concurrency Date: Sat Dec 21 13:09:08 CET 2019 Go over basic concurrent lit again. Distinguish CSP, Pi and Actors https://en.wikipedia.org/wiki/Actor_model_and_process_calculi_history https://en.wikipedia.org/wiki/%CE%A0-calculus Maybe this is an interesting opportuniy to constrast non-blocking and blocking architectures? Entry: Transforming packet processors Date: Sun Dec 22 11:04:45 CET 2019 Small experience report for converting a blocking recursive algorithm into a non-blocking variant. 1. Converting to explicit stack is very straightforward, and 2. The real problem is properly handling resume points. I know of two ways to do this: keep the algorithm's structure, but use computed gotos. I.e. sm.h Or split it up into explicit continuation functions. The latter leaves the code less recognizable, but probably makes for more readable C code. Let's go for the latter. Make the continuations explicit. Morale: write CPS. EDIT: Keep the code in blocking form until all blocking calls are lifted out into a main "next loop". EDIT: Variable management becomes a problem when all variables are tucked into a single struct. It would be nice to be able to use real lexical scope here. EDIT: This lack of sum types is the single most annoying point in C. Each state really should be its own type, which ultimately derives from the lexical scope of the continuations. I think this is then also the way to make a transformation from a high level description into C: the flattening of all the continuation types into a single C struct, attempting to re-use shared state. Entry: How everything is related Date: Sun Dec 22 14:14:14 CET 2019 When you don't do pre-emptive scheduling: - The core idea is definitely state machines and transition functions. - Partial CPS-transformation then links tasks to state machines by introducing pre-emption points. - Representation of the continuations makes the bridge to "monolythic" state machines. I.e. a sum type of the different state context types. I really don't like pre-emptive multitasking, but Eric comes to the conclusion that it is necessary, and that many priority levels are necessary. It's probably ok to combine the two: use the static state machine tasks with a scheduler. Actually the ideas are really orthogonal. My point is really about the representation of the tasks, not the scheduling. One remark: I found it easier to write state machines in a language like Erlang, with the availability of more appropriate types for representing state, and the ability to pattern-match. In raw C this is a real pain. Entry: The case against REST Date: Wed Dec 25 13:59:39 CET 2019 I think this is wrong. Communication is expensive, so always use incremental protocols. State isn't bad as long as it is really only cache. Entry: Why are protocols always so much work? Date: Wed Dec 25 16:38:41 CET 2019 Because next to "code", you also have the interpreter. And often, it is custom. Native code, or functions in a language do not have that disadvantage because they share a substrate. This shows how much information is actually encoded in the default interpreter. Entry: Always use external iteration in C Date: Wed Dec 25 17:42:46 CET 2019 Basically, callbacks and explicit context structs are a pain to use. Sometimes they are appropriate, but for small-grained stuff it's much easier to let the user perform the iteration in a particular context. Entry: Forth protocols Date: Wed Dec 25 19:11:36 CET 2019 To they are useful due to lack of intermediate data structures. However they likely have an imperative implementation where it is more work to guarantee constraints. It makes sense to do this for 1) protocols that are already very imperative or incremental 1by nature, and for 2) low-memory targets, which also tend to be very imperative I think the conclusion is that structured data types are expensive. The world is made out of state machines. Stick to that level and code is going to be small. It seems possible though to generate these things. Generate differential protocols. But really, why does it work so well? Because explicit data structures are replaced by folds. Morale: REPLACE STRUCTURE BY ITERATION Entry: Reverse constructors Date: Wed Dec 25 20:15:00 CET 2019 Let's look at this a bit more. A list in RPN is nil 2 cons 1 cons A binary tree is 1 leaf 2 leaf tree 3 leaf 4 leaf tree tree Entry: Forth as Protocol Date: Fri Dec 27 01:22:12 CET 2019 Posted something on Twitter then immediately realized that postfix isn't always good because you can't "choose parsers early". Or maybe you can? Just send a specific word to pick the sublanguage, and let that set up the context of the parse. Entry: Machines Date: Fri Dec 27 19:20:54 CET 2019 Often I really want tasks that are as efficient as state machines. As established before, there are two problems: - split code into resume points - represent local stack frame for each resume point Tasks really are very convenient, precisely because they are already stacks. EDIT: Doing this in bare C: I ran into a case where an explicit stack + control loop made sense. Resume points appeared in places that made conversion to proper states very awkward. Sometimes you really just want a blocking call... If the type of data that is associated with the resume point is simple it makes sense just to return the resume address and let caller pass in that address again. Entry: Low level "functional" machines don't make sense. Date: Sun Dec 29 15:29:42 CET 2019 I'm really coming back from this. The most important attribute of low-level code is simplicitly. If that means "dumb state machine", then so be it. Typically the idea of "driver" makes sense: use a higher level language for configuration management, and convert to low-level state-setting commands. Entry: Events Date: Thu Jan 2 10:32:49 CET 2020 It's a lot of work to "introduce events" in a dumb system. And it gets worse as it gets more heterogeneous. I think I've arrived at the point where I am no longer writing code that isn't event-driven. Entry: So what went wrong in software evolution? Date: Thu Jan 2 13:23:06 CET 2020 Before watching https://www.youtube.com/watch?v=pW-SOdj4Kkk Here's my ideas: 1. Hardware became too cheap. There is no real constraint on memory usage and execution speed, and that allows for very sloppy coding. 2. Probably as a consequence, software became too cheap or free. This removed the ability for people to complain about quality. 3. Systems became too complex: this also removed accountability because assignment of blame became almost impossible. From the talk: the problem is to communicate from old hats to newer gen. So basically, software isn't all that important. We need to make sure we don't forget how to make computers. HA: 1 is mentioned. 2&3 sort of: we're used to the bugs. Entry: Cleanup previous crash Date: Thu Jan 2 14:26:44 CET 2020 There is one thing that I don't seem to realize: destructors don't work remotely. Failures always leave inconsistent state, so a restart needs to clean up the previous crash. The easiest way is to delete everything and start over. Entry: How to make improve software? Date: Thu Jan 2 15:27:32 CET 2020 1. Use less software: stop tolerating broken shit. Reduce stack, swap things out. 2. Use better a development methodology Practically, what does this look like? Create a "staging" system based on current available tech: PCs, linux, and programming languages. What can I do? First, get own shit under control - Security border: 1. internet software is Debian with frequent updates. 2. internal software only does updates when required for "app". the app is exo: integration layer for everything i use - Development tools: Bootstrap a new system on top of the old one Linux. Use tools wisely: - Erlang for system-wide network management - Rust / Haskell / OCaml for modules. - C for glue only when necessary - Embedded: Step 1: Keep Linux out of embedded. Step 2: Focus on state machines - PC: Use only decent hardware. And use a small OS that supports it and can run the standard C stack. Entry: focus on protocol design Date: Thu Jan 2 16:52:35 CET 2020 protocol design - single-ended messages - limit number of round-trips, e.g. send acks for a bulk transfer only. - focus on caching: most protocols are about re-creating the same object, locally. - translation between nested and stack-serialized representations is trivial. main difference is that smaller machines can deal with stack-serialized. for larger machines it can be completely abstracted into ADTs. does it make sense to get rid of TCP altogether and use only UDP? no. Entry: Configuration management is cache coherence / replication Date: Thu Jan 2 21:50:36 CET 2020 Basically: some central point (which itself can be in some kind of distributed CP/AP netwoerk) has a model of how the distributed system is configured. Edge nodes have a projection of this configuration, and their state is updated to bring them into coherence. Everything can be managed this way. All state, including code. ( All in the incremental lambda calculus framework. ) I don't think that incremental lambda calculus is a good approach for me at this time. Work on really understanding delta->delta maps at an intuitive level then maybe generalize. Entry: build as cache Date: Wed Jan 8 19:31:37 CET 2020 https://gbracha.blogspot.com/2020/01/the-build-is-always-broken.html https://lobste.rs/s/pf8tw3/build_is_always_broken @Gilad_Bracha Entry: Aging brain Date: Sat Jan 11 17:19:24 CET 2020 Not really softarch, but a reason to use modularity. I've found it more difficult to load new context. It seems I can do this once or twice per day for large switches, but then it's done. Modularity really helps here, as it makes it possible to zoom in on a small part of the program. Entry: Cleaning up Date: Tue Jan 21 13:49:17 EST 2020 I'm starting to think that the most useful skill in software is cleaning up: seeing how something is too complex, and simplifying it. Entry: FPGA Date: Fri Jan 24 09:40:13 EST 2020 I need more intuition. I can see some tradeoffs between stored program approach and small high-speed machines, but I can't use it yet to actually start writing things down. Entry: Make research more predictable Date: Fri Jan 24 09:52:52 EST 2020 This pattern keeps coming back: 1. Some idea surfaces out of the blue. I work it out a bit, pushing things forward, terminating at more questions but definitely providing a good vantage point.+https://www.google.com/ 2. Some practical problem needs solving, and while solving, it gets matched to wone of those theoretical ideas that were activated to linger in the backround. So I guess just keep doing that? Really keep track of the next.org document. Entry: Heterogeneous deployment: chain of caches Date: Fri Jan 24 15:29:55 EST 2020 This might be a very special case, but it seems to work alright: - distributed highly-available store that gets resynchronized get, put and reconnect. this level is multi-directional (each node can update). - in parallel, a read-only content-addressable store. can be referenced by the objects in the former, but is lazy. - local updates to store can be watched, and can cause further unidirectional propagation, e.g. load from CAS, then wait for locally managed nodes to appear for deployment. For bluepill, I've noticed that running a trans-atlantic GDB RSP session is not a great idea. Move the file first, then run gdb locally. Entry: Abstract code Date: Mon Jan 27 08:55:48 EST 2020 The problem about thinking too abstractly about code is that it is easy to run into the case where there are so many degrees of freedom that cannot be easily parameterized. I.e. "structural" parameters that would need some meta infrastructure to implement, such as a compiler or other data or code generator. Entry: Jonathan blow - On The Metal Date: Tue Jan 28 13:56:23 EST 2020 https://feeds.transistor.fm/on-the-metal-0294649e-ec23-4eab-975a-9eb13fd94e06 https://oxide.computer/blog/on-the-metal-9-jonathan-blow/ Takeaway arguments: - many abstractions are not worth it - systems are different for no reason - Rust is ok, I guess - data-oriented: operate on large collections to make things fast - who has courage to start over? Entry: Entity Component System in Rust Date: Tue Jan 28 16:04:34 EST 2020 https://www.youtube.com/watch?v=aKLntZcp27M - don't use OO - array(srtruct(...)) -> struct(array(...)) commutation - generational indexes (slotmap) - AnyMap (dynamic types) - type registry Entry: Ron Minnich - On The Metal Date: Tue Jan 28 19:21:03 EST 2020 https://share.transistor.fm/s/43499ab9 myrinet: source-routed network Entry: Redo initial dependency generation Date: Sat Feb 1 11:36:33 EST 2020 So I wonder.. If there are no dependencies yet, does the first occurance of redo-ifchange immediately abort the script? Or does it get suspended somehow? I think implementing this using a process per product actually makes a lot of sense. Going to make an Erlang model. Entry: schedulers Date: Sat Feb 1 20:22:55 EST 2020 - erlang-style actors - CSP - redo style "pull" frp with input polling and input-dependent dependencies. - "push" frp Entry: large teams make things complex Date: Tue Feb 4 05:51:17 EST 2020 So what about focusing on modules that fit in one head? Risky for a large company, but great for a small one. Entry: Jim Keller - Lex Fridman Date: Sat Feb 8 22:44:17 EST 2020 Interesting enough to watch it again and write down some ideas. Attributed to Elon musk: People are "how-constrained". I.e. I have this, how do I change it. But it is better to first find out what you want, and then try to build it. https://www.youtube.com/watch?v=Nb2tebYAaOA Entry: Static vs. Dynamic types Date: Mon Feb 10 18:43:49 EST 2020 Here's a simplification: If you have a lot of structure in your data, use types. Definitely. If you are mostly doing control, sequencing, protocol, ... I don't think it matters all that much. Entry: Haskell Seq and testing Date: Wed Feb 12 05:41:59 EST 2020 It works fairly well, but I can't shake the feeling that it is still more complicated than necessary. There are many differnent levels: - top level exo-ghcid.hs "current test" script - p_ predicate - x_ ad hoc test - e_ quickcheck quantification - t_ TH wrapper - test library - Seq test wrapper - Seq library code It seems these are all necessary. Maybe the problem is just that complicated. Maybe it is really about lifting something out of another world. Most of the load-beareing, non-recurring work is in: - x_ ad hoc test - p_ predicate - Seq test wrapper - Seq library code Those fork off reusable parts into: - test library - Seq library code Trivial but annoyting bits: - top level exo-ghcid.hs "current test" script - e_ quickcheck quantification So to simplify, streamline the writing _process_ of creating the load-bearing code, and make the trival bits actually trivial. Entry: one or more? Date: Sat Feb 15 12:39:21 EST 2020 So much of software architecture is about "do I need one or more", and then if it is more, abstract it away into one again. Entry: strangle instead of rewrite Date: Tue Feb 18 07:52:42 EST 2020 https://understandlegacycode.com/blog/avoid-rewriting-a-legacy-system-from-scratch-by-strangling-it/ Entry: Create "spaces" Date: Tue Feb 18 16:36:09 EST 2020 Our brain thinks in reference frames: https://www.youtube.com/watch?v=-EVqrDlAqYo So to make it feel at home, programs should be spaces? Entry: distributed systems and local memory Date: Fri Feb 21 07:11:34 EST 2020 Conclusion for now is that it's hard to design distributed systems without some form of local store, and really, you don't want to either. In general, communication performance cannot be ignored for distributed systems, so it is probably best to design it around tasks knowing the physical locations of the data they operate on. This breaks abstraction somewhat, but not doing this makes code very hard to debug. Entry: redo: restart manager Date: Fri Feb 21 09:44:30 EST 2020 So it was really good fortune to dig into this. I can now easily build a "restart manager" as part of this. Entry: Better logging? A two-level approach. Date: Sat Feb 22 10:21:05 EST 2020 This is a Goldilocks problem: too little and you don't have what you need, too much and dealing with volume is problematic. How to better encode logging? I like the idea of giving each log statement a tag. This way logging is not more than timestamp + tag + data, and it can go in a sqlite database. Many large volume problems benefit from a two-level approach: in this case: 1. sqlite tags can be used to limit the volume 2. ad-hoc structural filtering can be used at the finer scale. The problem I am trying to avoid is to have to design a query language. Sqlite is neat because the second level matching could be implemented as C predicates operating on encoded data strings. Entry: Two-level systems Date: Sat Feb 22 10:24:19 EST 2020 I run into this a lot. Flat systems are desireable but can be hard to manage due to data bulk when systems get large. Often just splitting implementations into two levels is enough. Examples: 1. logging (coarse, fine) 2. erlang epids 3. system design (integration, module) More levels usually are not worth it, as they create more non-uniformity. Important to note that this is about _implementation_ Semantically, these should be treated as flat whenever possible. Entry: Opaque vs. concrete state: can it just be cache? Date: Wed Feb 26 17:08:32 EST 2020 It is because Erlang's state machines have no magic. Their state is a single data structure. Putting it like that makes it easier to look at state in a different way: opaque vs. concrete state. Concrete state does not need to be "live". Spinning up a process with a particular concrete state can be done lazily, i.e. it is a cache. If combine that with a universal naming scheme, and perform just-in-time binding, then ALL state becomes concrete. To me, this is such an important concept: I grew up with mostly opaque state: a process' state is a bunch of early bound data structures. Entry: Holes Date: Fri Feb 28 07:41:26 EST 2020 I find that switching abstraction layers works well if you stay on one side for longer periods of time, and just add "holes" that can be added later. Entry: aging brain Date: Fri Feb 28 09:21:25 EST 2020 definitely, build an ide. Entry: why is the work never where you think it is? Date: Fri Feb 28 17:29:56 EST 2020 One thing is clear: decent infrastructure is expensive. Application development is usually imagined relative to decent infrastructure. Often, it just doesn't exist or isn't resource-friendly in existing forms. Why is it so easy to make the mistake to "forget" the infrastructure isn't there yet? Too high level thinking? Entry: moving from managing files to live state Date: Tue Mar 3 14:46:02 EST 2020 Implemented redo in Erlang, which makes it easier to manage "live" state, as services can be managed through Erlang proxy processes. This has changed my view on a lot of things. Entry: integration Date: Mon Mar 9 10:32:37 EDT 2020 This thing (exo) is irregular enough that I frequently bring together two distinct paths and end up with an impedance mismatch. That is where insight is necessary: to prune the accidental complexity. Entry: design is "such that" Date: Mon Mar 9 10:55:02 EDT 2020 I.e. it is very implicit: You create something such that the pieces fit together. Not all work is like that. If you are given the rules, you can just follow them. Design is making up rules such that the complexity turns out to be minimal. Much different. And often needs trial and error. Entry: naming is the central problem Date: Sat Mar 14 15:05:33 EDT 2020 It's not about code or data, but how everything connects. The main connection mechanism is implicit (late) binding using names. - filesystem: ordered list of strings - database: composite names (functions of 1 or more variables) Entry: exo: visceral programming Date: Sun Mar 22 10:26:28 EDT 2020 I have no term for this, so let's call it Visceral Programming. Programming by gut feeling. The principle is based on: - You're going to get it wrong if you design it. - Just start building the happy path and iterate. - Make it easy to make small changes. - Trust that something will emerge - Clean up when it is clear what is wrong This is only for "glue logic". Top level stuff. The thing that connects all the pieces but is in itself a little chaotic. Once patterns become clear, abstract them out. Entry: What if DNS had no root? Date: Sat Mar 28 15:04:40 EDT 2020 Basically, using a network topology, where each node has providers, and when "names" are exchanged between nodes, they are changed into routable addresses. Basically, a distributed set of name servers that can map symbolic names into routable addresses. E.g. considering the Ockam block chain: basically, instead of designing a single PKI, allow for delegation. Entry: paving over Date: Mon Mar 30 12:03:39 EDT 2020 Integration is difficult. Why? It is the part of development where there are a lot of opportunities to simplify, and if they are not taken when they present themself, complexity explodes. Basically if you find yourself paving over interface problems in a lower layer, you should fix them in the lower layer instead. If you don't do this, software tetris happens. Entry: Name resolution is routing Date: Thu Apr 2 10:11:21 EDT 2020 That is, if resolution maps to source-routable multihop address. Name resolution functions are a very powerful abstraction. Entry: environments, dynamic scope Date: Wed Apr 8 08:26:41 EDT 2020 When is it good? When you want to be able to configure things that are deep down. Cases where this is useful: - Drawing / rendering code - Modifying low level interfaces In some sense it always feels like giving up, as the ordinary composition has failed to express a requirement. Entry: Make local, incremental change easy Date: Mon Apr 13 12:01:43 EDT 2020 Basic observation: work that requires a lot of context takes a lot of energy. So optimize the development process towards supporting incremental local changes. To me, in embedded land, this means: set up a fully integrated on-target system as soon as possible. If that's not possible, create stubs against an api. DO NOT WORK WITHOUT LOW-FRICTION TESTS OR INTEGRATION LAYER. Entry: Log union Date: Wed Apr 15 09:23:25 EDT 2020 A recurring problem: how to handle log messages coming from many different subsystems? One way to handle it is to ensure that log messages are packaged at the source. E.g. do not send raw stderr chunks. That just makes it very hard to use. If that's not possible, add a framing mechanism as soon as possible. EDIT: While it's a nice idea to let devices only produce framed log messages, it does look like this puts the burden of abstraction in the wrong place. Log messages should be easy to use. They are ad-hoc. If they are not ad-hoc, they should be structured more than just moving from byte stream to line steam. So stick with raw streams, and add a line parser when merging. EDIT: The solution is to add line buffers for every stream that is merged. It doesn't matter where it is implemented, so implement it at the most convenient place. That might be all the way at the top. So let's add another tag for tagged logs. Entry: Create Code Places Date: Thu Apr 23 09:20:23 EDT 2020 Sometimes it's not easy to decide what runs where, so it is important to create a "place", a "host" that is very general such that code can be moved from one place to another. An example of this is a 3-tier architecture: - a bare-bones microcontroller for real-time work - a C (or Rust) program on a linux host, for performance critical work - some high level scripting language (Erlang, Lua, Scheme) for system management Once these are set up in a way that they can all talk to each other, it is often simple to decide where to put a thing based on all kinds of constraints: E.g any of the platform properties: real-time, not real-time but performance critical, or just development time. Entry: testability for low-level code Date: Mon Jun 22 13:02:47 EDT 2020 Trying to distill some lessons from last week's struggles, trying to go fast when developing low level code. The concrete context of this is a network of one "decent computer", i.e. a linux box that can basically run any kind of high level language to implement a test framework, and a bunch of low cost bare metal memory-constrained high traffic networked devices that are difficult to monitor. Some rules: 1. Don't try to go fast by cutting corners on automated testing. 2. Use ONE software-configurable test haredware setup. Do not build anything that requires manual intervention. Invest the time up-front to make it completely software configurable such that configuration can be recorded in source control, and the test system can be specified and duplicated. Time spent on this will be less than what would otherwise be lost in debugging anyway. 2. Spend the time to build assert tests. Don't just rely on visual inspection of test logs or scope traces. If necessary, add statistics gathering and special state queries to the bare metal devices, to allow assert testing on embedded device states. Same argument as 1 wrt. debugging time: this is expensive and slow, but it pays off. 3. Try to split development in two phases: feature append only, and refactoring. During append only work, also edit the append-only test suite. This will then provide a test for everything that was developed, and can later be used as a regression test during refactoring. A possible exception: it is allowed to make the test configurable to _temporarily_ disable some tests to be able to zoom in onto an implementation issue, or to make it use less time, but then TURN IT ALL BACK ON AGAIN. The idea is to have ONE test that is kept running in all circumstances and doesn't cost any effort to run. Simple pass/fail. 4. Writing the firmware is almost trivial if 1. you keep it simple and 2. you have good tests that catch issues immediately. 5. Create a mechanism that can encapsulate assertions so they can be sent back to host. I.e. each embedded device should have some kind of observer state machine of which the state needs to be queryable. 6. And last but not least: it is very easy to fool yourself. At least for me. Measure. Be explicit about properties. To boil it all down: refactoring is necessary. If your implementation subtrate can not catch simple programming errors, you need a test suite you can rely on to ensure that your application still works. Another trick is pipelined/feedforward asserts, i.e. send command, let controller change state and compute assert locally, send report upstream, fail test suite on failed report. Entry: Redo and "gathering patches" Date: Tue Jun 23 08:53:08 EDT 2020 I am very happy with the generalized redo approach. There is one thing though that does not work properly: gathering patches. Say I have a system (typically a db, or some other expensive state that is unacceptable to be "recomputed" on every change, like restarting a machine), and I want to let redo gather all the increments that are made to that system, but apply the patch only once. How do I go about that? The basic pattern seems to be that there is one central rule that takes as input all the "diffs", computes a new state possibly performing other side effects, and then computes a number of output diffs triggers to feed back into other systems. So a centralized diff in - diff out. Entry: Getting more annoyed with dynamic typing Date: Mon Jun 29 17:00:27 EDT 2020 Especially for "work", i.e. thing that have no intrinsic interest and just need to get done. There I really do not care about your fancy architecture. Just tell me what I'm doing wrong so I can get on with my day. Entry: Redo as a composition layer Date: Tue Jun 30 16:35:08 EDT 2020 It actually works quite well: leave the components to their protocols. Let them do whatever makes sense. Files are probably best. Then orchestrate a dataflow network around that. It is a bit of work to capture all that in data structures and build rules, but that work needs to happen anyway, so maybe best to use a good structure for it. One thing though: I am tired of dynamic typing errors. It would probably be a good idea to rewrite this in Haskell or something, or to create a layer where everyting is actually proxied in the file system using some kind of tokens, such that ordinary build tools can be used? E.g. a running daemon could be a file with its address (pid, socket, ...). Entry: Serialize / Deserialize Date: Sun Jul 5 13:46:07 EDT 2020 This is the biggest problem I face: pack/unpack messages without wasting a lot of memory and duplicating effort, and doing it in a polyglot way. Entry: What to do when tired? Date: Sun Jul 5 15:11:05 EDT 2020 I guess this is going to be more of a thing as time progresses. When I'm tired, I cannot hold a lot of details in my head, but I have relatively little problems doing local refactoring. If there is a type system that can catch silly mistakes, then this remains locally focused and is much less draining. Entry: Sequencing Date: Fri Jul 17 21:18:30 EDT 2020 What I want is async rust, but that is not going to happen. I don't want to mess with stack swiching, so I currently have two options: use sm.h, or use an interpreted language. Let's go for sm.h and create some more usable abstractions on top of it, and leave the road open to possible write a C compiler for it. Entry: Blocking on conditions Date: Fri Jul 17 23:37:31 EDT 2020 Context: implementing a simple multitasking system. The core decision for a simple system seems to be: a) round robin poll-based b) some kind of synchronous event system (e.g. CSP) The former is MUCH simpler to do, because events are implicit. You don't have to track when something becomes true, just that at some time in the future, somone will notice it has become true. So this is in essence asynchronous. Conditions seem much more general as opposed to synchronous event propagation. Also, callbacks seem to turn into CSP very naturally once they become recursive. Entry: Emulation, events Date: Sat Jul 18 08:40:52 EDT 2020 I want to be able to run sm.h tasks on Linux, then move them to the uc when ready. This should be possible when tasks are actually isolated and do not use "conditions" as explained in the previous task. So events really seem to be the proper abstraction mechanism. Reason is that conditions cannot be implemented without shared memory. Entry: Rendez-vous over network? Date: Sat Jul 18 10:50:30 EDT 2020 EDIT: Removed old notes because it appears they were wrong. Conclusion after trying to implement network-transparent rendez-vous is that it is not possible, because simultaneity does not exist across a network link. I.e. the proper way to model a network communication is to distingish send and receive as separate events. So. 1. Use synchronous CSP if simultaneity is available. It is a powerful abstraction mechanism in its own right, and its lack of buffering is easy to implement. 2. Asynchronous communication becomes essential when network links are introduced. They appear to introduce an essential non-simultaneity. 3. It is still possible to implement synchronization across the network, but it will NOT have the same properties as rendez-vous. So what is an asynchronous channel exactly? It is a channel that does not block on write. E.g. it looses the property of being able to block a task until the message is received. Entry: C protocol dispatch Date: Wed Jul 22 18:31:08 EDT 2020 This is what I end up with: - create a schema, describing all wire data structures - generate C structs + metadata necessary to perform serialization - create a generic ser/deser routine parameterized by metadata - generate a dispatcher that glues ser/deser to C routines taking plain C structs. - use bump/stack allocation for ser/deser Entry: Pattern matching Date: Wed Jul 29 22:39:04 EDT 2020 Lua doesn't have pattern matching. However, it is possible to replace pattern matchinging with generalized fold, where each alternative of the sum type corresponds to a function. Does that bring any notational convenience? It would if every alternative is named. So represent a match as a dictionary of case clauses, represented as n-ary functions, each named after the constructor. So for a cons list this would be: { nil = function() ... end cons = function(car, cdr) ... end } Entry: Implementing actors Date: Fri Jul 31 20:10:59 EDT 2020 Bookkeeping seems heavy. I find I need: - doubly linked list - backpointer to scheduler - status bit That's 4 words. Not very lightweight. Using indirect bookkeeping: 15 bits for task pointers, no backpointer to scheduler (store it elsewhere), and 2 bits left for flags. Is it worth the trouble? Probably not. EDIT: Worked on this a bit more. The main issues I end up with are related to memory allocation for task and message queue structs, and also it seems that blocking might be necessary, e.g. when a mailbox is full, a task needs to block. From these the conlusion is that actors might not be a great abstraction for low level implementation, but they would work much better if memory allocation wasn't such an ordeal. Practically though, I do not really have many cases where the tasks are going to be very dynamic. So let's not worry about that part. Entry: functional reactive programming Date: Sun Aug 2 19:14:09 EDT 2020 I made a lua and C implementation of a pure reactive network. Maybe now is a good time to revisit some of the ideas of purity, and why it would be beneficial as compared to an "event" or message based system. There is a difference between defining events as messages, and defining them as implementation details, i.e. updates that happen behind the scenes, with only an interface to the world where: - input messages are mapped to value changes, and might stop propagating if they do not change - output messages are derived from value changes, and will not happen if there is no change. So a pure setup like this feels simpler, as in being just an implementation of function evaluation and not a messaging system. But is it actually simpler to develop systems this way? Once example that has recently come up is in the context of a control panel, where two knobs have the same function. This might happen e.g. in a hardware device that also has a software interface, and turning a knob on the device should have the same effect as turning the knob on the PC GUI. This is not really well-defined, as in: what would happen when both controls are operated simultaneously? But if they are not, then it is obvious what the semantics is: the knob that is handled is enabled, and the other one is effectively disconnected. So how would something like that be implemented at the FRP level? Entry: Converting a condition machine into an event machine Date: Fri Aug 14 06:53:02 EDT 2020 Premise: it is much easier to design a round-robin condition polling machine, than it is to design an event driven machine. At least, it is in a bare-metal setting. A task in that context blocks on conditions, not events. E.g. wait for this uart transmit ready flag to be set. Is there a mechanical way to translate one into the other? Preferably by only changing the scheduling, and not the code? The scheduler should somehow know what condition is being blocked on, and only resume the task if there was a change. This smells like FRP, but isn't really beacause I really do have state. Maybe read Conal's paper again for inspiration. https://en.wikipedia.org/wiki/Functional_reactive_programming EDIT: The core problem: - convert the blocking condition into an event (differential) - sub/unsub: e.g. distinguish between being interested in the event, and not caring. Let's fix the polarity of the conditions for true to mean continue, and false to mean block. A task can do this: loop: 1. inspect a condition. if true, continue and don't register an event. 2. if false, register the false->true wakeup, and block To find a shape, implement it manually a couple of times and focus on how to convert conditions into events. It seems that this can be relaxed a lot by treating the events as optimizations, e.g. their only effect is to schedule a poll, so they could be implemented as a dumb poll. What I want to know is what kind of eventing mechanism this should be placed into? It is clearly not stateless, so FRP doesn't seem to be right. It seems that some kind of pub/sub would be more appropriate. In my specific case: interrupt routines and a broker. The broker will receive events from hardware, will filter them based on what tasks are interested. Then each task can send change notifications to the broker if it changed a condition. So this looks like a mix between actor and csp: - the relation from broker -> tasks is non-buffered. there is only a single channel that carries the "something might have changed" semantics. - the relation from event source -> broker is buffered. Let's zoom in on the concrete condition variables in the current application. For now I have the following actual condtions: - Token stream ready (uc_tools cbuf) - Timer expired (uc_tools 32 bit cycle_counter poll) - Hardware ready flag event (e.g. uart) - Software ready flag (e.g. some state machine has reached its final state. ) It's probably most appropriate to do this as part of sm.h because it is in that contect that I am faced with the need for opaque polling. These all need to be inverted, meaning that the entitiy that causes the change needs to schedule a notification somewhere. How to implement? - For cbuf writes, it could go straight into the cbuf data structure. Just add a callback that is invoked whenever there is data in the buffer after a write. - Software ready flag is also just a callback. Same case as cbuf. - Hardware ready flag can use peripheral interrupts + main loop wakeup. - Timer expiration needs an actual abstraction. I.e. a software timer driven by a hardware timer. So this is quite a bit of work to extract due to the variety of events. Hence: it is a lot eaasier to just poll if the energy consumption is not a problem. This is also the central issue: if power saving is necessary, then hardware event conversion to interrupt is necessary. Basically, conditions need to be inverted into events. EDIT: One remark wrt. timer: there is no consequential difference between letting the main loop poll, and letting the task poll. The only real change here is to turn timers into events, either using separate hardware timers, or using a single timer to implement a heap-based software timer. Entry: Mixing poll architecture with push events Date: Sun Aug 16 08:21:35 EDT 2020 I'm glad I worked out the CSP scheduler: the problem there was exactly the same: when you push an event into a channel, and nobody is listening on that channel, there is nothing else to do than to drop the event. But you always know that this is what is happening. The other solution is to buffer it, but a buffer is nothing more than a task that is always ready to receive. EDIT: The core idea is the following: - move some data into the context of a task - resume the task - get feedback over whether the data was consumed or not. Entry: Lazy timer infrastructure Date: Sun Aug 16 08:33:54 EDT 2020 Here's an idea: if a task is just waiting for a timeout, the poll loop can be optimized to automatically implement it by registering a reschedule. EDIT: Still half-baked, but I really need to implement a software timer to get a better handle on the complexity of the problem. Entry: Logging Date: Sun Aug 16 19:14:27 EDT 2020 The core problem I face atm is logging. I could use some fresh ideas there. I know apenwarr wrote about this, but it's not exactly what I'm looking for. https://apenwarr.ca/log/20190216 Interesting conclusion though: - you want structured events in your database. - you need to be able to produce them from unstructured logs. And once you can do that, exactly how those structured events are produced (either from logs or directly from structured trace output) turns out to be unimportant. I think my problem is much simpler: I just want proper time stamps. First, is there a way to automatically add time stamps? Prefix a timestamp for every character that is printed directly following a newline. That is possible if infof is patched. Maybe just add a hook there? Entry: poll events Date: Wed Aug 19 09:41:44 EDT 2020 So what is the relation between polling a condition (e.g. timer == timeout), and having a CSP event? There seems to be a subtle difference... First transformation that can be made is to move the condition from reader to writer. E.g. the system interface will produce an event when the condition happens. But what I do not quite understand is: what happens with events that are not read? In the condition architecture, this is implicit: control flow just happens, and does't poll certain conditions in certain execution paths. So it seems that the real issue is in moving the condition from reader to writer. e.g. the writer doesn't know whether the event needs to be produced or not. How to express that in csp? In the case of the timer, it would e.g. mean that the timer selects on all its writes, and if a later one happens, it can discard an earlier one? There seems to be a fundamental difference. Find a better way to represent this. The fundamental idea is causal relationships. In the case of condition synchronization, there is a shared world. Tasks can mutate the shared world, and tasks can wait for conditions computed form the shared world. When a task resumes operation, that can be seen as an event at that task's end, but it is not an event anywhere else. Event is just not the correct abstraction. What am I missing? To re-iterate: the thing causing a condition to become true is an event, but it is implicit. It is probably possible to represent the event through the derivative of the blocking condition. I.e. the operation of deriving makes the event explicit, but otherwise it has no explicit representation. Entry: Events vs. Conditions Date: Wed Aug 19 11:23:49 EDT 2020 Let's try again. - Conditions are more fundamental. It is easy to represent events as conditions, but the other way around seems to need the idea of derivative. - Conditions map better to dedicated sequential hardware: polling is a natural operations. Events need to be implemented on top of polling conditions. - The main advantage of events seems to be that it allows a time-sharing implementation and more efficient emulation and mathematical manipulation. - This smells like FRP, but in my mind I have sequential processes waiting on conditions, not just computing functions from inputs. Assume that derivation will produce events from conditions. How does this work in practice? Maybe find some examples. And second: FRP might be what I am actually looking for, if I can map time series to time series, instead of time instance to time instance. Entry: The condition to event spectrum Date: Wed Aug 19 11:33:46 EDT 2020 A little context: I have been exploring several multitasking scheduler architectures: - Actor - CSP - pull-FRP (redo) - push-FRP - conditions These have a strong affinity to particular problem and solution spaces. By feel: Actor is for garbage collected languages doing network things, and is fairly general. CSP is a variant that maps better to resource constrained problems. FRP works well if the model is somewhat physical, and condtions work well if the implementation is hardware-like, e.g. dedicated gateware or a CPU loop. The core implementation constraint boils down to events being expensive at the low level, but cheap at the high level (once the message API already exists). The border here is the introduction of time-sharing: a CPU or programmable datapath does more than one thing, v.s. dedicated hardware polling a single function. Right in the middle of the software complexity gradient is the transition from no scheduler (just a poll loop), to an inverted representation with events, tasks and the idea of using events to the process of task scheduling. Going up the ladder a bit more, there is the abstraction of memory: if unbounded queues are a possibility, the Actor model can be implemented. To rephrase: the availability of abstractions drastically change the engineering landscape, and there is no one-size-fits-all. From bottom up, these are the world-changing abstractions: - combinatorial logic - async logic (flip-flop) - clocked logic (clocked FF, RTL) - programmable datapath, CPU - software loop vs. scheduler - memory abstractions I think this settles the question: why are there so many different paradigms? This leaves the question: what does the semantic border between round-robin condition polling and an evented scheduler look like? Entry: From condtions to events Date: Wed Aug 19 11:49:59 EDT 2020 The practical problem I face is to bridge events (essentially, interrupt occured, data is available), with conditions: did a timeout expire, did a certain flag go fron 0->1? It seems that automatically deriving the condition poll into a schedulable event is not straightforward, but let's try anyway. 1. Did timeout expire? If that is the only blocking point, it is easy to translate the evaluation into the creation of an event: take the difference, set a timer, wake the task up again after timeout. 2. Did the other task read from a buffer? Or generically, set a flag? This can be done by intercepting the actual change. If it is a task, it should somehow register its output change to the scheduler. If it is hardware, it should enable an interrupt and go from there. It seems trivial. Why is this so hard in practice? Because it has to be done for each condition that might effect another machine. So the man benefits of polled conditional synchronization is not so much that the inputs are easier to specify, but that the outputs are implicit. To the problem that needs to be solved is to make outputs explicit. This can be done by adding notifications. This is a simple addition that "implements" power efficiency. I.e. polls can just remain. It is also possible to connect outputs to inputs directly with a data structure, but that doesn't seem possible without a code transformation. No matter how I turn this: events are just more concrete. There is no free stuff here, apart from program transformation that would turn a mutation into an event behind the scenes. Bottom line: 1. conditions, implemented via polling, seem to be much simpler to work with in my current setting, especially when a lot of timing decisions need to be made. 2. if power efficiency is needed, convert to some standard evented architecture. 3. if power efficiency is clear from the start, then use an evented architecture to begin with. better to start off with a properly engineered design. Entry: If time is a parameter, use condition variables Date: Wed Aug 19 12:31:08 EDT 2020 I'm honing in on the core problem: If the algorithm is strongly parameterized by time or time differences, it might make more sense to use conditions. E.g. do not invert the handling of time into events. If otoh the algorithm is strongly reactional without too much depedency on time, an event-based architecture makes a lot more sense. Converting from one to the other is probably going to be painful, but is not impossible. It is however possible to do an intermediate optimization where the poll loop is pruned. This requires some infrastructure where notifications are added. If the polling is kept, notifications can even be optimisitic: e.g. make it legal to notify without something actually happening. Entry: A language that does this automatically? Date: Wed Aug 19 12:41:45 EDT 2020 So with all this talk about event stuff and the realization that events need to be done explicitly in a low level C setting, is it possible to write a language that does all these conditional transformations automatically? This is probably a well-established insight in literature. It would be nice to express that as an actual derivative on program text. I.e. replace condition evaluation with condition evaluation + conditional scheduling of wakeup. EDIT: That reminds me of http://ceu-lang.org/ http://ceu-lang.org/publications.html Maybe time to summarize the ideas there. Entry: Conditions are the better abstraction Date: Thu Aug 20 07:37:44 EDT 2020 Some seed has been planted... Focus on finding a representation that can translate conditions into events. I'm stuck. Go back to Haskell. Find an abstract representation first, then map it to C.