CIP - calculations in parallel

Arjen Markus (7 january 2023) The break during the holiday season allowed me to pick up an old project again: a way to define a bunch of calculations and have them run partly simultaneously and partly sequentially, so that you use the computing resources efficiently. The main steering program was complete and even documented, but I thought it would be nice to have some more supporting programs:

  • A program to prepare the input files based on a simple table
  • A program to report on the variations
  • Various modes for the first program that the reporting program should act upon

Well, I was able to finish a lot of what I had in mind with the exception of the additional documentation. So, this page is mostly a placeholder :) at this moment. But I wanted to publish it now.

saito (6 january 2023) Sounds interesting. I can see how it relates to some of the previous comments on the Tasks page. It might be helpful to show a specific use case where this is used to solve a problem for you. Not necessarily the code you have developed but its usage, as in: calculation-a does this, calculation-b does this, this is how they are related and/or how they vary, and here is how they should all fit together, using your terminology above.

arjen - 2023-01-08 10:33:38

A quick answer to your question: one application of this would be the Morris method for examining the sensitivity of parameters in a numerical model. The idea is to run a set of simulations, each with a slightly different value for a parameter (but the variation accumulates). The set of programs I have makes this very easy, including the subsequent overview of the result: you can simply specify what parameters to change and the "cip-prepare" program generates the input. There are of course a bunch of conventions, but they should be straightforward to meet.

saito - 2023-01-10

Thank you for the comment - I see that you have also published the code. I can see the similarities and the differences between this and the Tasks, and what challenges you aimed to solve with your code. It is interesting to see how different people approach a basically identical problem and the resulting differences in their solution. In any case, this topic is an area of interest to me. I have done something very similar to this in my data flow code.

arjen - 2023-01-08 15:35:40

I published the code with documentation and an example in its current incomplete state on Github - .

arjen - 2023-01-11 08:07:52

With respect to the Tasks package, the original code I posted dates from 2014, so well before that package saw the light of the Wiki :). I am interested in learning more about your work.

saito - 2023-01-11

Of course- by similarity, I only meant the objective or functional similarity.

I added some technical info to my page at saito. Within the context of this CIP page, I have a node called Pipe that implements a similar functionality. It is based on threads, and takes care of data preparation between tasks. When you look at the screenshots, it looks like the old Yahoo Pipes. However, it predates it by almost a decade as I implemented the first version of it in mid 2000's.

The environment has a bunch of other features including creature comforts. For example, you can draw on an infinite canvas, create various shapes, images, and text. Which is actually useful for documenting what the DAG is doing or plan out your tasks in schematics as a guide.