Raw to Ready In 0 Clicks?! - The Power of WBPP Custom Steps (Free PixInsight Script)

taking WBP to the next level using the pipeline builder script This is part two of getting the most out of Pix Insight using these custom scripts If you haven't seen part one I'll link that up here That was the event script And I think it is a very good idea to start with that one first because this one is a little bit more complex and I think it's a good idea that we have a good understanding of how the event script works first At the end of the event script video I mentioned what if I told you that the magic of scripting WBPP doesn't actually stop with the event script It only starts there and then we can do so much more if we get to the pipeline builder script It is even more powerful and even more customizable So today we're going to have a look at how the pipeline builder script works how to set it up what we can do with it and in the second half of the video something extremely cool Let's get into it You're watching the Space Quala channel and my name is Luta With the event scripts we have seen that WBPP already does so much for us and we can get all this information out of it every time there is an event such as a registration starting integration ending all of those things all the steps starting and ending and their respective information frame information all the things that we have seen for the event script And then based on those events we could actually trigger our own operations and build cool things like we have done with the telegram notifications But what if instead of doing something because WBP has sent me an event update I want to do kind of the opposite I want to be the one that tells WBP what to do This is exactly what we can do with the pipeline builder Whereas last time we were focusing on the before and after operation with the yellow boxes today we're going to focus on the blue part which is essentially the specific operations that WBPP does So things like the calibration step things like registration step What can we possibly do with this I mean it already does everything Well once again we have full access to pre and post groups and anything else that we have seen before But the absolutely cool thing is we can actually add our own custom operations to it And you don't have to imagine it in a way that we're hacking the system and we're adding stuff to it Uh you know just for the heck of it because we can This is entirely supported by WBP It is set up in a way that whatever we set up as a custom operation can then become part of the integral pipeline for all intents and purposes What what does it mean We can set up the name that should show up um in the whole pipeline We can decide where it should be done between which two steps For example we can decide where it should be performed We can of course tell it what to do actually And these events can even be trackable via the event script with what we have seen in the event script So our custom operations can do and do do all the things that the native operations do Only downside of it is being so powerful Quite a bit of coding is actually required to do it Or is it Stick around to find out What could we possibly do with our own custom operations I mean WBP already does everything for us The limit is only our imagination here But here are some examples that I thought of We could for example do a little bit of quick autoprocessing just so by the time you get back to the computer you would actually have your color calibrated image for example That is one thing you could do but you could also do your full processing along with stretching and curves and all that just so you can have an idea of what the image will look like You could also do background extraction Of course Pixight offers so many ways to do background extraction but one thing it doesn't offer is doing a background extraction on the single sub Something that I know that some users like to use in serial for example So you could take your calibrated images and as long as you have enough signal to noise ratio you could argue that the gradients are so much simpler on the single sub So you might want to do background extraction there You could do downsampling in your images and this is actually my use case and this is the very reason that I started looking into all of this You could also do multi-format output I mean XISF is amazing and very and it contains all the information you could ever know and more But some external systems just don't accept it So what you could do is keep your nice pixight format but then you also export it as a TIFF or as a Fitz file which is then accepted in your further systems and you don't have to spend time doing this manual export If you are very particular about how you like to filter your frames you could actually do that after your measurements and you could also gather your own statistics just to understand where it goes wrong if it goes wrong where you're losing images and whatever you want really This is a screenshot of my WBPP and at a first glance it looks quite normal But if you take a closer look there are some steps here that you most likely don't have on your own computer These are my own custom steps that I have defined and these are now becoming an integral part of the pipeline from every point of view And today we're going to have a look at how we can do just this In this pipeline tab right under the field for the event script we have another script field called the pipeline builder and it takes once again another JavaScript file That JavaScript file is going to be the one that tells WBP how to build up the entire pipeline and this is where you can provide your custom steps if you will So what do we need to do if we want our custom operation First we have to define our operation and then we have to place it into the pipeline So today we're going to be working through one example here and this will be the down sampler because as I mentioned before this was my actual use case I know that a lot of people will ask why on earth I do it It's because I use a 61 62 megapixel full-frame camera Um many times I don't have the scenes to support that Many times I don't I'm not able to sample correctly but also I don't really need uh such a large image to look at it on my screen So I used to do this every time I would start WBPP wait for it to calibrate I would interrupt it I would downsample the files I would load them back into WPPP and start another session to finish the whole thing This is no longer necessary because it does the whole thing for me So today we're going to work through how this was implemented A word of warning we are going to be looking at quite a bit of code again So if you're not interested in that skip to the next chapter in the video But for those of you who are interested in it let's go through it step by step and let's see how we can define our custom downsampling operation The first thing we're going to do is we're going to declare a new step or down sampler We're going to say that this is based on a BPP operation And we're just going to say that this is a down sampler And since our function here takes two arguments one is our frame frame group and the second one is sampling factor We're just going to say that our operation is called down sample And then we will be executing it on the frame group And we will say that this is trackable What does trackable mean We already know what it means It means that our operation is actually accessible externally via the event script So if you do the telegram thing you could even get an update of how your down sampling went Next we are going to have to define a couple of helper functions to have the full functionality within WBPP The first one being calculating how much size is required by our operation on the hard disk This is then used by WBP You know when you click on start it tells you a warning of the total amount of space that will be be required and uh it will warn you if you don't have enough space Um for most operations realistically the size before and after will be the same but downsampling is a particular one So I this is why I want to mention this because by definition we're changing the size of the frame So the way we're going to calculate it is we're going to check the size before and we're just going to divide it by our sampling factor twice in a row So if we're downsampling by two we're going to divide it by two And then we're going to divide it by two again And this is how we get the correct new size Next we define the helper function that will be feeding our information into the event script if we choose to use it This will be most likely a copy paste one for every single operation you do You just change the name really And then everything else is what is being spit out by the engine And now we get to the exciting part the actual downsampling function We're going to start by saying that our active frames is going to be the active frames of the current frame group that we have passed into this function We're going to do some sanity checks before we go any further such as do we even have any frames in this group If not then obviously we're done for the day but otherwise we go further We define our output directory And this is a good example of how much help WBPP is providing for you to do this in an easy way because I think we're all familiar with the whole smart naming system that they do Um you know for example when you have different frame groups or different targets or different filters even it automatically creates the folders for you for each of those right we can take full advantage of that and do the same thing for our own custom operation I have created a subdirectory called resampled and within we are just letting WBP generate the folder structure for us depending on the groups that we have Here we finally get to the actual downsampler where we are just calling an instance of the integer resampling process from pixight You can call any process that you wish and we just have to provide the actual parameters that the process needs So in this case we need a zoom factor because we have to know by how much we're resampling here It will be the input number factor So two multiplied by minus one because we're down sampling not upsampling Here I've also specified the downsampling mode We want this to be average but you can choose And very importantly if your process has a parameter such as no guey messages or generate graphs you want to make sure that you do it in a way that doesn't create like a popup message I'm sure we've all used a process before that does this When you do a dynamic crop for example it will come up with a pop-up message saying a warning Look if you do this you're going to lose your plate solution data Um things like this which is fine when you're on the user interface but here we're trying to execute this on 200 frames potentially We really don't want that to be popping up It's going to halt the execution So we just define that we don't want that And then that's it for the actual resampling process Now let's gather some statistics We're going to initialize a few variables to count the number of successful versus um failed images And then since we have our active frames we can start going over them looping over them one by one So we get the first image We retrieve the file path We actually open this up into Pixight We check if the opening of this image was actually successful If it failed we're done with this frame We add one to the number of failed images We go on to the next one But if we did manage to open it working with these XISFS you may have seen that sometimes you open a single file but it opens up like two or three images A good example of this would be when you have a an integration file when you have a master file and it comes with a rejection map for example or the drizzle map Clearly we're not interested in those extra files We're only interested in our main window And that's what we're going to do here We're going to close every single window except the first one that we keep Now even this window could have multiple views So we're just going to say we're interested in the main view not any like previews or anything like that And we're just going to define the name of our output file that we're going to have We're actually just going to append the word resampled by the factor to the end of the file name And then we're simply going to execute our downsampler Now if our process has an output file we can conclude that it was successful So we actually try to save it We do the saving We check did it actually work If so then we count this as a successful frame And this is one of the most important lines here in my opinion We're going to call the processing succeeded method on our original input frame And we're going to tell it the processing calibration was successful And this is the output file Why do we put these specific parameters in there That is because I aim to execute the downsampling after the calibration step Right So if then my next step is registration then by default Pix inside would go back to the original calibrated frame because it wouldn't know that I've done a new version since then So this is the method that's going to tell the system that if you're looking for the calibrated file get this file So this forces it to get our new resampled file when it's doing the next steps And this is the whole point of what we're doing here It's embedding it in the native pipeline Of course if the file saving didn't work we're just going to count this as a failure And we're also going to tell WBB take this frame out of the active ones cuz it just didn't work Next we do a little bit of a cleanup We close the window we have just opened And we save this information to send it out to the event script later should it ask for an update Now we will be creating the actual block So before we declare the step now we will be creating the actual block the BPP operation block And all we have to do now is just decide where to put it in our pipeline This is our pipeline It could be a bit overwhelming but don't be afraid because most of it is not what we have done because most of it is the operations that we're all familiar with So we have the calibration linear pattern subtraction all of those and it's only the orange part that is actually the part where I've inserted the down sampler So this is obviously not the entire code of WBP This is just the pipeline where it tells the system now you go calibrate now you go register and so on So this is exactly what we have to do as well Say now you go down sample How do we do it We navigate to the correct parts of the pipeline We say okay we're going to start our custom step here And this part I think is quite a cool thing because if you remember the way we have defined the down sampler is that it was taking two parameters One was the frame group the other one was the sampling factor If I don't want to update the code every single time with my sampling factor I shouldn't have to touch this script file every single time And this is exactly what I've done Let's have a look at how So we have done the pre-calibration step We're going to loop through the groups We're going to retrieve the file items And then we're going to retrieve the grouping keywords that we have on this group Because what I've done is I have defined a custom grouping keyword RSF that stands for resampling factor which I can actually put in the names of my folders uh in the regular like the regular way that we do the grouping keywords in WBP So if I have something like RSF-2 it would automatically know that I want to resample these images postc calibration by a factor of two The way this is done is we are actually provided a method to just get out the keyword and value automatically So that's what we're going to do We have a default value of minus one Then we try to get the value out of the current group If we find a value and it is valid and so we have a factor that is not minus one that means we expect that we have to down sample So that's exactly what we're going to do We're going to initialize the the workspace the operation size counter We're going to start counting how much space we actually need Remember that we did define a helper function earlier So that's what we're going to call now with our actual frames because now we don't just know how to calculate it We have the actual frames to calculate it on So we create the operation we calculate the size and we add it to the queue and that's actually it Obviously it is extremely powerful but admittedly quite complex and maybe a lot of us don't have the time or the patience or the coding knowledge to to implement this So I was thinking this is such a powerful tool It would be so amazing if everyone could do it but without spending hours and hours or days on doing this code So how could I do it And I have come up with a way and therefore the no code pipeline builder was born which is the pipeline builder for the rest of us How does this work It works exactly the same way as we have done before but I have needed a way for the user to interact with WBP without having to write all that code But obviously we need to customize it because not everybody wants to do the same exact operation at the same exact place So I needed a handy way where we can provide a list of operations we want to do And turns out Pixonite already has a built-in tool that does just this And this is process containers So what we're going to have is a script that is provided externally in this case by myself And then we're going to have a process container into which you can simply drag and drop the operations you want to do The script provides for a number of predefined entry points These are entry points which to me made sense It even has support for your custom grouping keywords if you only want to do an operation on a certain subset of images and all of this with zero lines of code So let's have a look at how this works What you see here is a process container which currently is empty And this is what's going to be telling WBP which are the operations to apply when to apply them and on which groups Now which operations to apply is very simple because you just drag and drop the processes in there and everything else you can just provide in the description field How do we do it Here we have a description field that says step on calibration done Cam 6200 mm Once again this is my use case because I want to do the down sampling after calibration on the pictures that were taken with my full-frame camera But in this case CAM is just a custom keyword that I have provided into WBP So what are these predefined entry points Um this could change in the future but so far I have implemented these because I thought they made the most sense to me at least And we can see here which groups they're implemented on and in which case they're executed So for example things like on calibration end on registration start on registration end things that are linked to a specific native step of WBPP These will only be executed if that step is active Clearly it makes no sense to do something on registration end if we didn't actually register anything There are a couple of more generic entry points which are on pre-process end on postprocess start and on postprocess end These are always executed and special attention to the last one on postprocess end Since this is essentially at the very end of the entire pipeline it means that it makes no sense to do anything on the individual frames anymore So this is going to be executed on the on the master frames that we have generated We do have some naming rules to make sure that it works correctly The main process container that we use to define the steps must be called WBP When I say that the process container must be called that we create a process container create an a process icon out of it and we name that WBPP because this is what we can access from the code Then when a key such as step is present on the process container that will be implemented on all the processes or levels below it We have this hierarchy rule However if we redeclare the step on an individual process that will override the level above If you're not comfortable with these hierarchies and inheritance I think that the simplest thing to do is on every single process that you want to do you just step you just specify which step you want to do do it If you do that it is impossible that it goes wrong How does it work in practice We have a tree structure of a process container on the left here The pro process container is called WBP We have a process directly within it which is our down sampler that we have done before and on this one I have said that the step should be on calibration end Then we have a nested process container within the main process container because we can actually have nested process containers You can put them inside one another as many as you want So we have created a process container underneath and we have put description on that We have put the key on postprocess end So what's going to happen here is since I didn't define the step anywhere below that all those steps that I put within here are going to inherit that property So all of those processes are going to be executed on postprocess and as we already know it that means it's going to be done on the master Here is just a different way that accomplishes the same exact thing If I put unpost process n on the top level normally everything should inherit that value But on the integer sample I am overwriting it So the actual result is still exactly the same So every process will be applied at a step which has the corresponding step value that we have defined If we provide for further filtering criteria such as groups it will only apply on groups that do have those keywords And if we are doing something on the masters we're going to do that on every single variant of the master So if you have a regular and a drizzle and an autocrop then we would do it on all of those Now here we have an example the same thing as before but we have decided that we will define an additional filtering criterion on blur exterminator because we only want to do it on the frames that have the keyword camera equals 183 mm So everything will be the same as before with the exception that blur exterminator is only going to be done on those frames This is a lot to go through but let's have a look at how this works in practice I have once again recorded my screen and of course I have sped it up because otherwise it would take forever Let's have a look at how the no code pipeline builder works in practice We have our WBP process container here It's similar to the example we saw before So we have the integer sample with the step on pre-process end And let me just draw your attention to the fact that once again I have specified I do not want the GUI messages because we know that the pop-up is going to halt the operation Then we have a nested process container within here And on this process container I have defined step equals on postprocess end So it means that all the steps that come after this blur exterminator noise exterminator calibration all of those are going to be done on post-process and we know that means that it's going to be done on the masters Let's have a look at spectral photometry color calibration I have set all those values to make sure that it doesn't generate those popup for me I'm going to do an automatic background extractor at this point Also in this one I have customized it so that it's actually going to be replacing the target so that it doesn't generate a new image Then I've done a pixel mask that is just an auto stretch And finally some curves transformation that is applying a little bit of contrast and some saturation So we're ready to run this Let's go into WBP These are the lights that we're going to work on We have 36 frames We have astrometric solution active because of course that will be necessary for the color calibration We have enabled fast integration in this case And here we have all the steps You can see that we have all the custom steps that we have defined on the process container They have already been added automatically here just because I'm using the pipeline builder So all that's left is to start it And obviously this is being very sped up So this is very quick But we've already done the calibration the debearing We have already done our first custom operation after all that So we have our resampled files And now the system is working on those resampled files We are doing the fast integration And as soon as that is done we're ready to do the plate solution of the master frames and we start doing our custom steps with blur exterminator noise exterminator color calibration and all the other stuff that we have done And we're done So let's close WBP and then let's see what actually happened So I'm going to go into my file browser This is the working directory So see that the debayer files were 300 megabytes on pre-process end which is what was generated after our resampling They're much smaller So we know that this worked Then of course we have our master file We're just going to open that up and see what it looks like So we have a stack as it came out normally of WBP Of course we're not going to overwrite that But we also have our own master file that has been processed So let's have a look at that Check out that I don't even need to auto stretch this This is exactly the image that came out of WBPP with the custom script with our custom steps I think this image is about 90% there to what it would look like if I processed it myself Now I personally don't process my images in an automated way I don't have a fixed workflow but I still think that this is useful because it gives you a super good idea of what it could look like And then of course if you actually tune the parameters of the things that you do and according to the images needs you will get a better result But it already gives you a very realistic idea of how much is in there in that image And I also know for a fact that a lot of you like to automate stuff and you like to process many many images And I think that for for those people this will be a really convenient way to just do it It's repeatable every time and you get out an image that is essentially done without you having to click on anything So what do you need to do to actually get started Similarly to the event script the no code pipeline builder is available on my website the spacequala.com This is completely free and open So if you go on the website there is a section called Pix Insight and there you will find the documentation which will tell you essentially everything that I've covered in this video um of how to set it up how the hierarchies work how the grouping works and then you have the URI of my Pixight repository So you would just copy paste that text put that into your Pix Insight repositories list run um schedule an update get the files Now very important while this and the event script are technically speaking scripts they will not show up in the scripts menu because they cannot be run standalone like these scripts only make sense in the context of WBPP So you run the update you still get the file It will be in your Pix Inside scripts folder So you would just open WBP click on the browse feature and just navigate to the scripts folder find the one that is called no code pipeline builder.js load it in there create your process container and you're good to go It's going to work Now while in my opinion this is actually a super convenient way to do very very powerful things it is just one thing that you can do with the pipeline builder out of a million that you could So if you feel up for it I encourage you to write your own code and share it with us so that the community scripts can grow When you do so some of these Gen AI tools such as Chad GPT or Gemini can be super useful However there is a very important word of warning here Our Pixon site license does not allow us to just take that code and send it to their servers and just you know essentially publish it for them But it doesn't mean that we cannot use these tools What what this means is that we shouldn't use the intellectual property of pixight to improve the model of Ted GPT But there's a very very simple solution because in all of these tools if you actually go into the settings for example this is JGPT here you can turn off improve model for others So what this will do is it will still do everything you ask it to do but it will not be storing this code and the information it gets out of it on their actual model So this is only available to the specific user in this case you if you are the one uploading it So I highly recommend that everyone do that if you're going to do any of this stuff because first of all we don't want to breach the terms and conditions of the license but also it is in all of our interests that the code remains open so that we can actually do this because if people abuse the system they might just decide to lock down everything and we we will not have access to all the code So it will become so much more difficult for all of us to develop new scripts I hope at this point you're all convinced that this is the absolute epitome of WBP and many of you will try this out If you do let me know how it went or if you run into issues Also let me know Down in the comments below this video is most likely the best place to do it because if other people run into the same issue they can hopefully find their answers there This is a first version of this piece of code If you have suggestions on how to make it better uh please do let me know because I'm very excited to be working on this and being able to share it with the community So thank you very much for watching If you have enjoyed this uh hit the subscribe button so that you don't miss the updates that I plan to do on this and some other things Happy pre-processing and I wish you clear skies

2025-04-26 11:45

Other news