Sunday, June 19, 2016
UPDATE:ICAPS_0.0.7.4 - Mean, Median and Enfuse stacking, plus additional user options are provided with this release.
INFO: Link to HFOV calculator - enter focal length and sensor crop factor.
NOTE: ICAPS, while stable, is a work in progress. Given time and inspiration, updates may be frequent, sometimes daily. As a rule, odd numbered versions are stable and even numbers beta. To that end, subversions indicate minor changes or improvements to the main release.
ICAPS is an end-to-end preprocessing utility for use with RAW image data and uses an array of image processing tools available to most Linux distributions.
Please read the documentation. The Manual is a QRH or quick reference handbook - quick start guide with some helpful hints. The help file provides more detail and the section on file naming conventions is important reading.
This project began as an inquiry. Given the programs available is it possible to create a complete preprocessing utility for a Linux system and does the astrophotography world really need yet another image preprocessing program - inquisitiveness won the day.
The image processing tools chosen for the task are well known among the image processing community. They are maintained and utilised in commerce and by hobbyists alike. ICAPS merely orchestrates the order and function, each step of the way.
DCRAW converts and demosaics the RAW images. ImageMagick performs all the calibration maths (math) and Panotools aligns the processed images in the same manner as a Hugin session. ImageMagick or Enfuse produce the image stack. Note that demosaicing occurs prior to calibration. This was a bit risky…
Demosaicing or deBayering is normally performed post calibration and just before image alignment in traditional workflows, where image calibration is performed on monochrome images with the color filter array RGGB in place.
Short of implementing a demosaicing routine, as in traditional workflows, the opportunity to split images into their R G B monochrome components and calibrate each channel independently was too attractive to pass up. Using the demosaicing capability of DCRAW meant demosaicing first thing. Post processing does not reveal any adverse effects.
Unlike its predecessor rawprepro, ICAPS expects as a minimum bias flat and light frames. Dark frames are optional, though recommended unless the need is redundant because your DSLR has a sub-zero cooling system.
Overall, it will be interesting to see how ICAPS shapes up, making no claims whatsoever as to its capabilities, it does the job… further development is a function of developing additional capabilities.
As its name suggests, ICAPS separates and processes individual R G and B colour channels and then recombines to a single aligned and stacked image ready for postprocessing, once calibration is complete.
Zenity or Yad (v0.0.6 onwards) provides the user with an interactive interface, while DCRAW, ImageMagick, Panotools and Enfuse form the processing tool set. Hugin and MacroFusion are optional graphical interfaces for performing complex alignment and stacking tasks.
Except for Yad or Zenity, which should be installed prior to using ICAPS, ICAPS will prompt for the installation of other programs, should one not be installed.
ICAPS adopts much of the RAWPREPRO workflow. Files are arranged in folders according to their type, bias dark, flat and light frames. Calling on the main processing programs, ICAPS orders the workflow from beginning to end, step-by-step.
ICAPS expects a dithered set of light frames and alignment is superb, providing frames are not displaced or rotated too much.
ICAPS works in 16bit. Memory use is limited to approx 40% RAM. All but 1 processor core is used to avoid robbing the system of resources. Processing large sets of full frame images can take some time - an hour or so. This is the current behaviour which can be modified in the script under the ImageMagick resource section.
ICAPS presently has an image limit of 100 raw files per frame type; that is, 100 bias 100 dark, 100 flat and 100 light frames. This interim policy is under review as it may not suit alt/az users who typically takes hundreds of exposures.
Here is a sample 22 frame stacked using Hugin in ICAPS. The file has not been processed except for gamma 2.2 and sigmoidal contrast 3×50 in ImageMagick display.
ICAPS is free and licensed under GPLv3
ICAPS - v 0.0.7.4 ~ mean and median and enfuse stacking plus additional control options. It is stable but remains a subversion.
ICAPS - v 0.0.5 - stable - zenity interface
ICAPS manual - stable old
ICAPS Hugin Video
Version History and Updates:
ICAPS_0.0.7.4 - bug fixes
ICAPS_0.0.7.3 - fixed image restriction for stacking. Added stack control options. Updated manual v5.
ICAPS_0.0.7.2 stable - added mean and median stacking
ICAPS_0.0.7.1 stable - improvements in exposure fusion stacking
ICAPSMANUALv4 - for 0.0.7
ICAPS_0.0.7 stable - acquisition details may be added - optional. Changed handling of log files and system align and stack.
ICAPSMANUALv3 - for 0.0.6.4_beta
ICAPS_0.0.6.4_beta - consolidation of splash screen and log file options - explicit run icaps check box - log file handling
ICAPSMANUAL2 - for 0.0.6.3_beta
ICAPS_0.0.6.3_beta - addition of use log file option - to run the previous image set again with the same options
ICAPS_0.0.6.2_beta - help document amendments - run this beta version from folder
ICAPS_0.0.6.1_beta - fixes to system align and stack yad interface consolidation of entry dialogs - run this beta version from folder
ICAPS video update
version 0.0.5 - major fix to run from folder - documentation edits and improved file handling
ICAPS manual available for download
Version 0.0.4 - change to output file handling, help and info text files and normalize flats option fix.
Version 0.0.3 - On occasions flats may not be of ideal quality and this shows up in the final image as uneven illumination or dust mote shadows. A work around is to turn normalization OFF - default is ON.
Note: While this is often successful with poor quality flats it also increases the brightness of the final image.
Version 0.0.2 - fixed issue with Enfuse deleting the output.tif when duplicated from a previous session. output.tif is now saved in zip file. ICAPS will now display stack result on subsequent runs.
Tuesday, May 3, 2016
I have removed the hurriedly composed post about DSLR thermoelectric cooling and heatsink selection. It was long winded and lacked the detail the subject deserves. However, in the interests of safety to both operator and camera gear there are some basic guidelines for design and purchase of 3rd party units.
1. Proper selection of components.
The energy dissipation requirements of a thermoelectric cooling system are straight forward and basic rules of thumb can be applied. Particularly in relation to heatsink selection.
Heatsinks should be capable of dissipating twice the maximum power that will be applied to the TEC device, for safety and efficiency.
Note: for each watt of electrical energy conveyed through the system, a watt of heat energy will be transferred from the heat load to be dissipated by the heatsink.
Power supply, W = I*V
Heatsink power dissipation = W*2
If the selected TEC device has a rating of say 12V and 5A it will operate efficiently and safely with a heatsink capable of dissipating approximately 100 to 120 watts of energy. It will also act as a safety valve should the fan fail and not get too hot.
The thermal resistance value of the heatsink will be in the region of 0.3 to 0.2C/W. Note that many eBay heatsinks advertised for LED cooling quote dissipation values in watts. Heatsink manufacturers usually in degreesC/Watt.
2. Condensation protection - if the system you are designing or intend buying from a 3rd party does not have condensation protection applied internally or you haven’t thought about applying protection in critical places ,the camera will likely suffer damage due to corrosion.
There are many other considerations but for now these are the fundamental safety and performance considerations that will keep you out of trouble.
Friday, June 12, 2015
Pixinsight is a fabulous program and its preprocessing tools are superb. However, when first using PixInsight v1.6, DSLR preprocessing was not easily facilitated. Following several forum discussions scripts became available smoothing the way for DSLR preprocessing tasks. There was no documented workflow and what follows is a summary of the now DSLR RAW preprocessing thread - the alternate method, which works very well with DSLR RAW data.
The following is a copy and paste from the PI thread.
This thread is a description of an alternate method of DSLR RAW image reduction/preprocessing. It is partly intended for newcomers to astrophotography, and PI, who are using DSLR cameras, and hopefully it provides a degree of flexibility when using PIs excellent image calibration tools.
The method described here is a simpler option and is not intended to replace the standard PI dark optimization/scaling method of image calibration.
This guide assumes that DSLR RAW data is non-linear compared to dedicated astronomical imaging devices, for the following reasons; DSLR sensor data is modified by camera firmware to produce a photo quality image - this affects the linearity of DSLR RAW data (downloaded from the camera) - it is assumed that the various makes and models differ as to the degree of firmware modification.
“sRGB Colorspace Correction
Saving an image using a sRGB Colorspace is very similar to Gamma correcting an image, but is slightly more complicated so as to better reproduce the actual response of the human eye, specifically with very dark shades of color.” (Magick).
DSLR cameras usually have two colour format options, sRGB and AdobeRGB. sRGB is a standard format used in many applications. AdobeRGB increases the colour gamut to include more cyan and green hues. As a rule, sRGB is used for astrophotography.
DSLR cameras internally process the linear data obtained from the camera’s sensor (CMOS or CCD) to produce an image which is clean and within the response of the human eye. The RAW image from a DSLR camera is a modified version of what the sensor ’saw’. As such the image produced is just a rendition, rather than a faithful representation of the sensor (linear) data - which is to say, DSLR RAW data is not linear.
Despite the non-linearity of DSLR RAW data PI handles it very well, even when linear processing methods are applied.
A data set usually comprises, bias frames, dark frames, flat frames and light frames. However, because DSLR cameras are generally not cooled and because DSLR RAW data is not linear, poor acquisition technique almost always results in data which is not representative of the other data in the set - and invariably leads to preprocessing problems.
The alternate work flow is partly intended to offer an option which may assist producing a reasonable image - though, it is primarily intended to overcome the issues of DSLR RAW data non-linearity, and to some extent poor data acquisition technique.
It is very desirable to match dark and light frames in particular, with the following caveats.
For the alternate work flow described here, it is best to match dark and light frames to ISO, temperature and exposure time. If using the PI standard linear work flow, which involves dark optimization/scaling, then light frames and dark frames are matched to ISO. However, because dark noise is a function of exposure time and temperature, it is possible to match dark noise to light frames by changing one variable or the other. The dark optimization/scaling method is explained here, and is not discussed further.
In practice an alternate method of preprocessing DSLR RAW data in Pixinsight does not leave the user without options where one or the other is not producing the desired results. There are of course no silver bullets. Good acquisition techniques and learning the art of preprocessing PI style can produce excellent results with DSLR RAW data, despite its vagaries.
It cannot be overemphasized, that if the following workflow is not successful there is most likely a problem with acquisition of the frames. Acquisition is an aspect that doesn’t get enough attention - time spent getting acquisition right is time well spent and will avoid complications down stream.
Conventions - Menu toolbar selections; e.g., > View > Explorer Windows > Format Explorer > DSLR_RAW - this is first settings dialogue.
What follows is my method (long hand for illustration) of preprocessing DSLR_RAW data in PixInsight. It should be no problem to plug the same values into the BatchPreprocessing script, with the caveat that the master bias, dark and flat frames are created long hand. We can then load our master dark, master flat and light frames, tick the master dark and master flat boxes and press [ Run ]
Bottom line - the bias frame is not subtracted from the dark or light - only the flats are bias subtracted.
Note: If this method doesn’t produce good results in the final image, first check the quality of darks and / or flats. This method has been tested over and again.
6. Calibrate Light Frames with Master Dark and Master Flat and deBayer calibrated lights.
This shows light raw frames loaded for calibration - that’s OK, but it’s just as well to convert to tiff first.
Note: the optimize dark box is now NOT TICKED - this is NOT default. It is recommended to clear this box. However, try it both ways if you have time and compare the results.
Note: Cosmetic Correction may be applied at this point.
The result is a PixInsight .xisf file. The output format can be changed in the BatchPreProcessing script.
In summary, the DSLR RAW calibration has been simplified by subtracting the bias from the flats only - note: the bias is not subtracted from the dark or light frames. Next the master dark is subtracted from the lights and the result divided by the master flat. The calibrated/reduced light frames are debayered, registered and integrated to produce an integrated image for post processing.
A good set of frames properly acquired is essential. Dodgy frames will mess very badly with DSLR RAW data.
Here is the BatchPreProcessing script set up once bias, dark and flat master frames have been created long hand.
Light frames can be loaded RAW as shown. Alternatively, convert all files to 16bit .tiff
Thursday, May 1, 2014
This post has been rewritten substantially, and has its origins in an article by Craig Stark on DSLR RAW data non-linearity.
ICAPS is an extended development of RAWPREPRO and provides deBayer and Stacking. It provides end-to-end DSLR RAW processing and includes deBayer and stacking to produce an image ready for post processing.
Latest Beta rawprepro_2.16 - extends checkbox selection to all frames and sets a standard width for all progress, question and information windows.
Corrections applied to DSLR RAW data, before it leaves the camera, in particular, long exposures such as dark and light frames (irrespective of user selectable camera options), affects the linearity of the RAW files. As a result DSLR RAW data does not approach that of dedicated CCD/scientific cameras.
Differences show up during preprocessing, where the results obtained from DSLR RAW data, to which linear processing techniques have been applied, may not be consistent. This often amounts to data loss, and consequently, disappointment, frustratingly poor results and hours spent unnecessarily looking for solutions.
There are useful strategies to avoid data loss when working with DSLR RAW data, as there are one or two complications. As a rule DSLR RAW data reduction benefits from consistent temperature across the data set. In particular light and dark frames.
Temperature reduction overnight can be significant, and 5C is touted as an acceptable variation. Obviously, larger temperature variations require modification to image acquisition strategies. Dark scaling is not discussed here.
Jim Solomon provides a recipe for DLSR RAW data acquisition, and without going into a lot of unnecessary detail, providing lights and darks are taken at a reasonably consistent temperature the following strategy is safe and effective.
I have tested this procedure in AstroArt, Nebulosity, Pixinsight, Regim, and a bash script that uses DCRAW and ImageMagick to perform the data reduction process. In every case the results were nearly identical. The script is available for download for Linux and Mac users, further down the page.
I recommend reading the blog on dithering… which is DSLR salvation, in my view.
Please note, the term flat dark is interchangeable with bias, where flats are taken at shutter speeds of less than a second or two, as opposed to longer flat exposures where dark current may be a consideration.
The formula that relates these physical phenomenon, and the actual frames we’ll collect over a night of imaging, are as follows:
(1) Light = (Signal * Flat Signal) + Dark + Offset
where Signal is the image of the target object we wish we could collect under ideal circumstances, and Light is the image we actually captured. Rearranging the terms, we have:
(2) Signal = Light - (Dark + Offset)/Flat Signal
But realize that the Flats we capture with the camera will, in turn, be “polluted” by Darks and Offsets in their own right, and so we must subtract Flat Darks and Flat Offsets from the Flat Lights as follows:
(3) Flat Signal = Flat Light - (Flat Dark + Flat Offset)
So, plugging equation (3) into equation (2), yields this general formula:
(4) Signal = Light - (Dark + Offset)/Flat Light - (Flat Dark + Flat Offset)
Here, “Dark” refers to the thermal noise signal of the imaging camera; i.e., the noise signal that varies in proportion to temperature, ISO, and exposure length. Note, however, that any exposure we take with a digital camera contains the Offset, and “Darks” are no exception. So, if we define Dark’ to be an exposure of some length with the body cap in place, then Dark’ = Dark + Offset, and, similarly, Flat Dark’ = Flat Dark + Offset. Plugging these values into Equation 4 yields the following simplified form:
(5) Signal = Light - Dark’/Flat Light - Flat Dark’
And just to make things even simpler, let’s drop the prime indicators (the apostrophes) that we stuck on “Dark” and “Flat Dark”, and just remember that by “Dark” and “Flat Dark” we mean frames captured with the body cap in place but with the same ISO and exposure length as the Lights and Flat Lights, respectively. That gives us our final form:
(6) Signal = Light - Dark/Flat Light - Flat Dark (bias) - editor’s comment
Equation 6 gives us our marching orders for astrophotography, providing us with a set of Frames that must be captured for each imaging session. The actual order in which I choose to capture these frames is as follows, the reasons for which will be made clear in the acquisition section below:
Flat Darks (bias) editor’s comment
RAWPREPRO - DSLR RAW file conversion and data reduction/calibration script
RAWPREPRO is a Linux based DSLR RAW image conversion and data reduction/calibration script. It was originally intended to be nothing more than a means of converting DSLR RAW images to 16bit integer tiff (CFA monochrome, no flip/rotation) files for use in programs such as DSS.
RAWPREPRO has grown into a GUI based conversion and image calibration utility and uses DCRAW for conversion and ImageMagick to perform the image calibration/reduction/preprocessing steps - DCRAW and ImageMagick must be installed to take advantage of all preprocessing options.
Besides data conversion and calibration, RAWPREPRO orders the image preprocessing task through a folder structure, placing files in predictable places for easy access. It keeps things in one place, under a project name, renaming files accordingly. All DSLR RAW data files are preserved. Files can be added or removed, and the project run over and again, as required. Temporary files are deleted at the end of the script saving disk space.
The 3 main functions are as follows;
1. RAWPREPRO uses dcraw to convert DSLR RAW files to 16bit integer CFA monochrome tiff files (no flip) for calibration in RAWPREPRO, itself, or an external preprocessing program - Regim, DSS and others.
2. RAWPREPRO uses ImageMagick to perform the mathematical functions for the creation of master frames, which may be imported into an external preprocessing program.
3.RAWPREPRO uses ImageMagick for the calibration/reduction of light frames, using the safe and effective, bias-in-the-dark method of data reduction.
Note: RAWPREPRO does not perform cosmetic correction or pixel rejection.
Installation and Operation - very quick start:
Following extraction of the zip files, rawprepro may be installed by clicking the INSTALL.sh script - this will also create a desktop link if desired - if not run rawprepro from command line by typing rawprepro at the prompt.
Alternatively, rawprepro may be run, without installation, within the extracted folder by clicking the rawprepro file or by command line, typing ./rawprepro at the prompt.
The older command line has been updated, as of rawprepro_cl_0.9 for MacOS and Linux users. Presently, it is not as flexible as the Linux GUI version, but is reported to work. It performs the same main functions as the GUI version, but requires the user to load files into the respective folders manually. A cocoadialog version would be nice - anyone?
rawprepro_2.9 - no -auto-level
Command Line Version
rawprepro_cl_1.5 - auto-level
rawprepro_cl_1.4 - no -auto-level
I borrowed this crop from here. Thought it was worth including. While it may be amp glow, first check that the view finder is not the culprit. This artefact looks suspiciously like light leaking through the viewfinder. Tape up the view finder before imaging.