Friday, June 12, 2015

PixInsight DSLR RAW Pre Processing - long hand

PixInsight was my first astro image processing program. I find it irreplaceable for preprocessing; that is, image reduction. Perhaps because I am now comfortable with its use. On first encounter, v1.5, it was overwhelming - there seemed to be no guidance about how to use the various tools, and it seemed that unless you were an expert in the field of image processing, with superior insight (no pun intended), you were likely to struggle.

Not being one to give up, only to find that PI was never really designed with DSLR users in mind - well it might have been, but it didn’t facilitate the process, at that time - after much discussion, scripts became available that smoothed the way for DSLR preprocessing tasks. There was however, no documented workflow and it was up to the user to work it out, with help from other users in the same boat.

This was never satisfactory, and having very little experience or know-how I attempted to write some sort of procedure. Unfortunately, this caused more problems than it solved. And, it was some time before the light went on. The traditional workflow and scaling algorithms used by PI are ideal and very flexible for linear data. But! DSLR RAW data is not linear and this caused problems, particularly data loss.

Unfortunately, experience differed and was inconsistent. And in the absence of a sensible time efficient means of verifying globally distributed individual data sets among amateurs, there was a lot of head scratching.

What follows is a very conservative approach, and whether Jim Solomon new it or not, and I suspect he did, the process described in his “Astrophotography Cookbook” for use with IRIS, is ideal for DSLR RAW data. Mainly, it is safe and logical and can be applied to any DSLR camera, from what I can determine.

It is a moot point whether dark scaling is appropriate for DSLR RAW data, and rather than address the complexity, the following workflow does not involve scaling, and makes every attempt to keep the data as intact as possible.

I can still hear the cries on other forums - …”the bias is in the dark, the bias is in the dark, the bia…” thanks H., and so it is, and for DSLR RAW data it is better to keep it that way until acquiring significant “insight,” to handle the data otherwise.

This is the way I do it. It is intended to improve the experience of newcomers to PI, rather than see them give up in despair. However, I cannot overemphasize that if this doesn’t work there is most likely a problem with acquisition of the frames - please read on - acquisition is an area among beginners that doesn’t get enough attention - the time spent getting acquisition right is time well spent and will avoid complications down stream.

Conventions - Menu toolbar selections; e.g., > View > Explorer Windows > Format Explorer > DSLR_RAW - this is first settings dialogue.

What follows is my method (long hand for illustration) of preprocessing DSLR_RAW data in PixInsight. It should be no problem to plug the same values into the BatchPreprocessing script, with the caveat that the master bias, dark and flat frames are created long hand. We can then load our master dark, master flat and light frames, tick the master dark and master flat boxes and press [ Run ]

Note: If this method doesn’t produce good results in the final image there is a problem with acquisition. Most likely darks and / or flats. This method has been tested over and again.

1. > View > Explorer Windows > Format Explorer > DSLR_RAW
As with all the windows set the parameters as shown

2. > Scripts > Batch Processing > BatchFormatConversion
Convert all files to 16 bit integer tiff

3. > Process > ImageIntegration
Integrate Bias and Dark frames to create Master Bias and Master Dark frames

4. > Process > ImageCalibration
Calibrate Flat frames with Master Bias

5. > Process > Image Integration
Integrate calibrated flat frames to create Master Flat

6. Caibrate Light Frames with Master Dark and Master Flat and debayer calibrated lights.
This shows light raw frames loaded for calibration - that’s OK, but it’s just as well to convert to tiff first.

7. > Process > StarAlignment
Register debayered images

8. Integrate light registered/aligned images
Here pixel rejection is set according to the size of the image set

The result is a PixInsight .xisf file. The output format can be changed in the BatchPreProcessing script.

In summary, we have simplified the preprocessing/calibration methodology by subtracting the bias from the flats only - note: we did not subtract the bias from the dark or light frames. Next we subtracted the master dark from the lights and divided the result by the master flat. Then we debayered, registered and integrated our calibrated/preprocessed light frames to produce an integrated image for post processing.

First we need a good set of frames properly acquired. Dodgy frames will mess very badly with DSLR RAW data.

Here is the BatchPreProcessing script set up once bias, dark and flat master frames have been created long hand.

Master Dark

Master Flat

LIght Frames debayer selections - again this shows RAW files - or convert to tiff beforehand

Light Frames registration selections

Light Frames integration selections

Light frames can be loaded RAW as shown. Alternatively, convert all files to 16bit .tiff

Saturday, October 25, 2014


To put things in perspective. After what seems a lifetime subscribing to scientific naturalism - evolution - and upon examining the arguments about origins, I conclude that the universe must (by way of reason and probability) be a work of intelligent creation - The Creator and His creation, Man, the epicentre of its purpose.

Further concluding, that all arguments about origins can be categorised as Creator, no Creator (God, no God), worldview - there being no middle ground. Speculation about natural origins is, as I eventually discovered, wishful thinking - a calculated diversion from reality. An “escape from reason.” There is, I am convinced, no sound argument to the contrary, however well constructed.

Friday, October 24, 2014


Thursday, May 1, 2014

Astrophotography DSLR RAW Data Calibration: preprocessing

This post has been rewritten substantially, and has its origins in an article by Craig Stark on DSLR RAW data non-linearity.

Corrections applied to DSLR RAW data before it leaves the camera, in particular, long exposures such as dark and light frames (irrespective of user selectable camera options), affects the linearity of the RAW files. As a result DSLR RAW data does not approach that of dedicated CCD/scientific cameras.

Differences show up during preprocessing, where the results obtained from DSLR data to which linear processing tecniques have been applied, may not be consistent with linear data. This often amounts to data loss, dissapointment, frustratingly poor results and hours spent unecessarily looking for solutions.

There are useful strategies to avoid data loss when working with DSLR RAW data, as there are one or two complications. As a rule DSLR RAW data reduction benefits from consistent temperature across the data set. In particular light and dark frames.

Temperature reduction overnight can be significant, and 5C is touted as an acceptable variation. Obviously, larger temperature variations require modification to image acquisition strategies.

Jim Solomon provides a recipe for DLSR RAW data acquisition, and without going into a lot of unnecessary detail, providing lights and darks can taken at a reasonably consistent temperature the following strategy is safe and effective.

I have tested this procedure in AstroArt, Nebulosity, Pixinsight, Regim, and a bash script that uses DCRAW and ImageMagick to perform the data reduction process. In every case the results were nearly identical. The script is available for download for Linux and Mac users, further down the page.

I hope this goes some way to a better experience. Advanced techniques such as data scaling is outside the scope of this blog. I recommend reading the blog on dithering… which is DSLR salvation, in my view.

Please note, the term flat dark is interchangeable with bias, where flats are taken at shutter speeds of less than a second or two, as opposed to longer flat exposures where dark current may be a consideration.

The formula that relates these physical phenomenon, and the actual frames we’ll collect over a night of imaging, are as follows:

(1) Light = (Signal * Flat Signal) + Dark + Offset

where Signal is the image of the target object we wish we could collect under ideal circumstances, and Light is the image we actually captured. Rearranging the terms, we have:

(2) Signal = Light - (Dark + Offset)/Flat Signal

But realize that the Flats we capture with the camera will, in turn, be “polluted” by Darks and Offsets in their own right, and so we must subtract Flat Darks and Flat Offsets from the Flat Lights as follows:

(3) Flat Signal = Flat Light - (Flat Dark + Flat Offset)

So, plugging equation (3) into equation (2), yields this general formula:

(4) Signal = Light - (Dark + Offset)/Flat Light - (Flat Dark + Flat Offset)

Here, “Dark” refers to the thermal noise signal of the imaging camera; i.e., the noise signal that varies in proportion to temperature, ISO, and exposure length. Note, however, that any exposure we take with a digital camera contains the Offset, and “Darks” are no exception. So, if we define Dark’ to be an exposure of some length with the body cap in place, then Dark’ = Dark + Offset, and, similarly, Flat Dark’ = Flat Dark + Offset. Plugging these values into Equation 4 yields the following simplified form:

(5) Signal = Light - Dark’/Flat Light - Flat Dark’

And just to make things even simpler, let’s drop the prime indicators (the apostrophes) that we stuck on “Dark” and “Flat Dark”, and just remember that by “Dark” and “Flat Dark” we mean frames captured with the body cap in place but with the same ISO and exposure length as the Lights and Flat Lights, respectively. That gives us our final form:

(6) Signal = Light - Dark/Flat Light - Flat Dark (bias) - editor’s comment

Equation 6 gives us our marching orders for astrophotography, providing us with a set of Frames that must be captured for each imaging session. The actual order in which I choose to capture these frames is as follows, the reasons for which will be made clear in the acquisition section below:

Flat Darks (bias) editor’s comment
Flat Lights

RAWPREPRO - DSLR RAW file conversion and data reduction/calibration script

RAWPREPRO is a Linux based DSLR RAW image conversion and data reduction/calibration script. It was originally intended to be nothing more than a means of converting DSLR RAW images to 16bit integer tiff (CFA monochrome, no flip/rotation) files for use in programs such as DSS.

RAWPREPRO has grown into a GUI based conversion and image calibration utility. RAWPREPRO uses DCRAW for conversion and ImageMagick to perform the image calibration/reduction/preprocessing steps - DCRAW and ImageMagick must be installed to take advantage of all preprocessing options.

Besides data conversion and calibration, RAWPREPRO orders the image preprocessing task through a folder structure, placing files in predictable places for easy access. It keeps things in one place, under a project name, renaming files accordingly. All DSLR RAW data files are preserved. Files can be added or removed, and the project run over and again, as required. Temporary files are deleted at the end of the script saving disk space.

1. RAWPREPRO uses dcraw to convert DSLR RAW files to 16bit integer CFA monochrome tiff files (no flip) for calibration in RAWPREPRO, itself, or an external preprocessing program - Regim, DSS and others.

2. RAWPREPRO uses ImageMagick to perform the mathematical functions for the creation of master frames, which may be imported into an external preprocessing program.

3.RAWPREPRO uses ImageMagick for the calibration/reduction of light frames, using the safe and effective, bias-in-the-dark method of data reduction.

Note: RAWPREPRO does not perform cosmetic correction or pixel rejection, keeping the converted data as intact as possible.

Installation and Operation - very quick start:
Following extraction of the zip files, rawprepro may be installed by clicking the script - this will also create a desktop link if desired - if not run rawprepro from command line by typing rawprepro at the prompt.

Alternatively, rawprepro may be run, without installation, within the extracted folder by clicking the rawprepro file or by command line, typing ./rawprepro at the prompt.

The older command line version will be updated for MacOS users. Presently, it is not as flexible as the Linux GUI version, but is reported to work in MacOS. It performs the same main functions as the GUI version, but requires the user to load files into the respective folders manually. A cocoadialog version would be nice - anyone?

rawprepro_1.8 - Fixes - This latest version has all the tweaks since 1.5. Dark frame selection may be deselected for cooled DSLRs (nominally 0C and below) and rawprepro will use the master bias frame in place of a master dark. This is not really recommended for cameras without cooling.

rawprepro 0.8208 - old version - command line -

RAWPREPRO Quick Start Guide - smaller download
RAWPREPRO Quick Start Guide - slightly bigger download - refresh PDF if displayed in browser.

Acquisition notes:

I borrowed this crop from here. I thought it was worth including. While it may be amp glow, first check that the view finder is not the culprit. This artifact looks suspiciously like light leaking through the viewfinder. Tape up the view finder before imaging.


Thursday, October 31, 2013

Welcome to FlatPress!

This is a sample entry, posted to show you some of the features of FlatPress.

The more tag allows you to create a “jump” between an excerpt and the complete article.

[Read More…]