Thursday, May 1, 2014

Astrophotography DSLR RAW Data Calibration: preprocessing

This post has been rewritten substantially, and has its origins in an article by Craig Stark on DSLR RAW data non-linearity.

V2.7 and CL_1.2 are the current rawprepro script versions - recommended upgrade.

Please note: There has been a lot of updates to the software package further down the page. If you have used/are using the utility it has seen a lot of refinement and readers may wish to check occasionally. The latest version is mainly cosmetic fixes.

Corrections applied to DSLR RAW data, before it leaves the camera, in particular, long exposures such as dark and light frames (irrespective of user selectable camera options), affects the linearity of the RAW files. As a result DSLR RAW data does not approach that of dedicated CCD/scientific cameras.

Differences show up during preprocessing, where the results obtained from DSLR RAW data, to which linear processing tecniques have been applied, may not be consistent. This often amounts to data loss, and consequently, dissapointment, frustratingly poor results and hours spent unecessarily looking for solutions.

There are useful strategies to avoid data loss when working with DSLR RAW data, as there are one or two complications. As a rule DSLR RAW data reduction benefits from consistent temperature across the data set. In particular light and dark frames.

Temperature reduction overnight can be significant, and 5C is touted as an acceptable variation. Obviously, larger temperature variations require modification to image acquisition strategies. Dark scaling is not discussed here.

Jim Solomon provides a recipe for DLSR RAW data acquisition, and without going into a lot of unnecessary detail, providing lights and darks are taken at a reasonably consistent temperature the following strategy is safe and effective.

I have tested this procedure in AstroArt, Nebulosity, Pixinsight, Regim, and a bash script that uses DCRAW and ImageMagick to perform the data reduction process. In every case the results were nearly identical. The script is available for download for Linux and Mac users, further down the page.

I recommend reading the blog on dithering… which is DSLR salvation, in my view.

Please note, the term flat dark is interchangeable with bias, where flats are taken at shutter speeds of less than a second or two, as opposed to longer flat exposures where dark current may be a consideration.

The formula that relates these physical phenomenon, and the actual frames we’ll collect over a night of imaging, are as follows:

(1) Light = (Signal * Flat Signal) + Dark + Offset

where Signal is the image of the target object we wish we could collect under ideal circumstances, and Light is the image we actually captured. Rearranging the terms, we have:

(2) Signal = Light - (Dark + Offset)/Flat Signal

But realize that the Flats we capture with the camera will, in turn, be “polluted” by Darks and Offsets in their own right, and so we must subtract Flat Darks and Flat Offsets from the Flat Lights as follows:

(3) Flat Signal = Flat Light - (Flat Dark + Flat Offset)

So, plugging equation (3) into equation (2), yields this general formula:

(4) Signal = Light - (Dark + Offset)/Flat Light - (Flat Dark + Flat Offset)

Here, “Dark” refers to the thermal noise signal of the imaging camera; i.e., the noise signal that varies in proportion to temperature, ISO, and exposure length. Note, however, that any exposure we take with a digital camera contains the Offset, and “Darks” are no exception. So, if we define Dark’ to be an exposure of some length with the body cap in place, then Dark’ = Dark + Offset, and, similarly, Flat Dark’ = Flat Dark + Offset. Plugging these values into Equation 4 yields the following simplified form:

(5) Signal = Light - Dark’/Flat Light - Flat Dark’

And just to make things even simpler, let’s drop the prime indicators (the apostrophes) that we stuck on “Dark” and “Flat Dark”, and just remember that by “Dark” and “Flat Dark” we mean frames captured with the body cap in place but with the same ISO and exposure length as the Lights and Flat Lights, respectively. That gives us our final form:

(6) Signal = Light - Dark/Flat Light - Flat Dark (bias) - editor’s comment

Equation 6 gives us our marching orders for astrophotography, providing us with a set of Frames that must be captured for each imaging session. The actual order in which I choose to capture these frames is as follows, the reasons for which will be made clear in the acquisition section below:

Flat Darks (bias) editor’s comment
Flat Lights

RAWPREPRO - DSLR RAW file conversion and data reduction/calibration script

Video - no sound

RAWPREPRO is a Linux based DSLR RAW image conversion and data reduction/calibration script. It was originally intended to be nothing more than a means of converting DSLR RAW images to 16bit integer tiff (CFA monochrome, no flip/rotation) files for use in programs such as DSS.

RAWPREPRO has grown into a GUI based conversion and image calibration utility and uses DCRAW for conversion and ImageMagick to perform the image calibration/reduction/preprocessing steps - DCRAW and ImageMagick must be installed to take advantage of all preprocessing options.

Besides data conversion and calibration, RAWPREPRO orders the image preprocessing task through a folder structure, placing files in predictable places for easy access. It keeps things in one place, under a project name, renaming files accordingly. All DSLR RAW data files are preserved. Files can be added or removed, and the project run over and again, as required. Temporary files are deleted at the end of the script saving disk space.

The 3 main functions are as follows;

1. RAWPREPRO uses dcraw to convert DSLR RAW files to 16bit integer CFA monochrome tiff files (no flip) for calibration in RAWPREPRO, itself, or an external preprocessing program - Regim, DSS and others.

2. RAWPREPRO uses ImageMagick to perform the mathematical functions for the creation of master frames, which may be imported into an external preprocessing program.

3.RAWPREPRO uses ImageMagick for the calibration/reduction of light frames, using the safe and effective, bias-in-the-dark method of data reduction.

Note: RAWPREPRO does not perform cosmetic correction or pixel rejection.

Installation and Operation - very quick start:
Following extraction of the zip files, rawprepro may be installed by clicking the script - this will also create a desktop link if desired - if not run rawprepro from command line by typing rawprepro at the prompt.

Alternatively, rawprepro may be run, without installation, within the extracted folder by clicking the rawprepro file or by command line, typing ./rawprepro at the prompt.

The older command line has been updated, as of rawprepro_cl_0.9 for MacOS and Linux users. Presently, it is not as flexible as the Linux GUI version, but is reported to work. It performs the same main functions as the GUI version, but requires the user to load files into the respective folders manually. A cocoadialog version would be nice - anyone?

rawprepro_2.7 - important and significantly improved changes to flat processing and master flat creation.

rawprepro_cl_1.2 - command line version important and significantly improved changes to flat processing and master flat creation.

RAWPREPRO Quick Start Guide - smaller download
RAWPREPRO Quick Start Guide - slightly bigger download - refresh PDF if displayed in browser.

Acquisition notes:

I borrowed this crop from here. Thought it was worth including. While it may be amp glow, first check that the view finder is not the culprit. This artifact looks suspiciously like light leaking through the viewfinder. Tape up the view finder before imaging.


Thursday, October 31, 2013

Welcome to FlatPress!

This is a sample entry, posted to show you some of the features of FlatPress.

The more tag allows you to create a “jump” between an excerpt and the complete article.

[Read More…]

Saturday, December 29, 2012

The Black Generic: a Trout fly

For something completely different. “The Black Generic” (say that with a thick Scottish accent). Presented here for posterity. See further down, for a small snail pattern that was very effective in small dams, too.

“The Generic” (which never had a name, until yesterday) was an attempt to accentuate the form of nymphs - and it worked. The hook shank bent upwards, giving the body a curved posture, the long flared tail, tight skinny abdomen and bulbous bushy thorax were intentional. I noticed that variations were less successful. A thinly tied or short fiber thorax being the main flaws. It needs to look, curvy, leggy, cheeky and thoroughly provocative - hey! look at me, I’m here, come and get me!

My rough sketch doesn’t do it justice and I no longer have my fly tying gear.


Fished, in shallow runs, deep, sinking and on the rise, this fly caught plenty of fish in streams North of Melbourne, some 25 years ago, when I had the time to fish - Deep Creek and Jacksons Creek in particular, when they were healthy, before the drought devastated them. I understand that recent rains have regenerated the area and that fish are more plentiful.

The appearance is a long bulging thorax with a very bushy hackle to 2/3 of the body length. The abdomen is thinly wound single layer of silk with tightly wound silver wire segments and a long flared tail.


Hook size: 14-16 longshank - slightly bent upward ~2/3 from the eye.
Body : Black silk.
Tail: Soft black hen 6-9 curved fibres
Segment: Silver wire - anterior only.
Carapace: Black crow wing.
Hackle: 3 dark brown - black Ostrich herl with long fibres.
Head: Tie off crow wing with 3 half hitches and clip to cover eyelet - a flared finish.


Starting near the hook bend (not the 2/3 bend), tie in the tail, flaring with turns of silk. Tie in the wire and wind the body tightly (a single layer) to the bend (2/3 bend). Wind in narrow segments with the wire and tie off and clip at the 2/3 bend.

At and continuing from the 2/3 bend, tie in a good bunch of crow wing and several Ostrich herl. Wind silk to eye and half hitch. Wind Ostrich herl to within ~1/2mm of the eye and secure with half hitches. Bring crow wing over and tie off with half hitches immediately behind the eye and trim to cover the eye.


It may be necessary to tease out the hackle under the crow wing with a dubbing needle.

You may wax the thread, but I don’t. I prefer the fly waterlogged and sinking fast in moving or deep water.

Tying the crow wing shiny side up is an alternative presentation.

Small Snail pattern:

A second and very effective little fly, was a very small snail pattern. Hook size 16. Olive thread. A few winds of olive chenille with an olive carapace and a few turns of yellow chenille. About half and half.

Fished above weed beds in particular. Very easy fly to make.

Monday, September 24, 2012

Dithering: a DSLR astrophotographer’s best friend

There has been a lot of interest in this thread in recent times so I’ve provided a short summary, mainly for the benefit of newcomers to DSLR astrophotography - I hope it’s helpful.

Dithering, is in my view, one of the most useful and effective techniques applied to image acquisition for increasing SNR, applied to DSLR cameras, saving hours of processing time and frustratingly poor results.

There has been a bit of discussion on various forums about substituting dithering for calibration. Sounds attractive, but;

***dithering is not intended to be a substitute for calibration - I recommend reading this, as well***

Dithering and image reduction serve different purposes with the same aim - increased SNR. Temperature must be considered when using DSLRs, requiring dark subtraction. Dithering however, will hide the majority of temperature related calibration errors/inaccuracies as well as several other types of artifacts… read on.

You can also read about dithering in Berry and Burnell’s, “Handbook of Astronomical Image Processing,” where they recommend displacement of images by at least 12 pixels. There are several informative academic papers on-line, as well.

Backyard EOS has dithering capability, however, I have never used the program. My setup is Arduino based, controlling the RA or DEC axis between images, simply by slewing to present the camera to the target, displaced by 10 - 15 pixels or more between images - it’s that basic.

The comparison image, is intended to accentuate the underlying issues with the image on the right. No attempt has been made to minimize the effect with post processing. The image was stacked and stretched - please note that the red streaks were not evident in individual subs and only appeared after integration. In-fact, I naively spent hours trying to salvage that image - a complete waste of time. The image on the left was taken with the same camera, dithered.

Rather than spending time eradicating/covering up unsightly problems, time was spent lifting out detail, which in the image on the right was partly obliterated by poor acquisition - no dithering.


Here is the pattern I follow… it keeps the image within the sensor boundary. I use a look up table in the Arduino program to schedule the correct hand controller button activation.


More info

Struggling to produce a decent image, it became apparent that the physical and electronic characteristics of DSLR CMOS sensors demand more attention and an understanding of the limitations of the camera’s sensor was essential.

I went to the trouble of replacing the factory IR filter with a special purpose astronomical filter to increase the transmission of Ha wavelengths and built a sensor cooling system to reduce dark current/thermal noise - this all worked quite well. But, there is more to sensor technology and manufacture than meets the eye.

For instance, the anti-aliasing filter, which forms part of the dust reduction system in Canon DSLR cameras is designed to reduce Moire, an artifact produced by the Bayer (RGB) colour matrix. However, the reduction of moire tends to soften the image. Even with a focus mask at ‘perfect focus’ - consequently, the AA filter should be removed.

But that’s not the end of the story. Among the millions of pixels that make up the sensor light gathering matrix, a small percentage are dead (don’t work). Some are on all the time (hot) and overall, pixels differ in their ability to convert photons to electrons. For daylight photography this isn’t a problem, as a rule.

There’s more. The pixel matrix and associated electronics produce fixed pattern noise. Heating of the sensor during extended operation also increases noise. Random noise is a function of arriving photons and is different for every frame. The optical properties and cleanliness of the sensor, filters and lens also produce artifacts.

Combined, this all conspires to ruin the image to which you have dedicated copious amounts of time at the wrong end of the day, when perhaps, you should be sleeping.

So what’s the solution? In principle, reducing the effects of the deleterious electronic and physical influences inherent in the optical system is quite simple, in practice however, it involves another layer of complexity; that is, DITHERING!.

Dithering is an authentic solution because it addresses noise suppression, optical and sensor artifacts - dithering does not replace proper image calibration techniques. It will however, greatly improve results and avoid problems that no amount of calibration or sensible image processing can resolve.

So, what is dithering?

Dithering is the practice of shifting the sensor (camera) between images, so that each new image is slightly offset from the previous image. The image is sampled by different pixels; that is, the image moves, ideally, by 10 - 15 pixels, from the position of the preceding exposure. With careful management the target image remains well within the sensor boundary.

Dithering can be random using a hand controller and estimating the offset by timing the button push in DEC or RA. Better still, an automated system that dithers in a box shaped spiral, or maze pattern. The goal is to avoid a succession of images occupying the same or neighbouring pixels (which produces poor results) and to prevent the target object moving out of the FOV.


Dithering, particularly with DSLR cameras improves signal to noise ratio for very little effort. It can be win win for the astrophtographer.

Executed properly, dithering deals effectively with random noise, hides hot and cold pixels, improves flat fielding and sub-pixel sampling; that is, capturing the image over a range of pixels means that we are not sampling the same and possibly less efficient pixels repeatedly for the same object location.

Calibration is not always as effective as we would like with DSLR images. And even if the images weren’t calibrated, a dithered stack would produce pretty good results.

Monday, July 2, 2012

Unknown Arthropod: Pycnogonida or terrestrial spider?

For something completely different.

Edit: The latest information indicates that this is a terrestrial spider - it enjoys the seaside life! Could it be an Opilionid or Harvestman? Spiders generally have leg hair to one degree or another. The Australian Daddy Long Legs, which appears to have hairless legs, is a tangle web spider. However, Daddy Long Legs is also applied to the order Opiliones or Harvestman, which are Aracnids, but not spiders.

This single photo, unfortunately, doesn’t show what I recall as binocular vision typical of hunting spiders. Vividly, however, I do recall four bright blue eyes, two of which face forward.

There is a little fan shaped cove with rocky promontories at either end near Torquay (Victoria Australia). Walking this stretch of beach at low tide one warm sunny day in May 2011, I noticed a small creature, perhaps 5 - 6mm (leg tips,) hurrying along the waterline, occasionally covered by a light wash. The water receding, this leggy little organism, having resisted the flow of water, continued on its way.

A wisp of life, it seemed to be feeding near the low tide water line over smooth packed sand. Scurrying here and there, stopping suddenly, then moving on with equal energy. Carrying a camera with a macro lens was fortuitous, managing to squeeze off one shot in focus, while chasing this little fellow around the beach, losing sight of him, standing up to find him again, a bright little speck on the sand.

I made several inquiries to various institutions without much joy. Without a ’sample’ there was little to identify. Eventually, however, and almost a year later, it was suggested that the arthropod is most likely a species of Sea Spider or Pycnogonida, which inhabit the oceans from the shore to the deep, worldwide.

This was later modified to a species of terrestrial spider. Identification is a little difficult with only a photo. I have returned to the location several times with no further sightings.

I’m intrigued by what appears to be a proboscis attached by a socket to the front of a dorsal appendage sweeping backward over the body, like a trowel handle. The other legs appear to be arranged asymmetrically. The abdomen gives the appearance of a terrestrial Daddy Long Legs. The eyes, blue and arranged either side of the base of the dorsal appendage. Fascinating little guy!