Asterism

Astronomical exposure reduction (and) image stacking method

User Manual

Copyright 2017 -18 cheshire_r@fastmail.fm

Definitions

CDC – consumer digital camera / DSLR / mirror-less

OSC – one shot colour – refers to cameras with sensors overlaid with a Bayer matrix or Fovean filtered

RAW – RAW data or image data - The image file or data file straight from the camera in its RAW format.

Noise – unwanted signal generated by various sources.

Pre-processing – noise reduction, deBayer, align and stack an image set.

Light frame – an image of the subject – usually, multiple images, combined to improve signal to noise ratio (SNR).

Bias frame – applied to CDCs the bias is an image of the electronic ‘stamp’ of the camera image sensor, in operation - fixed pattern and predictable.

Dark frame – an image of the dark current / noise, generated by sensor operation – fixed pattern and predictable, at a given temperature, exposure time and ISO setting.

Flat frame – an image of the optical flaws (obstructions, dust, hair and other contaminants,) visible between the telescope objective / lens and the image sensor.

Master frame – a noise template representative off the bias, dark or flat signal - referred to as Master bias Master dark and Master flat.

Introduction and operation

Asterism, is a Linux based open source image processing utility, primarily for use with CDC RAW images (and other image formats) and makes use of freely available image manipulation programs.

From version 0.1.3.5 (onward), an applications menu entry and optional desktop link may be created, replacing the former system installation.

Installation and running Asterism

  1. The programs yad dcraw imagemagick and hugin-tools must be installed - hugin macrofusion and enfuse are optional.

    Note: Asterism will prompt for programs and selected functions that are not installed.

  2. Download the source asterism.zip file and extract the Asterism folder to the user's home directory.

  3. Optional - open the Asterism folder and run SETUP – to create an applications menu entry and / or desktop link / icon; or

  4. Launch Asterism with RUN; alternatively

  5. Open a terminal in the Asterism folder and type (without quotes) 'bash -x ./asterism' - Enter

Note: use SETUP to remove the menu entry and / or desktop link

Note: if using the GitHub version it may be necessary to set both RUN and SETUP as Run as program in the Properties Permissions window; alternatively

Open a terminal in the Asterism folder and type chmod +x RUN – same for SETUP

Asterism’s processes are mostly stand-alone and may be combined as required. Best results are obtained following the methods described in this manual.

CDC image sets tend to be of variable quality and data linearity may vary (that is, between bias dark, flat and light frames). Experimenting is inevitable and leads to better outcomes and time savings.

RAW image data, acquired at and below 0°C - +5°C , may be processed with Bias frames and Flat frames. RAW image data acquired above +5°C (rule-of-thumb) is better processed with a Defect map and flats.

Note: Because the “bias is in the dark”, a Defect map may be applied to cooled data, without detriment (as a rule).

Note: Flat frames must be calibrated with a master bias – a Superbias is preferred.

Image and calibration frame acquisition

This section is provided for general reference.

Light frame acquisition is subject to a range of variables, such as equipment and environmental and is an area of study itself.

Bias frames should be taken at the camera’s lowest ISO setting and fastest shutter speed, with the sensor in complete darkness.

Flat frames should also be taken at the camera’s lowest ISO with an even (flat) light source, at the same aperture as the light frames. Flat illumination is then a function shutter speed. Dark-flats are not usually required for CDC flats because shutter speeds are sufficiently high to preclude dark noise.

Note: Flat illumination should be approximately 50 – 70% (right side) of the camera live view histogram. 50% for dark skies and 70% for suburban skies.

Note: Asterism compensates for under and over exposed flats, as well as variations in light frame exposure to produce consistent and clean flat correction.

Dark frames are acquired with the sensor in complete darkness, at the same ISO, exposure time and temperature as light frames. There are various strategies to match dark and light frame temperature during acquisition, where cooling is not available.

Using Asterism – overview

A basic RAW processing work flow

  1. Choose or create a Project folder* – usually the folder containing the image files - but the image files may be located anywhere.
  2. Note: the project name is assigned the name of the project folder - must_be_one_word – no spaces or colons (:)

  3. Input format’ set to RAW (default).

  4. Select the required processes and options, as individual tasks or as a complete processing run.

  5. Accept all other default settings; and

  6. Select ‘Process’ - bottom right tool bar.

Asterism will perform the selected task/s and return to the main interface.

By default, all calibration frames are mean combined, however, median combination improves noise rejection. The preferred use of a Defect map or Superbias and Master flat (median combined or to which mean-shift noise reduction has been applied) minimises noise contamination of the light frames.

QRMin and QRMax noise reduction may be applied to light frames to set the minimum and /or maximum pixel values within the percentage of quantum range selected. This is similar to Windsorized sigma clipping.

Note: QR = quantum range; that is, 0 - 65535 or 65536 colour values (black to white) for a 16 bit image. Alternatively 0 – 1 for normalised values – there are 65536 values from 0 – 1.

Caution: Use QR settings with caution and avoid their use when preparing Bias and Dark frames.

Noise rejection in the light frames is further enhanced by polynomial stacking. ‘POLY’ is the preferred stacking algorithm.

A small image set of 3 bias 3 darks (optional), 3 flats and 3 lights (or as many as considered representative) is a convenient way to gauge the likely outcome of the full image set.

Processing examples

Create master frames – in this slide, Defect map, Superbias and Master flat with noise reduction – note the Noise-radius and Noise-amount settings. Previews are normalised to make the otherwise linear master files viewable.



...extend the process to include light frame calibration deBayering, image alignment and stacking – with noise reduction.

Note: the Noise-radius and Noise-amount settings are not applied to the light frames in this case. A prompt will appear to reselect noise reduction parameters to be applied to the stacked image, prior to develop and / or display.



In the next slide, calibrate light frames, deBayer align and stack and create a preview, using existing master frames. In this case a Defect map and Master flat.

Technical process options and other features

The remainder of the manual describes the various options processes and features available in Asterism.

Asterism’s processing model

This section is provided for completeness.

Asterism’s processing model assumes that CDC RAW data is not strictly linear.

The most basic algorithm, suitable for cooled image data is;

Signal = (Light Frames – Master Bias) / (Flat Frames – Master Bias)

Exchanging the Master Bias with a ‘Master dark’ modifies the algorithm for cameras without cooling; a Defect map is better used here.

Signal = (Light Frames – Master Dark) / (Flat Frames – Master Bias)

Main tab – Process and Options

Many of Asterism’s original functions have not changed. 3 channel RGB processing has been removed from Asterism 0.0.8.5 onward – use the deprecated version available at the homepage, for 3 channel RGB processing.

Select project (output) folder, processes and options

Process and options’ - the main tab from which all image processing originates.

Select camera matrix’ - apply ‘sRGB’ channel multipliers – unmodified cameras.

Add camera matrix’ - add up to 5 unique channel multiplier matrices.

Edit camera list’ - remove or modify channel multipliers.

'Project details' - record the details of the project.

'System settings' - remaining hard-drive space, memory and processing threads.

'Documentation' – links to documentation resources and version information.

Process and options tab

Note:*’ is a reminder that an entry is required in the field

Project folder*’ - select or create a project folder – the project name is assigned automatically.

Input format’ - RAW,

OTHER (e.g., jpeg png tiff - deBayered images),

OTHER BAYER - not RAW and not deBayered – e.g., fits tiff).

Output format’ enter an output extension – default tif (16bit no compression)

Colour depth’ - default 16 – range 8 – 64 in 8 bit steps

Master frames – create master frames or use template

Note: The indented options are selected with the main option; for example, to create a ‘Super-bias’, select ‘Bias frames’ and Super-bias’. Similarly, for median combined frames, select Median

Map Defects’ – ‘Defect map’ is created using Dark frames - a single dark frame ‘Defect map’ is superior to a single dark.

Bias frames- Create a mean combined Master bias

- ‘Super-bias’ – Create a (master) ‘Super-bias’ - significant random noise reduction in the Master bias frame, results in cleaner calibration of flat and light frames.

-Median’- Create a median filtered Master bias - ‘Super-bias’ is preferred.

Dark frames’- Create a mean combined Master dark.

- ‘Median’ - Create a median filtered Master dark, useful for minimising noise in noisy image sets and / or small calibration frame image sets - ‘Defect map’ is preferred.

Flat frames’- Create a mean combined Master flat. Setting a ‘Noise - radius’ = 7 and ‘Noise - amount’ = 0.1, creates a Master flat with similar standard deviation to a median combined Master flat.

- ‘Median’ - Create a median filtered Master flat, useful for minimising noise in noisy image sets and / or small calibration frame image sets – see above alternative application of ‘Noise reduction parameters’.

Flat correction- Automatic flat correctiondefault ‘5’ for automatic flat correction of all light frames, regardless of variations in flat or light exposure. This process rescues poor flats and mixed light image sets with different exposure times. The luminance of each light image is calculated and the flat frame adjusted to ensure reliable flat reduction. Set ‘0’ to turn off session-by-session.

Flat frame illumination is important. A 'darkish' flat will brighten the light frames, whereas, a brighter flat will tend to darken the light frames, in Asterism. Correcting flat illumination, for each light frame produces clean flat division.

Note: If a Master bias is not available to the Flat frames a file selection window will pop-up – select a Master bias – Super-bias preferred.

Note: Flats are assumed to be taken at each imaging session.

Defect map’ - select a previously created defect map

Master bias’ - select a previously created master bias / super-bias

Master dark’ - select a previously created master dark

Master flat’ - select a previously created master flat

Note: Master frames are located in the project_folder/master directory - miff files. miff is the ImageMagick native format.

Image reduction – calibrate light framesImage reduction

Light frames’ - Process / calibrate the light frames.

Image registration – alignment and stackingImage registration

DeBayer Align Stack’ - As described – deBayer align and stack the calibrated light images

Develop preview’ - Background extraction and automatic white balance. Basic – use PMStack for colour modification, post processing.

Noise filters

Noise – radius’ and ‘Noise – amount’- set noise reduction parameters for the optional ‘-mean-shift’ flat process.

Note: if ‘radius’ and / or ‘amount’ values >= 0 and / or 0.00 are selected (if flat frames were created in the same processing run) a will prompt will appear following stacking permitting reevaluation and adjustments to noise reduction parameters for the image stack.

Note: the ‘-mean-shift’ routine should be used judiciously. A large ‘Noise - radius’ increases processing time, whereas, ‘Noise – amount’ spreads the effect of the radius - keep this small; say 0.1 – 0.2 to preserve edges in the image.

QRMin' and ‘QRMax- set the minimum and maximum pixel values of ‘all’ processed image files, within the data range (as a percentage of quantum range).

Caution: Bias and Dark frames, in particular, are very sensitive to excessive ‘clipping’.

DeBayer options

DeBayer’- SUPERPIXELfast - produces half scale image - use BILINEAR if artefacts are generated at bright edges and / or the colour channels are swapped (this may occur with Canon 5D MKII images).

BILINEAR – few colour artefacts

MALVAR – use BILINEAR if artefacts are generated at bright edges

'Bayer matrix'RGGB (default - most CDCs) – check camera technical data

BGGR

GBRG

GRBG

Align and stack options

Interface’SYSTEM or select an optional graphical user interface.

Alignment’ –LINEAR - fast

MULTI – for more exacting control point selection – slower than LINEAR

ALL PAIRS - for more exacting control point selection slower than MULTI and LINEAR

Stack’Stacking options are;

POLY – default - a noise reduction stacking routine with excellent outlier rejection – remove satellite trails and bad pixels.

MEAN – best signal to noise ratio (SNR) – poor rejection properties. Combine with ‘Noise - radius’ and ‘Noise – amount’ and / or QRMax and QRMin.

MEDIAN - noise reduction at the expense of SNR.

ENFUSE - (exposure stacking, if installed) is useful for focus stacking macro images.


Satellite trail… before….


and after polynomial stacking

Please note – these are linear images; that is, no post processing has been performed.

Ransac iterations’2000 is a suggested upper limit – default 200.

Note: poor image alignment is infrequent and more often a result of widely displaced images or significant rotation between two or more images in sequence. Panotools may not find control points. Use ‘Inspect’ to remove offending images

HFOV’Horizontal Field of View. Panotools HFOV estimation is adequate in most cases – default 8.

The on-line calculator at http://www.scantips.com/lights/fieldofview.html is helpful. To calculate HFOV input focal length and sensor size; that is, 35mm (full frame), APS(-C) and 4/3 are typical CDC sensor dimensions.

Mask’Mask is a Panotools option. HARD to sharpen and SOFT to soften exposure fusion stacks – default NONE

Cache’Panotools image processing cache – set as required. Default 4096.

Colour processing

Colour processing – bracketed exposures / HDR align and stack multiple images

RAW colour images’ - select RAW images

Demosaic’ - DCRAW deBayer routines

Note: Develop preview produces a close representation of the post processed result.

Select camera matrix

RGB channel multipliers are not applied to RAW images in camera, unless saving in jpeg. In Asterism, channel multipliers may be applied before or after stacking. Select a camera model and method of application – PRESTACK or POSTSTACK – that is, before or after stacking. PRESTACK applies channel multipliers to all sub-frames. POSTSTACK applies channel multipliers to the final stacked image.

Eta Carina – Channel multipliers applied to an unmodified CDC image

For illustration purposes only. Image is overly saturated. Multiplier values may be manipulated to accommodate an overly Red response in modified and unmodified camera data.

Add CDC channel multipliers

Edit camera list – quick edit parameters

Acquisition details



System settings – session by session


 

Documentation and resource links



Advisory and error messages

Messages are generated in the following circumstances;

A Project folder is not selected

No process has been selected

Light frames are selected without generating or loading master frames.

Stack result ‘NO’ generates a message about possible options to improve image alignment.

Progress bars

Progress bars pop up with successive processes



As mentioned, if Develop was selected, Asterism displays a background extracted auto-white balanced preview. The preview may be processed and saved using the basic tools provided by the display program. A single mouse click opens the ImageMagick options menu. Saturation sigmoidal-contrast, despeckle and sharpen are useful tools.

The full resolution develop .tif file processes very easily with default settings in StarTools.



The final image is displayed for inspection and is characteristically linear. Closing the display presents the 'Stack result OK' question...

Stack question



Answering ‘No’ will generate a message. If alignment is not ideal select Hugin on the bottom tool bar. Hugin is a very powerful panoramic stitching and HDR utility.

LLRGB channel creation



Answering ‘Yes’ will progress to the LRGB L* channel creation question. Here the final image can be separated into its individual components with which to create an LLRGB image. LLRGB provides greater control over colour saturation and luminance. L* is the LAB lightness channel and is useful as a mask for various post-processing operations.

All channel images are found in the llrgb_name folder in the project directory.

Cleaning up advice – back to Asterism interface


 

All temporary files and folders are removed, including any master frames in the temporary .master directory.

File Location

All processed file output is found in the project folder.

The ‘acquisition_details_name’ file lists the 'Acquisition' tab information. The ‘log_name’ file is the process directive.

Please note:when entering text into fields, do not use colons. ‘:’ are field separators for awk and Asterism may do strange things or not work at all.

Bottom tool bar utilities

These are the stand-alone utilities.

Files

A file manager with links to the system file manager.

Convert

An image format conversion utility –Select images convert and resize. RAW images are converted to 16bit linear images. 

Inspect

Select images and create previews and inspect and reject unwanted image files.

The Inspect utility is used for inspecting and rejecting images. As a rule, images of suspect quality; subject to vibration, poor focus, under or over exposed, high noise levels or significantly displaced in distance and / or rotation from the general distribution of the image set, should be removed.

Window focus is important when usingInspect’. Ensure that the file window is ‘active’ prior to making selections and ensure that the display window is active before advancing or reversing the animation, with the space bar or backspace keys.



Ctrl-C + Click’ to select images for rejection. Selecting 'OK' will move unwanted images to the reject folder and Asterism will continue processing.

PMStack

PMStack is a stand-alone image alignment and stacking utility. PMStack uses Asterism’s image alignment and stacking method and provides a second opportunity to align and stack the deBayered images - should the user’s original Asterism settings not be successful.

Asterism’s default LINEAR alignment option seldom fails to align images properly. However, it may be necessary to use MULTI or ALL PAIRS and / or increase RANSAC iterations to align a ‘difficult’ image set.



Select a log file; created during a previous Asterism processing session (log_DSO or log_PMS in the jpeg example). PMStack automatically finds the images in the debayer_name, if present- or the previously aligned images in the process_name folder.

PMStack – align and stack options

Next, select appropriate image align and stack options. If PMStack is required, more sensitive settings may be required - depending on the size of the image set, alignment may take some time.

balance modifications and channel depth may be made here) and then present options similar to Asterism, including ‘LRGB L*’. If alignment is not successful with any setting, inspect the image set and remove any problem frames using ‘Inspect’.

The ‘Stack only’ option may be used to stack previously aligned images in the project_directory/process_dir folder and in the project_directory/debayer_name/align_name where previous alignment using PMStack was performed. PMStack will look for both directories and choose the first it finds.

‘Develop' is accessible from the PMStack window by selecting the Develop check box and will be performed following image stacking. Alternatively, Develop accessed independently from the Develop window is stand-alone – select an image to activate



ImageMagick 7 may be configured for HDRI, Q32 or Q64. In any case, 8 – 64 colour depth may be selected, however, 64 bit is not fully implemented in Asterism. ImageMagick 6+ Q16 is the current package in Ubuntu. Asterism is set up for Q16 defaults.

Note: PMStack is not confined to image stacks created by Asterism; and providing deBayered images are located in the debayer folder or previously aligned images in the align folder (if stacking only), PMStack will process them. A log file is required and it may be necessary to first generate one in Asterism.

Hugin

Hugin comes with a variety of alignment and optimisation algorithms – process image files as follows;

1. Add Images

2. Select alignment algorithm

3. Select Create Control Points

4. Optimise - Geometric and Photo-metric

Once control points have been created select the Stitch Tab

5. Projection - default

6. Select Optimise Field of View

7. Select Calculate Optimal Size

8. Select Fit Crop to Images

9. Deselect Exposure corrected, low dynamic range

Remapped Images

10. Select No exposure correction, low dynamic range

Combined Stacks – for exposure fusion

11. Select Exposure fused stacks

Processing

12. Nona Options - deselect Save cropped images

NOTE: Aligned images can be stacked using ImageMagick

do not select fused stack option if manual stacking is

to be performed

13. Enfuse Options - copy and paste the line below

-m 8192 -l auto -d 16

Use this option if desired, however, it is better to let Asterism stack the aligned Hugin output using PMStack, ‘Stack only’ option – as above.

14. Select Stitch

15. Hugin will prompt to save a '.pto' name; followed by

16. A request for an output prefix.

Histogram

Generate a single histogram or an animation of histograms. Caution it will generate and display as many files as are loaded. Histograms are deleted once the histogram display window is closed.

MacroFusion

Important: Please refer to the file naming convention section below.

MacroFusion defaults are usually adequate. The user must check the check boxes in the Align images field - bottom left. As a rule Hugin is a better choice. It is also recommend to increase the Preview size under the Configuration tab.

Note: The 'Preview' option is recommended prior to the 'Save' option.

Align - must be selected

Autocrop - ideal for previews

Optimise image centre shift - try previews with and without

Optimise field of view - try with and without.

Technical – Linux specific

Asterism is written in GNU Bash and requires yad (GUI) dcraw, imagemagick and hugin-tools

Enfuse Hugin and MacroFusion are optional; and if installed, may be selected through the Asterism interface.

Note: Asterism will prompt installation of required and optional programs, if selected and not installed

Yad was chosen as the graphical user interface for its flexible simplicity and neat looks.

DCRAW is used by many image processing applications for converting RAW image files.

ImageMagick is an expansive image processing utility and performs complex tasks.

Hugin-tools uses the panotools library and is used for image alignment and auto-cropping; and is the back end to programs such as Hugin.

Optional programs, Enfuse Hugin and MacroFusion, belong to the same family of image alignment, panorama stitching, HDR and exposure fusion utilities.

System requirements

Asterism was tested on a 4 core i5 8GB Linux (xubuntu) system. Memory management and multi-threading is provided by ImageMagick's inbuilt resources, configurable from the system tab, session-by-session. Presently, ImageMagick is configured to use all but 1 core on multi-processor systems and ~50% of system memory, which may be increased session by session – refer ‘System settings’ tab.

Note: Because Asterism uses several programs, processor threads and memory allocation may be set by the operating program.

Note: Asterism permissions are set to 777 (full access) and there should be no problem running bash -x ./asterism. However, if permission is denied, set user permissions across the Asterism folder by typing (in a terminal);

chmod 777 -R ~/Asterism

775 may be used in place of 777. This permits access to the user and the users group, while preventing write access to other users.

A work in progress…Please drop me a line at cheshire_r@fastmail.fm should you have questions about using Asterism.