Help! My ECW Has Speckled Edges!

8

They appear out of nowhere. They set up camp and act like they own the place. How did they get there, and who invited them? I’m not talking about your nosy neighbor, I’m talking about a different problem: those speckled edges. Also known as transparency remnants, removing collars, nodata transparency, edge patterns and I’ve even heard a customer refer to it as the “cow pattern” but let’s just stick with calling it the “speckled edge” for now as all GIS.SE questions here are talking about the same thing.

It’s a problem that continues to haunt users loading ECW files in a variety of applications. These ugly artifacts appear along the edge of the dataset and is particularly noticeable when you have gone through the painstaking process of mosaicking or trying to view underlying datasets:

ecw artifacts

Via GIS Stackexchange, display in Esri ArcGIS for Desktop

The problem with trying to solve these questions on various online communities is that they are confusing two completely separate concepts:

  • nodata or null pixel transparency
  • transparency / alpha / opacity masks (also called Opacity bands).

They are very different but continue to be used interchangeably which leads to at least part of the confusion.

So what can you do..?

Recommendation 1: Setting nodata transparent values is never recommended for ECW or JP2.

ecw artifacts lost shadows

Non-application specific, this time in GeoMedia 2014 but shows unintended consequences losing shadowed areas

Applications such as GDAL created nearblack and GeoMedia, QGIS, ArcGIS, FME supports multiple nodata values, but it’s arguably a workaround that will never work consistently because the degree of colour shifting in the background (white or black) will never be the same and you risk losing areas within your image such as shadowed areas.

It’s at best a quick fix but as soon as you compress to ECW, say after mosaicking, you will once again have the speckly edges return because it’s a symptom of the compression. The other problem is interoperability.  Whilst most applications support knocking out one nodata value—such as 0,0,0—not all support removing a range. This leads to an inconsistent user experience when viewing ECW’s in different applications.

no opacity

ECW with no opacity band, black background.

nodata transparency

Nodata transparency enabled in QGIS 2.2 for 0,0,0

Ecw with nodata

ECW display with nodata enabled and white background. Ugh

ecw with 5 nearblack values

ECW display with 5 near black nodata values. Double ugh!

Now I have hopefully convinced you that knocking out specific or a range of pixels is not a great scalable idea. The portable solution is to somehow persist this information back into the file. But how?

Recommendation #2:  Always compress using a transparency/opacity/alpha mask.

Using the ECWJP2 SDK terminology, a transparency mask is implemented as an opacity band called AllOpacity. Like other formats this mask controls whether underlying RGB bands will be transparent or opaque and nothing in between. This completely removes any compression artifacts and solves our other problem of portability since this information is stored within the ECW file. The last question then is application support which leads me to the other common problem that customers run into. Lack of a time travelling device!

ECW was designed during the 1990s and predates any use of Opacity bands in the industry. This means the first version of the publicly available decoding SDK, v3, has no support. Period. Around 2008 we added support for Opacity bands in the v4 SDK by appending the information in a backwards-compatible way and was first released in ERDAS IMAGINE 2010. But this turned out to be a double edged sword as it ensured everyone in the industry could continue reading ECW files but it was never clear to users why one application could view the alpha mask (v4) while others couldn’t (v3) since it really comes down to the version of the SDK being used internally by each application. Unfortunately, that is not something that Hexagon Geospatial can control (thus my desire for time travel).

Version First Released Last Released Opacity Support
ECW JP2 SDK v3.x 2004 2008  Not supported
ECW JP2 SDK v4.x 2009 2012 ECW v2 (Appended) supported
ECW JP2 SDK v5.x 2013 2015* ECW v2 (Appended) & ECW v3 (Interleaved)  supported

Our latest release of the v5 ECWJP2 SDK has full support for Opacity bands in ECW v2 and the new ECW v3 format. With all the other benefits of v5 such as extensive platform support and performance improvements, if you are still running into this speckled edge problem now is the time to upgrade or ask your application/data provider to get with the program! It hasn’t been an ECW format limitation for many years now.

Some examples of what an embedded ECW Opacity channel looks like in various applications are below.

band4alpha channel

Band4 interpreted as Alpha in QGIS 2.2

alpha in esri

Band_4 as Alpha band in Esri ArcGIS for Desktop 10.2

alpha in IMAGINE

Opacity layer = yes in ERDAS IMAGINE 2014 Image Metadata (auto-applied)

gdal alpha

GDALinfo 1.10 reporting

As a user of Application X, how are you supposed to know whether it supports reading/writing ECW Opacity bands?

Well if you are so inclined, you can browse to the Application binary location (typically /bin) and locate a file named either NCSEcw.dll, NCSEcw4.dll or NCSEcw4_RO.dll and view the file properties. Since v4, Hexagon Geospatial only provides signed, fully versioned DLL’s so for example Esri ArcGIS for Desktop 10.3 ships v5.0.1 by default, Globalmapper 16 v5.1.1, QGIS 2.6 v5.0.1 and ERDAS IMAGINE 2015 v5.2.0. Of course just because an application ships a version greater than v4.x still might require work at the application level to enable it but you can be 100% sure its not a limitation of the ECW format.

Pssst… since you’ve read this far down, the latest ECWJP2 SDK v5.2 release previewed here is now available!

 

Share.

About Author

Chris Tweedie

Chris Tweedie directs all stages of the product lifecycle for ERDAS APOLLO and ECW technologies. He has been with Hexagon Geospatial for 6 years in Technical Sales and Product Management roles for what is now the Provider Suite. Not one to shy away from getting his hands dirty, he enjoys squeezing the most value out of the products built in the Australian development office including ERDAS APOLLO, ECW/JP2 SDK, GeoCompressor and ECW Plugins.

8 Comments

  1. Samuel Bärisch on

    Hi Chris,

    interesting post. But apart from technical aspects: why is ERDAS Imagine 2015 still missing the functionality for using opacity when exporting to JP2?

    Cheers,

    Samuel

    • Hello Sam,
      ERDAS IMAGINE has focused on many ECW improvements for the last few releases. There was a lot of work to be done. ECW has formed the foundation for what we do in JP2, and many other file formats going forward. For 2016, we expect to address JP2 opacity layers (also act as NODATA layers) in Export JP2 and in Spatial Modeler.
      Regards,
      Paul

      • Samuel Bärisch on

        Hello Paul,

        thanks’ for the reply. Adding this functionality to ERDAS would be an important feature as many high resolution data sets (e.g. LANDSAT, Rapideye) are not strictly north oriented and many customers define individual, also non-rectangular, AOIs. So, using opacity with such datasets without having this speckled-edges problem would very much increase usability of JP2 file format in ERDAS and other GIS SW.
        Looking forward to the next release… 🙂

        Best regards,

        Samuel

  2. Great article! It answers a lot of questions but we are still have trouble displaying ECW’s in geomedia. Do you have any ideas on how to properly display ECW’s with transparency in Geomedia 2014? We are currently mosaiking images in ERDAS 2015, saving as ECW then opening them in Global mapper and exporting as ECW to remove the alpha channel. But that still leaves the black blotches.

    Thanks’
    Jon

    • Chris Tweedie
      Chris Tweedie on

      Hi Jon

      I would recommend contacting support for help working this through in GeoMedia. Certainly your workflow is overly complicated! Mosaicking in IMAGINE 2015 should all be thats required.

      • Thanks for quick response, although I was hoping for more insight now that Geomedia is in I bed with Hexagon. But your right, this is a very painfully long and complicated process! I’ll contact Geomedia and see what they have to say.

        Regards’
        Jon

  3. Excellent write up. Thank you for clearly describing the difference between Nodata pixels and Alpha/Opacity channel.

    One critical piece is missing though: how does one define or create the masked areas? (I know, it’s application specific and out of scope for the post, but you’re leaving us hanging!)

    • Chris Tweedie
      Chris Tweedie on

      Hi Matt,

      The interleaved opacity band is only supported at Write time of the file. The old appended opacity type can be added onto an existing ECW v2 but that is no longer recommended unless you absolutely have to keep using ECW v2 files. JPEG2000 is similar, writing AllOpacity is done during compression.

      ERDAS IMAGINE and GeoCompressor remain the best options for robust opacity band writing (typically via some sort of region file such as SHP). For other third party licensees its up to the implementation on how they support it

Leave A Reply