Aerial photography archives

/

Notes on where to find historical aerial imagery (thanks to Kass Green): The USDA has an archive of aerial imagery in Salt Lake City at APFOhttp://www.fsa.usda.gov/programs-and-services/aerial-photography/index.  There is a ArcGIS online map of the  tiles and dates of this photos. Search in ArcGIS online for the AFPO Historical Availability Tile Layer. USDA is in the process of scanning these photos, but you can order them through a manual process now (which can take a long time). 

The EROS data center in Sioux Falls also has an archive of high altitude photos for the US from the 1980s.  Also check out https://lta.cr.usgs.gov/NHAP  and https://lta.cr.usgs.gov/NAPP .  These photos are available digitally, but are not terrain corrected or georeferenced.

Great links from class today

/

Today was WebGIS and the Geoweb (I know, we could do a whole semester), and rounded up some nice resources. 

  1. Open Street Map interactions (from Vanessa):
    1. Here is Overpass Turbo, the OSM data filtering site. https://overpass-turbo.eu
    2. Here is Tag Info, where you can find the keys to query information on Overpass Turbo. https://taginfo.openstreetmap.org/
  2. Privacy (from Wyeth): Radiolab did a great piece on the intersection between GIS data and privacy.
    1. Link to the article: http://www.radiolab.org/story/update-eye-sky/ (this is the updated article after changes from the original broadcast in June 2015 [http://www.radiolab.org/story/eye-sky/] ) 
    2. Also, the company that developed from this: http://www.pss-1.com/

New digitization project: Soil-Vegetation Map Collection

/

Between the years 1949-1979 the Pacific Southwest research station branch of the U.S. Forest service published two series of maps: 1) The Soil-Vegetation Maps, and 2) Timber Stand Vegetation Maps. These maps to our knowledge have not been digitized, and exist in paper form in university library collections, including the UC Berkeley Koshland BioScience Library.

Index map for the Soil Vegetation MapsThe Soil-Vegetation Maps use blue or black symbols to show the species composition of woody vegetation, series and phases of soil types, and the site-quality class of timber. A separate legend entitled “Legends and Supplemental Information to Accompany Soil-Vegetation Maps of California” allow for the interpretation of these symbols in maps published 1963 or earlier. Maps released following 1963 are usually accompanied by a report including legends, or a set of “Tables”. These maps are published on USGS quadrangles at two scales 1:31,680 and 1:24,000. Each 1:24,000 sheet represents about 36,000 acres. 

The Timber Stand Vegetation Maps use blue or black symbols to show broad vegetation types and the density of woody vegetation, age-size, structure, and density of conifer timber stands and other information about the land and vegetation resources is captured. The accompanying “Legends and Supplemental Information to Accompany Timber Stand-Vegetation Cover Maps of California” allows for interpretation of those symbols. Unlike the Soil-Vegetation Maps a single issue of the legend is sufficient for interpretation. 

We found 22 quad sheets for Sonoma County in the Koshland BioScience Library at UC Berkeley, and embarked upon a test digitization project. 

Scanning. Using a large format scanner at UC Berkeley’s Earth Science and Map library we scanned each original quad at a standard 300dpi resolution. The staff at the Earth Science Library completes the scans and provides an online portal with which to download. 

Georeferencing. Georeferencing of the maps was done in ArcGIS Desktop using the georeferencing toolbar. For the Sonoma county quads which are at a standard 1:24,000 scale we were able to employ the use of the USGS 24k quad index file for corner reference points to manually georeference each quad. 

Error estimation. The georeferencing process of historical datasets produces error. We capture the error created through this process through the root mean squared error (RMSE). The min value from these 22 quads is 4.9, the max value is 15.6 and the mean is 9.9. This information must be captured before the image is registered. See Table 1 below for individual RMSE scores for all 22 quads. 

Conclusions. Super fun exercise, and we look forward to hearing about how these maps are used. Personally, I love working with old maps, and bringing them into modern data analysis. Just checking out the old and the new can show change, as in this snap from what is now Lake Sonoma, but was the Sonoma River in the 1930s.

Thanks Kelly and Shane for your work on this!

ISECCI historical ecology working group wrap-up

/

Last week Kelly and I with others travelled to the Sierra Nevada Aquatic Research Lab (SNARL) in the eastern Sierra Nevada, just south of Mono Lake for a research retreat. SNARL is part of the UC's Natural Reserve System, which is comprised of nearly 40 properties across the state. These are preserves that foster research, education and collaboration. They have much in common with ANR's REC system. I've been to a few of them now, and am very pleased to make more visits. I love the east side of the Sierra, and that iconic Highway 395. 

This trip was a retreat for the ISECCI historical ecology working group, led by the inspirational Peter Alagona from UCSB. We discussed our existing projects, including the VTM work (see figure below), and talked about potentials for more collaborative research and further integration between NRS and ANR. We have a list of wishes for digitization, and if anyone out there has ideas about pitching these to donors, please let me know. For example: 

  • Kelly and I want to digitize the Leiburg maps from the northern Sierra to add to the VTM stack;
  • We want to find a better way to index and view historical aerial photography state-wide. Something like this for historical maps: http://ngmdb.usgs.gov/maps/TopoView/help/

And we had a field trip looking at Mono Lake water issues. Great time spent!

Density of VTM features across the collections

Digitizing old maps for science: the Soil-Vegetation Map Series

/

As many of you know, my lab at Berkeley has been involved in rescuing, digitizing and sharing historical ecological data for about a decade. Our big push has been working with the Wieslander Vegetation Type Mapping (VTM) project in California. The VTM collection, created in the 1920s and 1930s, has been described as “the most important and comprehensive botanical map of a large area ever undertaken anywhere on the earth’s surface” (Jepson et al. 2000). It was pioneered by Albert E Wieslander, an employee of the Forest Service Forest and Range Experiment Station in Berkeley, CA. Overall, the collection covers about 28 million ha or just over a quarter of the state including natural areas exclusive of the deserts and the larger agricultural areas. The collection includes over 200 vegetation maps, 18,000 vegetation plots, over 3,000 photographs, and over 100 plant voucher specimens.  It is a detailed, extensive (although not complete), and multi-modal description of the vegetation of California in the early 20th century, and its availability in digital form presents multiple opportunities to examine, characterize and understand changes to California landscapes. Many groups around the state have helped in the digitization, and now the four parts of the collection are reunited in digital space. Here is a nice pic of their coverage in the state. 

The data has been used in many scientific publications, but one of the VTM projects under-sung roles has been to provide a foundation for many subsequent mapping efforts in the state. For example, the protocols developed by Wieslander and his crew became the foundation for the State Cooperative Soil-Vegetation surveys from that covered 4.6 million ha of land during 1947-1977. Those early surveys paved the way for many of the contemporary vegetation classification schemes used today in California, including the Manual for California Vegetation, the National Vegetation Classification System, and the California Gap Analysis Program.

The maps from the State Cooperative Soil-Vegetation Survey of California, which are the first state-wide program to use aerial photography interpretation to produce vegetation and timber maps, have not to our knowledge been digitized, and might add to the understanding of California flora in the post-war period. We are about to embark on a pilot project to examine the feasibility of digitizing and georeferencing these soil-vegetation maps, and adding them to the digital collection. I think the georeferencing process will have to be just an attribution of the marked corner coordinates. Stay tuned! Here are some snap shots so you know what we are up against.

Map examples: a) detail of veg polygons; b) index indicating year of aerial photo used; c) example from timber series of maps Fore more about the VTM project, see my blog posts here, and the vtm.berkeley.edu website. 

New datum for Australia: catch me if you can!

/

In the US, we have revised our geodetic datum over the yearrs. Most famously the switch from NAD27 to NAD83 as instrumention necessitated more measurements and resulted in a more accurate model. But in Australia, they are creating a new datum because the continent is MOVING 7cm a year. 

Read here: http://www.bbc.com/news/technology-36912700

Check it:

The Geocentric Datum of Australia, the country's local co-ordinate system, was last updated in 1994. Since then, Australia has moved about 1.5 metres north.

So on 1 January 2017, the country's local co-ordinates will also be shifted further north - by 1.8m.

The over-correction means Australia's local co-ordinates and the Earth's global co-ordinates will align in 2020.

I for one welcome our ESRI overloards: Wrap Up from the 2016 ESRI User Conference

/

What a full week! Here is my wrap-up from a great 2016 ESRI User Conference.

I haven't been here in many years, and I am glad I came. I learned much, and have some new ideas for workshops and classes, and how IGIS can be more of service to ANR, and I got started on ArcGIS Pro - ESRI's eventual replacement for ArcGIS Desktop. Pro has a completely new user interface that is very clear; you can visualize, edit, and perform analysis in both 2D and 3D; it is super fast via multithreading & 64-bit processing (finally), and it has new icons and a bunch of new processing tools. A bunch of very cool stuff comes with 1.3 soon. 

Day 1: Monday was spent in big-picture, inspirational talks from camera beautiful GIS people. 15,000 in the audience, 10 massive screens. I loved it, I felt like I was at a really intense but sedate rock concert. Note to ESRI: could you put the chairs any closer together? The highlight was speaker keynote Andrea Wulf, talking about her obsession with Alexander Von Humboldt.  Note to self: get the book. In addition to the big picture stuff, Day 1 is ESRI's chance to highlight this year's software improvements as we continue the voyage away from the desktop workflow: Pro, integrated 3D, green design, Apps, seamless integration with the web. 

Day 2: Tuesday focused on workshops. I picked four workshops from the Spatial Statistics Team at ESRI. These were led by Lauren Bennett and her crew (Flora Vale, Jenora D'Acosta). Uniformly fantastic. I had downloaded Pro the night before, and with some trepidation got started and followed along. I am happy to report that it seems very intuitive. I have read elsewhere about worries that there is loss of cartographic control, and I will look into that. I learned about the Spatial Stats toolbox in Pro, and some very new capacity in optimization of pattern analysis (you know how difficult it is to pick that distance kernel), and in the new space-time cube capabilities. The space-time capabilities make very complex analyses doable, and are very exciting, but still a bit constraining if you don't know how to update the code. Oh yeah, and space-time uses netCDF format. 

Day 3: For Wednesday's workshops I chose workshops that would help me update class labs: Python + Raster analysis; Site suitability in Pro; Cost Connectivity in Pro; and some crazy cool prediction tool called Empirical Baysien Kriging, which I will be checking out.  I guess EBK has been around for awhile, but now implemented in ESRI software. The new suite of tools in site suitability + connectivity are going to be key. Kevin M. Johnston and Elizabeth Graham led the Site Suitability and Connectivity, and Eric Krause led the Kriging workshop. 

Day 4: All day was spent in Pix4D immersion with the excellent Pix4D support/training team. Pix4D is the gold standard for drone imagery workflow; it also serves as the backend engine for ESRI's Drone2Map application, which I have not tried. Most of the morning was spent in basics: workflow basics, application domains, super wow factor examples like 0.5cm resolution imagery processing. We also looked at workflow and best practices, accuracy, and some example projects. The room was full of 50+ people, many with specific questions about a range of projects. Got to hang out a bit with Greg Crustinger, who is now at Parrot. Even more excited now about our new Sequoia cameras. 

Thoughts:

  • Little Italy has some great restaurants. 
  • We need to move to Pro soon. Probably not in time for Fall's class, but soon. GIF and IGIS workshops are going to have to get updated. 
  • I need to get more in touch with imagery analysis in Pro. Especially with the segmentation and classification part. 
  • I want to recreate the workflow for site suitability + locate regions + cost connectivity soon. 
  • The ability to perfom complex analyses in a GUI is increasingly easy, but is that a good thing? We have to be increasingly vigilant about training the fundamentals as well. 
  • One frustration about these workshops that I bet my workshop participants share - the data all is perfect, and ready to go. We need to keep talking about where to get data, and how to wrangle it into shape. 
  • Could drones mean the resurrection of photogrammetry? At least in the classroom?

Trends: 

  • Hyper granularity: how do we processes these increasingly fine resolution datasets? 
  • Global to local focus in modeling: GWR, optimized Getis-Ord, empirical baysien kriging all try to deal with and model local variability across a complex study area;
  • Incorporation of permutations and distribution functions in modeling has been made way easier;
  • Big Data, multidimensional data, temporal data: ESRI is really trying to be a better informatics platform for research;
  • ESRI seems to be increasingly relying on external and open standards for new data formats/products; this is a great trend;
  • Decision-making: all these analyses need to support decision-making; communication remains critical, tools for web-based interaction continue to expand.

Searching for patterns in high res imagery: template matching

/

From two friends in the space of a week! While I was away, this tool made the rounds: 

http://sf.terrapattern.com/: This is the alpha version of Terrapattern, a visual search tool for satellite imagery. The project provides journalists, citizen scientists, and other researchers with the ability to quickly scan large geographical regions for specific visual features.  

It is a great deal like some of the template matching routines in Definiens Ecognition among other proprietary software tools.  

Here is an article about it: http://techcrunch.com/2016/05/25/terrapattern-is-a-neural-net-powered-reverse-image-search-for-maps/ 

They say:

Terrapattern is a visual search engine that, from the first moment you use it, you wonder: Why didn’t Google come up with this 10 years ago? Click on a feature on the map — a baseball diamond, a marina, a roundabout — and it immediately highlights everything its algorithm thinks looks like it. It’s remarkably fast, simple to use and potentially very powerful. 

Go ahead and give it a try first to see how natural it is to search for something. How does that work? And how did a handful of digital artists and developers create it — and for under $35,000?

The secret, as with so many other interesting visual computing projects these days, is a convolutional neural network. It’s essentially an AI-like program that extracts every little detail from an image and looks for patterns at various levels of organization — similar to how our own visual system works, though the brain is infinitely more subtle and flexible.

National Park Maps All in One Place

/

Kudos to Matt Holly, a member of the National Park Service’s (NPS) National Resource Stewardship and Science Directorate.  Matt has been uploading all of the NPS maps into a single portal available online.  At the moment these maps are available in GIF, JPEG, and PDF...but maybe shapefiles will follow??  You can search the maps alphabetically by park, or by state.  Access the website here.  

Mapping the Housing Divide

/

The Washington Post, using data from Black Knight Financial Services, recently published an amazing series of maps showing disparities in the United States' housing recoveries.  They argue that these disparities have exacerbated inequality and have particularly worked against Americans of moderate means and minority neighborhoods.  Check the full article out here and explore the maps.  

Wrap up on the Hopland Bioblitz 2016

/

This text excerpted from the Hopland Newsletter:

Over 70 scientists and naturalists descended upon HREC from  April 8-10th in our first Hopland Bioblitz. During the weekend over 400 species from the recently discovered blind silverfish to the characterful kangaroo rat were observed and recorded on the HREC iNaturalist page. 

You can still get involved with our bioblitz efforts by logging onto our iNatualist page and seeing if you can help to identify any unknown species. Enjoy more of our discoveries by taking a look through our photography competition entries.

This bioblitz was supported by a grant from the University of California Agriculture and Natural Resources and was organized by Kip Will, Maggi Kelly, George Roderick, Rosemary Gillespie. IGIS's Shane Feirer helped set up the IT infrastructure for the day.

Hopland Bioblitz is on!

/

Our big Hopland scientific bioblitz is this weekend (9-10 April, with some events on the 8th) and I look forward to seeing many of you there. If you can't make it to HREC, there are many ways you can remotely help us and check out what is happening all weekend long.

HELP US OUT. http://www.inaturalist.org/ Many people will be using iNaturalist to make and share observations. Helping out the effort is easy. Look for observations at the iNaturalist site by searching for "Hopland" in the "Projects" pulldown menu and choose "Hopland Research Extension Center". Once there, you can browse the plants and animals needing identification and needing confirmation. Every identification counts toward our goal of massively increasing the knowledge of the HREC's flora and fauna.

VOTE ON IMAGES.  http://www.hoplandbioblitz.org/ We are hosting an image contest for the plants and animals of HREC. Great prizes will be given  for images that get the most votes(REI gift cards and a GoPro grand prize!). Please visit the site and vote for your favorites frequently during the weekend and share them and then sit back and what the slide show.  

CHECK US OUT. http://geoportal.ucanr.edu/# Our new app will graphically show you our progress for the bioblitz observations. Results will be updated every 15 minutes. See how your favorite groups are doing in the challenge to document as many species as possible.

Look for #HoplandBioblitz on Twitter and Instagram

Follow along on Facebook https://www.facebook.com/HoplandREC/

AAG 2016 wrap up

/
1870s-ish map of SFAside from all the sunshine, architecture, maps, and food, at the 2016 AAG conference Kelly and I participated in four organized sessions on Historical Ecology. On Saturday, we heard from a number of fantastic researchers, and here is my wrap-up.  (Alas, these sessions overlapped with sessions on CyberGIS, Who says you can't be interestd in both? cyber-historical-environmental-spatial-data-science.)
  • History: We heard from researchers working on data from the Holocene, to pre-history, to the 20th century.
  • Focus: Ecosystems included prairie, forests (Maryland, New York, California, Florida, Ohio); and Wetlands (China, California, etc.); Land Use and Agriculture (Mexico, Brazil); Fire (Arizona); and biological collections. 
  • Data included inventory (PLS system: land appraisal value; Cadastral surveys); Imagery (Landsat, aerial imagery); and biological (paleo; tree ring; resurveys; pollen records; bird census; and PLS system: witness trees, survey lines, FIA data). 
  • Methods: Comparison between past and present from existing inventory data, as well as comparison between historic and modern resurveys; digitization of multiple data sources; narrative analysis; ecological modeling; ecosystem services modeling; fire behavior modeling; OBIA of historic imagery; and some really neat modeling work. 
  • Emerging Themes from the sessions included: 
    • Data. Most people used digital data from an existing source - websites, clearinghouse, existing digital source. Many digitized their data. One person used an API. 
    • Accuracy. About half of speakers have thought about, or incorporated understanding of data quality or uncertainty in your work; but this is difficult to do quantitatively. Some people use the 'Multiple lines of evidence' from diverse datasets to increase confidence in results. 
    • Tools. We heard about a number of tools, including GIS as desktop tool, Open tools, Backcasting with landcover models, Complex modeling approaches, One paper used OBIA methods, and one paper discussed Big historic data (maybe moving toward the cyberGIS overlap). 
    • Theoretical frameworks: A few papers used resilience as a framework, social, ecological and coupled; and several papers used a landscape ecology framework. 
    • New terms: I learned a new term: “Terrageny”: a record of how a landscape became fragmented through time, containing information on the ‘ancestry’ of fragments and showing how an initially continuous landscape was progressively divided into fragments of decreasing size. Ewers et al. 2013. Gorgeous word. Must incorporate into cocktail party discussion. 

We also sent out a survey to the speakers prior to the talks, and here are some preliminary results. 

Question: What are the three top challenges that you see for historical ecology research?
  • Data/Logistical/Availability
    • The further back in time we look, the more sparse the data. 
  • Technical 
    • Lack of metadata: Current data deluge may attract attention/urgency away from the discovery and digitization of historical data;
    • Few models capable of incorporating human and environment interactions over long time scales. 
  • Theoretical
    • Maintaining perceived relevance in the context of the novel ecosystem/no-analog system conversation - not having historical ecology be the baby that is thrown out with the bathwater.
  • Operational 
    • Many respondants mentioned issues with funding - these projects are by nature interdisciplinary, often require large programs to fund at achievable levels, and not many funding sources exist.
  • Communication
    • We need to focus on communicating the importance of understanding past conditions to inspire and guide current design proposals. 
Question: What exciting future directions do you envision for historical ecology research?
  • The importance of historical data and analysis:
    • Historical data is essential: Multi- Inter-disciplinary research needs historical research, particularly so that we can understand 1) historical reference conditions, but also so that we can understand 2) when we might have novel interactions between species and ecosphere. 
  • Practicality of historical data and analysis: 
    • Historical ecology is critical for restoration projects and for studying climate change, and for its power to communicate through environmental education with the public.
  • New data/Big data/Data Fusion: 
    • Increase in digitally available historical sources (longer ecological and climate records and reconstructions), plus the availability of large, high-resolution datasets to assess change (thinking LiDAR, government reports, survey data...)  
    • There is also increasing sophistication of analysis and visualization tools.
    • But, the current data deluge may attract attention/urgency away from the discovery and digitization of historical data.

A fantastic time was had by all!

Spatial Data Science Bootcamp 2016!

/

Last week we held another bootcamp on Spatial Data Science. We had three packed days learning about the concepts, tools and workflow associated with spatial databases, analysis and visualizations. Our goal was not to teach a specific suite of tools but rather to teach participants how to develop and refine repeatable and testable workflows for spatial data using common standard programming practices.

2016 Bootcamp participants

On Day 1 we focused on setting up a collaborative virtual data environment through virtual machines, spatial databases (PostgreSQL/PostGIS) with multi-user editing and versioning (GeoGig). We also talked about open data and open standards, and moderndata formats and tools (GeoJSON, GDAL).  On Day 2 we focused on open analytical tools for spatial data. We focused on Python (i.e. PySAL, NumPy, PyCharm, iPython Notebook), and R tools.  Day 3 was dedicated to the web stack, and visualization via ESRI Online, CartoDB, and Leaflet. Web mapping is great, and as OpenGeo.org says: “Internet maps appear magical: portals into infinitely large, infinitely deep pools of data. But they aren't magical, they are built of a few standard pieces of technology, and the pieces can be re-arranged and sourced from different places.…Anyone can build an internet map."

All-in-all it was a great time spent with a collection of very interesting mapping professionals from around the country. Thanks to everyone!