blog authors

Welcome to the Kellylab blog

geospatial matters

Please read the UC Berkeley Computer Use Policy. Only members can post comments on this blog.

Monday
Mar232015

Satellites can be vulnerable to solar storms

I don't use ocean color data, but found this report of interest nonetheless. From the HICO website. HICO is the Hyperspectral Imager for the Coastal Ocean.

HICO Operations Ended. March 20, 2015

In September 2014 during an X-class solar storm, HICO’s computer took a severe radiation hit, from which it never recovered.  Over the past several months, engineers at NRL and NASA have attempted to restart the computer and have conducted numerous tests to find alternative pathways to communicate with it.  None of these attempts have been successful.  So it is with great sadness that we bid a fond farewell to HICO.

Yet we rejoice that HICO performed splendidly for five years, despite being built in only 18 months from non space-hardened, commercial-off-the-shelf parts for a bargain price.  Having met all its Navy goals in the first year, HICO was granted a two-year operations extension from the Office of Naval Research and then NASA stepped in to sponsor this ISS-based sensor, extending HICO’s operations another two years.  All told, HICO operated for 5 years, during which it collected approximately 10,000 hyperspectral scenes of the earth.

Most of the HICO scenes taken over sites worldwide are available now, and will remain accessible to researchers through two websites:  http://oceancolor.gsfc.nasa.gov/ and http://hico.coas.oregonstate.edu.  HICO will live on through research conducted by scientists using HICO data, especially studies exploring the complexities of the world’s coastal oceans.

Wednesday
Mar112015

Mapsense talk at BIDS for your viewing pleasure

Here is Erez Cohen's excellent talk from the BIDS feed: http://bids.berkeley.edu/resources/videos/big-data-mapping-modern-tools-geographic-analysis-and-visualization

Title: Big Data Mapping: Modern Tools for Geographic Analysis and Visualization

Speaker: Erez Cohen, Co-Founder and CEO of Mapsense

We'll discuss how smart spatial indexes can be used for performant search and filtering for generating interactive and dynamic maps in the browser over massive datasets. We'll go over vector maps, quadtree indices, geographic simplification, density sampling, and real-time ingestion. We'll use example datasets featuring real-time maps of tweets, California condors, and crimes in San Francisco. 

The BIDS Data Science Lecture Series is co-hosted by BIDS and the Data, Science, and Inference Seminar. 

About the Speaker

Erez is co-founder and CEO at Mapsense, which is builds software for the analysis and visualization of massive spatial datasets. Previously Erez was an engineer at Palantir Technologies, where he worked with credit derivatives and mortgage portfolio datasets. Erez holds a BS/MS from UC Berkeley's Industrial Engineer and Operations Research Department. He was a PhD candidate in the same department at Columbia University.

Wednesday
Mar112015

print 'Hello World (from FOSS4G NA 2015)'

FOSS4G NA 2015 is going on this week in the Bay Area, and so far, it has been a great conference.

Monday had a great line-up of tutorials (including mine on PySAL and Rasterio), and yesterday was full of inspiring talks.  Highlights of my day: PostGIS Feature Frenzy, a new geoprocessing Python package called PyGeoprocessing, just released last Thurs(!) from our colleagues down at Stanford who work on the Natural Capital Project, and a very interesting talk about AppGeo's history and future of integrating open source geospatial solutions into their business applications. 

The talk by Michael Terner from AppGeo echoed my own ideas about tool development (one that is also shared by many others including ESRI) that open source, closed source and commercial ventures are not mutually exclusive and can often be leveraged in one project to maximize the benefits that each brings. No one tool will satisfy all needs.

In fact, at the end of my talk yesterday on Spatial Data Analysis in Python, someone had a great comment related to this: "Everytime I start a project, I always wonder if this is going to be the one where I stay in Python all the way through..."  He encouraged me to be honest about that reality and also about how Python is not always the easiest or best option.

Similarly, in his talk about the history and future of PostGIS features, Paul Ramsey from CartoDB also reflected on how PostGIS is really great for geoprocessing because it leverages the benefits of database functionality (SQL, spatial querying, indexing) but that it is not so strong at spatial data analysis that requires mathematical operations like interpolation, spatial auto-correleation, etc. He ended by saying that he is interested in expanding those capabilities but the reality is that there are so many other tools that already do that.  PostGIS may never be as good at mathematical functions as those other options, and why should we expect one tool to be great at everything?  I completely agree.

Saturday
Feb282015

10-year anniversary for the GIF

I'm musing, contemplating and writing on the decade 2005-2015, as this is the GIF's 10-year anniversary. What a decade it was. Here I'll post and add to some of the key events that helped transform mapping (and the GIF) in the last 10 years.

Key background events

  • 1996. Mapquest launched.
  • 1997. Skynet becomes self-aware.
  • May 2000. Selective Availabilility on GPS turned off, leading the way for GPS in smartphones.
  • The Scan Line Corrector (SLC) on the Landsat 7 ETM+ instrument failed May 31, 2003.
  • 2004. Open Street Map founded.
  • March 2004. Yahoo! maps launched, first slippy maps (click and drag to pan and zoom the map).
  • 2004. NASA releases WorldWind.
  • October 2004. Google acquires Where 2 allowing AJAX map tiling to a desktop client.
  • October 2004. Google acquires Keyhole.

What made 2005 such a crazy year

  • Google Maps launches in February, and goes mobile in April.
  • The first mashup: Paul Rademacher's Housingmaps.org. His original post on Craigslist asking for feedback: https://forums.craigslist.org/?ID=26638141
  • Google Maps API launches in June.
  • NASA's Blue Marble Next Generation released.
  • Google Earth launches in June.
  • Hurricane Katrina hits in August. Simple webmaps for the disaster proliferate, and ESRI and GE get on the scene.
  • Kellylab's first blog post in September.
  • GIF launches and hosts our first GIS Day in November with Michael Jones, formerly of Keyhole.
  • The back-up solar array drive on Landsat 5 began failing and was not able to provide the power needed to charge the batteries. November 26.

Where we are in 2015

We've gone through a number of transitions in the world of mapping:

  • Data have transitioned from being siloed, and found in clearinghouses to being open and provided through APIs.
  • We’ve moved from desktop computing to cloud computing.
  • Webmaps have transitioned from using proprietary stacks to networks with multiple open and proprietary options.
  • We’ve moved from imagery gathered monthly or seasonally to daily; footprints are smaller, and our focus has shifted from local focus to global coverage.
  • Our planimetric 2D view is changing with lidar and radar sensors.
  • Visualization has moved from static cartography or simple animations to dynamic interactive visualization.
  • Finally, mapped content is no longer anonymous or regulated, but highly personal and narrative.

Key GiF milestones:

  • 2005 GIIF (Geospatial Imaging and Informatics Facility) launches
  • 2006 OakMapper changes from ArcIMS to Google Earth API
  • 2008 GIIF becomes GIF
  • 2008 OakMapper 2.0 launches
  • 2008 SNAMP website launches
  • 2011 Cal-Adapt goes live
  • 2013 EcoEngine/HOLOS goes live
  • 2014 LandCarbon launches
  • 2014 GIF and Cal-Adapt go to the White House
  • 2014 vtm.berkeley.edu goes live, built from the HOLOS API
  • 2015 Spatial Data Science bootcamp in May

Onwards and upwards!

Saturday
Feb282015

A brief history of the digital map

Nice consise history of mapping from his lecture "The Ubiquitous Digital Map" by Gary Gale, Director of Global Ccommunity Programs, HERE.

http://www.vicchi.org/2013/02/19/the-ubiquitous-digital-map-abridged/

Thursday
Feb262015

Lidar + hyperspectral, ortho and field data released by NEON

http://www.neoninc.org/data-resources/get-data/airborne-dataFrom LASTools list:

The National Ecological Observatory Network (NEON) published this week airborne remote sensing data including full waveform and discrete return LiDAR data and LiDAR derivatives (DTM, DSM, CHM) as well as corresponding hyperspectral data, orthophotos, and field data on vegetation strucutre, foliar chemistry and ASD field spectra.

NEON Airborne Data Set
.

Wednesday
Feb252015

Questions about the Spatial Data Science Bootcamp? Read on!

In May, the GIF will be hosting a 3-day bootcamp on Spatial Data Science.

What is the significance of Spatial Data Science?

We live in a world where the importance and availability of spatial data is ever increasing, and the value of Spatial Data Science: big data tools, geospatial analytics, and visualization is on the rise. There are many new and distributed tools available to the geospatial professional, and the ability to efficiently evaluate and integrate the wide array of options is a critical skill for the 21st century marketplace.  Spatial Data Science offers a modern workflow that includes the integration of data from multiple sources and scales; with open-source and web-based technology for robust data analysis and publication; with core spatial concepts and application of spatial analysis methods; and allows for the collaborations of people – companies, scientists, policy-makers, and the public.

Why come to the GIF to learn about it?

The Geospatial Innovation Facility (GIF) at UC Berkeley is the premier research and educational facility in the Bay Area that focuses on a broad vision of Spatial Data Science. The GIF has a decade-long history of successful GIS and remote sensing research projects. The GIF has also trained many students, researchers, and community members in geospatial techniques and applications through our popular workshop series and private consultation. With more recent advances in web-based mapping capabilities, the GIF has been at the forefront of complex web-based spatial data informatics (web-based data sharing and visualization), such as the Cal-Adapt  tool, which provides a wealth of data and information about California’s changing climate. Participants will get the benefit of our decade-long focus on Spatial Data Science: collaborative project development, rigorous spatial analysis methods, successful interaction with clients, and delivery of results to project managers, the public, and other stakeholders.

What are the key elements of the Bootcamp?

This Bootcamp is designed to familiarize participants with some of the major advances in geospatial technology today: big data wrangling, open-source tools, and web-based mapping and visualization. You will learn how and when to implement a wide range of modern tools that are currently in use and under development by leading Bay Area mapping and geospatial companies, as well as explore a set of repeatable and testable workflows for spatial data using common standard programming practices. Finally, you will learn other technical options that you can call upon in your day-to-day workflows. This 3-day intensive training will jump start your geospatial analysis and give you the basic tools you need to start using open source and web-based tools for your own spatial data projects.  

Interested in integrating open source and web-based solutions into your GIS toolkit? Come join us at our May 2015 Bootcamp: Spatial Data Science for Professionals. Applications due: 3/16/2015. Sign up here!

Monday
Feb232015

Information on the GIST Minor and Graduate Certificate

Hi all,

Our gis.berkeley.edu website had to be taken down. Information on the GIST Minor and Graduate Certificate can be found here:

 Thanks!

Friday
Feb202015

List of Online Geospatial Data

From Sean's IGIS workshops this week.

Base Layers

Land Cover and Wildlife Habitat

Imagery

Soils

Climate and Weather Data

California Geopolitical Boundaries

Digital Elevation Models

Thursday
Feb192015

Geolunch Seminar Talk: Spatial Data Analysis with Python

Thanks to all who attended today!  Here is a link (modified after FOSS4G NA 2015) to the slides, and below are more links for some of the free resources that were highlighted.

Code Academy (programming tutorials): http://www.codecademy.com/
Coursera (full courses): https://www.coursera.org/courses?query=python
Python wiki pages: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
https://docs.python.org/2/tutorial/
Python at Berkeley (DLab): http://python.berkeley.edu/learning_resources.html
Python Books and Training: http://pythonbooks.revolunet.com/
http://www.learnpython.org/
ArcPy tutorials from ESRI: http://training.esri.com/gateway/index.cfm?fa=catalog.
webCourseDetail&courseid=2520

http://training.esri.com/gateway/index.cfm?fa=catalog.webCourseDetail&courseid=2523

List of spatial Python packages: https://github.com/SpatialPython/spatial_python/blob/master/packages.md

If you are a graduate student, check out the graduate student-led workshops for Spring 2015: http://goo.gl/forms/rUceY1I67n