Content-Based Image Retrieval using Pivotal HD with HAWQ

featured-image-lake The number of images and videos captured by humans using digital cameras is staggering. It has been estimated that an average of 350 million photos are uploaded to Facebook daily, and about 100 hours of video are uploaded to Youtube every minute. This has resulted in enormous “image and video lakes,” a term we use to describe these ever-growing collections of images and videos. Such gigantic image lakes are not unique to consumer services —they are also found in domains such as healthcare and astronomy. In this post, we will demonstrate how a Content-Based Image Retrieval system can be easily and efficiently realized using Pivotal’s Hadoop Distribution (Pivotal HD) with HAWQ, our SQL engine for Hadoop.

Staging Changes to cf-release

featured-CF-logo-full This is the final post in our CI pipeline series. This time, we talk about how the pipeline stages loggregator-related changes to cf-release to other cf-release consumers (e.g. other teams) in a minimal-impact way. We also show how the pipeline uses git to rebase and merge the various repositories and branches that flow into it.

This Month In Data Science

featured-datascience-august The month of August saw a plethora of data science-related news and announcements, including a Wired feature on Kaggle founder Jeremy Howard's quest to improve health care, General Electric's success in bringing analytics-derived insights to its products for the railroad, airline, hospital, and utility industries, a provocative interview with OKCupid founder Christian Rudder, and much more. Here's our roundup of the major data science news of the month, both from Pivotal and the wider industry.

Pivotal and VMware Announce New Collaborations, Integrations, and Services at VMworld 2014

featured-vmworld-stars VMware kicked off VMworld, its annual week-long showcase in San Francisco, on Sunday with a number of significant product and partnership announcements. Aligned with Pivotal's goal to empower enterprises to develop and deliver modern applications on an open cloud platform, VMware is pushing for a new model of IT that supports hybrid cloud platforms and drives agile, mobile-first companies.

Platform Wind Tunnel, Part I: Heroku


For well over 100 years, engineers have utilized wind tunnels to study the effects of aerodynamic forces on airplanes, buildings, automobiles, and other structures. This testing is conducted as part of a full suite of analyses intended to demonstrate, among other things, the ability of these structures to withstand various forms of stress.

Read more »

Optimizing Bosh-Lite Deployment Times

featured-polycloud-green This post continues our CI pipeline series. In this edition, we talk about how we optimized the pipeline’s Cloud Foundry deployment step (into bosh-lite on AWS) from previously over one hour to about 20 minutes, resulting in shorter feedback cycles and therefore faster development.

Using Docker For CI Pipelines

featured-dcloud A previous post explained CI pipelines for loggregator, and this post zooms in on why the Cloud Foundry engineering team uses Docker and how. A background is given and covers the problems faced without Docker and the benefits. Then, one of our top engineers, Johannes Petzold, explains how we use it and the issues we've seen.

Docker Service Broker for Cloud Foundry

featured-clouds-whale Pivotal's Cloud Foundry team is a fan of Docker. In this post, learn how the Docker service broker for Cloud Foundry allows users to expose and provision services offerings that run inside a Docker container and bind applications to those services. One of Pivotal's expert Docker and Cloud Foundry engineers covers everything from the broker use cases for multi-database engine tests, log draining, known limitations, and information to get started using Docker and Cloud Foundry.

Session Replication on Cloud Foundry

featured-polycloud-green Recently, the Java Buildpack added support for session replication using Redis. This support complements the existing session affinity (i.e. sticky sessions) that Cloud Foundry already supports. So now, in addition to having sessions routed to the instance they were created on, sessions will also survive the loss of that instance. The Java Buildpack will auto-configure Redis-based Session Replication for you when it notices that you’ve bound to a specially identified Redis service. This post describes how this works.

Experience NO LIMITS: Join Pivotal at VMware VMworld US 2014

featured-vmworld-NoLimits This weekend Pivotal will join forces with VMware and our collective customers at VMworld 2014. Pivotal will lead six sessions covering various uses and areas of Cloud Foundry and Pivotal CF. Attendees will also be the first to get a preview of the new Pivotal CF Mobile Services which will accelerate how companies can develop mobile applications, by offering critical services for things like push notifications, API gateways and data sync that are built into the platform. Additionally, you can find Pivotal experts demoing Pivotal products such as the Pivotal Big Data Suite and Pivotal CF in booth #941, located towards the center by the VMware booths.