The Mid Week News 09/05/2018 edit   discuss  

It’s time for the news again. One one tech release this week, but we have some reading for you…

Technology updates (details are on the relevant technology pages):

Other technology news:

  • Google Cloud Composer, and orchestration service based on Apache Airflow is now available on the Google Cloud Platform - link; [Datanami view]9https://www.datanami.com/2018/05/01/apache-airflow-to-power-googles-new-workflow-service/)
  • Confluent now support running their Confluent Platform (bsaed on Kafka on Kubernetes, although it’s only available in their commercial version - link; Datanami view
  • I’d like to talk about how to test data pipelines in more detail at some point, but this is right up my ally - a new framework called Great Expectations for defining tests or checks that run as part of the pipeline - link
  • From Cloudera, how to backup and recover data in Apache Solr - link
  • From Datanami - DataTorrent, the company behind Apache Apex and their commercial version of it (DataTorrent RTS) has gone under - link
  • Again from Datanami - looks like Netflix has moved its Keystone data pipeline from Samza to Apache Flink
  • An update from Microsoft on CosmosDB - link