Blogs

Introduction In today’s digital era, businesses are increasingly adopting cloud computing to scale their operations, enhance flexibility, and reduce costs. Among the major cloud service providers, Amazon Web Services (AWS), Microsoft Azure, Google Cloud, and Oracle Cloud have emerged as dominant players in the market. Each offers a comprehensive cloud technology stack tailored to meet different business needs. In this blog, we’ll conduct a thorough comparison of these leading cloud technology stacks to help you make an informed decision when choosing the best-fit cloud provider for your organization.

Embracing Kubernetes, Goodbye Spring Cloud

in digest

June 19, 2021

I believe many developers, after familiarizing themselves with microservices, realized that they thought they had successfully built a microservices architecture empired with Spring Cloud. But after the popular of kubernetes (K8S), they were curious and exciting of creating the cloud native microservices serivces. The Era of Spring Boot and Cloud In October 2012, Mike Youngstrom created a feature request in Spring Jira to support a containerless web application architecture in the Spring Framework.

Spark SQL in Depth

in digest

April 28, 2021

In this article, we’ll look at how Spark SQL working on data queries in depth. Checking Execution Plan Data Preparing create database if not exists test; create table if not exists test.t_name (name string); insert into test.t_name values ('test1'),('test2'),('test3'); Test Code Preparing Below Scala code is used with testing with blocking at the standard input at the end. In this case, we can see more details from Spark WebUI.

Apache Spark 3.1.1 Released :)

in digest

March 10, 2021

Apache Spark 3.1.1 is released on March 2, 2021. It is milestone release for Spark in 2021. This version of spark keeps making it more efficient and stable. Below are highlighted new features and changes. Python usability ANSI SQL compliance Query optimization enhancements Shuffle hash join improvements History Server support of structured streaming Project Zen Project Zen was initiated in this release to improve PySpark’s usability in these three ways:

Apache Superset:)

in digest

January 22, 2021

On January 21, 2021, Apache’s official announced that Apache® Superset™ has become a top-level project. Apache® Superset™ is a modern big data exploration and visualization platform that allows users to build dashboards quickly and easily using a simple code-free visualization builder and the most advanced SQL editor. The project was launched on Airbnb in 2015 and entered the Apache incubator in May 2017. Apache Superset is a big data-related BI visualization tool.

Spark SQL Read/Write HBase

in digest

January 1, 2020

Apache Spark and Apache HBase are very commonly used big data frameworks. In many senarios, we need to use Spark to query and analyze the big volumn of data in HBase. Spark has wider support to read data as dataset from many kinds of data source. To read from HBase, Spark provides TableInputFormat, which as following disadvantages. There is only on scan triggerred in each task to read from HBase TableInputFormat does not support BulkGet Cannot leverage the optimization from Spark SQL catalyst Considering the above points above, there is another choice by using Hortonworks/Cloudera Apache Spark—Apache HBase Connector short for (SHC).

Apache Airflow Overview

in article

December 15, 2019

What is Airflow? Airflow is a platform to programmaticaly author, schedule and monitor workflows or data pipelines. It composes Directed Acyclic Graph (DAG) with multiple tasks which can be executed independently. The Airflow scheduler executes the tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

Apache Kafka Producers

in digest

November 2, 2019

Kafka producers send records to topics. The records are sometimes referred to as messages. The producer picks which partition to send a record to per topic. The producer can send records round-robin. The producer could implement priority systems based on sending records to certain partitions based on the priority of the record. Generally speaking, producers send records to a partition based on the record’s key. The default partitioner for Java uses a hash of the record’s key to choose the partition or uses a round-robin strategy if the record has no key.

Flink Windows Explained

in article

October 5, 2019

Overview Apache Flink supports data analysis over specific ranges in terms of windows. It supports two ways to create windows, time and count. Time window defines windows by specific time range. Count window defines windows by specifc number of envents. In addition, there are two windows time attributes. size: how long the window itsef last interval: how long between windows Whenever the window size = interval, this type of windows are called tumbling-window.

The Complete SQL Tuning

in article

September 2, 2019

The most practice comes for MySQL server, but it applies to other relational database as well. Aviod full table scan and try to create index on the columns used after where or order by. Aviod check null after where clause. You set set null as default value when creating tables. However, mostly we should use not null value or use special value, such as 0 or -1 for instead.

Leading Cloud Tech. Stack Comparison

Embracing Kubernetes, Goodbye Spring Cloud

Spark SQL in Depth

Apache Spark 3.1.1 Released :)

Apache Superset:)

Spark SQL Read/Write HBase

Apache Airflow Overview

Apache Kafka Producers

Flink Windows Explained

The Complete SQL Tuning

Search

Categories

Tags