Home Spark와 Data Engineering 관련 프로젝트 시작하기
Post
Cancel

Spark와 Data Engineering 관련 프로젝트 시작하기

What Is Spark?

Open-source distributed general-purpose cluster-computing framework.

How to start data engineering projects?

  1. Choose any framework, let’s say Kafka.
  2. Write some codes using that framework.
  3. Keep Expanding (try adding other technology)

Project Idea : Creating Real Time REST API

  1. crawls data from popular websites like Twitter,Forex.
  2. store them in buffer (by producer in Kafka).
  3. store them in MySQL database (by consumer in Kafka).
  4. Web server provides real-time REST API.
This post is licensed under CC BY 4.0 by the author.

Apache Storm

Order of execution of a Query (쿼리의 실행순서)

Comments powered by Disqus.

Trending Tags