What Is Spark?
Open-source distributed general-purpose cluster-computing framework.
How to start data engineering projects?
- Choose any framework, let’s say Kafka.
- Write some codes using that framework.
- Keep Expanding (try adding other technology)
Project Idea : Creating Real Time REST API
- crawls data from popular websites like Twitter,Forex.
- store them in buffer (by producer in Kafka).
- store them in MySQL database (by consumer in Kafka).
- Web server provides real-time REST API.