Posts

Showing posts with the label Python

SSIS - Import data from python Rest API into SQL Server

Question: I am trying to take a Python script I wrote that makes  GET   requests utilizing a REST API and returns data in the form of JSON and then have that data be inserted into a SQL server that I will have to create. This job will need to run each day at least once. I am not familiar with creating tables in MSSQL let alone creating a SSIS package or working with ETL. I would appreciate some direction as to how to do this and how realistic it is for somebody with little actual experience, but a good understanding of the process itself conceptually. My end goal is to import this data into a Power BI file for live reporting. Answer: I don't think you will get a very detailed answer, since you are not showing any trial you have done and it looks like you need an answer from scratch. What i will do is to mention some of the approaches you can use to achieve your goal and i will provide some links for each approach to get more information on it (1) Why SSIS? I di...

Working With Apache Spark, Python and PySpark

Image
1. Environment ·           Hadoop  Version: 3.1.0 ·           Apache Kafka Version: 1.1.1 ·           Operating System: Ubuntu 16.04 ·           Java Version: Java 8 2. Prerequisites Apache Spark requires Java. To ensure that Java is installed, first update the Operating System then try to install it: sudo  apt-get update sudo  apt-get –y upgrade sudo  add-apt-repository -y ppa:webupd8team/java sudo  apt-get install oracle-java8-installer  3. Installing Apache Spark 3.1. Download and install Spark First, we need to create a directory for apache Spark. sudo   mkdir  /opt/spark Then, we need to download apache spark binaries package. wget  “ http://www-eu.apache.org/dist/spark/spark-2.3.1/spark-2.3.1-bin-hadoop2.7.tgz ” Next,...