PySpark Schema Generator - A simple tool to generate PySpark schema from JSON data

 Hi Folks,

I built a small tool that solves a problem for a data engineer while dealing with JSON data. As we know JSON data is semi-structured and we always ingest them and denormalize them to smaller tables properly for further processing.


In my case I had to generate PySpark Schema from JSON to ingest the data and the JSON structure often gets changed.

The JSON I was dealing was very complex but let me give you an example about the tool, what problem it solves.

For example we have a JSON coming from Kafka like below

{
  "name": "PREETish ranjan",
  "dob": "2022-03-04T18:30:00.000Z",
  "status": "active",
  "isActive": true,
  "id": 102,
  "address": {
    "city": "Bhubaneswar",
    "PIN": 500016
  },
  "mobiles": ["8989898989", "5656565656"],
  "id_cards": [1, 2, 3, 4, 5]
}

The output i need is like this,

StructType([
    StructField('name', StringType(), True),
    StructField('dob', StringType(), True),
    StructField('status', StringType(), True),
    StructField('isActive', BooleanType(), True),
    StructField('id', IntegerType(), True),
    StructField('address', StructType([StructField('city',
                StringType(), True), StructField('PIN', IntegerType(),
                True)]), True),
    StructField('mobiles', ArrayType(StringType()), True),
    StructField('id_cards', ArrayType(IntegerType()), True),
    ])

If the JSON gets more complex and big its quite difficult to generate the schema if you are dealing the data like this. My tool is a simple javascript tool that generates this for you. It has few bugs and limitations but it works.

Checkout the GitHub: PySpark Schema Generator

Here is the tool:



Thanks for reading!!!

Comments

Post a Comment

Popular posts from this blog

Use SCSS with ASP.NET Core 5.x or 3.X

Image
Here we will see how we can use SCSS or SASS with ASP.NET Core 3.X or 5.X Here is the step to step guide for using SCSS in ASP.NET Core. Step 1: Install sass as global using npm: npm install --global sass Bash Copy Step 2: Create a new ASP.NET Core project , choose MVC or Razor Pages Add a SCSS file to the Web app, here I have added test.scss to the wwwroot/css folder. Add the following lines to the .csproj file to enable SCSS to CSS compilation in the web app. For  node-sass  and  ruby-sass  write this: sass --no-source-map $( ProjectDir ) wwwroot/css/test2.scss $( ProjectDir ) wwwroot/css/test2.css Bash Copy If you're using  dart-sass If you're using dart-sass the usage is --no-source-map sass --no-source-map $( ProjectDir ) wwwroot/css/test2.scss $( ProjectDir ) wwwroot/css/test2.css Bash Copy   < Target Name = " ScsstoCss " BeforeTargets = " Build " > < Exec Command = " mkdir $(ProjectDir)wwwroot/css " Condition = " !Exist
Read more

Building a Login Flow with .NET MAUI

Image
​ Let's build a Login Flow with .NET MAUI with Shell. Authentication in any mobile app is very common. Lets get started with this. Its obvious that it should ask for login only if it isn't authenticated. We will check for authentication , if not there we will move to Login page if login is success we will move to the Home page. For this example we will override the backbutton pressed event to quit the application but you can customize accordingly as per your need. For this post I am using a simple authentication but you can use JWT or any method you want.  Here is the example of the login flow:       All the pages that has to be used needs  to be registered with the Shell. If you are a bit familier with the Shell navigation the first content page is the one which is displayed after startup. So we need to structure the shell accordingly in order. The pages we are using here for the example: LoadingPage LoginPage HomePage SettingsPage Here is the AppShell.xaml <
Read more

Generate PySpark Schema dynamically in Python from JSON Sample

 Hi Folks, If you need to genarate pyspark schema from JSON you can always my tool here  https://preetranjan.github.io/pyspark-schema-generator/ but if you need to do it in Python then here is the code snippet for it. It takes a python dictionary as input and generates the PySpark schema. import json from pyspark.sql.types import * def GeneratePySparkSchema ( json ):     fields = []     for key , value in json.items ():         if isinstance ( value , dict ):             field = StructField ( key , GeneratePySparkSchema ( value ), True )         elif isinstance ( value , list ):             if len ( value ) == 0 :                 field = StructField ( key , ArrayType ( StringType ()), True )             elif isinstance ( value [ 0 ], dict ):                 field = StructField (                     key , ArrayType ( GeneratePySparkSchema ( value [ 0 ]), True )                 )             else :                 field = StructField ( key , ArrayType ( GetSparkData
Read more