Pyspark - converting json string to DataFrame

Question

I have a test2.json file that contains simple json:

{  \"Name\": \"something\",  \"Url\": \"https://stackoverflow.com\",  \"Author\": \"jangcy\",  \"BlogEntr         


        

Answer 1

You can do the following

newJson = '{"Name":"something","Url":"https://stackoverflow.com","Author":"jangcy","BlogEntries":100,"Caller":"jangcy"}'
df = spark.read.json(sc.parallelize([newJson]))
df.show(truncate=False)

which should give

+------+-----------+------+---------+-------------------------+
|Author|BlogEntries|Caller|Name     |Url                      |
+------+-----------+------+---------+-------------------------+
|jangcy|100        |jangcy|something|https://stackoverflow.com|
+------+-----------+------+---------+-------------------------+

Answer 2

It is apprently part of the "ingestion pipeline" help section,

Therefore, renaming the field @ indexing time, not querying time