Automating Your Data Science Tasks In Python (importing CSV files to database AUTOMATION TUTORIAL)

Опубликовано: 18 Февраль 2025
на канале: StrataScratch

24,679

678

This is part 2 on automating the import of CSV files to a database - a common data science task . In this video, we'll take our python script from part 1, which manually imports a CSV/Excel file to a postgres database, and completely automate the script so that it can import multiple files without the need to edit the code. You'd essentially be saving hours of work.

Importing CSV files to a database is a common task needed for data science and analytics and it can be done completely with python using pandas dataframes, numpy, os library, and the posgres database wrapper psycopg2.

This video focuses on automation. If you're interested in build a script to import files to a database, check out the first video in our series: • Solve Data Science Tasks In Python (S... .

Topics Covered:
Automatically identifying your CSVs and handling multiple files using for loops, bash commands, and the os library
Automatically cleaning the table name and column headers using methods like replace() and lower() using format() and %s variables
Automatically converting your pandas dataframe to a CSV file using for loops and dictionaries
Automatically creating multiple database tables and importing your data to a postgres database on Amazon Web Services (AWS) in python

______________________________________________________________________
Resources:

How to setup Anaconda: https://docs.anaconda.com/anaconda/in...
How to install psygopg2: https://anaconda.org/anaconda/psycopg2

👉 Subscribe to my channel: https://bit.ly/2GsFxmA
👉 Practice real data science interview questions: https://platform.stratascratch.com/co...

______________________________________________________________________
Timestamps:

Intro: (0:00)
Identifying CSVs automatically: (2:20)
Automatically making a new directory and moving files: (6:55)
Converting CSV files to pandas dataframes: (13:13)
Automating the cleaning of table names and headers: (18:17)
Automatically connect to database and import files: (25:17)

______________________________________________________________________

Contact:

If you have any questions, comments, or feedback, please leave them here!
Feel free to also email me at [email protected]
______________________________________________________________________