This folder is the Data Acquisition part of the Stock Price Aggregation Model. Since the data is stored in the local relational database, this folder is storing the pipeline files that ingest data to the database. The relational database is Postgre. You may find the tables in the Data folder.
There are 6 Python files and 1 csv file. The following list are the files in this folder:
- IngestSP500_init.py
- IngestSP500CompanyPrice_init.py
- IngestSP500Divisor.py
- IngestSP500ShareOutstanding_init.py
- IngestSP500ShareOutstanding_forerrors.py
- ShareOutstanding_ForErrors.csv
- InsertData.py
Package used:
- pandas
- datetime
- pasycopg2
- yfinance
This is a helper code for insert data into the database. It consists of 7 functions:
- insert_price() - function to insert data to stockprice table
- insert_meta() - function to insert data to stockmeta table
- insert_share() - function to insert data to stockshareoutstanding table
- insert_pred() - function to insert data to insert prediction data
- insert_result() - function to insert predictive model accuracy and meta data
- insert_divisor() - function to insert data to sp500divisor table
- insert_sp500_db - function to insert data to pred_dy_sp500 table
This pipeline obtains the historical closing of S&P 500 from Yahoo Finance and store the entries in stockprice table. Once the data is downloaded from Yahoo Finance, it adds the ticker of S&P 500 in Yahoo Finance format and insert into the table. It relies on insert_price() in InsertData.py. The detail of the database table can be found in the Data folder.
This pipline obtains the historical stock prices of 505 stock prices and stock meta data from Yahoo Finance and store the entries in stockprice table and stockmeta table. The detail of the database table can be found in the Data folder. Once the data is downloaded from Yahoo Finance, it adds the ticker of the companies in Yahoo Finance format and insert into the tables. It relies on insert_price() and insert_meta() in InsertData.py. Beside inserting the entries into the tables, it also returns the error log and timer log which time the work of each stock.
This pipeline obtains the divisor of S&P 500 from Y Chart and store the quarterly divisor into the sp500divisor table. The detail of the database table can be found in the Data folder. It relies on insert_divisor() in the InsertData.py.
This pipeline obtains the floating shares of all S&P 500 components from Yahoo Finance in May, 2020 and store the entries in stockshareoutstanding table. Once the data is downloaded from Yahoo Finance, it adds the entries to the table. It relies insert_share() in InsertData.py. Beside inserting the entries into the tables, it also returns the error log. The detail of the database table can be found in the Data folder.
This pipeline obtains the floating shares of the stocks which is not sucessfully insert in the IngestSP500ShareOutstanding_init.py. ShareOutstanding_ForErrors.csv is the csv files that contains data that is obtained manually that is needed to be inserted into the stockshareoutstanding table. This pipeline takes this csv file and adds the entries into the table. It adds the entries to the table. It relies insert_share() in InsertData.py. Beside inserting the entries into the tables, it also returns the error log. The detail of the database table can be found in the Data folder.
The next step is to train the baseline model with approach 1, you may find more detail in the Baseline Model folder