Inserting data from NYT API into DB

Hi all,

my first post here. I am working on a personal test project building a MVC framework to insert NYT Book API data into a database and display the content in the front end. I am using XAMPP for this project.

Currently the categories are loaded onto the database, however I wanted to include the buy links for each book as well by adding a foreign key book_id and populating it with the collected id from lastInsertId() of the book. However I am running into maximum execution exceeded errors.

What am I doing wrong?

How would I also limit the uploads to the DB? Since currently with each refresh it’s reloading the data in the DB.

    public function getBooks()
    {

        $apiKey = 'apikey';

        $genresURL = "https://api.nytimes.com/svc/books/v3/lists/names.json?api-key=" . $apiKey;
        $response = file_get_contents($genresURL);
        $genresData = json_decode($response, true);

        $genres = [];

        foreach ($genresData['results'] as $genre) {
            $genres[] = str_replace(' ', '-', $genre['list_name']);
        }

        $sqlGenre = "INSERT INTO genres (name) VALUES (:name) ON DUPLICATE KEY UPDATE name = name;";

        foreach ($genres as $genre) {
            $this->db->query($sqlGenre);
            $this->db->bind(':name', $genre);

            try {
                $this->db->execute();
            } catch (Exception $e) {
                echo "Error inserting genre: " . $e->getMessage();
            }
        }

        foreach ($genres as $genre) {
            $genresApi = "https://api.nytimes.com/svc/books/v3/lists/current/" . urlencode($genre) . ".json?api-key=" . $apiKey;
            $resp = file_get_contents($genresApi);
            $booksData = json_decode($resp, true);

            if (!empty($booksData) && isset($booksData['results']['books'])) {
                foreach ($booksData['results']['books'] as $bookDetail) {

                    $publishedDate = $booksData['results']['published_date'];

                    $sql = "INSERT INTO books (title, isbn13, isbn10, author, description, publisher, published_date, cover_image_url, page_count, language, average_rating, rating_count, availability_status, availability_count, format, edition, contributor, price, age_group, book_image_url, amazon_product_url, rank, rank_last_week, weeks_on_list, book_image_width, book_image_height, book_uri)
                        VALUES (:title, :isbn13, :isbn10, :author, :description, :publisher, :published_date, :cover_image_url, :page_count, :language, :average_rating, :rating_count, :availability_status, :availability_count, :format, :edition, :contributor, :price, :age_group, :book_image_url, :amazon_product_url, :rank, :rank_last_week, :weeks_on_list, :book_image_width, :book_image_height, :book_uri)
                    
                 
                    $this->db->query($sql);                    

                    $this->db->bind(':title', $bookDetail['title'] ?? '');
                    $this->db->bind(':isbn13', $bookDetail['primary_isbn13']);
                    $this->db->bind(':isbn10', $bookDetail['primary_isbn10']);
                    $this->db->bind(':author', $bookDetail['author']);
                    $this->db->bind(':description', $bookDetail['description']);
                    $this->db->bind(':publisher', $bookDetail['publisher']);
                    $this->db->bind(':published_date', $publishedDate);
                    $this->db->bind(':rank', $bookDetail['rank']);
                    $this->db->bind(':rank_last_week', $bookDetail['rank_last_week']);
                    $this->db->bind(':weeks_on_list', $bookDetail['weeks_on_list']);
                    $this->db->bind(':book_image_width', $bookDetail['book_image_width']);
                    $this->db->bind(':book_image_height', $bookDetail['book_image_height']);
                    $this->db->bind(':book_uri', $bookDetail['book_uri']);
                    $this->db->bind(':cover_image_url', $bookDetail['cover_image_url'] ?? '');
                    $this->db->bind(':page_count', $bookDetail['page_count'] ?? '');
                    $this->db->bind(':language', $bookDetail['language'] ?? '');
                    $this->db->bind(':average_rating', $bookDetail['average_rating'] ?? '');
                    $this->db->bind(':rating_count', $bookDetail['rating_count'] ?? '');
                    $this->db->bind(':availability_status', $bookDetail['availability_status'] ?? '');
                    $this->db->bind(':availability_count', $bookDetail['availability_count'] ?? '');
                    $this->db->bind(':format', $bookDetail['format'] ?? '');
                    $this->db->bind(':edition', $bookDetail['edition'] ?? '');
                    $this->db->bind(':contributor', $bookDetail['contributor']);
                    $this->db->bind(':price', $bookDetail['price']);
                    $this->db->bind(':age_group', $bookDetail['age_group']);
                    $this->db->bind(':book_image_url', $bookDetail['book_image'] ?? '');

                    try {
                        $this->db->execute();
                        $last_id = $this->db->lastInsertId();
                                     
                        foreach ($bookDetail['buy_links'] as $buyLink) {
                            $buyLink_sql = "INSERT INTO buy_links (book_id, name, url) VALUES (:book_id, :name, :url)";
                            $this->db->query($buyLink_sql);
                
                            $this->db->bind(':book_id', $last_id);
                            $this->db->bind(':name', $buyLink['name']);
                            $this->db->bind(':url', $buyLink['url']);
                
                            $this->db->execute(); 
                        }
                    } catch (Exception $e) {
                        echo "Error inserting book: " . $e->getMessage();
                    }

                }
            }

            sleep(15);
        }
    }

The current api documentation doesn’t show any buy links data. However, the example code I’m about to post shows how you would do this if there is related data.

Is the 15 second sleep between genres necessary? If there are just 2 genres, this would exceed the default max execution time of 30 seconds.

The normal way of handling the timeout for long executing scripts is to call set_time_limit(), with a reasonably small value, inside the main loop to keep resetting the time limit. This will limit the time per loop, but allow the script to continue to run until it completes.

I assume you mean this is inserting duplicate data into the books table? The correct way of doing this is to define a column, for one of the pieces of data that will always have a value that won’t ever change after it has been inserted, to be a unique index, then use an INSERT … ON DUPLICATE KEY UPDATE … query so that new data will get inserted, and any existing data will get updated. You can get the id of the row that get updated by using the MySql LAST_INSERT_ID() function in the UPDATE part of the query.

Related to the above paragraph, the existing INSERT … ON DUPLICATE KEY UPDATE … query for the genres table should instead be an INSERT IGNORE … query, since there’s only one column involved.

Some other points for the posted code -

  1. The data that this code is operating on doesn’t match the current api documentation. Either the documentation is wrong or this code is based on some other version of the api.
  2. If you have more than about 2-3 data fields, you should use a data-driven design to dynamically operate on the data, where you have a defining data structure (array) that controls what the code does. This will eliminate writing out line after line of code for every field.
  3. Using a data-driven design will let you match the api data simply by editing the defining array.
  4. Your query has a column for amazon_product_url, but there is no data being bound for this column (there is data in the json response.) Using a data-driven design will eliminate these type of mistakes, because you will be dynamically building the sql query and supplying the data based on the defining array.
  5. There can be more than 20 books per list. There is a num_results value in the response that indicates how many books are in the list. You need to use pagination to get all of them if there is more than 20. You would add &offset=[0,20,40] to the query string and loop until you have gotten all the books in a list.
  6. One of the points of using a prepared query is the sql query statement gets sent to the database server only once, where it is parsed and its execution is planed only once. You should prepare each query only once, before the start of any looping, then just execute the query inside the loop with each set of input data. By re-preparing and binding inputs inside the looping, you are not benefiting from this feature of prepared queries.
  7. If you are not already doing so, when you make the database connection, you should set the emulated prepared query setting to false (you want to run real prepared queries.)
  8. By using a wrapper class for the PDO extension, you cannot prepare more than one query at a time. The PDO extension is already a well written collection of three class. There’s no good reason to create a wrapper class for it.
  9. Explicit binding of input data only occurs within the php driver and is a waste of processing time. Use implicit binding and simply supply an array of the input values to the ->execute([…]) call.
  10. If you use simple ? place-holders, it saves a lot of typing.
  11. The only time you should catch and handle database statement errors in your code are for user recoverable errors, such as when inserting/updating duplicate user submitted data. Since this data isn’t being submitted by a user, there’s no point in trying to handle any of the exceptions in your code. Just let php catch and handle any database exceptions.
  12. You can operate on sets of data using php array functions. For the genre code, just use array_column() to get all the β€˜list_name’ values at once. Also, str_replace() can directly operate on an array of values.

Here’s some example code that should (tested only with made up data) work -

// this example code directly uses the PDO extension, with the connection in $pdo

getBooks($pdo);

function getBooks($pdo)
{
	$apiKey = 'apikey';
	
	// data-driven design
	// define the database column to data input mapping
	// if the value (on the right) is an empty string, the db column name will be used as the input name, i.e. only put a data input name/value if it is different than the column name
	$db_col['title'] = '';
	$db_col['isbn13'] = 'primary_isbn13';
	$db_col['isbn10'] = 'primary_isbn10';
	$db_col['author'] = '';
	$db_col['description'] = '';
	$db_col['publisher'] = '';
	$db_col['published_date'] = '';
	$db_col['rank'] = '';
	$db_col['rank_last_week'] = '';
	$db_col['weeks_on_list'] = '';
// the commented out entries don't exist in the current api documentation
//	$db_col['book_image_width'] = '';
//	$db_col['book_image_height'] = '';
//	$db_col['book_uri'] = '';
//	$db_col['cover_image_url'] = '';
//	$db_col['page_count'] = '';
//	$db_col['language'] = '';
//	$db_col['average_rating'] = '';
//	$db_col['rating_count'] = '';
//	$db_col['availability_status'] = '';
//	$db_col['availability_count'] = '';
//	$db_col['format'] = '';
//	$db_col['edition'] = '';
	$db_col['contributor'] = '';
	$db_col['price'] = '';
	$db_col['age_group'] = '';
//	$db_col['book_image_url'] = ''; // there is a book_image
	$db_col['amazon_product_url'] = '';

	// get all list names
	$genresURL = "https://api.nytimes.com/svc/books/v3/lists/names.json?api-key=" . $apiKey;
	$response = file_get_contents($genresURL);
	$genresData = json_decode($response, true);

	// extract list_name (genre)
	$genres = array_column($genresData['results'], 'list_name');
	// convert space to -
	$genres = str_replace(' ', '-', $genres);
	
	$sql = "INSERT IGNORE INTO genres (name) VALUE (?)";
	$stmt = $pdo->prepare($sql);
	foreach($genres as $genre)
	{
		$stmt->execute([$genre]);
	}

	// build and prepare queries for books and buy_links
	// foreach column (that can be updated) need a column = VALUES(column) term
	$update_terms = [];
	// add the term to get the id as the last insert id if the row that was updated
	$update_terms[] = "`id`=LAST_INSERT_ID(`id`)";
	// add the data terms
	foreach(array_keys($db_col) as $col)
	{
		$update_terms[] = "`$col`=VALUES(`$col`)";
	}

	$sql = "INSERT INTO books (`".implode('`, `',array_keys($db_col))."`) VALUE (".implode(', ',array_fill(0,count($db_col),'?')).")
		ON DUPLICATE KEY UPDATE
		".implode(', ',$update_terms);
	$stmt_books = $pdo->prepare($sql);
		
	// buy link query
	$sql = "INSERT INTO buy_links (book_id, name, url) VALUE (?, ?, ?)";
	$stmt_buy_links = $pdo->prepare($sql);

	// get current books data for each list_name (genre)
	foreach ($genres as $genre) {
		$genresApi = "https://api.nytimes.com/svc/books/v3/lists/current/" . urlencode($genre) . ".json?api-key=" . $apiKey;
		$resp = file_get_contents($genresApi);
		$booksData = json_decode($resp, true);

		// ['num_results'] number of books in the list. if there is more than 20, use a loop and pagination to get the rest of them
		// ['results']['published_date'] date the list was published
		// ['results']['books'] actual book data

		// if there is book data
		if (!empty($booksData) && isset($booksData['results']['books'])) {
			foreach ($booksData['results']['books'] as $bookDetail) {
				// special handling for the published_date
				$bookDetail['published_date'] = $booksData['results']['published_date'];
				
				// build input parameters
				$params = [];
				foreach($db_col as $key=>$input)
				{
					// if the input definition is empty, use the key as the input element name
					$in = empty($input) ? $key : $input;
					$params[] = $bookDetail[$in];
				}
				$stmt_books->execute($params);
				
				$last_id = $pdo->lastInsertId();

				// the current api documentation doesn't list 'buy_links' elements
				// if it did, you would build and prepare this query above the start of the main foreach loop
				foreach ($bookDetail['buy_links'] as $buyLink) {
					$stmt_buy_links->execute([ $last_id, $buyLink['name'], $buyLink['url'] ]);
				}
			}
		}
		sleep(15);
	} // end of foreach genre
}
1 Like

hey @phdr I must thank you for your detailed and extensive answer, it does help to make things a lot clearer. I did implement your suggestion and switched to using cURL. Even though I would need to read up on it some more to understand it better. However I realized that in my MVC app structure this function does not belong in the Book Model. It should be moved as a helper or in the libraries folder, since ones the DB is populated, I intend on making monthly updates, for new content possibly.
I will work on this some more in the coming days and share a working class hopefully.

This is the buy_links array that I intended to fetch and connect to each book’s id…

["buy_links"]=> array(6) { [0]=> array(2) { ["name"]=> string(6) "Amazon" ["url"]=> string(52) "https://www.amazon.com/dp/1649374178?tag=NYTBSREV-20" }
root_dir
β”‚
β”œβ”€β”€ app
β”‚   β”œβ”€β”€ config
β”‚   β”‚   └── config.php
β”‚   β”œβ”€β”€ controllers
β”‚   β”‚   └── Pages.php
β”‚   β”œβ”€β”€ helper
β”‚   β”œβ”€β”€ libraries
β”‚   β”‚   β”œβ”€β”€ Controller.php
β”‚   β”‚   β”œβ”€β”€ Core.php
β”‚   β”‚   β”œβ”€β”€ Database.php
β”‚   β”‚   
β”‚   β”œβ”€β”€ models
β”‚   β”‚   └── Book.php
β”‚   β”œβ”€β”€ services
β”‚   β”‚   
β”‚   └── views
β”‚       β”œβ”€β”€ inc
β”‚       β”‚   β”œβ”€β”€ footer.php
β”‚       β”‚   └── header.php
β”‚       └── pages
β”‚           └── index.php
β”œβ”€β”€ bootstrap.php
└── public
    β”œβ”€β”€ css
    β”œβ”€β”€ js
    └── img

Hey @phdr, here is the final draft named BooksApi that was moved into the services folder and I’ve created a task in the task scheduler for a monthly update of the data.

Thanks again for the help and explanation of the dynamic building of a sql query.
Regarding point 6. I already have a query method in the DB class. Provided below. Or maybe this is what you point out as a wrapper class and shouldn’t be used?
point 7: Isn’t the emulated prepared query set to false by default? I haven’t used that setting to set it to true.

I am however still unsuccessful in pulling the $last_id for each book_id in order to connect each book to the buy_links.

    public function query($sql)
    {
        $this->stmt = $this->dbh->prepare($sql);

        return $this->stmt;
    }
public function fetchData($urls)
    {
        $ch = curl_multi_init();
        $requests = [];
        $response = [];


        foreach ($urls as $i => $url) {
            $requests[$i] = curl_init($url);
            curl_setopt($requests[$i], CURLOPT_URL, $url);
            curl_setopt($requests[$i], CURLOPT_RETURNTRANSFER, true);
            curl_setopt($requests[$i], CURLOPT_TIMEOUT, 10);
            curl_setopt($requests[$i], CURLOPT_CONNECTTIMEOUT, 10);
            curl_setopt($requests[$i], CURLOPT_SSL_VERIFYHOST, false);
            curl_setopt($requests[$i], CURLOPT_SSL_VERIFYPEER, false);
            curl_multi_add_handle($ch, $requests[$i]);
        }


        $active = null;
        do {
            curl_multi_exec($ch, $active);
        } while ($active > 0);


        foreach ($requests as $request) {
            $httpStatus = curl_getinfo($request, CURLINFO_HTTP_CODE);
            if ($httpStatus == 200) {

                if ($httpStatus == 200) {

                    $content = curl_multi_getcontent($request);
                    $response[] = json_decode($content, true);
                } else {

                    echo "Error fetching data for URL: " . curl_getinfo($request, CURLINFO_EFFECTIVE_URL) . " - HTTP status: " . $httpStatus;
                }

                curl_multi_remove_handle($ch, $request);
                curl_close($request);
            }

            curl_multi_close($ch);
            return $response;
        }
    }



    public function getBooks()
    {
        $booksApi = new BooksApi();

        $apiKey = 'apikey';

        $db_col['title'] = '';
        $db_col['isbn13'] = 'primary_isbn13';
        $db_col['isbn10'] = 'primary_isbn10';
        $db_col['author'] = '';
        $db_col['description'] = '';
        $db_col['publisher'] = '';
        $db_col['published_date'] = '';
        $db_col['rank'] = '';
        $db_col['rank_last_week'] = '';
        $db_col['weeks_on_list'] = '';
        $db_col['contributor'] = '';
        $db_col['price'] = '';
        $db_col['age_group'] = '';
        $db_col['book_image_url'] = 'book_image';
        $db_col['amazon_product_url'] = '';


        // $response = file_get_contents($genresURL);
        $genresURL = "https://api.nytimes.com/svc/books/v3/lists/names.json?api-key=" . $apiKey;
        $genresData = $booksApi->fetchData([$genresURL]);


        $genres = array_column($genresData[0]['results'], 'list_name');
        $genres = str_replace(' ', '-', $genres);


        $sqlGenre = "INSERT IGNORE INTO genres (name) VALUE (:name)";

        foreach ($genres as $genre) {

            $this->db->query($sqlGenre);
            $this->db->bind(':name', $genre);

            $this->db->execute();
        }

        $update_terms = [];
        $update_terms[] = "`id` = LAST_INSERT_ID(`id`)";


        foreach (array_keys($db_col) as $col) {
            $update_terms[] = "`$col` = VALUES(`$col`)";
        }

        $sql = "INSERT INTO books (`" . implode('`, `', array_keys($db_col)) . "`) VALUE (" . implode(', ', array_fill(0, count($db_col), '?')) . ")
		ON DUPLICATE KEY UPDATE
		" . implode(', ', $update_terms);

        $stmt = $this->db->query($sql);

        $buyLink_sql = "INSERT INTO buy_links (book_id, name, url) VALUES (:book_id, :name, :url)";

        $stmt_buy_links = $this->db->query($buyLink_sql);


        foreach ($genres as $genre) {

            $offset = 0;
            $total_books = 0;
            $max_per_page = 20;

            do {
                $genresApi = "https://api.nytimes.com/svc/books/v3/lists/current/" . urlencode($genre) . ".json?api-key=" . $apiKey . "&offset=" . $offset;
                $booksData = $booksApi->fetchData([$genresApi]);
                
                if (!empty($booksData) && isset($booksData['results']['books'])) {
                    foreach ($booksData['results']['books'] as $bookDetail) {
                        $bookDetail['published_date'] = $booksData['results']['published_date'];

                        $params = [];

                        foreach ($db_col as $key => $input) {
                            $in = empty($input) ? $key : $input;
                            $params[] = $bookDetail[$in];
                        }

                        
                            $stmt->execute($params);
                            $last_id = $this->db->lastInsertId();
                            
                            foreach ($bookDetail['buy_links'] as $buyLink) {
                                print_r($buyLink    );
                                $stmt_buy_links->execute([
                                    'book_id' => $last_id,
                                    'name' => $buyLink['name'],
                                    'url' => $buyLink['url']
                                ]);
                            }
                    }

                    $total_books = $booksData['num_results'] ?? 0;
                    $offset += $max_per_page;
                }
            } while ($offset < $total_books);
        }
    }
}

Re point #6:
In your original code and for the genres query in the last code, you are calling the -.query() (prepare) method and explicitly re-binding the input data, inside the looping. This is re-preparing the query and re-binding the input data over and over, when the only thing that is changing for each pass through the loop is the data. Each call to the ->prepare() method sends the sql query statement to the database server, which requires a round-trip communication between php and the database server, where the query gets parsed again and its execution gets planed again each pass through the loop.

Your post of the class ->query() method, is basically a single line of code, as is probably the case for the class ->bind(), and class ->execute() methods. This is what is referred to as a wrapper class. All it is doing is adding more work for the programmer in writing this code and in keeping track of different names and calling syntax for existing methods and since it is not general-purpose, requires even more methods or in this case returning the PDOStatement object so that the PDO execute method can be called directly. Doing all this extra typing and code execution is not adding anything useful, is not general-purpose, and is not what OOP is about.

Re point #7:
Past documentation has indicated that the default setting is a true value, i.e. to always use emulated prepared queries. You can test the operation with a MySQL/MariaDB database, which does support prepared queries, by introducing an sql syntax error into a prepared query and seeing where the syntax error is reported. If it is reported at the ->prepare() call, a true prepared query is being used. If it is reported at the ->execute() call, an emulated prepared query is being used, because nothing is sent to the database server at the ->prepare() call and the actual sql query statement with the values inserted directly into it is sent at the ->execute() call.

Is there a column in the books table, that will always contain a value that doesn’t change, define as a unique index, so that the INSERT … ON DUPLICATE KEY UPDATE … query inserts new data and updates existing data?

What value is in $last_id?

Or does this statement mean that no data is being inserted into the buy_links table? As already stated, there is no β€˜buy_links’ element shown in the api documentation. What does using var_dump($bookDetail) show?

Hey @phdr thanks again for your support so far. It took me some time to get this to work, however I did rework the Database class to get rid of the unnecessary wrapper classes, this is functioning fine now.

namespace App\Libraries;
use PDO;
use PDOException;


class Database
{

    private $dbh;


    public function __construct()
    {
        // set DSN 
            $dsn = 'mysql:host=' . DB_HOST . ';dbname=' . DB_NAME;        
            $options = [
            PDO::ATTR_ERRMODE            => PDO::ERRMODE_EXCEPTION,
            PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC,
            PDO::ATTR_EMULATE_PREPARES   => false,
        ];

        try {
            $this->dbh = new PDO($dsn, DB_USER, DB_PASS, $options);
        } catch (PDOException $e) {
            die("DB connection failed: " . $e->getMessage());
        }
    }

    public function getConnection() {
        return $this->dbh;
    }

}

This next class is in the Book Model and is what I use to check if the getBooks class is working properly, before applying the code in the BookApi class. I did modifications to it in order to finally fetch that buy_links array with name and links and also created another table book_genre to manage the relationship between book_id and genre_id for future categorization. I am not sure if this is needed, any feedback would be appreciated.

    public function getBooks()
    {


        $apiKey = $_ENV['NYT_API_KEY'];

        $db_col['title'] = '';
        $db_col['isbn13'] = 'primary_isbn13';
        $db_col['isbn10'] = 'primary_isbn10';
        $db_col['author'] = '';
        $db_col['description'] = '';
        $db_col['publisher'] = '';
        $db_col['published_date'] = '';
        $db_col['contributor'] = '';
        $db_col['price'] = '';
        $db_col['age_group'] = '';
        $db_col['book_image_url'] = 'book_image';
        $db_col['amazon_product_url'] = '';
        $db_col['rank'] = '';
        $db_col['rank_last_week'] = '';
        $db_col['weeks_on_list'] = '';
        

    
        $genresURL = "https://api.nytimes.com/svc/books/v3/lists/names.json?api-key=" . $apiKey;
        $genresData = $this->fetchData([$genresURL]);

        $genres = array_column($genresData[0]['results'], 'list_name');
        $genres = str_replace(' ', '-', $genres);
       

        $update_terms = [];
        $update_terms[] = "`id` = LAST_INSERT_ID(`id`)";


        foreach (array_keys($db_col) as $col) {
            $update_terms[] = "`$col` = VALUES(`$col`)";
        }
        

        $sql_genre = "INSERT IGNORE INTO genres (name) VALUE (?)";

        $stmt_genre = $this->db->prepare($sql_genre);


        $sql_book_genre = "INSERT IGNORE INTO book_genre (book_id, genre_id) VALUES (?, ?)";

        $stmt_book_genre = $this->db->prepare($sql_book_genre);


        $sql = "INSERT INTO books (`" . implode('`, `', array_keys($db_col)) . "`) VALUE (" . implode(', ', array_fill(0, count($db_col), '?')) . ")
		ON DUPLICATE KEY UPDATE
		" . implode(', ', $update_terms);


        $stmt_books = $this->db->prepare($sql);

        $buyLink_sql = "INSERT INTO buy_links (book_id, name, url) VALUES (?, ?, ?)";

        $stmt_links = $this->db->prepare($buyLink_sql);


        foreach ($genres as $genre) {
            
            $offset = 0;
            $total_books = 0;
            $max_per_page = 20;
           var_dump($genre);
           $stmt_genre->execute([$genre]);
           $genre_last_id = $this->db->lastInsertId();
           var_dump($genre_last_id);
            do {
                $genresApi = "https://api.nytimes.com/svc/books/v3/lists/current/" . urlencode($genre) . ".json?api-key=" . $apiKey . "&offset=" . $offset;
                $booksData = $this->fetchData([$genresApi]);
                // var_dump($booksData);
                if (!empty($booksData) && isset($booksData[0]['results']['books'])) {

                    foreach ($booksData[0]['results']['books'] as $bookDetail) {
                        // print_r($bookDetail);
                        $bookDetail['published_date'] = $booksData[0]['results']['published_date'];

                        $params = [];

                        foreach ($db_col as $key => $input) {
                            $in = empty($input) ? $key : $input;
                            $params[] = $bookDetail[$in];
                        }

                        $stmt_books->execute($params);
                        $last_id = $this->db->lastInsertId();

                        $stmt_book_genre->execute([
                            $last_id, $genre_last_id
                        ]);
    

                        foreach ($bookDetail['buy_links'] as $buyLink) {                           
                                $stmt_links->execute([
                                    $last_id, $buyLink['name'], $buyLink['url']
                                ]);
                            
                        }
                    }
                 
                    $total_books = $booksData['num_results'] ?? 0;
                    $offset += $max_per_page;
                    sleep(10);
                }

            } while ($offset < $total_books);
        }
    }

I have an issue with the BookApi and instantiating the Database class in order to work… Whenever I run the task in the scheduler I get the following error. I cannot figure out why.

[01-Dec-2023 14:52:23 Europe/Berlin] PHP Fatal error:  Uncaught Error: Class "App\Libraries\Database" not found in C:\xampp\htdocs\nyt_book_list\app\services\BookApi.php:190
Stack trace:
#0 {main}
  thrown in C:\xampp\htdocs\nyt_book_list\app\services\BookApi.php on line 190
<?php

namespace App\Services;
use App\Libraries\Database;


class BookApi
{

    private $db;

    public function __construct()
    {
        $this->db = (new Database())->getConnection();
    }

// The rest is the same as getBooks...     

}


$database = new Database(); 
var_dump($database);
$booksApi = new BookApi($database->getConnection()); 
var_dump($booksApi);
$booksApi->getBooks();

Why not simply do this β†’

    public function __construct(PDO $db)
    {
        $this->db = $db;
    }
    // more code

then

$db = new Database(); 
var_dump($db);
$booksApi = new BookApi($db); 
var_dump($booksApi);
$booksApi->getBooks();

I had to include the bootstrap and autoloader in this file to finally work. Maybe the fact that I am running this from Task Scheduler has something to do with the execution…

In any case this wasn’t part of the original question, I will be closing this post.
Thanks to everyone that contributed, much appreciated.

Sponsor our Newsletter | Privacy Policy | Terms of Service