
blog
How to Efficiently Ingest HubSpot Data into Amazon S3 and Build a Production-Grade Lakehouse with Iceberg and AWS Glue
Writing a Python script to fetch data from an API is the easy part. Any junior developer can write a script that pulls HubSpot deals and dumps them into a CSV file. But what happens six months later? The file grows to 10GB. The script crashes because of memory issues. You have duplicate records everywhere because the API connection reset halfway through. Suddenly,