The info in this doc only applies to Redshift data warehouses.
In version 2 of the Amazon Redshift destination, realized tables replaced views as the interface for accessing your data. When new data is detected and ready to be loaded into these tables, Stitch does a few things depending on the replication method being used.
Each table inside the integration schemas in your data warehouse contain a few columns prepended with
Stitch uses these columns to replicate your data. They're only used by us, so don't worry too much about the values in them.
Note that if any columns in your data source are prepended with
_sdc, Stitch will NOT replicate them to your data warehouse. Stitch reserves all columns with this naming convention for internal use.
During the initial replication of a table using Full Table replication, you might see what looks like an “incomplete” table. This is because Stitch is still processing and replicating your data. Once the replication is complete, the table will be marked with a version 1 stamp.
During the next replication attempt, a new version of the table will be created. Each batch is marked with a version stamp as Stitch picks it up. At this point:
VACUUMcommand is executed.
If using Incremental Replication, new and updated data will be appended to the same table version.
The only time Stitch will create a new version for an incrementally replicated table is if the underlying structure has changed. In this case, an
ALTER command is issued to change the table’s structure.
When the data can’t be loaded losslessly into a realized table, Stitch will modify the structure of the table. This can happen for a few reasons:
You’ll experience some missing data on your end. To troubleshoot, we recommend taking a look at some of the more common causes for dropped (or missing) data:
__bigint, etc. These are reserved names for Stitch.
If you’re stumped or need some help pinpointing the issue, please reach out to support.